Copyright © 2014-2015 Avery Lee
a8rawconv is a utility for converting between raw and decoded disk images, taking advantage of recent availability of high-quality devices for reading and writing raw flux-level images of floppy disks. This allows for easy and fast methods of creating usable images of floppy disks without a modified disk drive, particularly unusually encoded or even copy protected floppy disks. Decoded disk images are better suited for testing in an emulator or on real hardware using a disk simulator, and provide a way to verify whether a disk image is valid before archiving.
The converter is provided in binary form as a Win32 command line utility called a8rawconv.exe, which can be run on Windows XP SP2 or later, on either a 32-bit or 64-bit system. To use it, open a Command Prompt and run the program from the command line. This allows command-line arguments to be specified and the output to be seen.
Source code is also provided so that the converter can be rebuilt or modified. Visual Studio 2010 or newer is required.
Although binaries are not provided for other platforms, the source code is included so that it can be compiled. The core is designed to be portable to most systems that have at least a C++11 compiler, with the serial device functionality also available for Linux systems.
To compile the converter, build compileall.cpp with a C++ compiler in C++11 or C++14 mode. This file is a bulk file that includes all the other files, avoiding the need for a build system. This is the same build process used for the Release build on Windows.
The main use of a8rawconv is to convert from a raw disk image produced by imaging hardware to a more usable decoded disk image. This can be done by running a8rawconv with the input and output names:
a8rawconv track00.0.raw mydisk.atx
a8rawconv mydisk.scp mydisk.atx
Raw images can be produced from physical floppy disks either by KryoFlux or SuperCard Pro imaging hardware. By default, the input format, decoding parameters and output format are determined by the extension on the input and output filenames; .atx selects the VAPI/ATX format with Atari 8-bit decoding parameters. Similarly, .nib selects the NIB disk format with Apple II 5.25" decoding parameters. These can be overridden with the -if, -of, and -d parameters.
When using KryoFlux raw streams (track00.0.raw, track02.0.raw, etc.), a 96 tracks per inch (TPI) drive and an 70/80 track dump are assumed by default. Thus, the converter will dump even tracks. When using a 48 TPI drive and 35/40 track dumps, the -tpi switch must be used to change the interpretation to 48 TPI.
a8rawconv can also be used to convert from one decoded disk image to another. For instance, if a disk encoded in the ATX format turns out to not have any special encodings or protection on it, it can be converted to the ATR format as follows:
a8rawconv mydisk.atx mydisk.atr
When converting from a more flexible format to a less flexible one, the converter carries over data that can be preserved and warns about any unusual characteristics in the disk image that cannot be preserved. For a conversion from ATX to ATR, timing information is dropped, all sectors are converted to readable sectors, and warnings are issued if any sectors are missing, have phantom copies, or have errors. For RAW to SCP, partial disk revolutions are dropped and the flux timing samples are resampled from 40ns to 25ns resolution.
When converting from a less flexible format to a more flexible format, additional needed information is created based on a standard disk format. For ATR to ATX, the sectors are laid out timing-wise with interleave and track-to-track skew comparable to a standard Atari 1050 disk drive.
Typically, the first step is to create a raw image of a floppy disk using the software that comes with the imaging hardware. There are a few things that should be kept in mind when doing this:
a8rawconv supports both KryoFlux raw streams (*.raw) and SuperCard Pro images (*.scp). Although there are differences between both the KryoFlux and SuperCard Pro hardware and software, both produce images suitable for conversion and neither has a significant advantage over the other in image quality or read reliability.
When archiving, it is always recommended that you keep the original raw images of a disk for archival purposes even if the converter successfully creates a fully functional decoded disk. This is because the decoded disk does not contain all of the information of the original disk. If it turns out that something is missing or the converter is updated, the raw source can be used to reconvert without using the physical disk again. Raw images compress well using standard compression tools.
In non-archival cases, this can be skipped. If you are decoding a disk containing a BASIC program you wrote as a kid, there won't be any copy protection on the disk and it's unlikely you'll care about the exact sector timing. Therefore, once you verify that the decoded disk is good, you don't need to keep the raw image. The raw images are so small in modern terms, though, that you might consider still keeping them around.
Note: Some versions of the SuperCard Pro software have a buggy profile for Atari 8-bit disks and will attempt to create an image using a single index-aligned revolution. This will result in lost sectors on most disks and a8rawconv will issue a warning if it sees this in the image. The imaging parameters must be manually adjusted to at least two revolutions.
a8rawconv can also use a SuperCard Pro device to directly read from a floppy disk. The SuperCard Pro software must be installed first to install the USB communications driver, after which the converter can access the hardware using special filenames:
Therefore, a physical disk can be imaged as follows:
a8rawconv scp0:96tpi test.atx
On Windows, a8rawconv will automatically search for the first COM port that is a USB connection to a SuperCard Pro device. On Linux, the serial device name must be manually specified, and you may also need to run with elevated (superuser) privileges.
Reading directly from a floppy disk can be useful when doing testing or transferring data. It is not recommended for old floppy disks since a raw image is not created during the process, and repeating the decoding process with different settings requires re-reading the disk, which is slower and puts more stress on the disk. Archiving should always be done by creating a preservation-level raw image once and then re-decoding the raw image as necessary.
The converter has a few features to assess the quality of a decoded image. The simplest is just how many sectors were extracted. For Atari 8-bit single density disks — the most common format — the converter should report no missing sectors or sectors with errors:
Writing ATX file: e:\test.atx 0 missing sectors, 0 phantom sectors, 0 sectors with errors
On the other hand, you might see something like this:
Writing ATX file: e:\test.atx
WARNING: Track 20, sector 16: Multiple sectors found at the same position
0.81 but different bad data. Encoding weak sector at offset 25.
WARNING: Track 20: Missing sectors: 18.
1 missing sector, 0 phantom sectors, 1 sector with errors
If your disk is supposed to be just a normal DOS disk, this means you have some problems. If it's a commercial release, and especially if it is copy protected, then this might be expected. In the latter case, some experience with copy protection techniques is helpful and testing the decoded disk image is recommended.
a8rawconv is designed to have good defaults so that most of the time conversion can be performed without a string of command-line switches. However, sometimes tweaking is necessary. One useful switch is the -p switch, which tweaks the default bit cell period used by the decoder, as a percentage of the normal value. This can be used to shrink or stretch the expected length of a bit slightly to nudge the decoder toward success when it is having trouble with a marginal disk:
a8rawconv -p 98 disk.scp disk.atx
The recommended approach is to try small adjustments in the range of 95-105 and watch the number of errors to zero in on the best value. Don't try crazy values like 70 or 130 to start with; you're likely to just get complete failure to decode the track. Watch out for when the decoder starts missing sectors entirely, as past that point the error count may drop because the decoder stops seeing some of the sectors.
The -t switch is also useful when tweaking decoding settings. It limits decoding to a single track on the disk, filtering out errors from tracks other than the one you're focusing on:
a8rawconv -t 27 disk.scp disk.atx
Note that this also causes only that track to be emitted into the output file, so this should not be used to produce the final disk image file.
When multiple revolutions are available for a track, a8rawconv compares all copies of each physical sector to see whether they are the same. If they are not, it attempts to separate them into good and bad copies, and reports when this occurs:
WARNING: Track 38, sector 1: 1/3 bad sector reads discarded at position 0.25.
For a standard unprotected disk, this is a bad sign, as it means that the disk or either the imaging hardware is marginal, and could mean that either the disk is going bad or the imaging drive needs to be cleaned. Fortunately, the Cyclic Redundancy Check (CRC) stored with each sector allows the converter to tell if a sector is good, and this message means that the converter was able to recover the sector. This is one reason that five disk revolutions is recommended — the extra images of each sector increase the changes that the converter will be able to recover sectors from marginal disks.
When tweaking the decoder timing with -p, the ratio of good to bad sector reads can be used to aid in determining whether a change in timing is helping or hurting.
Note that in this case, "good" means that the sector was read correctly. Protected disks often have sectors that were written with deliberate CRC errors such that they are always read by the disk drive with errors; the converter can isolate good reads of these sectors in the same way, as both the CRC and data will be stable even if the CRC is incorrect.
In the event that the converter can't find at least two reads of the same sector that match, it will salvage as much common data as it can and then encode a weak sector:
WARNING: Track 38, sector 4: Multiple sectors found at the same position
0.65 but different bad data. Encoding weak sector at offset 44.
For a regular disk, this is a bad sign as it means that the converter was not able to get a clean read of the sector. For protected disks, though, this can be valid as some disks were deliberately created with weak sectors that changed every time they were read by the disk drive. Typically there are only a small handful of these at most. When weak sectors are found, it is recommended that the decode be retried with different -p settings to try to determine whether these are truly weak sectors or bad reads of a stable sector.
The -l switch causes the converter to display a crude representation of the sectors within each track. This is useful to identify unusual disk layouts:
0 (18) | 6 8 10 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 1 (18) | 8 10 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 2 (18) | 8 10 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 3 (18) | 8 10 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 4 (18) | 10 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 5 (18) | 10 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 6 (18) | 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 10 7 (18) | 12 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 10 8 (18) | 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 9 (18) | 14 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 10 (18) | 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 11 (18) | 16 18 1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 12 (18) | 18 1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 16
The left columns indicate the track number and number of physical sectors. The remainder of the line shows all sectors in the track, where the relative position of each sector number indicates the angular position of that sector. Remember that floppy disk tracks are a continuous circle, so the start and end of the line are the same position on the disk.
For a disk formatted on a standard Atari 810 or 1050 drive, even and odd sectors will be separated due to sector interleave, a reordering of the sectors to improve read/write performance. Disk drives with high-speed capability can format with a different sector interleave for better high-speed performance. Some commercial disks were written with a non-standard interleave and read slower than usual on a real drive.
The sectors on successive tracks will also be offset due to skew between the tracks. This is related to the delay when stepping between tracks during initial formatting. It is also possible for tracks to lack such skew and be aligned to the index mark:
0 (18) | 18 7 14 3 10 17 6 13 2 9 16 5 12 1 8 15 4 11 1 (18) | 18 7 14 3 10 17 6 13 2 9 16 5 12 1 8 15 4 11 2 (18) | 18 7 14 3 10 17 6 13 2 9 16 5 12 1 8 15 4 11 3 (18) | 18 7 14 3 10 17 6 13 2 9 16 5 12 1 8 15 4 11 4 (18) | 18 7 14 3 10 17 6 13 2 9 16 5 12 1 8 15 4 11 5 (18) | 18 7 14 3 10 17 6 13 2 9 16 5 12 1 8 15 4 11 6 (18) | 18 7 14 3 10 17 6 13 2 9 16 5 12 1 8 15 4 11
Such a disk will read and write fine, but generally indicates that the disk was not created in an Atari drive. This example also shows a different sector interleave, which can cause the Atari to read the disk more slowly than usual.
Highly unusual patterns typically indicate copy protection, such as this track with 34 physical sectors:
2 (34) | 2 34 56 7 8 9 10112 114 116 11 23 45 67 89 10111213115 117 1
Any track on a single-density Atari disk with more than 18 physical sectors on it has more than one duplicated or phantom sectors, where two or more physical sectors have the same sector number. This is used for protection schemes where the timing of the sector read determines which of the phantom sectors is read. Such a disk cannot be created on a stock Atari drive, although it can be created with a modified drive or with a hardware imager such as SuperCard Pro.
Finally, there is the possibility that the disk is just simply something else:
0 ( 9) | 1 2 3 4 5 6 7 8 9 1 ( 9) | 1 2 3 4 5 6 7 8 9 2 ( 9) | 1 2 3 4 5 6 7 8 9 ... Writing ATX file: e:\test.atx WARNING: Track 0, sector 1: Long sector of 512 bytes found. WARNING: Track 0, sector 2: Long sector of 512 bytes found. WARNING: Track 0, sector 3: Long sector of 512 bytes found.
For Atari disk formats, the converter attempts both FM and MFM decoding on each track, which means that it can pick up even non-Atari disk formats. In this case, the disk has 9 index-aligned sectors per track with 512 bytes per track instead of 18 sectors per track of 128 bytes each, meaning that it is likely a 360K MS-DOS disk, not an Atari disk.
Some software shipped on "flippy" disks that could be flipped upside down to access another program variant or more data on the other side of the disk. Imaging these disks is problematic for a couple of reasons:
It is possible to modify a floppy disk drive to circumvent one of these two problems. If the drive is modified to realign the second head, reading the disk image will produce track streams that are backwards. In that case, the -r option should be passed to a8rawconv so that it reads the track images backwards to match.
Ingenious fellows in the past found ways to create floppy disks that booted on more than one computer, by creatively interleaving sectors from the different formats on the same disk. a8rawconv does not directly support mixed-format disks, but can aid in detecting such disks and extracting out platform-specific versions. Unusual sector sizes or missing sectors/tracks in the layout map, particularly on tracks 1 or 18, are a possible sign of such a disk. In that case, conversion can be attempted using output formats for all platforms.
It is highly recommended that preservation-quality images of such multi-format disks be kept, as they are uncommon and currently can't be converted to a single decoded image.
For experts dealing with troublesome disks, the -v option causes the converter to report additional information about the decoding process. There are four levels of verbosity:
00 FF 00 FF 00 FF 00 FF 00 FF 00 FF 00 FF 00 FF | 15.64 00 FF 00 FF 00 FF 00 FF 00 FF 00 FF 00 FF 00 FF | 15.71 00 FF 00 FF 01 FF 03 FE 07 FC 0F F8 1F F1 3E E3 | 15.64 7D C7 FB 8F F6 1F ED 3F DB 7F B7 FF 6F FF DF FF | 15.79 BF FF 7F FF FF FF FF FF FF FF FF FF FF FF FF FF | 15.74 FF FF FE FF FD FF FB FF F7 FF EF FF DF FF BF FF | 15.63 7F FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF | 15.65 FF FF FE FF FC FF F9 FF F2 FF E4 FF C8 FF 91 FF | 15.86
Listed on the left is the complete dump of the shift register states, and on the right is the number of bit cell times measured for each group of 16 bits, composed of 8 clock bits and 8 data bits interleaved together.
Because the relationship between clock and data bits and the byte alignment is not known until a sync byte is found, the bit-level dump simply shows all possible offsets. The layout of 16 shifts per line ensures that once the sync byte is found, successive bytes will be in a column. Above, highlighted in red, is the Data Address Mark (DAM) with $C7 clock pattern and $FB data byte, followed by the first four data bytes of the sector.
The higher verbosity levels can produce very large amounts of output, so restricting decoding to a single track with -t is highly recommended.
It is possible for the converter to believe it has successfully read a sector when it hasn't. The CRC16 used by the Atari disk format and other 177x compatible disk formats makes this unlikely. Unfortunately, 5.25" Apple II disks use a weak 8-bit XOR checksum algorithm that is more prone to false negatives and can fail to detect errors. This has been seen in practice with old disks that had many marginal sectors when read.
The only truly reliable way to verify a disk is to examine it manually, either by attempting to boot the decoded image or checking the files. The latter case can be annoying as the Apple II standard for text has the high bit set. The -I switch will invert bit 7 on decoded data so that text files can be more easily read in a hex editor.
With a SuperCard Pro device, a8rawconv can also write images back to physical disks. This is not typically useful for distribution as no one distributes software on physical floppy disks anymore, but can still be useful for testing or if floppy disk is the only way to get software onto a real computer.
To do so, simply specify the SuperCard Pro as the output device instead of the source:
a8rawconv test.atr scp0:96tpi
The converter will then encode the disk image to physical format and write them to the physical disk through the SCP. Pre-erasing or preformatting the disk is not necessary. Both Atari FM (single) and MFM (enhanced) formats can be encoded and written back to disk. Note that on an 80 track (96 tpi) drive, only the even tracks are written.
While it is possible to write out a raw image and then write that to physical disk using the imaging software that came with the hardware, doing this with a8rawconv can produce better results. This is because the converter can use its knowledge of the Atari disk format to better choose a suitable splice point for beginning and ending track writes. The .scp file format does not include such information, and so the imaging software can write out a broken track if it selects the wrong splice point.
Note: The image written back to disk will not be the same as the original, even though it will have the same data content. This is because the converter re-lays out the track according to the standard format expected by Atari disk drives, which can cause small timing differences from the original. If an accurate duplicate is required, it should be produced from the raw stream image using the duplication software supplied with the hardware.
Plain disk images with standard geometry typically write out to a physical disk without problems. For protected disks, the converter has support for writing sectors with CRC errors, long sectors, phantom sectors, deleted sectors, and even weak sectors, and can write out usable versions of many protected disks. However, some types of encoding can still pose a problem for the encoder.
The main problem that can occur when writing out a protected disk is long tracks:
WARNING: Track 2, sectors 4 and 2 overlapped during encoding. Encoded track may not work.
This happens when there are phantom sectors on the track, and more sectors than can fit onto the track with standard encoding. When this happens, the -p option can be used to record bits onto the track at a faster rate, squeezing more data onto the track. This should be done with care as it reduces the read margin and can cause the disk to read unreliably or not at all. Also, as with decoding, small increments should be used when adjusting the period; starting with -p 50 will simply produce an unusable disk.
When writing long sectors, a8rawconv will write the sectors with their intended length but truncate the sectors at 128 bytes. This saves space on the track, but produces the same result as the standard disk interface only reports the first 128 bytes.
Occasionally, a disk image may have a track with an impossibly high number of sectors in it, such as 36 sectors per track, that can't be written out in usable form even with tight -p settings. This almost always means that the track was crafted to have overlapping sectors, where the data field of one sector overlaps the address field and even the data field of the next sector. This is possible because the floppy drive controller (FDC) does not validate the clock bits of data bytes. Unfortunately, decoded image formats do not store where sectors were overlapped, and a8rawconv does not currently have analysis algorithms to encode overlapping sectors.
The author would like to thank Jim Drew for providing SuperCard Pro hardware and support, and fellow AtariAge board members such as Farb for testing, encouragement, and suggestions.
a8rawconv is released under the GNU General Public License, version 2 or later. A copy of the license is included in the file COPYING, and the source code is included within the same archive.
While the converter is licensed under the GNU GPL, the converter does not put any part of itself into disk images processed by the converter. Therefore, there are no restrictions on the license of any data or software on disks processed by the converter, nor does processing disks through the converter impose any licensing restrictions on the disk image or data/software within that disk image.