Implementing virtual memory on cassette tape

I'm working on a retro computing project, a 6502 Forth implementation for the Commodore PET 2001. https://github.com/chitselb/pettil if you're curious. The goal is for the language to be fast, tight, and capable of running on the actual hardware. For development I'm using the viceteam.org PET emulator with the xa65 cross-assembler, on Linux.

Since most of us back then (1980) didn't have disk drives, I am going to use the cassette tape for mass storage. These are a few ways I'm considering:

1) Simulate random access using two cassette decks and copy/merge

The PET cassette had two file types, sequential(data) and program.
a) For program files, there's a long tone followed by a short header block containing the filename, and then a shorter tone followed by one continuousblock of memory (two byte load address followed by the data)
b) for data files, there's the same long tone/file name header, followed byzero or more short tone/192-byte data blocks

On the PET (not the VIC-20 or C=64) there were two datassette ports, and I have two drives. Using the sequential file format and both decks, FLUSH would copy the entire virtual memory from one tape to the other in 1024-byte blocks (preceded by a 16-bit unsigned block number), inserting and replacing blocks from the memory buffers. Then rewind both tapes and go the other way. Slow, tedious, cumbersome. Welcome to my world in 1980.

2) Historically accurate

Some Forth implementations from back then implemented tape storage. I havebeen unable to locate one for the PET but yesterday I found tape images for Datatronic Forth on the C=64 and another thing called "C=64 Forth". Both of these appear to implement some type of mass storage on tape.

I'd be very interested to know what other Forth implementations of that eradid as far as tape storage. What Forth words, what did they do, etc...

3) Save source code as sequential files

Using native named files instead of blocks. Not very Forth-like, but possibly the most expedient.

I'm very grateful for the help of this community with my earlier design considerations (circa 2010) on this project, particularly the hashed dictionary and the incredibly fast inner interpreter. Check the project link above if you're curious to see how those parts turned out.

Charlie

chitselb <chitselb@gmail.com> wrote:

> I'm working on a retro computing project, a 6502 Forth
> implementation for the Commodore PET 2001.
> https://github.com/chitselb/pettil if you're curious. The goal is
> for the language to be fast, tight, and capable of running on the
> actual hardware. For development I'm using the viceteam.org PET
> emulator with the xa65 cross-assembler, on Linux.
>
> Since most of us back then (1980) didn't have disk drives, I am
> going to use the cassette tape for mass storage.

I've ported a Forth for PET.

> These are a few ways I'm considering:
>
> 1) Simulate random access using two cassette decks and copy/merge
>
> The PET cassette had two file types, sequential(data) and program.
> a) For program files, there's a long tone followed by a short header
> block containing the filename, and then a shorter tone followed by
> one continuous block of memory (two byte load address followed by
> the data)
> b) for data files, there's the same long tone/file name header,
> followed by zero or more short tone/192-byte data blocks
>
> On the PET (not the VIC-20 or C=64) there were two datassette ports,
> and I have two drives. Using the sequential file format and both
> decks, FLUSH would copy the entire virtual memory from one tape to
> the other in 1024-byte blocks (preceded by a 16-bit unsigned block
> number), inserting and replacing blocks from the memory buffers.
> Then rewind both tapes and go the other way. Slow, tedious,
> cumbersome. Welcome to my world in 1980.

God, no. I used floppies at the time, and they worked OK. Cassette
storage and Forth do not work at all well together.

Andrew.

chitselb <chitselb@gmail.com> writes:
>I'm working on a retro computing project, a 6502 Forth implementation for t=
>he Commodore PET 2001. https://github.com/chitselb/pettil if you're curiou=
>s. The goal is for the language to be fast, tight, and capable of running =
>on the actual hardware. For development I'm using the viceteam.org PET emul=
>ator with the xa65 cross-assembler, on Linux.
>
>Since most of us back then (1980) didn't have disk drives, I am going to us=
>e the cassette tape for mass storage.

On my C64, I used the datasette for half a year, then got a floppy
drive. The datasette was not working reliably in the end anyway.

>1) Simulate random access using two cassette decks and copy/merge

The datasette was already unbearably slow with sequential access,
simulated random access would be even slower.

>3) Save source code as sequential files
>
>Using native named files instead of blocks. Not very Forth-like, but possi=
>bly the most expedient.

Blocks were designed for disk drives. They are not very appropriate
for the datasette. Using named files where the underlying system
provides them seems more Forth-like to me than using disk-inspired
blocks on a tape-based system.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2012: http://www.euroforth.org/ef12/

On Aug 7, 2:21*pm, chitselb <chits...@gmail.com> wrote:
> I'm working on a retro computing project, a 6502 Forth implementation forthe Commodore PET 2001. *https://github.com/chitselb/pettilif you're curious. *The goal is for the language to be fast, tight, and capable of running on the actual hardware. For development I'm using the viceteam.org PETemulator with the xa65 cross-assembler, on Linux.
>
> Since most of us back then (1980) didn't have disk drives, I am going to use the cassette tape for mass storage. *These are a few ways I'm considering:
>
> 1) Simulate random access using two cassette decks and copy/merge
>
> The PET cassette had two file types, sequential(data) and program.
> a) For program files, there's a long tone followed by a short header block containing the filename, and then a shorter tone followed by one continuous block of memory (two byte load address followed by the data)
> b) for data files, there's the same long tone/file name header, followed by zero or more short tone/192-byte data blocks
>
> On the PET (not the VIC-20 or C=64) there were two datassette ports, and I have two drives. *Using the sequential file format and both decks, FLUSH would copy the entire virtual memory from one tape to the other in 1024-byte blocks (preceded by a 16-bit unsigned block number), inserting and replacing blocks from the memory buffers. *Then rewind both tapes and go the other way. *Slow, tedious, cumbersome. *Welcome to my world in 1980.
>
> 2) Historically accurate
>
> Some Forth implementations from back then implemented tape storage. *I have been unable to locate one for the PET but yesterday I found tape images for Datatronic Forth on the C=64 and another thing called "C=64 Forth". Both of these appear to implement some type of mass storage on tape.
>
> I'd be very interested to know what other Forth implementations of that era did as far as tape storage. *What Forth words, what did they do, etc....
>
> 3) Save source code as sequential files
>
> Using native named files instead of blocks. *Not very Forth-like, but possibly the most expedient.
>
> I'm very grateful for the help of this community with my earlier design considerations (circa 2010) on this project, particularly the hashed dictionary and the incredibly fast inner interpreter. *Check the project link above if you're curious to see how those parts turned out.
>
> Charlie

Hmm... This sounds like a really bad idea to me! Sorry to sound so
negative!

If I were FORCED and GUN-POINT to implement a block storage on
cassette, I would take the Forth approach and keep it very simple...

"You want an 80 block cassette based block system? Okay, give me 80 5
minute cassette tapes. One tape per block. Screen prompts will tell
you which cassette to put in the tape deck. Now, please stop pointing
that gun at me."

On Tue, 7 Aug 2012 06:21:33 -0700 (PDT), chitselb <chitselb@gmail.com> wrote:
>
> I'd be very interested to know what other Forth implementations of that era
> did as far as tape storage. What Forth words, what did they do, etc...

I ran a converted ZX81 with a Skywave Forth ROM for a while. I'll try
and find the manual and see how it worked.

Skywave Forth was a multi-tasking, windowing Forth. Multiple
applications running in seperate windows on a ZX-81 was something to
behold! Still got the computer and it still works, except the add-on
keyboard has grown a fault. (16K ram, 16 in/out lines and 8-port a2d
converter too...)

--
Cheers,
Stan Barr plan.b .at. dsl .dot. pipex .dot. com

The future was never like this!

On 7 Aug 2012 15:30:36 GMT, Stan Barr <plan.b@dsl.pipex.com> wrote:
> On Tue, 7 Aug 2012 06:21:33 -0700 (PDT), chitselb <chitselb@gmail.com> wrote:
>>
>> I'd be very interested to know what other Forth implementations of that era
>> did as far as tape storage. What Forth words, what did they do, etc...
>
> I ran a converted ZX81 with a Skywave Forth ROM for a while. I'll try
> and find the manual and see how it worked.
>
> Skywave Forth was a multi-tasking, windowing Forth. Multiple
> applications running in seperate windows on a ZX-81 was something to
> behold! Still got the computer and it still works, except the add-on
> keyboard has grown a fault. (16K ram, 16 in/out lines and 8-port a2d
> converter too...)
>

Couldn't locate the manual, but see here...

http://www.dibsco.co.uk/index.php/sk...orth-general/\
forth-skywave/73

watch out for the line wrap.
--
Cheers,
Stan Barr plan.b .at. dsl .dot. pipex .dot. com

The future was never like this!

You can assume that for the sake of the simulation, that you didn't get your computer from K-Mart or Toys-R-Us and didn't have enough to pay for a disk drive so went with the tape drive. LOL

Jason Damisch <jasondamisch@yahoo.com> writes:
> You can assume that for the sake of the simulation, that you didn't
> get your computer from K-Mart or Toys-R-Us and didn't have enough to
> pay for a disk drive so went with the tape drive. LOL

The PET and microcomputer cassette interfaces were before my time but
looking at

http://en.wikipedia.org/wiki/Datassette

was interesting. It was apparently a standard cassette recorder with
some adc/dac's and a special edge connector. In particular the article
doesn't say whether the computer could start and stop the tape transport
or rewind the tape under software control. Do you know if that was
possible?

If it wasn't, I doubt think the datasette was likely usable for much
other than a program loader. In particular, the usual things one did
with magtape, such as merge sorting, required controlling the transport.
Merge sorting involved reading blocks of data from 2 drives, merging
them, and writing the output to a third drive. That meant you had to
write to the output drive twice as fast as you were reading from the
individual input drives, so you had to control the drive speed or be
able to start and stop the tape when you read a block.

I did see some 9-track magtape drives back in the day and they had
powerful motors and tape slack mechanisms to be able to start and stop
the tape quickly. I don't think audio recorders were built anything
like that.

Paul Rubin <no.email@nospam.invalid> writes:
> Merge sorting involved reading blocks of data from 2 drives, merging
> them, and writing the output to a third drive. That meant you had to
> write to the output drive twice as fast as you were reading from the
> individual input drives,

Should have added, of course there were fancier methods with more than 3
drives. Maybe there were ways to do it with 4 drives, reading from two
and simultaneously writing to two. A lot of Knuth vol 3 is about how to
do stuff like this. The most recent edition says something like: nobody
uses those methods any more, but they're still in the book just in case
they come back into use someday, and because the methods were so
interesting that it's worth preserving the info.

Paul Rubin wrote:
> The most recent edition says something like:
> nobody uses those methods any more, but they're still in the book just
> in case they come back into use someday, and because the methods were
> so interesting that it's worth preserving the info.

It should be noted that current hard disks already have somewhat similar
performance to these old tape drives: In the time you can do *one* seek,
you can also read/write a megabyte of data.

--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://bernd-paysan.de/

Paul Rubin <no.email@nospam.invalid> writes:
>The PET and microcomputer cassette interfaces were before my time but
>looking at
>
> http://en.wikipedia.org/wiki/Datassette
>
>was interesting. It was apparently a standard cassette recorder with
>some adc/dac's and a special edge connector. In particular the article
>doesn't say whether the computer could start and stop the tape transport
>or rewind the tape under software control. Do you know if that was
>possible?

IIRC it was not.

- anton
--
M. Anton Ertl http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
New standard: http://www.forth200x.org/forth200x.html
EuroForth 2012: http://www.euroforth.org/ef12/

Bernd Paysan <bernd.paysan@gmx.de> writes:
> It should be noted that current hard disks already have somewhat similar
> performance to these old tape drives: In the time you can do *one* seek,
> you can also read/write a megabyte of data.

There is a more recent saying, "flash is disk, disk is tape, tape is
dead". I think disk is still "disk-like" in important ways, though. At
least on serious tape drives, "seeking" (spinning) a tape from one end
to the other was at most a few times faster than actually reading the
whole tape. Seeking across a disk is many orders of magnitude faster
than reading the disk.

Also, you can do quite a lot of disk seeks in the time it would take to
fill memory from disk transfers, on today's computers with gigabytes of
ram. That means you can treat a disk as "multiple" tape drives for
external sort/merge: in the merge phase you'd read a few hundred
contiguous megabytes of data into ram, then seek to antoher run
elsewhere on the disk, read that into ram, do the same with several more
runs until ram is full, then merge from ram to a new combined run on the
disk.

I saw in some storage newsletter earlier today that tape is essentially
not used any more even for backup, which surprised me. It's apparently
only used for massive archiving (I guess meaning long term retention of
data that probably won't be accessed anytime soon, as distinct from
backups which are for short term use in case a disk crashes).

On Aug 7, 8:39*pm, Paul Rubin <no.em...@nospam.invalid> wrote:
> Jason Damisch <jasondami...@yahoo.com> writes:
> > You can assume that for the sake of the simulation, that you didn't
> > get your computer from K-Mart or Toys-R-Us and didn't have enough to
> > pay for a disk drive so went with the tape drive. LOL
>
> The PET and microcomputer cassette interfaces were before my time but
> looking at
>
> *http://en.wikipedia.org/wiki/Datassette
>
> was interesting. *It was apparently a standard cassette recorder with
> some adc/dac's and a special edge connector. *In particular the article
> doesn't say whether the computer could start and stop the tape transport
> or rewind the tape under software control. *Do you know if that was
> possible?
>
> If it wasn't, I doubt think the datasette was likely usable for much
> other than a program loader. *In particular, the usual things one did
> with magtape, such as merge sorting, required controlling the transport.
> Merge sorting involved reading blocks of data from 2 drives, merging
> them, and writing the output to a third drive. *That meant you had to
> write to the output drive twice as fast as you were reading from the
> individual input drives, so you had to control the drive speed or be
> able to start and stop the tape when you read a block.
>
> I did see some 9-track magtape drives back in the day and they had
> powerful motors and tape slack mechanisms to be able to start and stop
> the tape quickly. *I don't think audio recorders were built anything
> like that.

It could start and stop the tape, but nothing else. No forward and
rewing via software. That had to be done by a human. Loading from
cassette tape was a big part of my computing life as a young child. I
shudder when I think of it now!

On Aug 8, 8:27*am, Paul Rubin <no.em...@nospam.invalid> wrote:
> Bernd Paysan <bernd.pay...@gmx.de> writes:
> > It should be noted that current hard disks already have somewhat similar
> > performance to these old tape drives: In the time you can do *one* seek,
> > you can also read/write a megabyte of data.
>
> There is a more recent saying, "flash is disk, disk is tape, tape is
> dead". *I think disk is still "disk-like" in important ways, though. *At
> least on serious tape drives, "seeking" (spinning) a tape from one end
> to the other was at most a few times faster than actually reading the
> whole tape. *Seeking across a disk is many orders of magnitude faster
> than reading the disk.
>
> Also, you can do quite a lot of disk seeks in the time it would take to
> fill memory from disk transfers, on today's computers with gigabytes of
> ram. *That means you can treat a disk as "multiple" tape drives for
> external sort/merge: in the merge phase you'd read a few hundred
> contiguous megabytes of data into ram, then seek to antoher run
> elsewhere on the disk, read that into ram, do the same with several more
> runs until ram is full, then merge from ram to a new combined run on the
> disk.
>
> I saw in some storage newsletter earlier today that tape is essentially
> not used any more even for backup, which surprised me. *It's apparently
> only used for massive archiving (I guess meaning long term retention of
> data that probably won't be accessed anytime soon, as distinct from
> backups which are for short term use in case a disk crashes).

That's interesting. Is there any evidence to suggest that tape is less
susceptible to bit-rot than say, flash?

On Aug 8, 8:27*am, Paul Rubin <no.em...@nospam.invalid> wrote:
> Bernd Paysan <bernd.pay...@gmx.de> writes:
> > It should be noted that current hard disks already have somewhat similar
> > performance to these old tape drives: In the time you can do *one* seek,
> > you can also read/write a megabyte of data.
>
> There is a more recent saying, "flash is disk, disk is tape, tape is
> dead". *I think disk is still "disk-like" in important ways, though. *At
> least on serious tape drives, "seeking" (spinning) a tape from one end
> to the other was at most a few times faster than actually reading the
> whole tape. *Seeking across a disk is many orders of magnitude faster
> than reading the disk.

I think you may have that back to front?

>
> Also, you can do quite a lot of disk seeks in the time it would take to
> fill memory from disk transfers, on today's computers with gigabytes of
> ram.

Err, no. Disk seeks are to be avoided, as they are horrendously time
expensive; they're measured in low milliseconds at best. The bandwidth
still lags main memory on data transfer, but it's not far behind;
arrays that can transfer 10s to 100s of GBytes/sec (that's bytes, not
bits) are commonplace.

> That means you can treat a disk as "multiple" tape drives for
> external sort/merge: in the merge phase you'd read a few hundred
> contiguous megabytes of data into ram, then seek to antoher run
> elsewhere on the disk, read that into ram, do the same with several more
> runs until ram is full, then merge from ram to a new combined run on the
> disk.
>

It's a long time since I've seen a merge sort as you describe here.
It's bottlenecked on the write, and isn't practical. Building an index
is cheaper and provides the needed parallelism.

> I saw in some storage newsletter earlier today that tape is essentially
> not used any more even for backup, which surprised me. *It's apparently
> only used for massive archiving (I guess meaning long term retention of
> data that probably won't be accessed anytime soon, as distinct from
> backups which are for short term use in case a disk crashes).

Tape is not dead; far from it. It has some distinct advantages over
other media, foremost of which is a very low cost per bit and no power
requirements when not being accessed (ignoring disk spin down, which
in contrast is pretty much a dead technology). When the archive has to
be "deep and cold", tape wins. Disk costs are just about bearable for
medium term storage. Flash memory technologies are just too expensive
for petabyte scale backup and archive. There are new tape file systems
being developed; if you're interested, take a look at LTFS.