Implement compression for patterns

[SuperDisk]

As it stands, the contents of patterns can be fairly repetitive and not benefit from any sort of compression.

There are a couple solutions I've been thinking about:

• Remove the restriction that patterns need to be 64 rows, and let patterns be any size, with some sort of terminator at the end to signify when it's over. This would allow musicians to write short phrases, and repeat them with only 2 bytes in the order table. Basslines and arps would benefit greatly from this sort of thing, and it's how AHX and Klystrack do it, I think. We could even just do pattern recognition/repetition logic in the tracker code itself so the user doesn't have to think about song size and can just focus on using it like a MOD tracker (4x64 patterns with lots of repetition)

• Leave the 64 row restriction in place and simply do pattern repetition recognition in the tracker code, then implement "loop points," similar to ProTracker's E6x command. There's a lot of unused note constant values, since only 71 are used and there's a range up to 255.

I'm partial to the first option since I don't really like the idea of having 2 "methods" of breaking up music (patterns/loop areas), and the second option would require some more logic to handle looping in each channel, whereas the first option fits pretty nicely in with the current structure of the code.

To be honest, this can probably be put off until after the first release or whatever since what currently exists is (afaik) better size-wise than everything out there already.

[RichardULZ]

If it's just for compression sake,

• There's the GameBoyTracker method as we talked about, which has 1, 2, or 3 bytes of data, jump commands likely have to catch up by reading bytes but not processing them up to our step, but it has the advantage of only needing 1 pointer for 4 channels.

• Decompressing into ram. Takes ram space, and causes similar cpu load as a long jump for every pattern change, on the fly decompression might spread the load out.

• RLE compression as you suggested. Not sure how far you were thinking, but

• RLE zero, would combine both methods, knowing the space between new data packets. Reducing 1 or many steps of blank data to 1 byte instead of 3+, much smaller than GBT ever could, and smaller than normal RLE that needs a length, and byte to repeat.

• Doesn't have to be run length either! You can say the exact step the next data will be for, between 0-63.

Thinking through how it might work
For jump commands, they would still need to "Catch up" By reading bytes until the pattern is at the right step, and keeping tabs on where we last read data from, could be 1 byte added to the pattern pointer.

The note byte bit7 would define whether the note/normal 3 bytes of data are here, or what step to wait for new data.
Check if the first byte has bit 7 set by subtracting 0x80, jump if we carry or not.

Jump, Has 3 bytes of data.
If we're not catching up, read the 3 bytes into ram.
If we are catching up, increment our current step counter for this channel, if we're not at the right step, skip the next 2 bytes, and load another.

Jump, Has next step with data,
If we're not catching up, store the next real step, when loading a new byte, now every step we have to check if the current step is equal to next real step, only then will we read the next byte.
If we are catching up, store the next real step, then math 'desired position' - next real step, see if it carried to determine where we are.

That all thought through out of my head, it's still debatable how slow decompression could be, as the simplicity of knowing it is always 3 bytes per step helps a lot, perhaps things will change when we start loading from other banks with pesky bank switching.

Edit: Now that i read through your ideas again, this is pretty similar to option 2 where the tracker loops through places.