Data file format and compatibility

[PaulFreund]

Introduction (V1)

I'd like to start a conversation around the data files on the SD card. This is mainly intended for developers of the application and anyone that directly works with the XML file contents currently found on the deluge.

Situation

Here are a few details about the thought process and where we are currently:

• Currently the official firmware and the community firmware are still mostly using the same file format with the same attributes.
• The official firmware has a mechanism that prevents older firmware to load newer files because it might not understand them properly
• There is a high chance that users will want to switch between official and community firmware, especially to have a known situation for live performances while the community firmware matures
• Adding new attributes in community will make the files incompatible with official. Either because it is blocked because of the newer firmware name written in the files or by discarding any new attributes that were unknown if the firmware version in the file is changed back to make it seemingly compatible.
• As a background the current official and community firmware are using XML reading and writing routines that do not store unknown attributes and that can refuse to work if unknown XML is found (only a very small subet is supported)

Requirements

I see the following requirements:

• Users must be informed about how the two firmware interact with each other and should see it before downloading the firmware if possible
• Users must be able to switch between community and official firmware without crashing and without destroying any files
• Users must be able to import their existing files
• Users should be able to export their files to be compatible with official firmware knowing that they might lose unsupported features
• 3rd party developers must be able and mostly be willing to work with the new standard we come up with going forward

Proposed solution

• Add an Import/Export submenu to the main settings menu (Shift+Selection encoder click) that allows importing all and exporting all previously known attributes from the /to the root of the SD card
• All new configuration files are stored in a subfolder of the SD card (e.g. /deluge/) to make sure they are not mixed up
• Switch to the JSON format for all files. Rationales are:
  • There are very good embedded JSON libraries like cJSON
  • MIDI2.0 includes JSON support so runtime objects could be serialized the same for transport and storage
  • A lot of 3rd party Deluge applications use JSON internally (downrush converts XML to JSON, delugian and Delurg and some others are based on web technology and interface very well with JSON
• Put the type in the file extension, eg. ...song.json, ...kit.json, ...synth.json

I dont' feel strongly about the proposal so feel free to suggest something else and I also don't know if I get to work on it myself so please step in if you are interrested but I felt it should be written down and also linked to all the 3rd party app creators to see what they have in mind.

[weavermedia]

I feel like a move to JSON would certainly help third-party app developers.

I just created a very simple kit maker app in Ruby. I really didn't want to bring in something quite heavy like Nokogiri for parsing and working with the XML.I ended up just doing simple string substitutions on the template XML. Not ideal but faster and easier than relying on a heavy gem.

If the files were JSON they would be very easy to work with in many ways - creation, manipulation, reading/writing, etc. and avoid the need to convert to/from JSON internally.

[jamiefaye]

A quick look around the storageManager and the calls made to same show places in the code where unknown attributes or tags are tolerated. There are also places where syntax errors and unbalanced tags cause the system to go into __builtin_unreachable();

It would be nice if unknown attributes and tags (to a particular version of our firmware) did not go to __builtin_unreachable() if the XML syntax is correct. Instead we should either just ignore the attributes or tags, or if they are deemed important enough, they can have default values.

Figuring out if there are unexpected pathways to __builtin_unreachable() caused by missing or unknown attributes and tags will likely require a systematic review of the code we have. Usually the code just skips to the next matching closing tag or past the attribute value without freaking out.

I call this "The Show Must Go On" policy. It won't preserve all values across round-trips between versions, but hopefully won't leave somebody stuck.

[jamiefaye]

I just ran a test where I sprinkled a few extra attributes and tags in a test file and tried loading that into the current community edition. It seemed to load OK.