SQL init commands / duckdb version compatibility

[jwhitaker-gridcog]

Hey, sort of continuing in a tangent from #1059 :
I can successfully specify an existing duckdb database file in my malloy connection, however when I try to "connect" to it, I get an error.
This error is because my local duckdb database is from duckdb 0.10, and the version bundled with malloy is 0.9.2.

In general DuckDB note that they don't try to keep disk format compatibility between versions. However, this means that my local workflow would be continually locked to whatever version malloy bundles, which I have no control over. This wouldn't be a good workflow.

DuckDB instead recommend that to manage differing versions, I should run 'export database ./dump' from my local project, and then run import database ./dump from the target duckdb (here, malloy).

Does Malloy give me any way to do this? I can run a SQL select clause as part of a source but this seems ill-suited for this sort of once-off init command. I guess it's similar to wanting to run a create table as as part of some init process, which is something I think I've seen come up in issues/discussions as well.

[whscullin]

As of 0.10.0, duckdb is promising backwards compatibility. There are a few things blocking us updating to 0.10.0 (MotherDuck support primarily), but ideally this issue will be resolved once we can update to 0.10.0

[jwhitaker-gridcog]

they don't promise forwards compat though, right?

For future DuckDB versions, our goal is to ensure that any DuckDB version released after can read files created by previous versions, starting from this release.

Forward compatibility is provided on a best effort basis. While stability of the storage format is important – there are still many improvements and innovations that we want to make to the storage format in the future. As such, forward compatibility may be (partially) broken on occasion.

Would this mean I will sometimes be unable to use a newer version of DuckDB than what's bundled in Malloy?