Support Writing Iceberg metadata

[zhousun]

What feature are you requesting?

Currently pg_mooncake is wring delta metadata to the object store. But it would be ideal to also write iceberg version.

Why are you requesting this feature?

Fit better into the iceberg ecosystem.

What is your proposed implementation for this feature?

#[cxx::bridge]
mod ffi {
    extern "Rust" {
        fn DeltaInit();

        fn DeltaCreateTable(
            table_name: &CxxString,
            path: &CxxString,
            options: &CxxString,
            column_names: &CxxVector<CxxString>,
            column_types: &CxxVector<CxxString>,
        ) -> Result<()>;

        fn DeltaModifyFiles(
            path: &CxxString,
            options: &CxxString,
            file_paths: &CxxVector<CxxString>,
            file_sizes: &CxxVector<i64>,
            is_add_files: &CxxVector<i8>,
        ) -> Result<()>;
    }
}

Implement the corresponding functions for iceberg, with same semantics.
Catalog support for iceberg is more important than delta, but for MVP it is fine to just write to filesystem.

[umeshkacha]

is there any ETA for this?

[dpxcc]

It's on our shortlist, but we don't have a concrete ETA yet.

The challenges are:

1. There isn't a good library for writing Iceberg metadata, except by spinning up a JVM. We're hoping that iceberg-rust will soon make this easier so we can adopt it. Alternatively, we could manually write the metadata ourselves since we only use a small subset of the Iceberg spec, though that's not ideal.
2. Writing Iceberg metadata is more expensive than Delta Lake because it's based on snapshots. This would require batched writes and compaction to deliver a better user experience.