[HUDI-5769] Ensure partitions created by async indexer are not deleted by regular writers

[codope]

Change Logs

Ensure partitions created by async indexer are not deleted by regular writers.

• Add a new metadata config to declare what metadata partitions to delete.
• Fix the condition for checking whether MDT partition needs to be deleted in HoodieTable
• Add a test in TestHoodieIndexer

Let's understand the interplay between index enable/disable boolean flag and the new config. For illustration prupose, we will assume, files and column_statsare enabled by ingestion writer, andrecord_index` is created using async indexer.

Existing boolean configs: hoodie.metadata.enable, hoodie.metadata.index.column.stats.enable and hoodie.metadata.record.index.enable.
New string config: hoodie.metadata.index.drop (this takes the index name/MDT partition name as value)

1. Regular ingestion writer has hoodie.metadata.enable and hoodie.metadata.index.column.stats.enable set to true. And, files and column_stats index are present.
2. Async indexer starts with setting only hoodie.metadata.enable and hoodie.metadata.record.index.enable to true. It checks the available partitions and automatically sets the columns stats config to true as well -

   hudi/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieIndexer.java

   Line 182 in c049e9b

   props.setProperty(ENABLE_METADATA_INDEX_COLUMN_STATS.key(), "true");

3. Indexer continues to build the record_index and that is now part of inflight metadata partitions in table config.
4. Ingestion writer continue to log updates for files and column_stats, while indexer catches up with ingestion writer commit. Note that at this point, regular ingestion writer does not set hoodie.metadata.index.drop config, so even though the hoodie.metadata.record.index.enable is false in ingestion writer config, the record_index is not deleted.
5. Indexer has caught up. Ingestion writer notices that in addition to files and column_stats, we have record_index fully available (not inflight) in the table config. It logs updates for all 3 indexes now.
6. In case users want to delete record_index in future, they need to set not only hoodie.metadata.record.index.enable=false but also hoodie.metadata.index.drop=record_index.

To summarise, there is a boolean flag for enabling.disabling an index, and there is a string config to explicitly specify what index to delete. The interplay between these two decides finally, whether index needs to be disabled/dropped or not.

(boolean, string) -> enabled
true, null or non-matching -> true
false, null or no-matching -> true
true, record_index -> true
false, record_index -> false

Impact

Ensure partitions created by async indexer are not deleted by regular writers.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[nsivabalan]

thanks for the PR desc, it helps.

btw, for bullet 4 in your PR desc

4. Ingestion writer continue to log updates for files and column_stats, while indexer catches up with ingestion writer commit. Note that at this point, regular ingestion writer does not set hoodie.metadata.index.drop config, so even though the hoodie.metadata.record.index.enable is false in ingestion writer config, the record_index is not deleted.

woudn't the ingestion writer also log updates for RLI, since the table config's inflight metadata partitions will say RLI.

[nsivabalan]

can you create a follow up ticket for doc updates. This def needs to go into some runbook in our docs.

[nsivabalan]

guess this was not the design.
the regular ingestion writer should log updates for partitions fully available and for partitions that are inflight.
if not, the async indexer has to do full catch up. The catch up is meant just for a race condition to ensure we do not miss any updates, but in most cases, the catch up phase should not have any work to do.
lets discuss if our understanding is different

[nsivabalan]

:) I also had to fix this in my partition stats patch.

[hudi-bot]

CI report:

• 0c0d9b4 Azure: SUCCESS
• 179fc3b Azure: PENDING

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build