Narwhal

Iceberg tables maintenance helper

Configuration

hive:
  name: hive
  uris: thrift://localhost:9083
  warehouse: /tmp
  access_key: ...
  secret_key: ...

rewrite_files:
  enabled: true
  policy:
    type: FILE_SKEW
    config: 
      target_file_size_bytes: 536870912
      file_size_skew_threshold: 0.2

  integration:
    type: SQS
    config:
      access_key: ...
      secret_key: ...
      region: eu-west-1
      queue: https://sqs.eu-west-1.amazonaws.com/<account-id>/<queue>
      batch_size: 10

Icon made by Freepik from flaticon