Dp 2977

[murdo-moj]

• Copy curated data in s3 between versions upon table deletion

[MatMoore]

I thought this wasn't needed because it was already handled by the athena query in CuratedDataCopier.

Rather than copy the all the parquet files in s3 we have an unload query that creates the new parquet from the latest load timestamp in the previous version. Am I misunderstanding how this works?

           UNLOAD (
                SELECT
                    *
                FROM {previous_major_database}.{curated_table.name}
                WHERE load_timestamp = (
                    SELECT MAX(load_timestamp)
                    FROM {previous_major_database}.{curated_table.name}
                )
            )
            TO '{self.table_path}'
            WITH(
                format='parquet',
                compression = 'SNAPPY',
                partitioned_by=ARRAY['load_timestamp']
            )

trigger from https://github.com/ministryofjustice/data-platform/blob/main/containers/daap-python-base/src/var/task/curated_data/curated_data_loader.py#L215

[github-actions]

This pull reuest is being marked as stale because it has been open for 30 days with no activity. Remove stale label or comment to keep the pull reuest open.

[github-actions]

This pull reuest is being closed because it has been open for a further 7 days with no activity. If this is still a valid pull reuest, please reopen it, Thank you!