Replace Athena load table partition refresh function

[LavMatt]

This currently uses a MSCK REPAIR TABLE sql query, in function refresh_table_partitions(), which is quick and easy, but it is not efficient and will become noticeably slower as and when the number of partitions increase. Also only adds new partitions, rather than removing any that don’t exist anymore.

Definition of done

• Implement more efficient function to update table partitions

[LavMatt]

Moving this back to the backlog for now as has some added complexities and will be better addressed when the MVP is more mature. See confluence writeup for further details https://dsdmoj.atlassian.net/wiki/spaces/DataPlatform/pages/4490035348/Data+Product+-+table+partition+refreshes

[github-actions]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.

[github-actions]

This issue is being closed because it has been open for a further 7 days with no activity. If this is still a valid issue, please reopen it, Thank you!