📖 Create architecture documentation to support Airflow upgrade

[simon-pope]

User Story

As an AP team member upgrading Airflow
I need complete and agreed technical documentation for the upgrade
So that I know what is in scope for the upgrade and how it will be carried out

Value / Purpose

Technical documentation will provide the opportunity for the team to review and clarify how the Airflow upgrade will be completed
Technical documentation will allow for a clear view on what is in scope of this work, and what is not
Technical documentation will help breakdown the tasks in to sprint ready user stories

Useful Contacts

No response

User Types

No response

Hypothesis

If we have complete and signed off technical documentation for the Airflow Upgrade
Then we will improve our understanding within the team of how this will be achieved, and minimise the risks of rework and misunderstandings

Proposal

This story will cover the creation and sign off of an architecture document to support the Airflow upgrade from Airflow 2.4.3 to 2.10.x
This will include:

• overview of the upgrade process including tasks, dependencies and indicative overall timeline
• using SSO to provide CP with identity within Airflow
• logging, monitoring and observability (both user facing and platform)
• identify which tasks do not have any dependencies on users
• exploration of opportunities within this migration to improve future work etc
• agreement on which areas of the wider Airflow work is in scope, and what is out of scope, and a proposal/exploration on how out of scope could be delivered

Additional Information

Work complete by @jacobwoffenden - 🧪 initial local development #4

Feature Request to be included as requirement in this epic: Enable MWAA to read ECR in data-engineering-prod #5645

Definition of Done

• Documentation provides a comprehensive overview of upgrade process
• Documentation has been reviewed by relevant team members
• Tasks/activities that are dependent on users are identifiable
• Testing method, both internal and involving users, is agreed
• Communication required to support users is understood

[jacobwoffenden]

First pass HLD https://drive.google.com/file/d/15hzBjxqN-M3cqp4NlVYUVoESgGrzG0qG/view?usp=drive_link

[jacobwoffenden]

Request for shared services account ministryofjustice/modernisation-platform#8621