🌆 Retire airflow-dev EKS

[jacobwoffenden]

User Story

As a…
I need/want/expect to…
So that…

Value / Purpose

No response

Useful Contacts

No response

User Types

No response

Hypothesis

If we... [do a thing]
Then... [this will happen]

Proposal

No response

Additional Information

https://github.com/ministryofjustice/analytical-platform/tree/main/terraform/aws/analytical-platform-data-production/airflow

Definition of Done

Example - [ ] Documentation has been written / updated

• README has been updated
• User docs have been updated
• Another team member has reviewed
• Tests are green

[BrianEllwood]

Steps to take - is this whats needed

1. Delete all pods
2. Drain nodes
3. Delete deployments
4. Kubectl delete -f manifest-file ? where is the manifest file ?
5. Remove all code referencing dev from files under analytical-platform/terraform/aws/analytical-platform-data-production/airflow
6. Check terraform plan and if as expected apply

Do we need to back anything up ?

[jacobwoffenden]

Steps to take - is this whats needed

1. Delete all pods
2. Drain nodes
3. Delete deployments
4. Kubectl delete -f manifest-file ? where is the manifest file ?
5. Remove all code referencing dev from files under analytical-platform/terraform/aws/analytical-platform-data-production/airflow
6. Check terraform plan and if as expected apply

Do we need to back anything up ?

you can probably skip 1 through 4 because deleting the cluster will clean all that up anyway

regarding step 5, only anything related to EKS

finally, I would post some comms telling people they need to migrate their dev workloads because we're going to shut it down next week

[BrianEllwood]

Created a draft pr above deleting as much code/objects as i could for the Dev EKS cluster as i could without getting any errors, it was suggested to do the delete in phases.

I have also posted a scheduled maintenance notice as follows

On 20/01/25 we will be shutting down the the Analytical Platform Airflow Dev Cluster, please ensure you migrate any Dev workloads prior to this date. See user guidance -https://user-guidance.analytical-platform.service.justice.gov.uk/tools/airflow/migration.html#migration for further information.

[BrianEllwood]

These are the 17 19 objects in the plan to be destroyed

aws_eks_node_group.dev_node_group_high_memory will be destroyed
aws_eks_node_group.dev_node_group_standard will be destroyed
aws_iam_openid_connect_provider.airflow_dev will be destroyed
aws_security_group.airflow_dev_cluster_additional_security_group will be destroyed
aws_security_group.airflow_dev_cluster_node_security_group will be destroyed
helm_release.kyverno_dev will be destroyed
kubectl_manifest.kyverno_policy_disallow_escalation_dev will be destroyed
kubectl_manifest.kyverno_policy_run_as_non_root_dev will be destroyed
kubectl_manifest.kyverno_policy_run_as_non_root_user_dev will be destroyed
kubernetes_config_map.dev_aws_auth_configmap will be destroyed
kubernetes_namespace.cluster_autoscaler_system will be destroyed
kubernetes_namespace.dev_airflow will be destroyed
kubernetes_namespace.dev_kube2iam will be destroyed
kubernetes_namespace.kyverno_dev will be destroyed
kubernetes_service_account.dev_airflow will be destroyed
module.airflow_dev_monitoring_iam_role.aws_iam_role.this[0] will be destroyed
module.airflow_dev_monitoring_iam_role.aws_iam_role_policy_attachment.this["policy"] will be destroyed

aws_iam_role.airflow_dev_cluster_autoscaler_role will be destroyed
aws_launch_template.dev_high_memory will be destroyed

[jacobwoffenden]

Not really blocked but pending 20/01/25 for shutdown