AWS EKS Jobs not completing

[SanjayVas]

Many Jobs in EKS are not entering the completed state. For example, the Pod for a Mill Job will complete its work but the Job status will still indicate that there is 1 active and 0 completed Pods. These stuck Jobs will eventually fill up all of the concurrency slots and stop computatations from progressing at that Duchy.

The current workaround for Halo test environments is to delete the stuck Jobs manually.

[SanjayVas]

The current plan of record for dealing with this is to remove the AWS Duchy from Halo test environments. The code will continue to be exercised via in-process integration tests, but the AWS Duchy will be considered unsupported until/unless we can prioritize the work to maintain it.