Problem with using Airflow chart 8.7.1 with DARE-Airflow base container apache/airflow:2.7.2-python3.10

[mikej888]

I updated the DARE-Teleport chart version from 1.10.1 to 1.13.6. This meant I got an updated Airflow configuration from charts/teleport/values.yaml:

appAirflow:
  enabled: true
  cluster: ""
  namespace: airflow
  source:
    repoURL: https://airflow-helm.github.io/charts
    chart: airflow
    # renovate: datasource=helm depName=airflow registryUrl=https://airflow-helm.github.io/charts
    targetRevision: 8.7.1

to 1.13.6 charts/teleport/values.yaml:

appAirflow:
  enabled: true
  cluster: ""
  namespace: airflow
  source:
    repoURL: https://airflow-helm.github.io/charts
    chart: airflow
    # renovate: datasource=helm depName=airflow registryUrl=https://airflow-helm.github.io/charts
    targetRevision: 8.7.1
  values:
    airflow:
      image:
        repository: harbor.ukserp.ac.uk/dare/airflow/base
        tag: 1.1.3

On deployment via ArgoCD, the airflow app toggled between Healthy and Progressing. There was a problem with a pod airflow-sync-users-789b8c9b64-5vb6t. EVENTS recorded:

Back-off restarting failed container sync-airflow-users in pod airflow-sync-users-789b8c9b64-m6xwp_dare-airflow(b003810c-711a-437a-9bec-04b4fc7c705b)

LOGS recorded:

/home/airflow/.local/lib/python3.10/site-packages/flask_limiter/extension.py:336 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
Traceback (most recent call last):
  File "/mnt/scripts/sync_users.py", line 146, in <module>
    from airflow.www.fab_security.sqla.models import User, Role
ModuleNotFoundError: No module named 'airflow.www.fab_security.sqla.models'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/scripts/sync_users.py", line 148, in <module>
    from flask_appbuilder.security.sqla.models import User, Role
  File "/home/airflow/.local/lib/python3.10/site-packages/flask_appbuilder/security/sqla/models.py", line 48, in <module>    assoc_permissionview_role = Table(  File "<string>", line 2, in __new__
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
    return fn(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 596, in __new__
    raise exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Table 'ab_permission_view_role' is already defined for this MetaData instance.  Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

The problematic pod cited image harbor.ukserp.ac.uk/dare/airflow/base:1.1.3 so the problem is the container which wasn't previously specified in values.

[mikej888]

Within DARE-Airflow, tag base-1.1.3 has containers/airflow/base/Dockerfile:

FROM apache/airflow:2.7.2-python3.10 as build
...

A change from the use of base image apache/airflow:2.6.3-python3.10 to apache/airflow:2.7.2-python3.10 arose between tag base-1.1.2 and base-1.1.3.

apache/airflow issue Error deploying Airflow Helm Chart 2.7.0-python3.8: sync-airflow-users pod issue with 'ab_permission_view_role' is already defined #33635 explains the problem relates to Airflow 2.7.0 changes and cites a fix in airflow-helm/charts as pull request fix: sync-users for airflow 2.7.0 #772 summarised as "Updates this import path to be Airflow 2.7.0-friendly". This pull request was merged on 29/08/23 and is present in release 8.8.0.

Updating the https://airflow-helm.github.io/charts airflow chart 8.7.1 version to 8.8.0 means that this problem no longer arises.