Can not remove AMF with error cannot destroy application "amf": application is used by 2 consumers

[gatici]

Bug Description

Deploy Charmed Ather SD-Core, all the charms were active.
Then I attached a custom image to AMF which put the charm in error error state.
Then I resolved the charm and tried to remove AMF. It failed with ERROR removing application amf failed: cannot destroy application "amf": application is used by 2 consumers.
I forced to remove it. I got the same error.
I removed all the K8s objects belongs to AMF, then tried to remove charm again.
I got the same error.
Now unit is in unknow status and charm hanged in Waiting status with a message Waiting for AMF service to start.

To Reproduce

Follow the tutorial to deploy the environment https://canonical-charmed-oai-ran.readthedocs-hosted.com/en/latest/tutorials/getting_started/

Environment

juju 3.5.5
Microk8s 1.31 classic

Relevant log output

gatici@gaticipc:~/charmed5g/sdcore-amf-operator$ juju remove-application amf 
WARNING This command will perform the following actions:
will remove application amf
- will remove unit amf/0
- will detach storage certs/20
- will detach storage config/22

Continue [y/N]? y
ERROR removing application amf failed: cannot destroy application "amf": application is used by 2 consumers

gatici@gaticipc:~/charmed5g/sdcore-amf-operator$ juju remove-application amf  --force 
WARNING This command will perform the following actions:
will remove application amf
- will remove unit amf/0
- will detach storage certs/20
- will detach storage config/22

Continue [y/N]? y
ERROR removing application amf failed: cannot destroy application "amf": application is used by 2 consumers

gatici@gaticipc:~/charmed5g/sdcore-amf-operator$ juju remove-application amf  --force --no-wait
WARNING This command will perform the following actions:
will remove application amf
- will remove unit amf/0
- will detach storage certs/20
- will detach storage config/22

Continue [y/N]? y
ERROR removing application amf failed: cannot destroy application "amf": application is used by 2 consumers

gatici@gaticipc:~/charmed5g/sdcore-amf-operator$ kubectl delete statefulset.apps/amf  service/amf-external   service/amf-endpoints  service/amf pod/amf-0   -n sdcore 
statefulset.apps "amf" deleted
service "amf-external" deleted
service "amf-endpoints" deleted
service "amf" deleted
pod "amf-0" deleted

gatici@gaticipc:~/charmed5g/sdcore-amf-operator$ juju status
Model   Controller                  Cloud/Region                Version  SLA          Timestamp
sdcore  microk8s-classic-localhost  microk8s-classic/localhost  3.5.5    unsupported  13:27:37+03:00

SAAS  Status  Store  URL
cu    active  local  admin/ran.cu

App                       Version  Status   Scale  Charm                     Channel        Rev  Address         Exposed  Message
amf                       1.5.1    waiting    0/1  sdcore-amf-k8s            1.5/stable     834  10.152.183.164  no       Waiting for AMF service to start
ausf                      1.5.1    active       1  sdcore-ausf-k8s           1.5/stable     645  10.152.183.21   no       
grafana-agent             0.40.4   blocked      1  grafana-agent-k8s         latest/stable   80  10.152.183.175  no       Missing ['grafana-cloud-config']|['logging-consumer'] for logging-provider; ['grafana-cloud-config']|['send-remote-wr...
mongodb                            active       1  mongodb-k8s               6/stable        61  10.152.183.231  no       
nms                       1.0.0    active       1  sdcore-nms-k8s            1.5/stable     741  10.152.183.90   no       
nrf                       1.5.2    active       1  sdcore-nrf-k8s            1.5/stable     720  10.152.183.53   no       
nssf                      1.5.1    active       1  sdcore-nssf-k8s           1.5/stable     614  10.152.183.152  no       
pcf                       1.5.2    active       1  sdcore-pcf-k8s            1.5/stable     650  10.152.183.195  no       
router                             active       1  sdcore-router-k8s         1.5/stable     424  10.152.183.232  no       
self-signed-certificates           active       1  self-signed-certificates  latest/stable  155  10.152.183.252  no       
smf                       1.6.2    active       1  sdcore-smf-k8s            1.5/stable     745  10.152.183.89   no       
traefik                   2.11.0   active       1  traefik-k8s               latest/stable  203  10.152.183.114  no       Serving at 10.0.0.4.nip.io
udm                       1.5.1    active       1  sdcore-udm-k8s            1.5/stable     605  10.152.183.155  no       
udr                       1.6.1    active       1  sdcore-udr-k8s            1.5/stable     597  10.152.183.140  no       
upf                       1.5.0    active       1  sdcore-upf-k8s            1.5/stable     691  10.152.183.171  no       

Unit                         Workload  Agent  Address      Ports  Message
amf/0                        unknown   lost   10.1.146.11         agent lost, see 'juju show-status-log amf/0'
ausf/0*                      active    idle   10.1.146.43         
grafana-agent/0*             blocked   idle   10.1.146.51         Missing ['grafana-cloud-config']|['logging-consumer'] for logging-provider; ['grafana-cloud-config']|['send-remote-wr...
mongodb/0*                   active    idle   10.1.146.4          Primary
nms/0*                       active    idle   10.1.146.12         
nrf/0*                       active    idle   10.1.146.55         
nssf/0*                      active    idle   10.1.146.10         
pcf/0*                       active    idle   10.1.146.29         
router/0*                    active    idle   10.1.146.20         
self-signed-certificates/0*  active    idle   10.1.146.46         
smf/0*                       active    idle   10.1.146.13         
traefik/0*                   active    idle   10.1.146.19         Serving at 10.0.0.4.nip.io
udm/0*                       active    idle   10.1.146.31         
udr/0*                       active    idle   10.1.146.48         
upf/0*                       active    idle   10.1.146.28         

Offer  Application  Charm           Rev  Connected  Endpoint  Interface  Role
amf    amf          sdcore-amf-k8s  834  1/1        fiveg-n2  fiveg_n2   provider
upf    upf          sdcore-upf-k8s  691  0/0        fiveg_n3  fiveg_n3   provider
gatici@gaticipc:~/charmed5g/sdcore-amf-operator$ juju remove-application amf  --force  --no-wait
WARNING This command will perform the following actions:
will remove application amf
- will remove unit amf/0
- will detach storage certs/20
- will detach storage config/22

Continue [y/N]? y
ERROR removing application amf failed: cannot destroy application "amf": application is used by 2 consumers

### Additional context

_No response_

[gatici]

The charm is removed after running juju remove-offer admin/sdcore.amf --force