Playbook fails due to roles using ansible-galaxy command

[myllynen]

Both filetree_create and object_diff do this:

	- name: "Check if the collection ansible.controller is installed"
          ansible.builtin.set_fact:
            ansible_controller_collection_installed: "{{ lookup('ansible.builtin.pipe',
              'ansible-galaxy collection list | grep -i ansible.controller || echo NOTINSTALLED') }}"
          failed_when: ansible_controller_collection_installed is match('NOTINSTALLED')

Ideally dependencies would be defined and thus automatically solved by defining them in the collection's galaxy.yml but here the intention seems to be supporting using either ansible.controller or awx.awx.

This causes problems when using a custom ansible.cfg which configures Ansible home or collections directory, think of for example fully isolated CI/CD runs or any kind of development/execution setup using non-default directories.

If a playbook using these roles is in a different directory than ansible.cfg, say, there is ./ansible.cfg and ./playbooks/test.yml, then running ansible-playbook ./playbooks/test.yml fails since the ansible-galaxy command invoked by the role runs in the playbooks dir, not in the current working directory where ansible-playbook is run and where ansible.cfg is located, and then fails to detect that a required collection is in fact already installed. This can also be confusing for the user when ansible-galaxy collection list shows the collection installed but the role complains the role not being installed.

Moving playbooks to the same directory would solve the issue but with repositories with lots of content keeping playbooks in a dedicated directory is helpful.

[myllynen]

Perhaps it could be considered that the user could define before use which collection to use to avoid this auto-detection and only in case not specified the role would fall-back to this current approach.

[ivarmu]

Most of times, it has no sense having more than one ansible.cfg file in the working tree, so the most easy way to avoid having troubles with this is to simply symlinking the ansible.cfg file into the playbooks directory, so you'll only have one copy of it, but it will be used from every place you need. For example, your current situation is as follows:

/tmp/ansibletest $ tree -L 2                 
.
├── ansible.cfg
├── collections
│   └── ansible_collections
├── playbooks
│   └── playbook.yaml
└── playbook.yaml

4 directories, 3 files

You can work consistently running the following commands (playbooks/ansible.cfg can also be a symbolic link, as you preffer):

/tmp/ansibletest $ cd playbooks 
/tmp/ansibletest/playbooks $ ln -s ../collections .
/tmp/ansibletest/playbooks $ ln ../ansible.cfg .
/tmp/ansibletest/playbooks $ cd ..
/tmp/ansibletest $ tree -L 2
.
├── ansible.cfg
├── collections
│   └── ansible_collections
├── playbooks
│   ├── ansible.cfg
│   ├── collections -> ../collections
│   └── playbook.yaml
└── playbook.yaml

5 directories, 4 files

That way, you'll get the same collections both from /tmp/ansibletest and from /tmp/ansibletest/playbooks:

/tmp/ansibletest $ cat playbook.yaml 
---
- hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - debug:
        msg:
          - "{{ lookup('pipe', 'ansible-galaxy collection list') }}"
    - copy:
        dest: /tmp/a.out
        content: "{{ lookup('pipe', 'ansible-galaxy collection list') }}"
...
/tmp/ansibletest $ diff playbook.yaml playbooks/playbook.yaml 
10c10
<         dest: /tmp/a.out
---
>         dest: /tmp/b.out
/tmp/ansibletest $ ansible-playbook playbook.yaml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *****************************************************************************************************************************************************************************************************************************

TASK [debug] *********************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        "\n# /usr/lib/python3.11/site-packages/ansible_collections\nCollection                    Version\n----------------------------- -------\namazon.aws                    5.5.1  \nansible.netcommon             4.1.0  \nansible.posix                 1.5.4  \nansible.utils                 2.10.3 \nansible.windows               1.14.0 \narista.eos                    6.0.1  \nawx.awx                       21.14.0\nazure.azcollection            1.16.0 \ncheck_point.mgmt              4.0.0  \nchocolatey.chocolatey         1.4.0  \ncisco.aci                     2.6.0  \ncisco.asa                     4.0.1  \ncisco.dnac                    6.7.2  \ncisco.intersight              1.0.27 \ncisco.ios                     4.6.1  \ncisco.iosxr                   4.1.0  \ncisco.ise                     2.5.12 \ncisco.meraki                  2.15.1 \ncisco.mso                     2.4.0  \ncisco.nso                     1.0.3  \ncisco.nxos                    4.4.0  \ncisco.ucs                     1.8.0  \ncloud.common                  2.1.3  \ncloudscale_ch.cloud           2.3.1  \ncommunity.aws                 5.5.0  \ncommunity.azure               2.0.0  \ncommunity.ciscosmb            1.0.6  \ncommunity.crypto              2.14.0 \ncommunity.digitalocean        1.23.0 \ncommunity.dns                 2.5.5  \ncommunity.docker              3.4.7  \ncommunity.fortios             1.0.0  \ncommunity.general             6.6.2  \ncommunity.google              1.0.0  \ncommunity.grafana             1.5.4  \ncommunity.hashi_vault         4.2.1  \ncommunity.hrobot              1.8.0  \ncommunity.libvirt             1.2.0  \ncommunity.mongodb             1.6.0  \ncommunity.mysql               3.7.2  \ncommunity.network             5.0.0  \ncommunity.okd                 2.3.0  \ncommunity.postgresql          2.4.2  \ncommunity.proxysql            1.5.1  \ncommunity.rabbitmq            1.2.3  \ncommunity.routeros            2.8.2  \ncommunity.sap                 1.0.0  \ncommunity.sap_libs            1.4.1  \ncommunity.skydive             1.0.0  \ncommunity.sops                1.6.2  \ncommunity.vmware              3.7.0  \ncommunity.windows             1.13.0 \ncommunity.zabbix              1.9.3  \ncontainers.podman             1.10.2 \ncyberark.conjur               1.2.0  \ncyberark.pas                  1.0.19 \ndellemc.enterprise_sonic      2.0.0  \ndellemc.openmanage            6.3.0  \ndellemc.os10                  1.1.1  \ndellemc.os6                   1.0.7  \ndellemc.os9                   1.0.4  \ndellemc.powerflex             1.6.0  \ndellemc.unity                 1.6.0  \nf5networks.f5_modules         1.25.0 \nfortinet.fortimanager         2.2.0  \nfortinet.fortios              2.3.0  \nfrr.frr                       2.0.2  \ngluster.gluster               1.0.2  \ngoogle.cloud                  1.1.3  \ngrafana.grafana               1.1.1  \nhetzner.hcloud                1.11.0 \nhpe.nimble                    1.1.4  \nibm.qradar                    2.1.0  \nibm.spectrum_virtualize       1.12.0 \ninfinidat.infinibox           1.3.12 \ninfoblox.nios_modules         1.5.0  \ninspur.ispim                  1.3.0  \ninspur.sm                     2.3.0  \njunipernetworks.junos         4.1.0  \nkubernetes.core               2.4.0  \nlowlydba.sqlserver            1.3.1  \nmellanox.onyx                 1.0.0  \nmicrosoft.ad                  1.2.0  \nnetapp.aws                    21.7.0 \nnetapp.azure                  21.10.0\nnetapp.cloudmanager           21.22.0\nnetapp.elementsw              21.7.0 \nnetapp.ontap                  22.7.0 \nnetapp.storagegrid            21.11.1\nnetapp.um_info                21.8.0 \nnetapp_eseries.santricity     1.4.0  \nnetbox.netbox                 3.13.0 \nngine_io.cloudstack           2.3.0  \nngine_io.exoscale             1.0.0  \nngine_io.vultr                1.1.3  \nopenstack.cloud               1.10.0 \nopenvswitch.openvswitch       2.1.1  \novirt.ovirt                   2.4.1  \npurestorage.flasharray        1.19.1 \npurestorage.flashblade        1.11.0 \npurestorage.fusion            1.5.0  \nsensu.sensu_go                1.13.2 \nsplunk.es                     2.1.0  \nt_systems_mms.icinga_director 1.33.1 \ntheforeman.foreman            3.11.0 \nvmware.vmware_rest            2.3.1  \nvultr.cloud                   1.8.0  \nvyos.vyos                     4.1.0  \nwti.remote                    1.0.5  \n\n# /tmp/ansibletest/collections/ansible_collections\nCollection        Version\n----------------- -------\ncommunity.general 8.1.0"
    ]
}

TASK [copy] **********************************************************************************************************************************************************************************************************************************
ok: [localhost]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

/tmp/ansibletest $ ansible-playbook playbooks/playbook.yaml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *****************************************************************************************************************************************************************************************************************************

TASK [debug] *********************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        "\n# /usr/lib/python3.11/site-packages/ansible_collections\nCollection                    Version\n----------------------------- -------\namazon.aws                    5.5.1  \nansible.netcommon             4.1.0  \nansible.posix                 1.5.4  \nansible.utils                 2.10.3 \nansible.windows               1.14.0 \narista.eos                    6.0.1  \nawx.awx                       21.14.0\nazure.azcollection            1.16.0 \ncheck_point.mgmt              4.0.0  \nchocolatey.chocolatey         1.4.0  \ncisco.aci                     2.6.0  \ncisco.asa                     4.0.1  \ncisco.dnac                    6.7.2  \ncisco.intersight              1.0.27 \ncisco.ios                     4.6.1  \ncisco.iosxr                   4.1.0  \ncisco.ise                     2.5.12 \ncisco.meraki                  2.15.1 \ncisco.mso                     2.4.0  \ncisco.nso                     1.0.3  \ncisco.nxos                    4.4.0  \ncisco.ucs                     1.8.0  \ncloud.common                  2.1.3  \ncloudscale_ch.cloud           2.3.1  \ncommunity.aws                 5.5.0  \ncommunity.azure               2.0.0  \ncommunity.ciscosmb            1.0.6  \ncommunity.crypto              2.14.0 \ncommunity.digitalocean        1.23.0 \ncommunity.dns                 2.5.5  \ncommunity.docker              3.4.7  \ncommunity.fortios             1.0.0  \ncommunity.general             6.6.2  \ncommunity.google              1.0.0  \ncommunity.grafana             1.5.4  \ncommunity.hashi_vault         4.2.1  \ncommunity.hrobot              1.8.0  \ncommunity.libvirt             1.2.0  \ncommunity.mongodb             1.6.0  \ncommunity.mysql               3.7.2  \ncommunity.network             5.0.0  \ncommunity.okd                 2.3.0  \ncommunity.postgresql          2.4.2  \ncommunity.proxysql            1.5.1  \ncommunity.rabbitmq            1.2.3  \ncommunity.routeros            2.8.2  \ncommunity.sap                 1.0.0  \ncommunity.sap_libs            1.4.1  \ncommunity.skydive             1.0.0  \ncommunity.sops                1.6.2  \ncommunity.vmware              3.7.0  \ncommunity.windows             1.13.0 \ncommunity.zabbix              1.9.3  \ncontainers.podman             1.10.2 \ncyberark.conjur               1.2.0  \ncyberark.pas                  1.0.19 \ndellemc.enterprise_sonic      2.0.0  \ndellemc.openmanage            6.3.0  \ndellemc.os10                  1.1.1  \ndellemc.os6                   1.0.7  \ndellemc.os9                   1.0.4  \ndellemc.powerflex             1.6.0  \ndellemc.unity                 1.6.0  \nf5networks.f5_modules         1.25.0 \nfortinet.fortimanager         2.2.0  \nfortinet.fortios              2.3.0  \nfrr.frr                       2.0.2  \ngluster.gluster               1.0.2  \ngoogle.cloud                  1.1.3  \ngrafana.grafana               1.1.1  \nhetzner.hcloud                1.11.0 \nhpe.nimble                    1.1.4  \nibm.qradar                    2.1.0  \nibm.spectrum_virtualize       1.12.0 \ninfinidat.infinibox           1.3.12 \ninfoblox.nios_modules         1.5.0  \ninspur.ispim                  1.3.0  \ninspur.sm                     2.3.0  \njunipernetworks.junos         4.1.0  \nkubernetes.core               2.4.0  \nlowlydba.sqlserver            1.3.1  \nmellanox.onyx                 1.0.0  \nmicrosoft.ad                  1.2.0  \nnetapp.aws                    21.7.0 \nnetapp.azure                  21.10.0\nnetapp.cloudmanager           21.22.0\nnetapp.elementsw              21.7.0 \nnetapp.ontap                  22.7.0 \nnetapp.storagegrid            21.11.1\nnetapp.um_info                21.8.0 \nnetapp_eseries.santricity     1.4.0  \nnetbox.netbox                 3.13.0 \nngine_io.cloudstack           2.3.0  \nngine_io.exoscale             1.0.0  \nngine_io.vultr                1.1.3  \nopenstack.cloud               1.10.0 \nopenvswitch.openvswitch       2.1.1  \novirt.ovirt                   2.4.1  \npurestorage.flasharray        1.19.1 \npurestorage.flashblade        1.11.0 \npurestorage.fusion            1.5.0  \nsensu.sensu_go                1.13.2 \nsplunk.es                     2.1.0  \nt_systems_mms.icinga_director 1.33.1 \ntheforeman.foreman            3.11.0 \nvmware.vmware_rest            2.3.1  \nvultr.cloud                   1.8.0  \nvyos.vyos                     4.1.0  \nwti.remote                    1.0.5  \n\n# /tmp/ansibletest/playbooks/collections/ansible_collections\nCollection        Version\n----------------- -------\ncommunity.general 8.1.0"
    ]
}

TASK [copy] **********************************************************************************************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

/tmp/ansibletest $ diff /tmp/{a,b}.out
2c2
< # /tmp/ansibletest/collections/ansible_collections
---
> # /tmp/ansibletest/playbooks/collections/ansible_collections

Does this makes sense for you @myllynen?

[myllynen]

Thanks for looking into this.

I think the cleanest approach to solve this with the current code would be to simply use the ANSIBLE_CONFIG environment variable to point to the wanted ansible.cfg file. Especially in the case of CI/CD runs mentioned earlier there's usually some scripting involved so setting ANSIBLE_CONFIG should be no issue. I've also verified this works in the environment where I saw this earlier.

However, I wonder would it make sense to provide a configuration option for the role to allow defining whether to use ansible.controller, awx.awx, or auto detection. That way the user would be sure only the wanted collection will be used/tried and in case of issues there's no chance to fallback without the user knowing about it. This would also remove the need to set ANSIBLE_CONFIG and avoid the not so elegant trick to run ansible-galaxy inside a role.

But of course the most important thing would be to avoid the issue mentioned in #741 as that now prevents using the role in altogether with recent Ansible versions, perhaps the eventual fix for that could take this issue to consideration as well.

Thanks.