Irrevelant NotADirectoryError in modules bump-versions

[heuermh]

Description of the bug

nf-core modules bump-versions odgi fails with an irrevelant NotADirectoryError in cellranger.

Command used and terminal output

$ git clone https://github.com/nf-core/modules.git nf-core-modules
$ cd nf-core-modules

... create pyenv virtualenv, install nf-core/tools

(.venv) nf-core-modules master
$ nf-core -v modules bump-versions odgi


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.1.1 - https://nf-co.re


Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/Users/heuermh/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)        cmd.py:1253
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/Users/heuermh/.config/nfcore/nf-core/modules, stdin=None,             cmd.py:1253
         shell=False, universal_newlines=True)
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.0%DEBUG    Popen(['git', 'cat-file', '--batch-check'], cwd=/Users/heuermh/.config/nfcore/nf-core/modules, stdin=<valid stream>, shell=False,      cmd.py:1253
         universal_newlines=False)
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/Users/heuermh/.config/nfcore/nf-core/modules, stdin=None, shell=False,                       cmd.py:1253
         universal_newlines=False)
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/Users/heuermh/.config/nfcore/nf-core/modules, stdin=None, shell=False,                   cmd.py:1253
         universal_newlines=False)
DEBUG    Got '/Users/heuermh/.config/nfcore/nf-core/modules' as path                                                                           utils.py:280
DEBUG    Found a config cache, loading: /Users/heuermh/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json                        utils.py:314
DEBUG    Using config file: /Users/heuermh/.config/nfcore/nf-core/modules/.nf-core.yml                                                        utils.py:1361
DEBUG    Got '.' as path                                                                                                                       utils.py:280
DEBUG    Found a config cache, loading: /Users/heuermh/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json                        utils.py:314
DEBUG    Using config file: .nf-core.yml                                                                                                      utils.py:1361
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/heuermh/working/nf-core-modules/.venv/bin/nf-core:8 in <module>                           │
│                                                                                                  │
│   5 from nf_core.__main__ import run_nf_core                                                     │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(run_nf_core())                                                                  │
│   9                                                                                              │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/__main__.py:18 │
│ 2 in run_nf_core                                                                                 │
│                                                                                                  │
│    179 │   │   │   log.debug(f"Could not check latest version: {e}")                             │
│    180 │   │   stderr.print("\n")                                                                │
│    181 │   # Launch the click cli                                                                │
│ ❱  182 │   nf_core_cli(auto_envvar_prefix="NFCORE")                                              │
│    183                                                                                           │
│    184                                                                                           │
│    185 @tui(                                                                                     │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/rich_click/rich_comman │
│ d.py:367 in __call__                                                                             │
│                                                                                                  │
│   364 │   │   # Include this here because I run into a false warning                             │
│   365 │   │   # in the PyCharm IDE otherwise; for some reason PyCharm doesn't                    │
│   366 │   │   # seem to think RichGroups are callable. (No issues with Mypy, though.)            │
│ ❱ 367 │   │   return super().__call__(*args, **kwargs)                                           │
│   368                                                                                            │
│   369                                                                                            │
│   370 class RichCommandCollection(CommandCollection, RichGroup):                                 │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/click/core.py:1161 in  │
│ __call__                                                                                         │
│                                                                                                  │
│   1158 │                                                                                         │
│   1159 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1160 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1161 │   │   return self.main(*args, **kwargs)                                                 │
│   1162                                                                                           │
│   1163                                                                                           │
│   1164 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/rich_click/rich_comman │
│ d.py:152 in main                                                                                 │
│                                                                                                  │
│   149 │   │   try:                                                                               │
│   150 │   │   │   try:                                                                           │
│   151 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                   │
│ ❱ 152 │   │   │   │   │   rv = self.invoke(ctx)                                                  │
│   153 │   │   │   │   │   if not standalone_mode:                                                │
│   154 │   │   │   │   │   │   return rv                                                          │
│   155 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                                │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/click/core.py:1697 in  │
│ invoke                                                                                           │
│                                                                                                  │
│   1694 │   │   │   │   super().invoke(ctx)                                                       │
│   1695 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1696 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1697 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1698 │   │                                                                                     │
│   1699 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1700 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/click/core.py:1697 in  │
│ invoke                                                                                           │
│                                                                                                  │
│   1694 │   │   │   │   super().invoke(ctx)                                                       │
│   1695 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1696 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1697 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1698 │   │                                                                                     │
│   1699 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1700 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/click/core.py:1443 in  │
│ invoke                                                                                           │
│                                                                                                  │
│   1440 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1441 │   │                                                                                     │
│   1442 │   │   if self.callback is not None:                                                     │
│ ❱ 1443 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1444 │                                                                                         │
│   1445 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1446 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/click/core.py:788 in   │
│ invoke                                                                                           │
│                                                                                                  │
│    785 │   │                                                                                     │
│    786 │   │   with augment_usage_errors(__self):                                                │
│    787 │   │   │   with ctx:                                                                     │
│ ❱  788 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    789 │                                                                                         │
│    790 │   def forward(__self, __cmd: "Command", *args: t.Any, **kwargs: t.Any) -> t.Any:        │
│    791 │   │   """Similar to :meth:`invoke` but fills in default keyword                         │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/click/decorators.py:33 │
│ in new_func                                                                                      │
│                                                                                                  │
│    30 │   """                                                                                    │
│    31 │                                                                                          │
│    32 │   def new_func(*args: "P.args", **kwargs: "P.kwargs") -> "R":                            │
│ ❱  33 │   │   return f(get_current_context(), *args, **kwargs)                                   │
│    34 │                                                                                          │
│    35 │   return update_wrapper(new_func, f)                                                     │
│    36                                                                                            │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/__main__.py:13 │
│ 40 in command_modules_bump_versions                                                              │
│                                                                                                  │
│   1337 │   Bump versions for one or more modules in a clone of                                   │
│   1338 │   the nf-core/modules repo.                                                             │
│   1339 │   """                                                                                   │
│ ❱ 1340 │   modules_bump_versions(ctx, tool, directory, all, show_all)                            │
│   1341                                                                                           │
│   1342                                                                                           │
│   1343 # nf-core subworkflows click command                                                      │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/commands_modul │
│ es.py:352 in modules_bump_versions                                                               │
│                                                                                                  │
│   349 │   │   │   ctx.obj["modules_repo_branch"],                                                │
│   350 │   │   │   ctx.obj["modules_repo_no_pull"],                                               │
│   351 │   │   )                                                                                  │
│ ❱ 352 │   │   version_bumper.bump_versions(module=tool, all_modules=all, show_uptodate=show_al   │
│   353 │   except ModuleExceptionError as e:                                                      │
│   354 │   │   log.error(e)                                                                       │
│   355 │   │   sys.exit(1)                                                                        │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/modules/bump_v │
│ ersions.py:79 in bump_versions                                                                   │
│                                                                                                  │
│    76 │   │   │   )                                                                              │
│    77 │   │                                                                                      │
│    78 │   │   # Get list of all modules                                                          │
│ ❱  79 │   │   _, nfcore_modules = nf_core.modules.modules_utils.get_installed_modules(self.dir   │
│    80 │   │                                                                                      │
│    81 │   │   # Load the .nf-core.yml config                                                     │
│    82 │   │   _, self.tools_config = nf_core.utils.load_tools_config(self.directory)             │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/modules/module │
│ s_utils.py:86 in get_installed_modules                                                           │
│                                                                                                  │
│   83 │   if local_modules_dir:                                                                   │
│   84 │   │   local_modules = [os.path.join(local_modules_dir, m) for m in local_modules]         │
│   85 │                                                                                           │
│ ❱ 86 │   nfcore_modules = [                                                                      │
│   87 │   │   NFCoreComponent(                                                                    │
│   88 │   │   │   m,                                                                              │
│   89 │   │   │   "nf-core/modules",                                                              │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/modules/module │
│ s_utils.py:87 in <listcomp>                                                                      │
│                                                                                                  │
│   84 │   │   local_modules = [os.path.join(local_modules_dir, m) for m in local_modules]         │
│   85 │                                                                                           │
│   86 │   nfcore_modules = [                                                                      │
│ ❱ 87 │   │   NFCoreComponent(                                                                    │
│   88 │   │   │   m,                                                                              │
│   89 │   │   │   "nf-core/modules",                                                              │
│   90 │   │   │   Path(nfcore_modules_dir, m),                                                    │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/components/nfc │
│ ore_component.py:98 in __init__                                                                  │
│                                                                                                  │
│    95 │   │   │   self.test_yml = None                                                           │
│    96 │   │   │   self.test_main_nf = None                                                       │
│    97 │   │                                                                                      │
│ ❱  98 │   │   self.process_name: str = self._get_process_name()                                  │
│    99 │                                                                                          │
│   100 │   def __repr__(self) -> str:                                                             │
│   101 │   │   return f"<NFCoreComponent {self.component_name} {self.component_dir} {self.repo_   │
│                                                                                                  │
│ /Users/heuermh/working/nf-core-modules/.venv/lib/python3.11/site-packages/nf_core/components/nfc │
│ ore_component.py:174 in _get_process_name                                                        │
│                                                                                                  │
│   171 │   │   return included_components                                                         │
│   172 │                                                                                          │
│   173 │   def _get_process_name(self):                                                           │
│ ❱ 174 │   │   with open(self.main_nf) as fh:                                                     │
│   175 │   │   │   for line in fh:                                                                │
│   176 │   │   │   │   if re.search(r"^\s*process\s*\w*\s*{", line):                              │
│   177 │   │   │   │   │   return re.search(r"^\s*process\s*(\w*)\s*{.*", line).group(1) or ""    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
NotADirectoryError: [Errno 20] Not a directory: 'modules/nf-core/cellranger/Dockerfile/main.nf'

System information

$ python --version
Python 3.11.11

$ nextflow -version

      N E X T F L O W
      version 24.10.3 build 5933
      created 16-12-2024 15:34 UTC (09:34 CDT)
      cite doi:10.1038/nbt.3820
      http://nextflow.io

$ nf-core --version


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.1.1 - https://nf-co.re

$ uname -a
Darwin xxx 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 31 21:38:20 PDT 2024; root:xnu-8796.141.3.707.4~1/RELEASE_ARM64_T8103 arm64

[awgymer]

This is a bug in nf_core.modules.module_utils.get_installed_modules.

This method appears to be designed to work with both pipeline and module repos, however it is only ever called in the codebase by the ModuleVersionBumper.bump_versions and only ever in the context of a modules dir.

There are a couple of issues with this method.

• It can never work for non nf-core org repos, because of hardcoding of nf-core.
• It's logic for finding main.nf files assumes there can never be anything other than <tool>/<subtool>

This second point is the actual bug in this instance, because cellranger has a Dockerfile (and also .gitignore and README.md at the <tool> subdir level.

(It's unclear whether this violation of the <tool>/<subtool> expectation causes other issues in the nf-core/tools code)

A possible fix is to use a method for finding modules and their main.nf more akin to that used in the SyncedRepo.get_avail_components:

avail_component_names = [
            str(Path(dirpath).relative_to(directory)) for dirpath, _, files in os.walk(directory) if "main.nf" in files
        ]