Tiled CLI register needs container to exist to register objects with a prefix

[edbarnard]

Use case:
Have a datasets organized by subdirectories in /storage/. Each dataset is a subdir ie /storage/dataset1. I would like to register sub-directories individually like (as new datasets arrive):

tiled register http://127.0.0.1:8000/ /storage/dataset1/ --prefix=/dataset1

This fails because the Container /dataset1 does not exist.

There is no current way to create the container via the CLI (currently have to use create_container in python API)

From discussion with @dylanmcreynolds

[edbarnard]

Here is the error that I get. It looks like key_from_filename is None during some part of the registration when --prefix is set. See tiled/client/register.py:166

Interestingly settings appears to have a non-None key_from_filename function. I think that this is bug and key_from_filename should be pulled from settings

# tiled register http://localhost:8000/ /mf-storage-prod/034bqac92h0kz0r89cdd330jv0/ --api-key=XXX --prefix=/034bqac92h0kz0r89cdd330jv0

...

│                                                                                                                                        │
│ /opt/venv/lib/python3.12/site-packages/tiled/client/register.py:166 in register                                                        │
│                                                                                                                                        │
│   163 │   for segment in prefix_parts:                                                                                                 │
│   164 │   │   child_node = await anyio.to_thread.run_sync(node.get, segment)                                                           │
│   165 │   │   if child_node is None:                                                                                                   │
│ ❱ 166 │   │   │   key = key_from_filename(segment)                                                                                     │
│   167 │   │   │   await create_node_or_drop_collision(                                                                                 │
│   168 │   │   │   │   node,                                                                                                            │
│   169 │   │   │   │   structure_family=StructureFamily.container,                                                                      │
│                                                                                                                                        │
│ ╭────────────────────────────────────────────────────────────── locals ──────────────────────────────────────────────────────────────╮ │
│ │    adapters_by_mimetype = {}                                                                                                       │ │
│ │              child_node = None                                                                                                     │ │
│ │                  filter = None                                                                                                     │ │
│ │       key_from_filename = None                                                                                                     │ │
│ │ mimetype_detection_hook = None                                                                                                     │ │
│ │   mimetypes_by_file_ext = {}                                                                                                       │ │
│ │                    node = <Container {'231219_171929_picam_readout'}>                                                              │ │
│ │               overwrite = True                                                                                                     │ │
│ │          parsed_walkers = [<function group_image_sequences at 0x7fffb96822a0>, <function one_node_per_item at 0x7fffb9682160>]     │ │
│ │                    path = PosixPath('/mf-storage-prod/034bqac92h0kz0r89cdd330jv0')                                                 │ │
│ │                  prefix = '/034bqac92h0kz0r89cdd330jv0'                                                                            │ │
│ │            prefix_parts = ['034bqac92h0kz0r89cdd330jv0']                                                                           │ │
│ │                 segment = '034bqac92h0kz0r89cdd330jv0'                                                                             │ │
│ │                settings = Settings(                                                                                                │ │
│ │                           │   adapters_by_mimetype=ChainMap({}, <OneShotCachedMap({'image/tiff': <lazy>,                           │ │
│ │                           'multipart/related;type=image/tiff': <lazy>, 'image/jpeg': <lazy>, 'multipart/related;type=image/jpeg':  │ │
│ │                           <lazy>, 'text/csv': <lazy>, 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet': <lazy>, │ │
│ │                           'application/x-hdf5': <lazy>, 'application/x-netcdf': <lazy>, 'application/x-parquet': <lazy>,           │ │
│ │                           'application/x-parquet;structure=sparse': <lazy>, 'application/x-zarr': <lazy>,                          │ │
│ │                           'application/x-awkward-buffers': <lazy>, 'application/vnd.apache.arrow.file': <lazy>})>),                │ │
│ │                           │   mimetypes_by_file_ext=ChainMap({}, {'.h5': 'application/x-hdf5', '.nxs': 'application/x-hdf5',       │ │
│ │                           '.hdf': 'application/x-hdf5', '.hdf5': 'application/x-hdf5', '.csv': 'text/csv', '.zarr':                │ │
│ │                           'application/x-zarr'}),                                                                                  │ │
│ │                           │   mimetype_detection_hook=None,                                                                        │ │
│ │                           │   key_from_filename=<function strip_suffixes at 0x7ffffd068ae0>,                                       │ │
│ │                           │   filter=<function default_filter at 0x7fffb96819e0>                                                   │ │
│ │                           )                                                                                                        │ │
│ │                 walkers = None                                                                                                     │ │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'NoneType' object is not callable

[edbarnard]

The following diff fixes this immediate problem, but returns me back to the original problem of the container not existing.

--- a/tiled/client/register.py
+++ b/tiled/client/register.py
@@ -163,7 +163,7 @@ async def register(
     for segment in prefix_parts:
         child_node = await anyio.to_thread.run_sync(node.get, segment)
         if child_node is None:
-            key = key_from_filename(segment)
+            key = settings.key_from_filename(segment)
             await create_node_or_drop_collision(
                 node,
                 structure_family=StructureFamily.container,

Now the problem is container does not exist:

tiled register http://localhost:8000/ /mf-storage-prod/034bqac92h0kz0r89cdd330jv0/ --api-key=XXX --prefix=/034bqac92h0kz0r89cdd330jv0

....

│                                                                                                                                        │
│ /opt/venv/lib/python3.12/site-packages/tiled/client/utils.py:84 in handle_error                                                        │
│                                                                                                                                        │
│    81 │   │   │   │   # instead of dumping HTML into the terminal.                                                                     │
│    82 │   │   │   │   detail = response.reason_phrase                                                                                  │
│    83 │   │   │   message = f"{exc.response.status_code}: " f"{detail} " f"{exc.request.url}"                                          │
│ ❱  84 │   │   │   raise ClientError(message, exc.request, exc.response) from exc                                                       │
│    85 │   │   else:                                                                                                                    │
│    86 │   │   │   raise                                                                                                                │
│    87                                                                                                                                  │
│                                                                                                                                        │
│ ╭───────────────────────────────────────────── locals ─────────────────────────────────────────────╮                                   │
│ │   detail = "No such entry: ['034bqac92h0kz0r89cdd330jv0']"                                       │                                   │
│ │  message = "404: No such entry: ['034bqac92h0kz0r89cdd330jv0'] http://localhost:8000/api/v1/"+35 │                                   │
│ │ response = <Response [404 Not Found]>                                                            │                                   │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯                                   │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ClientError: 404: No such entry: ['034bqac92h0kz0r89cdd330jv0'] http://localhost:8000/api/v1/register/034bqac92h0kz0r89cdd330jv0

Server Side logs associated:

[d6fe2a8a5cfd1281] 127.0.0.1:46388 (public) - "GET /api/v1/ HTTP/1.1" 200 OK
[4f987cee67076f0b] 127.0.0.1:46388 ('service') - "GET /api/v1/metadata/?include_data_sources=false HTTP/1.1" 200 OK
[d2e746223ebbe2cc] 127.0.0.1:46388 ('service') - "GET /api/v1/metadata/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 404 Not Found
[2dbea10da5000201] 127.0.0.1:46388 ('service') - "POST /api/v1/metadata/ HTTP/1.1" 200 OK
[a81b532ab7cb2653] 127.0.0.1:46388 ('service') - "GET /api/v1/metadata/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 200 OK
[931b1fa74f340146] 127.0.0.1:46388 ('service') - "DELETE /api/v1/nodes/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 200 OK
[913ac4e9384e68df] 127.0.0.1:46388 ('service') - "POST /api/v1/register/034bqac92h0kz0r89cdd330jv0 HTTP/1.1" 404 Not Found
[79c6c9563072345c] 127.0.0.1:46388 ('service') - "GET /api/v1/search/?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 200 OK
[32c7f71c1fed9637] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[2f933fb09d19d476] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[9802a15df9cddf4d] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[9775beeb9dd1dba0] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[e75821f5d133ba33] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[a5edec636e7166d8] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[65418d54f667db21] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found
[b7d4dbbdd31d4a17] 127.0.0.1:46388 ('service') - "GET /api/v1/search/034bqac92h0kz0r89cdd330jv0?page%5Boffset%5D=0&fields=&sort= HTTP/1.1" 404 Not Found