Pipelime v1.9.1

What's Changed

• 91% test coverage
• 🩹 fixed ShellCommand
• 🚸 if LOGURU_LEVEL is not set as env var, then -v sets the logger level from WARNING (no -v) to TRACE (-vvv or more)
• 🗑️ WatcherNodesGraphExecutor has been deprecated (use DirectTrackCallback instead)

Full Changelog: v1.9.0...v1.9.1

Pipelime 1.9.0

What's Changed

🎉 New Features!

• 🚸 last pipelime call (with no --ckpt option) is saved and can be replayed with pipelime resume (❗TUI partial inputs included❗)
  • 👉 actually, last 3 calls are cached: try this pipelime resume +ckpt 2 to replay the second-last
• ✨ cat (both pipe and command) accepts multiple sequences with optional interleave flag
• ✨ batched/unbatched pipes to group/ungroup multiple samples
• ✨ InputDatasetInterface and OutputDatasetInterface accept a custom pipe; now you can:
  • ⚙️ add a custom processing on input/output dataset
  • 🔀 completely replace the data source with your own
  • ✏️ skip the default output underfolder

🪛 Many Fixes And UX Improvements!
*🪄 interactive context filling when, eg, some $vars are missing
*🧵 multiprocessing should now eat almost everything, lambda functions included

• 🧑‍💻 better error reporting in case of malformed @command or failing imports
• 📝 pipelime lc lists DAG commands in a separate section
• 🚸 when pipelime help finds a pydantic Field with expand_help=True, such field will be expanded if the type is a BaseModel
• 🐛 fixed split command with size 1

Full Changelog: v1.8.1...v1.9.0

Pipelime 1.8.1

What's Changed

• stub for SamplesSequence.from_images had an incorrect default
• fixed error when loading extra .py files multiple times
• fixed some use-cases in dag creation and drawing
• fixed missing variable in FileListenerCallback
• fixed pipelime list that was not showing anything

Full Changelog: v1.8.0...v1.8.1

Pipelime 1.8.0

What's Changed

• 🎉 Big news:
  • 🛸 Resume from checkpoint! -> especially when you run a DAG
  • 🧑‍💻 When you import a python symbol, eg, $symbol, CallableDef etc, now you can even use plain python code or lambda -> and it is fully compatible with a multi-process grabber!
  • 🧑‍💻 CLI: no need to repeat a +/@ option when you want to append a value -> see the docs
• 🕸️ DAG news:
  • 🛸 Resume/Replay a DAG
  • ✏️ Advanced node selection -> see the docs
  • 🐍 Python DAGs:
    • 📝 Checkout the docs
    • 🧑‍💻 create_graph can return a list of (lazy) PipelimeCommands: node names will be auto-generated
    • 🗑️ Temp folder is not automatically delete if you use --keep-tmp
• ✨ Commands, stages and pipes:
  • stage-time command to measure the average time spent by each stage of a pipeline
  • resume command to load and run a Pipelime checkpoint
  • map-if command to apply a stage only on a subset of samples
  • repeat now accepts floating point values to repeat a sequence a fractional number of times
• 🚶 General improvements:
  • 🧑‍💻 PipelimeTmp.make_session_dir() now always returns a subfolder of the Pipelime session temp folder
  • 🐛 @command now correctly manages submodel parameters
  • ✨ Input/Output interface now have a default piper port
  • ✨ The pydantic type NewPath can check or set the extension for you

Full Changelog: v1.7.0...v1.8.0

Pipelime 1.7.0

What's Changed

• fix: 🐛 Choixe now supports non-string dictionary keys
• fix: 🐛 Input/Output command ports can now be mappings
• feat: 🧑‍💻 Decoupled DAG command and DAG creation
• feat: ✨ Remove cached data after serialization

Full Changelog: v1.6.2...v1.7.0

Pipelime 1.6.2

What's Changed

• hotfix: 🚑️ Missing imports

Full Changelog: v1.6.1...v1.6.2

Pipelime 1.6.1

What's Changed

• build: 📦️ Missing css files in output package

Full Changelog: v1.6.0...v1.6.1

Pipelime 1.6.0

What's Changed

• 🚸 A new incredible wizard for our CLI! --> just type pipelime <command>
• 🧑‍💻 Write DAG as python code --> extend DagBaseCommand to start
• 🚸 New commands:
  • free temporary folders with tmp
  • set custom metadata with set-meta
  • copy root/shared items with copy-shared-items
  • remove duplicated samples with filter-duplicates
• 🚸 Load an image folder tree as SamplesSequence with from_images
• 🚸 New stages:
  • crop-and-pad-images
  • resize-images
  • set-meta
  • share-items
  • sample-hash
• 🐛 Notable bug fixes and other improvements:
  • the progress bar is now adaptive
  • XConfig can parse non-dict inputs
  • new --command-outputs CLI option

Full Changelog: v1.5.0...v1.6.0

Pipelime 1.5.0

What's Changed

• 🧑‍💻 new temp folder management, checkout the tutorial --> https://pipelime-python.readthedocs.io/en/v1.5.0/tutorials/temp_data/toc.html
• ➕ replaced multiprocessing with billiard
• ✨ new command and stage set-meta to add or change metadata
• ✨ new XConfig.unsafe_process() to generate any object from XConfig
• 🐛 fixed Choixe bugs: string bundles unparsing, no-case $switch
• 🐛 skip_empty option on input dataset now working in multiprocessing
• 🐛 pipelime --output-ctx now works as expected
• 🐛 fixed shell command output
• 🚸 show a clear error message in case of context recursion, eg, @myvar=myvar

Full Changelog: v1.4.1...v1.5.0

Pipelime 1.4.1

What's Changed

• ✨ New GLTFModel3DItem to handle GLTF scene in datasets
• ✨ New piper listener writing progress to yaml/json
• 🚸 When exists_ok is False, the check is now performed before actually running the pipelime command
• ✨ Faster and more reliable progress tracking when watching from the same shell of the running DAG

Full Changelog: v1.3.0...v1.4.1