diff --git a/FEATURES.md b/FEATURES.md deleted file mode 100644 index 8b24d26d..00000000 --- a/FEATURES.md +++ /dev/null @@ -1,101 +0,0 @@ -# Functionalities - -VIDEO -- example - 1. to showcase how a generic video module works. - - INPUT: - 1. a list of video file paths (.mp4, .avi) - 2. a service with a name - - PROCESSING: nothing. - - OUTPUT: a json object placeholder as a response - -- input/output: - 1. to extract the audio (all channels) from the given videos. - - INPUT: - 1. a list of video file paths (.mp4, .avi) - 2. a service with a name for extracting the audio from a video - - PREPROCESSING: - 1. check files exist - 2. check their format is ok - - PROCESSING: - 1. extract audio channels from video - 2. embed those into an HF dataset object - - OUTPUT: - 1. a HF datasets object with the audio recordings in the "audio" column - -AUDIO -- example - 1. to showcase how the module works. - - INPUT: - 1. a list of audio file paths (.mp3, .wav) - 2. a service with a name - - PROCESSING: nothing. - - OUTPUT: a json object placeholder as a response - -- input/output - 1. to read some audio recordings from disk. - - INPUT: - 1. a list of audio file paths (.mp3, .wav) - 2. a service with a name (the default is "soundfile") - - PREPROCESSING: - 1. check files exist - 2. check their format is ok - - PROCESSING: - 1. extract audio channels from video - 2. embed the extracted audio channels into an HF dataset object - - OUTPUT: a datasets object with the audio recordings in the "audio" column - - 2. to save HF dataset object to disk - - INPUT: - 1. a datasets object with the audio recordings, - 2. the output path - - PREPROCESSING: - 1. check if the dataset exists already (TODO: decide how to manage this scenario) - 2. create all folders to path if they don't exist - - PROCESSING: - 1. save HF dataset to disk - - OUTPUT: - - - 3. to upload HF dataset object to the remote platform - - INPUT: - 1. a datasets object with the audio recordings - 2. the remote ID (and maybe the version?) - - PROCESSING: - 1. upload the dataset to the remote platform - - OUTPUT: - - -- speech to text - 1. to transcribe speech into text - - INPUT: - 1. a datasets object with the audio recordings in the "audio" column (optionally including the language spoken in the audio) - 2. the speech to text service to use (including the name, the version, optionally the revision, and - for some services only and sometimes it's optional - the language of the transcription model we want to use) - - PREPROCESSING: - 1. adapt the language to the service format - 2. organize the dataset into batches - - PROCESSING: - 1. transcribe the dataset - - POSTPROCESSING: - 1. formatting the transcripts to follow a standard organization - - OUTPUT: - 1. a new dataset including only the transcripts of the audios in a standardized json format (plus an index?) - - 2. to compute word error rate - - INPUT: - 1. a dataset object with the "transcript" and the "groundtruth" columns - 2. a service with a name - - PROCESSING: - 1. computing the per-row WER between the 2 columns - - OUTPUT: - 1. a dataset with the "WER" column - -[TODO] -- raw signal processing -- speaker diarization - -- data_augmentation -- data_representation -- speech emotion recognition -- speech enhancement -- speech verification -- text to speech -- voice conversion \ No newline at end of file diff --git a/FEATURES.tmp.md b/FEATURES.tmp.md new file mode 100644 index 00000000..4c4bb1de --- /dev/null +++ b/FEATURES.tmp.md @@ -0,0 +1,38 @@ +# Functionalities +This file is here just as a support for development. + +AUDIO + +[TODO]: +- speech to text + 1. to transcribe speech into text + - INPUT: + 1. a datasets object with the audio recordings in the "audio" column + 2. the audio column (default = "audio") + 3. the speech to text service to use (including the name, the version, the revision, and - for some services only and sometimes it's optional - the language of the transcription model we want to use) + - PREPROCESSING: + 1. adapt the language to the service format + 2. organize the dataset into batches + - PROCESSING: + 1. transcribe the dataset + - POSTPROCESSING: + 1. formatting the transcripts to follow a standard organization + - OUTPUT: + 1. a new dataset including only the transcripts of the audios in a standardized json format (plus an index?) + - TESTS: + 1. test input errors (a field is missing, the audio column exists and contains audio objects, params missing) + 2. test the transcript of a test file is ok + 3. test the language is supported (and the tool handles errors) + + 2. to compute word error rate + - INPUT: + 1. a dataset object with the "transcript" and the "groundtruth" columns + 2. a service with a name (default is jitter) + - PROCESSING: + 1. computing the per-row WER between the 2 columns + - OUTPUT: + 1. a dataset with the "WER" column + - TESTS: + 1. test input errors (a field is missing, fields missing, the 2 columns don't contain strings) + 2. test output is ok + diff --git a/README.md b/README.md index 11bc1379..5ccb13cc 100644 --- a/README.md +++ b/README.md @@ -10,19 +10,32 @@ [![pages](https://img.shields.io/badge/api-docs-blue)](https://sensein.github.io/pipepal) -Welcome to the ```pipepal``` repo! This is a Python package for doing incredible stuff with speech and voice. +Welcome to the ```pipepal``` repo! This is a Python package for streamlining the processing and analysis of behavioral data, such as voice and speech patterns, with robust and reproducible methodologies. **Caution:**: this package is still under development and may change rapidly over the next few weeks. ## Features -- A few -- Cool -- Things -- These may include a wonderful CLI interface. +- **Modular design**: Utilize a variety of task-specific transformations that can be easily integrated or used standalone, allowing for flexible data manipulation and analysis strategies. -## Installation -Install this package via : +- **Pre-built pipelines**: Access pre-configured pipelines combining multiple transformations tailored for common analysis tasks, which help in reducing setup time and effort. + +- **Reproducibility**: Ensures consistent outputs through the use of fixed seeds and version-controlled processing steps, making your results verifiable and easily comparable. + +- **Easy integration**: Designed to fit into existing workflows with minimal configuration, `pipepal` can be used alongside other data analysis tools and frameworks seamlessly. + +- **Extensible**: Open to modifications and contributions, the package can be expanded with custom transformations and pipelines to meet specific research needs. Do you want to contribute? Please, reach out! + +- **Comprehensive documentation**: Comes with detailed documentation for all features and modules, including examples and guides on how to extend the package for other types of behavioral data analysis. + +- **Performance Optimized**: Efficiently processes large datasets with optimized code and algorithms, ensuring quick turnaround times even for complex analyses. +- **Interactive Examples**: Includes Jupyter notebooks that provide practical examples of how `pipepal` can be implemented to derive insights from real-world data sets. + +Whether you're researching speech disorders, analyzing customer service calls, or studying communication patterns, `pipepal` provides the tools and flexibility needed to extract meaningful conclusions from your data. + + +## Installation +Install this package via: ```sh pip install pipepal @@ -42,5 +55,20 @@ hello_world() ``` ## To do: -- [ ] A -- [ ] lot \ No newline at end of file +- [ ] Integrating more audio tasks and moving functions from b2aiprep package: + - [ ] data_augmentation + - [ ] data_representation + - [x] example_task + - [x] input_output + - [ ] raw_signal_processing + - [ ] speaker_diarization + - [ ] speech emotion recognition + - [ ] speech enhancement + - [ ] speech_to_text + - [ ] text_to_speech + - [ ] voice conversion +- [ ] Integrating more video tasks: + - [x] input_output + +- [ ] Preparing some pipelines with pydra +- [ ] Populating the CLI diff --git a/data_for_testing/audio_48khz_mono_16bits.wav b/data_for_testing/audio_48khz_mono_16bits.wav new file mode 100644 index 00000000..3d3f3f70 Binary files /dev/null and b/data_for_testing/audio_48khz_mono_16bits.wav differ diff --git a/data_for_testing/audio_48khz_stereo_16bits.wav b/data_for_testing/audio_48khz_stereo_16bits.wav new file mode 100644 index 00000000..9a578f60 Binary files /dev/null and b/data_for_testing/audio_48khz_stereo_16bits.wav differ diff --git a/data_for_testing/output_dataset/data-00000-of-00001.arrow b/data_for_testing/output_dataset/data-00000-of-00001.arrow new file mode 100644 index 00000000..c2ff86f3 Binary files /dev/null and b/data_for_testing/output_dataset/data-00000-of-00001.arrow differ diff --git a/data_for_testing/output_dataset/dataset_info.json b/data_for_testing/output_dataset/dataset_info.json new file mode 100644 index 00000000..b63f23ee --- /dev/null +++ b/data_for_testing/output_dataset/dataset_info.json @@ -0,0 +1,16 @@ +{ + "citation": "", + "description": "", + "features": { + "pokemon": { + "dtype": "string", + "_type": "Value" + }, + "type": { + "dtype": "string", + "_type": "Value" + } + }, + "homepage": "", + "license": "" +} \ No newline at end of file diff --git a/data_for_testing/output_dataset/state.json b/data_for_testing/output_dataset/state.json new file mode 100644 index 00000000..1844d528 --- /dev/null +++ b/data_for_testing/output_dataset/state.json @@ -0,0 +1,13 @@ +{ + "_data_files": [ + { + "filename": "data-00000-of-00001.arrow" + } + ], + "_fingerprint": "57821607a631abce", + "_format_columns": null, + "_format_kwargs": {}, + "_format_type": null, + "_output_all_columns": false, + "_split": null +} \ No newline at end of file diff --git a/data_for_testing/video_48khz_stereo_16bits.mp4 b/data_for_testing/video_48khz_stereo_16bits.mp4 new file mode 100644 index 00000000..d97b067d Binary files /dev/null and b/data_for_testing/video_48khz_stereo_16bits.mp4 differ diff --git a/poetry.lock b/poetry.lock index 4d7a7fc1..2413e9c1 100644 --- a/poetry.lock +++ b/poetry.lock @@ -110,6 +110,35 @@ files = [ [package.dependencies] frozenlist = ">=1.1.0" +[[package]] +name = "appnope" +version = "0.1.4" +description = "Disable App Nap on macOS >= 10.9" +optional = false +python-versions = ">=3.6" +files = [ + {file = "appnope-0.1.4-py2.py3-none-any.whl", hash = "sha256:502575ee11cd7a28c0205f379b525beefebab9d161b7c964670864014ed7213c"}, + {file = "appnope-0.1.4.tar.gz", hash = "sha256:1de3860566df9caf38f01f86f65e0e13e379af54f9e4bee1e66b48f2efffd1ee"}, +] + +[[package]] +name = "asttokens" +version = "2.4.1" +description = "Annotate AST trees with source code positions" +optional = false +python-versions = "*" +files = [ + {file = "asttokens-2.4.1-py2.py3-none-any.whl", hash = "sha256:051ed49c3dcae8913ea7cd08e46a606dba30b79993209636c4875bc1d637bc24"}, + {file = "asttokens-2.4.1.tar.gz", hash = "sha256:b03869718ba9a6eb027e134bfdf69f38a236d681c83c160d510768af11254ba0"}, +] + +[package.dependencies] +six = ">=1.12.0" + +[package.extras] +astroid = ["astroid (>=1,<2)", "astroid (>=2,<4)"] +test = ["astroid (>=1,<2)", "astroid (>=2,<4)", "pytest"] + [[package]] name = "async-timeout" version = "4.0.3" @@ -364,6 +393,23 @@ files = [ {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, ] +[[package]] +name = "comm" +version = "0.2.2" +description = "Jupyter Python Comm implementation, for usage in ipykernel, xeus-python etc." +optional = false +python-versions = ">=3.8" +files = [ + {file = "comm-0.2.2-py3-none-any.whl", hash = "sha256:e6fb86cb70ff661ee8c9c14e7d36d6de3b4066f1441be4063df9c5009f0a64d3"}, + {file = "comm-0.2.2.tar.gz", hash = "sha256:3fd7a84065306e07bea1773df6eb8282de51ba82f77c72f9c85716ab11fe980e"}, +] + +[package.dependencies] +traitlets = ">=4" + +[package.extras] +test = ["pytest"] + [[package]] name = "coverage" version = "7.4.4" @@ -475,6 +521,37 @@ tests = ["Pillow (>=6.2.1)", "absl-py", "apache-beam (>=2.26.0)", "elasticsearch torch = ["torch"] vision = ["Pillow (>=6.2.1)"] +[[package]] +name = "debugpy" +version = "1.8.1" +description = "An implementation of the Debug Adapter Protocol for Python" +optional = false +python-versions = ">=3.8" +files = [ + {file = "debugpy-1.8.1-cp310-cp310-macosx_11_0_x86_64.whl", hash = "sha256:3bda0f1e943d386cc7a0e71bfa59f4137909e2ed947fb3946c506e113000f741"}, + {file = "debugpy-1.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dda73bf69ea479c8577a0448f8c707691152e6c4de7f0c4dec5a4bc11dee516e"}, + {file = "debugpy-1.8.1-cp310-cp310-win32.whl", hash = "sha256:3a79c6f62adef994b2dbe9fc2cc9cc3864a23575b6e387339ab739873bea53d0"}, + {file = "debugpy-1.8.1-cp310-cp310-win_amd64.whl", hash = "sha256:7eb7bd2b56ea3bedb009616d9e2f64aab8fc7000d481faec3cd26c98a964bcdd"}, + {file = "debugpy-1.8.1-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:016a9fcfc2c6b57f939673c874310d8581d51a0fe0858e7fac4e240c5eb743cb"}, + {file = "debugpy-1.8.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd97ed11a4c7f6d042d320ce03d83b20c3fb40da892f994bc041bbc415d7a099"}, + {file = "debugpy-1.8.1-cp311-cp311-win32.whl", hash = "sha256:0de56aba8249c28a300bdb0672a9b94785074eb82eb672db66c8144fff673146"}, + {file = "debugpy-1.8.1-cp311-cp311-win_amd64.whl", hash = "sha256:1a9fe0829c2b854757b4fd0a338d93bc17249a3bf69ecf765c61d4c522bb92a8"}, + {file = "debugpy-1.8.1-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:3ebb70ba1a6524d19fa7bb122f44b74170c447d5746a503e36adc244a20ac539"}, + {file = "debugpy-1.8.1-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a2e658a9630f27534e63922ebf655a6ab60c370f4d2fc5c02a5b19baf4410ace"}, + {file = "debugpy-1.8.1-cp312-cp312-win32.whl", hash = "sha256:caad2846e21188797a1f17fc09c31b84c7c3c23baf2516fed5b40b378515bbf0"}, + {file = "debugpy-1.8.1-cp312-cp312-win_amd64.whl", hash = "sha256:edcc9f58ec0fd121a25bc950d4578df47428d72e1a0d66c07403b04eb93bcf98"}, + {file = "debugpy-1.8.1-cp38-cp38-macosx_11_0_x86_64.whl", hash = "sha256:7a3afa222f6fd3d9dfecd52729bc2e12c93e22a7491405a0ecbf9e1d32d45b39"}, + {file = "debugpy-1.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d915a18f0597ef685e88bb35e5d7ab968964b7befefe1aaea1eb5b2640b586c7"}, + {file = "debugpy-1.8.1-cp38-cp38-win32.whl", hash = "sha256:92116039b5500633cc8d44ecc187abe2dfa9b90f7a82bbf81d079fcdd506bae9"}, + {file = "debugpy-1.8.1-cp38-cp38-win_amd64.whl", hash = "sha256:e38beb7992b5afd9d5244e96ad5fa9135e94993b0c551ceebf3fe1a5d9beb234"}, + {file = "debugpy-1.8.1-cp39-cp39-macosx_11_0_x86_64.whl", hash = "sha256:bfb20cb57486c8e4793d41996652e5a6a885b4d9175dd369045dad59eaacea42"}, + {file = "debugpy-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:efd3fdd3f67a7e576dd869c184c5dd71d9aaa36ded271939da352880c012e703"}, + {file = "debugpy-1.8.1-cp39-cp39-win32.whl", hash = "sha256:58911e8521ca0c785ac7a0539f1e77e0ce2df753f786188f382229278b4cdf23"}, + {file = "debugpy-1.8.1-cp39-cp39-win_amd64.whl", hash = "sha256:6df9aa9599eb05ca179fb0b810282255202a66835c6efb1d112d21ecb830ddd3"}, + {file = "debugpy-1.8.1-py2.py3-none-any.whl", hash = "sha256:28acbe2241222b87e255260c76741e1fbf04fdc3b6d094fcf57b6c6f75ce1242"}, + {file = "debugpy-1.8.1.zip", hash = "sha256:f696d6be15be87aef621917585f9bb94b1dc9e8aced570db1b8a6fc14e8f9b42"}, +] + [[package]] name = "decorator" version = "5.1.1" @@ -526,6 +603,37 @@ files = [ [package.extras] test = ["pytest (>=6)"] +[[package]] +name = "executing" +version = "2.0.1" +description = "Get the currently executing AST node of a frame, and other information" +optional = false +python-versions = ">=3.5" +files = [ + {file = "executing-2.0.1-py2.py3-none-any.whl", hash = "sha256:eac49ca94516ccc753f9fb5ce82603156e590b27525a8bc32cce8ae302eb61bc"}, + {file = "executing-2.0.1.tar.gz", hash = "sha256:35afe2ce3affba8ee97f2d69927fa823b08b472b7b994e36a52a964b93d16147"}, +] + +[package.extras] +tests = ["asttokens (>=2.1.0)", "coverage", "coverage-enable-subprocess", "ipython", "littleutils", "pytest", "rich"] + +[[package]] +name = "ffmpeg-python" +version = "0.2.0" +description = "Python bindings for FFmpeg - with complex filtering support" +optional = false +python-versions = "*" +files = [ + {file = "ffmpeg-python-0.2.0.tar.gz", hash = "sha256:65225db34627c578ef0e11c8b1eb528bb35e024752f6f10b78c011f6f64c4127"}, + {file = "ffmpeg_python-0.2.0-py3-none-any.whl", hash = "sha256:ac441a0404e053f8b6a1113a77c0f452f1cfc62f6344a769475ffdc0f56c23c5"}, +] + +[package.dependencies] +future = "*" + +[package.extras] +dev = ["Sphinx (==2.1.0)", "future (==0.17.1)", "numpy (==1.16.4)", "pytest (==4.6.1)", "pytest-mock (==1.10.4)", "tox (==3.12.1)"] + [[package]] name = "filelock" version = "3.13.3" @@ -666,6 +774,17 @@ smb = ["smbprotocol"] ssh = ["paramiko"] tqdm = ["tqdm"] +[[package]] +name = "future" +version = "1.0.0" +description = "Clean single-source support for Python 3 and 2" +optional = false +python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*" +files = [ + {file = "future-1.0.0-py3-none-any.whl", hash = "sha256:929292d34f5872e70396626ef385ec22355a1fae8ad29e1a734c3e43f9fbc216"}, + {file = "future-1.0.0.tar.gz", hash = "sha256:bd2968309307861edae1458a4f8a4f3598c03be43b97521076aebf5d94c07b05"}, +] + [[package]] name = "huggingface-hub" version = "0.22.2" @@ -736,6 +855,96 @@ files = [ {file = "iniconfig-2.0.0.tar.gz", hash = "sha256:2d91e135bf72d31a410b17c16da610a82cb55f6b0477d1a902134b24a455b8b3"}, ] +[[package]] +name = "ipykernel" +version = "6.29.4" +description = "IPython Kernel for Jupyter" +optional = false +python-versions = ">=3.8" +files = [ + {file = "ipykernel-6.29.4-py3-none-any.whl", hash = "sha256:1181e653d95c6808039c509ef8e67c4126b3b3af7781496c7cbfb5ed938a27da"}, + {file = "ipykernel-6.29.4.tar.gz", hash = "sha256:3d44070060f9475ac2092b760123fadf105d2e2493c24848b6691a7c4f42af5c"}, +] + +[package.dependencies] +appnope = {version = "*", markers = "platform_system == \"Darwin\""} +comm = ">=0.1.1" +debugpy = ">=1.6.5" +ipython = ">=7.23.1" +jupyter-client = ">=6.1.12" +jupyter-core = ">=4.12,<5.0.dev0 || >=5.1.dev0" +matplotlib-inline = ">=0.1" +nest-asyncio = "*" +packaging = "*" +psutil = "*" +pyzmq = ">=24" +tornado = ">=6.1" +traitlets = ">=5.4.0" + +[package.extras] +cov = ["coverage[toml]", "curio", "matplotlib", "pytest-cov", "trio"] +docs = ["myst-parser", "pydata-sphinx-theme", "sphinx", "sphinx-autodoc-typehints", "sphinxcontrib-github-alt", "sphinxcontrib-spelling", "trio"] +pyqt5 = ["pyqt5"] +pyside6 = ["pyside6"] +test = ["flaky", "ipyparallel", "pre-commit", "pytest (>=7.0)", "pytest-asyncio (>=0.23.5)", "pytest-cov", "pytest-timeout"] + +[[package]] +name = "ipython" +version = "8.23.0" +description = "IPython: Productive Interactive Computing" +optional = false +python-versions = ">=3.10" +files = [ + {file = "ipython-8.23.0-py3-none-any.whl", hash = "sha256:07232af52a5ba146dc3372c7bf52a0f890a23edf38d77caef8d53f9cdc2584c1"}, + {file = "ipython-8.23.0.tar.gz", hash = "sha256:7468edaf4f6de3e1b912e57f66c241e6fd3c7099f2ec2136e239e142e800274d"}, +] + +[package.dependencies] +colorama = {version = "*", markers = "sys_platform == \"win32\""} +decorator = "*" +exceptiongroup = {version = "*", markers = "python_version < \"3.11\""} +jedi = ">=0.16" +matplotlib-inline = "*" +pexpect = {version = ">4.3", markers = "sys_platform != \"win32\" and sys_platform != \"emscripten\""} +prompt-toolkit = ">=3.0.41,<3.1.0" +pygments = ">=2.4.0" +stack-data = "*" +traitlets = ">=5.13.0" +typing-extensions = {version = "*", markers = "python_version < \"3.12\""} + +[package.extras] +all = ["ipython[black,doc,kernel,matplotlib,nbconvert,nbformat,notebook,parallel,qtconsole]", "ipython[test,test-extra]"] +black = ["black"] +doc = ["docrepr", "exceptiongroup", "ipykernel", "ipython[test]", "matplotlib", "setuptools (>=18.5)", "sphinx (>=1.3)", "sphinx-rtd-theme", "sphinxcontrib-jquery", "stack-data", "typing-extensions"] +kernel = ["ipykernel"] +matplotlib = ["matplotlib"] +nbconvert = ["nbconvert"] +nbformat = ["nbformat"] +notebook = ["ipywidgets", "notebook"] +parallel = ["ipyparallel"] +qtconsole = ["qtconsole"] +test = ["pickleshare", "pytest (<8)", "pytest-asyncio (<0.22)", "testpath"] +test-extra = ["curio", "ipython[test]", "matplotlib (!=3.2.0)", "nbformat", "numpy (>=1.23)", "pandas", "trio"] + +[[package]] +name = "jedi" +version = "0.19.1" +description = "An autocompletion tool for Python that can be used for text editors." +optional = false +python-versions = ">=3.6" +files = [ + {file = "jedi-0.19.1-py2.py3-none-any.whl", hash = "sha256:e983c654fe5c02867aef4cdfce5a2fbb4a50adc0af145f70504238f18ef5e7e0"}, + {file = "jedi-0.19.1.tar.gz", hash = "sha256:cf0496f3651bc65d7174ac1b7d043eff454892c708a87d1b683e57b569927ffd"}, +] + +[package.dependencies] +parso = ">=0.8.3,<0.9.0" + +[package.extras] +docs = ["Jinja2 (==2.11.3)", "MarkupSafe (==1.1.1)", "Pygments (==2.8.1)", "alabaster (==0.7.12)", "babel (==2.9.1)", "chardet (==4.0.0)", "commonmark (==0.8.1)", "docutils (==0.17.1)", "future (==0.18.2)", "idna (==2.10)", "imagesize (==1.2.0)", "mock (==1.0.1)", "packaging (==20.9)", "pyparsing (==2.4.7)", "pytz (==2021.1)", "readthedocs-sphinx-ext (==2.1.4)", "recommonmark (==0.5.0)", "requests (==2.25.1)", "six (==1.15.0)", "snowballstemmer (==2.1.0)", "sphinx (==1.8.5)", "sphinx-rtd-theme (==0.4.3)", "sphinxcontrib-serializinghtml (==1.1.4)", "sphinxcontrib-websupport (==1.2.4)", "urllib3 (==1.26.4)"] +qa = ["flake8 (==5.0.4)", "mypy (==0.971)", "types-setuptools (==67.2.0.1)"] +testing = ["Django", "attrs", "colorama", "docopt", "pytest (<7.0.0)"] + [[package]] name = "jinja2" version = "3.1.3" @@ -799,6 +1008,48 @@ files = [ [package.dependencies] referencing = ">=0.31.0" +[[package]] +name = "jupyter-client" +version = "8.6.1" +description = "Jupyter protocol implementation and client libraries" +optional = false +python-versions = ">=3.8" +files = [ + {file = "jupyter_client-8.6.1-py3-none-any.whl", hash = "sha256:3b7bd22f058434e3b9a7ea4b1500ed47de2713872288c0d511d19926f99b459f"}, + {file = "jupyter_client-8.6.1.tar.gz", hash = "sha256:e842515e2bab8e19186d89fdfea7abd15e39dd581f94e399f00e2af5a1652d3f"}, +] + +[package.dependencies] +jupyter-core = ">=4.12,<5.0.dev0 || >=5.1.dev0" +python-dateutil = ">=2.8.2" +pyzmq = ">=23.0" +tornado = ">=6.2" +traitlets = ">=5.3" + +[package.extras] +docs = ["ipykernel", "myst-parser", "pydata-sphinx-theme", "sphinx (>=4)", "sphinx-autodoc-typehints", "sphinxcontrib-github-alt", "sphinxcontrib-spelling"] +test = ["coverage", "ipykernel (>=6.14)", "mypy", "paramiko", "pre-commit", "pytest", "pytest-cov", "pytest-jupyter[client] (>=0.4.1)", "pytest-timeout"] + +[[package]] +name = "jupyter-core" +version = "5.7.2" +description = "Jupyter core package. A base package on which Jupyter projects rely." +optional = false +python-versions = ">=3.8" +files = [ + {file = "jupyter_core-5.7.2-py3-none-any.whl", hash = "sha256:4f7315d2f6b4bcf2e3e7cb6e46772eba760ae459cd1f59d29eb57b0a01bd7409"}, + {file = "jupyter_core-5.7.2.tar.gz", hash = "sha256:aa5f8d32bbf6b431ac830496da7392035d6f61b4f54872f15c4bd2a9c3f536d9"}, +] + +[package.dependencies] +platformdirs = ">=2.5" +pywin32 = {version = ">=300", markers = "sys_platform == \"win32\" and platform_python_implementation != \"PyPy\""} +traitlets = ">=5.3" + +[package.extras] +docs = ["myst-parser", "pydata-sphinx-theme", "sphinx-autodoc-typehints", "sphinxcontrib-github-alt", "sphinxcontrib-spelling", "traitlets"] +test = ["ipykernel", "pre-commit", "pytest (<8)", "pytest-cov", "pytest-timeout"] + [[package]] name = "lazy-loader" version = "0.3" @@ -944,6 +1195,20 @@ files = [ {file = "MarkupSafe-2.1.5.tar.gz", hash = "sha256:d283d37a890ba4c1ae73ffadf8046435c76e7bc2247bbb63c00bd1a709c6544b"}, ] +[[package]] +name = "matplotlib-inline" +version = "0.1.6" +description = "Inline Matplotlib backend for Jupyter" +optional = false +python-versions = ">=3.5" +files = [ + {file = "matplotlib-inline-0.1.6.tar.gz", hash = "sha256:f887e5f10ba98e8d2b150ddcf4702c1e5f8b3a20005eb0f74bfdbd360ee6f304"}, + {file = "matplotlib_inline-0.1.6-py3-none-any.whl", hash = "sha256:f1f41aab5328aa5aaea9b16d083b128102f8712542f819fe7e6a420ff581b311"}, +] + +[package.dependencies] +traitlets = "*" + [[package]] name = "mpmath" version = "1.3.0" @@ -1207,6 +1472,17 @@ files = [ {file = "mypy_extensions-1.0.0.tar.gz", hash = "sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782"}, ] +[[package]] +name = "nest-asyncio" +version = "1.6.0" +description = "Patch asyncio to allow nested event loops" +optional = false +python-versions = ">=3.5" +files = [ + {file = "nest_asyncio-1.6.0-py3-none-any.whl", hash = "sha256:87af6efd6b5e897c81050477ef65c62e2b2f35d51703cae01aff2905b1852e1c"}, + {file = "nest_asyncio-1.6.0.tar.gz", hash = "sha256:6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe"}, +] + [[package]] name = "networkx" version = "3.2.1" @@ -1544,6 +1820,21 @@ sql-other = ["SQLAlchemy (>=2.0.0)", "adbc-driver-postgresql (>=0.8.0)", "adbc-d test = ["hypothesis (>=6.46.1)", "pytest (>=7.3.2)", "pytest-xdist (>=2.2.0)"] xml = ["lxml (>=4.9.2)"] +[[package]] +name = "parso" +version = "0.8.4" +description = "A Python Parser" +optional = false +python-versions = ">=3.6" +files = [ + {file = "parso-0.8.4-py2.py3-none-any.whl", hash = "sha256:a418670a20291dacd2dddc80c377c5c3791378ee1e8d12bffc35420643d43f18"}, + {file = "parso-0.8.4.tar.gz", hash = "sha256:eb3a7b58240fb99099a345571deecc0f9540ea5f4dd2fe14c2a99d6b281ab92d"}, +] + +[package.extras] +qa = ["flake8 (==5.0.4)", "mypy (==0.971)", "types-setuptools (==67.2.0.1)"] +testing = ["docopt", "pytest"] + [[package]] name = "pdoc" version = "14.4.0" @@ -1563,6 +1854,20 @@ pygments = ">=2.12.0" [package.extras] dev = ["hypothesis", "mypy", "pdoc-pyo3-sample-library (==1.0.11)", "pygments (>=2.14.0)", "pytest", "pytest-cov", "pytest-timeout", "ruff", "tox", "types-pygments"] +[[package]] +name = "pexpect" +version = "4.9.0" +description = "Pexpect allows easy control of interactive console applications." +optional = false +python-versions = "*" +files = [ + {file = "pexpect-4.9.0-py2.py3-none-any.whl", hash = "sha256:7236d1e080e4936be2dc3e326cec0af72acf9212a7e1d060210e70a47e253523"}, + {file = "pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f"}, +] + +[package.dependencies] +ptyprocess = ">=0.5" + [[package]] name = "pillow" version = "10.3.0" @@ -1718,6 +2023,73 @@ nodeenv = ">=0.11.1" pyyaml = ">=5.1" virtualenv = ">=20.10.0" +[[package]] +name = "prompt-toolkit" +version = "3.0.43" +description = "Library for building powerful interactive command lines in Python" +optional = false +python-versions = ">=3.7.0" +files = [ + {file = "prompt_toolkit-3.0.43-py3-none-any.whl", hash = "sha256:a11a29cb3bf0a28a387fe5122cdb649816a957cd9261dcedf8c9f1fef33eacf6"}, + {file = "prompt_toolkit-3.0.43.tar.gz", hash = "sha256:3527b7af26106cbc65a040bcc84839a3566ec1b051bb0bfe953631e704b0ff7d"}, +] + +[package.dependencies] +wcwidth = "*" + +[[package]] +name = "psutil" +version = "5.9.8" +description = "Cross-platform lib for process and system monitoring in Python." +optional = false +python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*" +files = [ + {file = "psutil-5.9.8-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:26bd09967ae00920df88e0352a91cff1a78f8d69b3ecabbfe733610c0af486c8"}, + {file = "psutil-5.9.8-cp27-cp27m-manylinux2010_i686.whl", hash = "sha256:05806de88103b25903dff19bb6692bd2e714ccf9e668d050d144012055cbca73"}, + {file = "psutil-5.9.8-cp27-cp27m-manylinux2010_x86_64.whl", hash = "sha256:611052c4bc70432ec770d5d54f64206aa7203a101ec273a0cd82418c86503bb7"}, + {file = "psutil-5.9.8-cp27-cp27mu-manylinux2010_i686.whl", hash = "sha256:50187900d73c1381ba1454cf40308c2bf6f34268518b3f36a9b663ca87e65e36"}, + {file = "psutil-5.9.8-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:02615ed8c5ea222323408ceba16c60e99c3f91639b07da6373fb7e6539abc56d"}, + {file = "psutil-5.9.8-cp27-none-win32.whl", hash = "sha256:36f435891adb138ed3c9e58c6af3e2e6ca9ac2f365efe1f9cfef2794e6c93b4e"}, + {file = "psutil-5.9.8-cp27-none-win_amd64.whl", hash = "sha256:bd1184ceb3f87651a67b2708d4c3338e9b10c5df903f2e3776b62303b26cb631"}, + {file = "psutil-5.9.8-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:aee678c8720623dc456fa20659af736241f575d79429a0e5e9cf88ae0605cc81"}, + {file = "psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:8cb6403ce6d8e047495a701dc7c5bd788add903f8986d523e3e20b98b733e421"}, + {file = "psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d06016f7f8625a1825ba3732081d77c94589dca78b7a3fc072194851e88461a4"}, + {file = "psutil-5.9.8-cp36-cp36m-win32.whl", hash = "sha256:7d79560ad97af658a0f6adfef8b834b53f64746d45b403f225b85c5c2c140eee"}, + {file = "psutil-5.9.8-cp36-cp36m-win_amd64.whl", hash = "sha256:27cc40c3493bb10de1be4b3f07cae4c010ce715290a5be22b98493509c6299e2"}, + {file = "psutil-5.9.8-cp37-abi3-win32.whl", hash = "sha256:bc56c2a1b0d15aa3eaa5a60c9f3f8e3e565303b465dbf57a1b730e7a2b9844e0"}, + {file = "psutil-5.9.8-cp37-abi3-win_amd64.whl", hash = "sha256:8db4c1b57507eef143a15a6884ca10f7c73876cdf5d51e713151c1236a0e68cf"}, + {file = "psutil-5.9.8-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:d16bbddf0693323b8c6123dd804100241da461e41d6e332fb0ba6058f630f8c8"}, + {file = "psutil-5.9.8.tar.gz", hash = "sha256:6be126e3225486dff286a8fb9a06246a5253f4c7c53b475ea5f5ac934e64194c"}, +] + +[package.extras] +test = ["enum34", "ipaddress", "mock", "pywin32", "wmi"] + +[[package]] +name = "ptyprocess" +version = "0.7.0" +description = "Run a subprocess in a pseudo terminal" +optional = false +python-versions = "*" +files = [ + {file = "ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35"}, + {file = "ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220"}, +] + +[[package]] +name = "pure-eval" +version = "0.2.2" +description = "Safely evaluate AST nodes without side effects" +optional = false +python-versions = "*" +files = [ + {file = "pure_eval-0.2.2-py3-none-any.whl", hash = "sha256:01eaab343580944bc56080ebe0a674b39ec44a945e6d09ba7db3cb8cec289350"}, + {file = "pure_eval-0.2.2.tar.gz", hash = "sha256:2b45320af6dfaa1750f543d714b6d1c520a1688dec6fd24d339063ce0aaa9ac3"}, +] + +[package.extras] +tests = ["pytest"] + [[package]] name = "pyarrow" version = "15.0.2" @@ -1843,6 +2215,23 @@ pytest = ">=4.6" [package.extras] testing = ["fields", "hunter", "process-tests", "pytest-xdist", "virtualenv"] +[[package]] +name = "pytest-mock" +version = "3.14.0" +description = "Thin-wrapper around the mock package for easier use with pytest" +optional = false +python-versions = ">=3.8" +files = [ + {file = "pytest-mock-3.14.0.tar.gz", hash = "sha256:2719255a1efeceadbc056d6bf3df3d1c5015530fb40cf347c0f9afac88410bd0"}, + {file = "pytest_mock-3.14.0-py3-none-any.whl", hash = "sha256:0b72c38033392a5f4621342fe11e9219ac11ec9d375f8e2a0c164539e0d70f6f"}, +] + +[package.dependencies] +pytest = ">=6.2.5" + +[package.extras] +dev = ["pre-commit", "pytest-asyncio", "tox"] + [[package]] name = "python-dateutil" version = "2.9.0.post0" @@ -1868,6 +2257,29 @@ files = [ {file = "pytz-2024.1.tar.gz", hash = "sha256:2a29735ea9c18baf14b448846bde5a48030ed267578472d8955cd0e7443a9812"}, ] +[[package]] +name = "pywin32" +version = "306" +description = "Python for Window Extensions" +optional = false +python-versions = "*" +files = [ + {file = "pywin32-306-cp310-cp310-win32.whl", hash = "sha256:06d3420a5155ba65f0b72f2699b5bacf3109f36acbe8923765c22938a69dfc8d"}, + {file = "pywin32-306-cp310-cp310-win_amd64.whl", hash = "sha256:84f4471dbca1887ea3803d8848a1616429ac94a4a8d05f4bc9c5dcfd42ca99c8"}, + {file = "pywin32-306-cp311-cp311-win32.whl", hash = "sha256:e65028133d15b64d2ed8f06dd9fbc268352478d4f9289e69c190ecd6818b6407"}, + {file = "pywin32-306-cp311-cp311-win_amd64.whl", hash = "sha256:a7639f51c184c0272e93f244eb24dafca9b1855707d94c192d4a0b4c01e1100e"}, + {file = "pywin32-306-cp311-cp311-win_arm64.whl", hash = "sha256:70dba0c913d19f942a2db25217d9a1b726c278f483a919f1abfed79c9cf64d3a"}, + {file = "pywin32-306-cp312-cp312-win32.whl", hash = "sha256:383229d515657f4e3ed1343da8be101000562bf514591ff383ae940cad65458b"}, + {file = "pywin32-306-cp312-cp312-win_amd64.whl", hash = "sha256:37257794c1ad39ee9be652da0462dc2e394c8159dfd913a8a4e8eb6fd346da0e"}, + {file = "pywin32-306-cp312-cp312-win_arm64.whl", hash = "sha256:5821ec52f6d321aa59e2db7e0a35b997de60c201943557d108af9d4ae1ec7040"}, + {file = "pywin32-306-cp37-cp37m-win32.whl", hash = "sha256:1c73ea9a0d2283d889001998059f5eaaba3b6238f767c9cf2833b13e6a685f65"}, + {file = "pywin32-306-cp37-cp37m-win_amd64.whl", hash = "sha256:72c5f621542d7bdd4fdb716227be0dd3f8565c11b280be6315b06ace35487d36"}, + {file = "pywin32-306-cp38-cp38-win32.whl", hash = "sha256:e4c092e2589b5cf0d365849e73e02c391c1349958c5ac3e9d5ccb9a28e017b3a"}, + {file = "pywin32-306-cp38-cp38-win_amd64.whl", hash = "sha256:e8ac1ae3601bee6ca9f7cb4b5363bf1c0badb935ef243c4733ff9a393b1690c0"}, + {file = "pywin32-306-cp39-cp39-win32.whl", hash = "sha256:e25fd5b485b55ac9c057f67d94bc203f3f6595078d1fb3b458c9c28b7153a802"}, + {file = "pywin32-306-cp39-cp39-win_amd64.whl", hash = "sha256:39b61c15272833b5c329a2989999dcae836b1eed650252ab1b7bfbe1d59f30f4"}, +] + [[package]] name = "pyyaml" version = "6.0.1" @@ -1928,6 +2340,111 @@ files = [ {file = "PyYAML-6.0.1.tar.gz", hash = "sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"}, ] +[[package]] +name = "pyzmq" +version = "25.1.2" +description = "Python bindings for 0MQ" +optional = false +python-versions = ">=3.6" +files = [ + {file = "pyzmq-25.1.2-cp310-cp310-macosx_10_15_universal2.whl", hash = "sha256:e624c789359f1a16f83f35e2c705d07663ff2b4d4479bad35621178d8f0f6ea4"}, + {file = "pyzmq-25.1.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:49151b0efece79f6a79d41a461d78535356136ee70084a1c22532fc6383f4ad0"}, + {file = "pyzmq-25.1.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d9a5f194cf730f2b24d6af1f833c14c10f41023da46a7f736f48b6d35061e76e"}, + {file = "pyzmq-25.1.2-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:faf79a302f834d9e8304fafdc11d0d042266667ac45209afa57e5efc998e3872"}, + {file = "pyzmq-25.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7f51a7b4ead28d3fca8dda53216314a553b0f7a91ee8fc46a72b402a78c3e43d"}, + {file = "pyzmq-25.1.2-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:0ddd6d71d4ef17ba5a87becf7ddf01b371eaba553c603477679ae817a8d84d75"}, + {file = "pyzmq-25.1.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:246747b88917e4867e2367b005fc8eefbb4a54b7db363d6c92f89d69abfff4b6"}, + {file = "pyzmq-25.1.2-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:00c48ae2fd81e2a50c3485de1b9d5c7c57cd85dc8ec55683eac16846e57ac979"}, + {file = "pyzmq-25.1.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:5a68d491fc20762b630e5db2191dd07ff89834086740f70e978bb2ef2668be08"}, + {file = "pyzmq-25.1.2-cp310-cp310-win32.whl", hash = "sha256:09dfe949e83087da88c4a76767df04b22304a682d6154de2c572625c62ad6886"}, + {file = "pyzmq-25.1.2-cp310-cp310-win_amd64.whl", hash = "sha256:fa99973d2ed20417744fca0073390ad65ce225b546febb0580358e36aa90dba6"}, + {file = "pyzmq-25.1.2-cp311-cp311-macosx_10_15_universal2.whl", hash = "sha256:82544e0e2d0c1811482d37eef297020a040c32e0687c1f6fc23a75b75db8062c"}, + {file = "pyzmq-25.1.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:01171fc48542348cd1a360a4b6c3e7d8f46cdcf53a8d40f84db6707a6768acc1"}, + {file = "pyzmq-25.1.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bc69c96735ab501419c432110016329bf0dea8898ce16fab97c6d9106dc0b348"}, + {file = "pyzmq-25.1.2-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3e124e6b1dd3dfbeb695435dff0e383256655bb18082e094a8dd1f6293114642"}, + {file = "pyzmq-25.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7598d2ba821caa37a0f9d54c25164a4fa351ce019d64d0b44b45540950458840"}, + {file = "pyzmq-25.1.2-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:d1299d7e964c13607efd148ca1f07dcbf27c3ab9e125d1d0ae1d580a1682399d"}, + {file = "pyzmq-25.1.2-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:4e6f689880d5ad87918430957297c975203a082d9a036cc426648fcbedae769b"}, + {file = "pyzmq-25.1.2-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:cc69949484171cc961e6ecd4a8911b9ce7a0d1f738fcae717177c231bf77437b"}, + {file = "pyzmq-25.1.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:9880078f683466b7f567b8624bfc16cad65077be046b6e8abb53bed4eeb82dd3"}, + {file = "pyzmq-25.1.2-cp311-cp311-win32.whl", hash = "sha256:4e5837af3e5aaa99a091302df5ee001149baff06ad22b722d34e30df5f0d9097"}, + {file = "pyzmq-25.1.2-cp311-cp311-win_amd64.whl", hash = "sha256:25c2dbb97d38b5ac9fd15586e048ec5eb1e38f3d47fe7d92167b0c77bb3584e9"}, + {file = "pyzmq-25.1.2-cp312-cp312-macosx_10_15_universal2.whl", hash = "sha256:11e70516688190e9c2db14fcf93c04192b02d457b582a1f6190b154691b4c93a"}, + {file = "pyzmq-25.1.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:313c3794d650d1fccaaab2df942af9f2c01d6217c846177cfcbc693c7410839e"}, + {file = "pyzmq-25.1.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1b3cbba2f47062b85fe0ef9de5b987612140a9ba3a9c6d2543c6dec9f7c2ab27"}, + {file = "pyzmq-25.1.2-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:fc31baa0c32a2ca660784d5af3b9487e13b61b3032cb01a115fce6588e1bed30"}, + {file = "pyzmq-25.1.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:02c9087b109070c5ab0b383079fa1b5f797f8d43e9a66c07a4b8b8bdecfd88ee"}, + {file = "pyzmq-25.1.2-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:f8429b17cbb746c3e043cb986328da023657e79d5ed258b711c06a70c2ea7537"}, + {file = "pyzmq-25.1.2-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:5074adeacede5f810b7ef39607ee59d94e948b4fd954495bdb072f8c54558181"}, + {file = "pyzmq-25.1.2-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:7ae8f354b895cbd85212da245f1a5ad8159e7840e37d78b476bb4f4c3f32a9fe"}, + {file = "pyzmq-25.1.2-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:b264bf2cc96b5bc43ce0e852be995e400376bd87ceb363822e2cb1964fcdc737"}, + {file = "pyzmq-25.1.2-cp312-cp312-win32.whl", hash = "sha256:02bbc1a87b76e04fd780b45e7f695471ae6de747769e540da909173d50ff8e2d"}, + {file = "pyzmq-25.1.2-cp312-cp312-win_amd64.whl", hash = "sha256:ced111c2e81506abd1dc142e6cd7b68dd53747b3b7ae5edbea4578c5eeff96b7"}, + {file = "pyzmq-25.1.2-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:7b6d09a8962a91151f0976008eb7b29b433a560fde056ec7a3db9ec8f1075438"}, + {file = "pyzmq-25.1.2-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:967668420f36878a3c9ecb5ab33c9d0ff8d054f9c0233d995a6d25b0e95e1b6b"}, + {file = "pyzmq-25.1.2-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5edac3f57c7ddaacdb4d40f6ef2f9e299471fc38d112f4bc6d60ab9365445fb0"}, + {file = "pyzmq-25.1.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:0dabfb10ef897f3b7e101cacba1437bd3a5032ee667b7ead32bbcdd1a8422fe7"}, + {file = "pyzmq-25.1.2-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:2c6441e0398c2baacfe5ba30c937d274cfc2dc5b55e82e3749e333aabffde561"}, + {file = "pyzmq-25.1.2-cp36-cp36m-musllinux_1_1_i686.whl", hash = "sha256:16b726c1f6c2e7625706549f9dbe9b06004dfbec30dbed4bf50cbdfc73e5b32a"}, + {file = "pyzmq-25.1.2-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:a86c2dd76ef71a773e70551a07318b8e52379f58dafa7ae1e0a4be78efd1ff16"}, + {file = "pyzmq-25.1.2-cp36-cp36m-win32.whl", hash = "sha256:359f7f74b5d3c65dae137f33eb2bcfa7ad9ebefd1cab85c935f063f1dbb245cc"}, + {file = "pyzmq-25.1.2-cp36-cp36m-win_amd64.whl", hash = "sha256:55875492f820d0eb3417b51d96fea549cde77893ae3790fd25491c5754ea2f68"}, + {file = "pyzmq-25.1.2-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:b8c8a419dfb02e91b453615c69568442e897aaf77561ee0064d789705ff37a92"}, + {file = "pyzmq-25.1.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8807c87fa893527ae8a524c15fc505d9950d5e856f03dae5921b5e9aa3b8783b"}, + {file = "pyzmq-25.1.2-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5e319ed7d6b8f5fad9b76daa0a68497bc6f129858ad956331a5835785761e003"}, + {file = "pyzmq-25.1.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:3c53687dde4d9d473c587ae80cc328e5b102b517447456184b485587ebd18b62"}, + {file = "pyzmq-25.1.2-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:9add2e5b33d2cd765ad96d5eb734a5e795a0755f7fc49aa04f76d7ddda73fd70"}, + {file = "pyzmq-25.1.2-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:e690145a8c0c273c28d3b89d6fb32c45e0d9605b2293c10e650265bf5c11cfec"}, + {file = "pyzmq-25.1.2-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:00a06faa7165634f0cac1abb27e54d7a0b3b44eb9994530b8ec73cf52e15353b"}, + {file = "pyzmq-25.1.2-cp37-cp37m-win32.whl", hash = "sha256:0f97bc2f1f13cb16905a5f3e1fbdf100e712d841482b2237484360f8bc4cb3d7"}, + {file = "pyzmq-25.1.2-cp37-cp37m-win_amd64.whl", hash = "sha256:6cc0020b74b2e410287e5942e1e10886ff81ac77789eb20bec13f7ae681f0fdd"}, + {file = "pyzmq-25.1.2-cp38-cp38-macosx_10_15_universal2.whl", hash = "sha256:bef02cfcbded83473bdd86dd8d3729cd82b2e569b75844fb4ea08fee3c26ae41"}, + {file = "pyzmq-25.1.2-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:e10a4b5a4b1192d74853cc71a5e9fd022594573926c2a3a4802020360aa719d8"}, + {file = "pyzmq-25.1.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:8c5f80e578427d4695adac6fdf4370c14a2feafdc8cb35549c219b90652536ae"}, + {file = "pyzmq-25.1.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:5dde6751e857910c1339890f3524de74007958557593b9e7e8c5f01cd919f8a7"}, + {file = "pyzmq-25.1.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ea1608dd169da230a0ad602d5b1ebd39807ac96cae1845c3ceed39af08a5c6df"}, + {file = "pyzmq-25.1.2-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:0f513130c4c361201da9bc69df25a086487250e16b5571ead521b31ff6b02220"}, + {file = "pyzmq-25.1.2-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:019744b99da30330798bb37df33549d59d380c78e516e3bab9c9b84f87a9592f"}, + {file = "pyzmq-25.1.2-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:2e2713ef44be5d52dd8b8e2023d706bf66cb22072e97fc71b168e01d25192755"}, + {file = "pyzmq-25.1.2-cp38-cp38-win32.whl", hash = "sha256:07cd61a20a535524906595e09344505a9bd46f1da7a07e504b315d41cd42eb07"}, + {file = "pyzmq-25.1.2-cp38-cp38-win_amd64.whl", hash = "sha256:eb7e49a17fb8c77d3119d41a4523e432eb0c6932187c37deb6fbb00cc3028088"}, + {file = "pyzmq-25.1.2-cp39-cp39-macosx_10_15_universal2.whl", hash = "sha256:94504ff66f278ab4b7e03e4cba7e7e400cb73bfa9d3d71f58d8972a8dc67e7a6"}, + {file = "pyzmq-25.1.2-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:6dd0d50bbf9dca1d0bdea219ae6b40f713a3fb477c06ca3714f208fd69e16fd8"}, + {file = "pyzmq-25.1.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:004ff469d21e86f0ef0369717351073e0e577428e514c47c8480770d5e24a565"}, + {file = "pyzmq-25.1.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:c0b5ca88a8928147b7b1e2dfa09f3b6c256bc1135a1338536cbc9ea13d3b7add"}, + {file = "pyzmq-25.1.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2c9a79f1d2495b167119d02be7448bfba57fad2a4207c4f68abc0bab4b92925b"}, + {file = "pyzmq-25.1.2-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:518efd91c3d8ac9f9b4f7dd0e2b7b8bf1a4fe82a308009016b07eaa48681af82"}, + {file = "pyzmq-25.1.2-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:1ec23bd7b3a893ae676d0e54ad47d18064e6c5ae1fadc2f195143fb27373f7f6"}, + {file = "pyzmq-25.1.2-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:db36c27baed588a5a8346b971477b718fdc66cf5b80cbfbd914b4d6d355e44e2"}, + {file = "pyzmq-25.1.2-cp39-cp39-win32.whl", hash = "sha256:39b1067f13aba39d794a24761e385e2eddc26295826530a8c7b6c6c341584289"}, + {file = "pyzmq-25.1.2-cp39-cp39-win_amd64.whl", hash = "sha256:8e9f3fabc445d0ce320ea2c59a75fe3ea591fdbdeebec5db6de530dd4b09412e"}, + {file = "pyzmq-25.1.2-pp310-pypy310_pp73-macosx_10_9_x86_64.whl", hash = "sha256:a8c1d566344aee826b74e472e16edae0a02e2a044f14f7c24e123002dcff1c05"}, + {file = "pyzmq-25.1.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:759cfd391a0996345ba94b6a5110fca9c557ad4166d86a6e81ea526c376a01e8"}, + {file = "pyzmq-25.1.2-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7c61e346ac34b74028ede1c6b4bcecf649d69b707b3ff9dc0fab453821b04d1e"}, + {file = "pyzmq-25.1.2-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4cb8fc1f8d69b411b8ec0b5f1ffbcaf14c1db95b6bccea21d83610987435f1a4"}, + {file = "pyzmq-25.1.2-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:3c00c9b7d1ca8165c610437ca0c92e7b5607b2f9076f4eb4b095c85d6e680a1d"}, + {file = "pyzmq-25.1.2-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:df0c7a16ebb94452d2909b9a7b3337940e9a87a824c4fc1c7c36bb4404cb0cde"}, + {file = "pyzmq-25.1.2-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:45999e7f7ed5c390f2e87ece7f6c56bf979fb213550229e711e45ecc7d42ccb8"}, + {file = "pyzmq-25.1.2-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:ac170e9e048b40c605358667aca3d94e98f604a18c44bdb4c102e67070f3ac9b"}, + {file = "pyzmq-25.1.2-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d1b604734bec94f05f81b360a272fc824334267426ae9905ff32dc2be433ab96"}, + {file = "pyzmq-25.1.2-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:a793ac733e3d895d96f865f1806f160696422554e46d30105807fdc9841b9f7d"}, + {file = "pyzmq-25.1.2-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:0806175f2ae5ad4b835ecd87f5f85583316b69f17e97786f7443baaf54b9bb98"}, + {file = "pyzmq-25.1.2-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:ef12e259e7bc317c7597d4f6ef59b97b913e162d83b421dd0db3d6410f17a244"}, + {file = "pyzmq-25.1.2-pp38-pypy38_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:ea253b368eb41116011add00f8d5726762320b1bda892f744c91997b65754d73"}, + {file = "pyzmq-25.1.2-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1b9b1f2ad6498445a941d9a4fee096d387fee436e45cc660e72e768d3d8ee611"}, + {file = "pyzmq-25.1.2-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:8b14c75979ce932c53b79976a395cb2a8cd3aaf14aef75e8c2cb55a330b9b49d"}, + {file = "pyzmq-25.1.2-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:889370d5174a741a62566c003ee8ddba4b04c3f09a97b8000092b7ca83ec9c49"}, + {file = "pyzmq-25.1.2-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9a18fff090441a40ffda8a7f4f18f03dc56ae73f148f1832e109f9bffa85df15"}, + {file = "pyzmq-25.1.2-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:99a6b36f95c98839ad98f8c553d8507644c880cf1e0a57fe5e3a3f3969040882"}, + {file = "pyzmq-25.1.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4345c9a27f4310afbb9c01750e9461ff33d6fb74cd2456b107525bbeebcb5be3"}, + {file = "pyzmq-25.1.2-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:3516e0b6224cf6e43e341d56da15fd33bdc37fa0c06af4f029f7d7dfceceabbc"}, + {file = "pyzmq-25.1.2-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:146b9b1f29ead41255387fb07be56dc29639262c0f7344f570eecdcd8d683314"}, + {file = "pyzmq-25.1.2.tar.gz", hash = "sha256:93f1aa311e8bb912e34f004cf186407a4e90eec4f0ecc0efd26056bf7eda0226"}, +] + +[package.dependencies] +cffi = {version = "*", markers = "implementation_name == \"pypy\""} + [[package]] name = "referencing" version = "0.34.0" @@ -2512,6 +3029,25 @@ numpy = "*" docs = ["linkify-it-py", "myst-parser", "sphinx", "sphinx-book-theme"] test = ["pytest"] +[[package]] +name = "stack-data" +version = "0.6.3" +description = "Extract data from python stack frames and tracebacks for informative displays" +optional = false +python-versions = "*" +files = [ + {file = "stack_data-0.6.3-py3-none-any.whl", hash = "sha256:d5558e0c25a4cb0853cddad3d77da9891a08cb85dd9f9f91b9f8cd66e511e695"}, + {file = "stack_data-0.6.3.tar.gz", hash = "sha256:836a778de4fec4dcd1dcd89ed8abff8a221f58308462e1c4aa2a3cf30148f0b9"}, +] + +[package.dependencies] +asttokens = ">=2.1.0" +executing = ">=1.2.0" +pure-eval = "*" + +[package.extras] +tests = ["cython", "littleutils", "pygments", "pytest", "typeguard"] + [[package]] name = "sympy" version = "1.12" @@ -2812,6 +3348,26 @@ torch = "2.2.2" [package.extras] scipy = ["scipy"] +[[package]] +name = "tornado" +version = "6.4" +description = "Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed." +optional = false +python-versions = ">= 3.8" +files = [ + {file = "tornado-6.4-cp38-abi3-macosx_10_9_universal2.whl", hash = "sha256:02ccefc7d8211e5a7f9e8bc3f9e5b0ad6262ba2fbb683a6443ecc804e5224ce0"}, + {file = "tornado-6.4-cp38-abi3-macosx_10_9_x86_64.whl", hash = "sha256:27787de946a9cffd63ce5814c33f734c627a87072ec7eed71f7fc4417bb16263"}, + {file = "tornado-6.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f7894c581ecdcf91666a0912f18ce5e757213999e183ebfc2c3fdbf4d5bd764e"}, + {file = "tornado-6.4-cp38-abi3-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e43bc2e5370a6a8e413e1e1cd0c91bedc5bd62a74a532371042a18ef19e10579"}, + {file = "tornado-6.4-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f0251554cdd50b4b44362f73ad5ba7126fc5b2c2895cc62b14a1c2d7ea32f212"}, + {file = "tornado-6.4-cp38-abi3-musllinux_1_1_aarch64.whl", hash = "sha256:fd03192e287fbd0899dd8f81c6fb9cbbc69194d2074b38f384cb6fa72b80e9c2"}, + {file = "tornado-6.4-cp38-abi3-musllinux_1_1_i686.whl", hash = "sha256:88b84956273fbd73420e6d4b8d5ccbe913c65d31351b4c004ae362eba06e1f78"}, + {file = "tornado-6.4-cp38-abi3-musllinux_1_1_x86_64.whl", hash = "sha256:71ddfc23a0e03ef2df1c1397d859868d158c8276a0603b96cf86892bff58149f"}, + {file = "tornado-6.4-cp38-abi3-win32.whl", hash = "sha256:6f8a6c77900f5ae93d8b4ae1196472d0ccc2775cc1dfdc9e7727889145c45052"}, + {file = "tornado-6.4-cp38-abi3-win_amd64.whl", hash = "sha256:10aeaa8006333433da48dec9fe417877f8bcc21f48dda8d661ae79da357b2a63"}, + {file = "tornado-6.4.tar.gz", hash = "sha256:72291fa6e6bc84e626589f1c29d90a5a6d593ef5ae68052ee2ef000dfd273dee"}, +] + [[package]] name = "tqdm" version = "4.66.2" @@ -2832,6 +3388,21 @@ notebook = ["ipywidgets (>=6)"] slack = ["slack-sdk"] telegram = ["requests"] +[[package]] +name = "traitlets" +version = "5.14.2" +description = "Traitlets Python configuration system" +optional = false +python-versions = ">=3.8" +files = [ + {file = "traitlets-5.14.2-py3-none-any.whl", hash = "sha256:fcdf85684a772ddeba87db2f398ce00b40ff550d1528c03c14dbf6a02003cd80"}, + {file = "traitlets-5.14.2.tar.gz", hash = "sha256:8cdd83c040dab7d1dee822678e5f5d100b514f7b72b01615b26fc5718916fdf9"}, +] + +[package.extras] +docs = ["myst-parser", "pydata-sphinx-theme", "sphinx"] +test = ["argcomplete (>=3.0.3)", "mypy (>=1.7.0)", "pre-commit", "pytest (>=7.0,<8.1)", "pytest-mock", "pytest-mypy-testing"] + [[package]] name = "transformers" version = "4.39.3" @@ -2982,6 +3553,17 @@ platformdirs = ">=3.9.1,<5" docs = ["furo (>=2023.7.26)", "proselint (>=0.13)", "sphinx (>=7.1.2)", "sphinx-argparse (>=0.4)", "sphinxcontrib-towncrier (>=0.2.1a0)", "towncrier (>=23.6)"] test = ["covdefaults (>=2.3)", "coverage (>=7.2.7)", "coverage-enable-subprocess (>=1)", "flaky (>=3.7)", "packaging (>=23.1)", "pytest (>=7.4)", "pytest-env (>=0.8.2)", "pytest-freezer (>=0.4.8)", "pytest-mock (>=3.11.1)", "pytest-randomly (>=3.12)", "pytest-timeout (>=2.1)", "setuptools (>=68)", "time-machine (>=2.10)"] +[[package]] +name = "wcwidth" +version = "0.2.13" +description = "Measures the displayed width of unicode strings in a terminal" +optional = false +python-versions = "*" +files = [ + {file = "wcwidth-0.2.13-py2.py3-none-any.whl", hash = "sha256:3da69048e4540d84af32131829ff948f1e022c1c6bdb8d6102117aac784f6859"}, + {file = "wcwidth-0.2.13.tar.gz", hash = "sha256:72ea0c06399eb286d978fdedb6923a9eb47e1c486ce63e9b4e64fc18303972b5"}, +] + [[package]] name = "xxhash" version = "3.4.1" @@ -3205,4 +3787,4 @@ multidict = ">=4.0" [metadata] lock-version = "2.0" python-versions = "^3.10" -content-hash = "3c3904d05963c1301b59f8d6a8d3bc027b752503f1de70d8089d8046d7b44ece" +content-hash = "86f88ae39d0a14a9241af53ec88b829968dbd059f46ddedd7ce23af8f090718a" diff --git a/pyproject.toml b/pyproject.toml index d2dac94d..9dc68785 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,8 @@ torchaudio = "^2.2.2" transformers = "^4.39.3" librosa = "^0.10.1" soundfile = "^0.12.1" +ffmpeg-python = "^0.2.0" +ipykernel = "^6.29.4" [tool.poetry.group.dev] optional = true @@ -42,6 +44,7 @@ mypy = "^1.9.0" pre-commit = "^3.7.0" pytest-cov = "^5.0.0" ruff = "^0.3.4" +pytest-mock = "^3.14.0" [tool.poetry.group.docs] optional = true diff --git a/src/pipepal/audio/tasks/__init__.py b/src/pipepal/audio/tasks/__init__.py index 55df3a94..36ddaacd 100644 --- a/src/pipepal/audio/tasks/__init__.py +++ b/src/pipepal/audio/tasks/__init__.py @@ -2,6 +2,5 @@ from .example_task import Interface as ExampleTask from .input_output import Interface as IOTask -from .raw_signal_processing import Interface as RawSignalProcessingTask -__all__ = ['ExampleTask', 'IOTask', 'RawSignalProcessingTask'] +__all__ = ['ExampleTask', 'IOTask'] diff --git a/src/pipepal/audio/tasks/example_task/interface.py b/src/pipepal/audio/tasks/example_task/interface.py index a8592565..e2be79e4 100644 --- a/src/pipepal/audio/tasks/example_task/interface.py +++ b/src/pipepal/audio/tasks/example_task/interface.py @@ -1,4 +1,4 @@ -"""This module defines an API for the task.""" +"""This module defines an API for managing services related to the example task.""" import os from typing import Any, Dict @@ -9,32 +9,68 @@ class Interface(AbstractComponent): - """A factory class for creating and managing service instances. - - It ensures a single instance per service type based on a unique key. + """A factory class for creating and managing instances of services related to the example task. + + This class facilitates the retrieval and singleton management of service instances based on + a unique identifier derived from the service data. The main functionality is provided through + the `run` method which processes input through a comprehensive workflow including + preprocessing, processing, and postprocessing phases, using the specified service instance. + + Attributes: + _instances (Dict[str, Any]): A class-level dictionary that caches service instances + to ensure they are singleton per type, indexed by a unique key. + + Examples: + >>> exampleTask = Interface() + >>> example_response = exampleTask.run({ + ... "service": { + ... "service_name": "ExampleService", + ... "model_checkpoint": "model.ckpt", + ... "model_version": "1.0", + ... }, + ... "data": { + ... "hello": "world" + ... } + ... }) + >>> print(example_response) + The output from the ExampleService after processing the input data. """ - _instances: Dict[str, Any] = {} # Class attribute for shared instance cache + _instances: Dict[str, Any] = {} # Cache to store unique service instances def __init__(self) -> None: - """Initialize the Interface class with the path to the base directory.""" + """Initialize the Interface class with the path to the directory where this file is located.""" # noqa: E501 super().__init__(os.path.dirname(__file__)) @classmethod - def get_service(cls, service_data: Dict[str, Any]) -> Any: + def get_service(cls, service_data: Dict[str, Any]) -> Any: # noqa: ANN401 """Retrieves or creates a service instance based on the provided service data. + This method ensures that each service type, identified by a composite key + (including the service name, model checkpoint, and version), has only one instance. + Parameters: - service_data (Dict[str, Any]): Data required to identify or create the service instance. + service_data (Dict[str, Any]): A dictionary containing the service configuration, + which must include 'service_name', and may include 'model_checkpoint' and 'model_version' + for specific service setups. Returns: Any: An instance of the requested service. Raises: - ValueError: If the service name is unsupported. + ValueError: If the 'service_name' in service_data is unsupported or not recognized. + + Examples: + >>> service_data = { + ... "service_name": "ExampleService", + ... "model_checkpoint": "model.ckpt", + ... "model_version": "1.0" + ... } + >>> service = Interface.get_service(service_data) + >>> print(service) + Instance of ExampleService configured with model checkpoint 'model.ckpt' and version 1.0 """ - # Use a composite key to uniquely identify instances - key: str = cls.get_data_uuid(service_data) + key: str = f"{service_data.get('service_name')}|{service_data.get('model_checkpoint')}|{service_data.get('model_version')}" # noqa: E501 if key not in cls._instances: if service_data["service_name"] == ExampleService.NAME: @@ -45,17 +81,37 @@ def get_service(cls, service_data: Dict[str, Any]) -> Any: @AbstractComponent.get_response_time @AbstractComponent.schema_validator - def run(self, input: Dict[str, Any]) -> Any: - """Processes input through a workflow: preprocessing, processing, and postprocessing. + def run(self, input_obj: Dict[str, Any]) -> Any: # noqa: ANN401 + """Processes input through a workflow of preprocessing, processing, and postprocessing. + + This method uses a service instance, which is fetched based on the service details provided + in 'input', to run the given data through the service's workflow. This includes preprocessing + the data, processing it according to the service's logic, and then postprocessing results. Parameters: - input (Dict[str, Any]): Input data containing service information and data to process. + input_obj (Dict[str, Any]): A dictionary containing: + - 'service': A dictionary with the service configuration. + - 'data': The data to be processed by the service. Returns: - Any: The postprocessed output from the service. + Any: The output of the service after the workflow has been applied to the input. + + Examples: + >>> input_data = { + ... "service": { + ... "service_name": "ExampleService", + ... "model_checkpoint": "model.ckpt", + ... "model_version": "1.0" + ... }, + ... "data": {"hello": "world"} + ... } + >>> exampleTask = Interface() + >>> output = exampleTask.run(input_data) + >>> print(output) + The processed data. """ - service = self.get_service(input["service"]) - preprocessing_output = service.preprocess(input["data"]) + service = self.get_service(input_obj["service"]) + preprocessing_output = service.preprocess(input_obj["data"]) processing_output = service.process(preprocessing_output) postprocessing_output = service.postprocess(processing_output) return postprocessing_output \ No newline at end of file diff --git a/src/pipepal/audio/tasks/example_task/schemas/input.json b/src/pipepal/audio/tasks/example_task/schemas/run/input.json similarity index 100% rename from src/pipepal/audio/tasks/example_task/schemas/input.json rename to src/pipepal/audio/tasks/example_task/schemas/run/input.json diff --git a/src/pipepal/audio/tasks/example_task/schemas/run/output.json b/src/pipepal/audio/tasks/example_task/schemas/run/output.json new file mode 100644 index 00000000..23d695a7 --- /dev/null +++ b/src/pipepal/audio/tasks/example_task/schemas/run/output.json @@ -0,0 +1,13 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "properties": { + "output": { + "type": "string", + "description": "The output from the ExampleService", + "pattern": "^ExampleService output$" + } + }, + "required": ["output"], + "additionalProperties": true + } diff --git a/src/pipepal/audio/tasks/example_task/services/example_service/service.py b/src/pipepal/audio/tasks/example_task/services/example_service/service.py index 6e6696b0..fbcc61eb 100644 --- a/src/pipepal/audio/tasks/example_task/services/example_service/service.py +++ b/src/pipepal/audio/tasks/example_task/services/example_service/service.py @@ -1,52 +1,110 @@ -"""This module implements an example service for the task.""" - +"""This module implements an example service class which extends the AbstractService. + +The service is designed to demonstrate a typical implementation setup where +preprocessing, processing, and postprocessing steps are defined to handle data +in a manner specific to the service's requirements. + +Example: + Demonstrate how to use the ExampleService: + + >>> from path.to.this.module import Service + >>> input_obj = { + "data": { + "hello": "world" + }, + "service": { + "service_name": "ExampleService", + "model_checkpoint": "model.ckpt", + "model_version": "1.0", + } + } + >>> service = Service() + >>> preprocessing_output = service.preprocess(input_obj["data"]) + >>> processing_output = service.process(preprocessing_output) + >>> postprocessing_output = service.postprocess(processing_output) + >>> print(postprocessing_output) + {'output': 'ExampleService output'} + +Attributes: + NAME (str): A class attribute which gives the service a name, used internally. +""" + +import os from typing import Any, Dict from ...abstract_service import AbstractService class Service(AbstractService): - """Example service that extends AbstractService.""" + """Example service that extends AbstractService to demonstrate custom processing steps. + + This service class exemplifies a basic structure of a service in a system designed + for processing data through computational steps: preprocess, process, and postprocess. + + Attributes: + NAME (str): The public name of the service, intended for identification in registries. + """ NAME: str = "ExampleService" - def __init__(self, configs: Dict[str, Any]) -> None: # noqa: ANN401 - """Initialize the service with given configurations. + def __init__(self, configs: Dict[str, Any]) -> None: + """Initialize the service class with the path to the base directory. + + The initialization involves setting up the base directory and potentially other + configurations necessary for the service's operations. Args: - configs: A dictionary of configurations for the service. + configs (Dict[str, Any]): The configs dictionary for the service. """ super().__init__() - - def preprocess(self, data: Any) -> Any: # noqa: ANN401 - """Preprocess input data. Implementation can be customized. + + def preprocess(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Preprocess the input data to fit the requirements of this service's processing step. Args: - data: The input data to preprocess. + data (Dict[str, Any]): The input data to preprocess, expected to be in dictionary format. Returns: - The preprocessed data. + Dict[str, Any]: The preprocessed data, adjusted according to the service's needs. This + implementation simply passes the data through without modification. + + Example: + >>> service.preprocess({"hello": "world"}) + {'hello': 'world'} """ - return super().preprocess(data) + return data + + def process(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Process the input data to produce a service-specific output. - def process(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Process input data. Custom implementation for ExampleService. + This method is the core of the service where the main data manipulation happens. The current + implementation outputs a placeholder dictionary for demonstration purposes. Args: - data: The input data to process. + data (Dict[str, Any]): The preprocessed data ready for processing. Returns: - A dictionary containing 'output' key with a sample output. + Dict[str, Any]: A dictionary containing 'output' key with a string value representing + the result of data processing. + + Example: + >>> service.process({"hello": "preprocessed world"}) + {'output': 'ExampleService output'} """ return {"output": "ExampleService output"} - def postprocess(self, data: Any) -> Any: # noqa: ANN401 - """Postprocess processed data. Implementation can be customized. + def postprocess(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Postprocess the processed data to format the output as required by downstream services or storage solutions. Args: - data: The data to postprocess. + data (Dict[str, Any]): The data to postprocess after it has been processed. Typically involves + final adjustments before sending the data to the next step or storing it. Returns: - The postprocessed data. + Dict[str, Any]: The postprocessed data, which in this case is the same as the input data. + + Example: + >>> service.postprocess({'output': 'ExampleService output'}) + {'output': 'ExampleService output'} """ - return super().postprocess(data) \ No newline at end of file + return data diff --git a/src/pipepal/audio/tasks/input_output/abstract_service.py b/src/pipepal/audio/tasks/input_output/abstract_service.py index d8683235..3b34377e 100644 --- a/src/pipepal/audio/tasks/input_output/abstract_service.py +++ b/src/pipepal/audio/tasks/input_output/abstract_service.py @@ -1,41 +1,84 @@ -"""This module defines an abstract service for the task.""" +"""This module defines an abstract service for the audio IO task.""" from abc import ABC, abstractmethod from typing import Any, Dict class AbstractService(ABC): - """Abstract base class for services. + """Abstract base class for audio IO services. - This class provides a template for services with preprocess, process, - and postprocess methods. + This class provides a template for services that handle audio input/output operations, + ensuring that all essential methods such as reading audio from disk, saving datasets to disk, + and uploading datasets to the Hugging Face Hub are implemented. """ - def preprocess(self, data: Any) -> Any: # noqa: ANN401 - """Preprocess input data. + + @abstractmethod + def read_audios_from_disk(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Reads audio files from disk and creates a dataset. + + Args: + data (Dict[str, Any]): A dictionary with a key 'files' which is a list of file paths to audio files. + + Returns: + Dict[str, Any]: A dictionary with a single key 'output', containing the dataset created from the audio files. + + Raises: + ValueError: If the 'files' key is not in the input dictionary. + FileNotFoundError: If any audio files listed in the 'files' key do not exist. + """ + pass + + @abstractmethod + def save_HF_dataset_to_disk(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Saves a Hugging Face `Dataset` object to disk. Args: - data: The input data to preprocess. + input_obj (Dict[str, Any]): A dictionary containing: + - 'dataset': A `Dataset` object to save. + - 'output_path': A string representing the path to save the dataset to. Returns: - The preprocessed data. + Dict[str, Any]: A dictionary confirming the output status. """ - return data + pass @abstractmethod - def process(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Process input data. + def upload_HF_dataset_to_HF_hub(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Uploads a Hugging Face `Dataset` object to the Hugging Face Hub. Args: - data: The input data to process. + input_obj (Dict[str, Any]): A dictionary containing: + - 'dataset': A `Dataset` object to upload. + - 'output_uri': A string representing the URI to the remote directory where the dataset will be uploaded. Returns: - A dictionary with the processed output. + Dict[str, Any]: A dictionary confirming the upload status. """ - - def postprocess(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Postprocess the processed data. + pass + - :param data: The data to postprocess. - :return: The postprocessed data. + @abstractmethod + def read_local_HF_dataset(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Loads a Hugging Face `Dataset` object from a local directory. + + Args: + input_obj (Dict[str, Any]): A dictionary containing: + - path (str): The file path to the local directory containing the dataset. + + Returns: + Dict[str, Any]: A dictionary with a key 'output', containing the loaded `Dataset` object. + """ + pass + + @abstractmethod + def read_HF_dataset_from_HF_hub(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Loads a Hugging Face `Dataset` object from the Hugging Face Hub. + + Args: + input_obj (Dict[str, Any]): A dictionary containing: + - uri (str): The URI to the dataset on the Hugging Face Hub. + + Returns: + Dict[str, Any]: A dictionary with a key 'output', containing the loaded `Dataset` object. """ - return data \ No newline at end of file + pass \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/interface.py b/src/pipepal/audio/tasks/input_output/interface.py index b923a6c3..aaef9d1d 100644 --- a/src/pipepal/audio/tasks/input_output/interface.py +++ b/src/pipepal/audio/tasks/input_output/interface.py @@ -5,7 +5,6 @@ from pipepal.utils.abstract_component import AbstractComponent -from .services import ExampleService from .services import DatasetsService @@ -38,9 +37,7 @@ def get_service(cls, service_data: Dict[str, Any]) -> Any: key: str = cls.get_data_uuid(service_data) if key not in cls._instances: - if service_data["service_name"] == ExampleService.NAME: - cls._instances[key] = ExampleService(service_data) - elif service_data["service_name"] == DatasetsService.NAME: + if service_data["service_name"] == DatasetsService.NAME: cls._instances[key] = DatasetsService(service_data) else: raise ValueError(f"Unsupported service: {service_data['service_name']}") @@ -48,8 +45,8 @@ def get_service(cls, service_data: Dict[str, Any]) -> Any: @AbstractComponent.get_response_time @AbstractComponent.schema_validator - def run(self, input: Dict[str, Any]) -> Any: - """Processes input through a workflow: preprocessing, processing, and postprocessing. + def read_audios_from_disk(self, input: Dict[str, Any]) -> Any: + """Processes input through a workflow. Parameters: input (Dict[str, Any]): Input data containing service information and data to process. @@ -58,7 +55,77 @@ def run(self, input: Dict[str, Any]) -> Any: Any: The postprocessed output from the service. """ service = self.get_service(input["service"]) - preprocessing_output = service.preprocess(input["data"]) - processing_output = service.process(preprocessing_output) - postprocessing_output = service.postprocess(processing_output) - return postprocessing_output \ No newline at end of file + output = service.read_audios_from_disk(input["data"]) + return output + + @AbstractComponent.get_response_time + @AbstractComponent.schema_validator + def save_HF_dataset_to_disk(self, input: Dict[str, Any]) -> Dict[str, Any]: + """Saves HF dataset to disk. + + Parameters: + input (Dict[str, Any]): Input data containing service information and data to process. + + Returns: + Any: The postprocessed output from the service. + + Todo: + - This method is not audio specific and may be moved out of this class. + """ + service = self.get_service(input["service"]) + output = service.save_HF_dataset_to_disk(input["data"]) + return output + + @AbstractComponent.get_response_time + @AbstractComponent.schema_validator + def upload_HF_dataset_to_HF_hub(self, input: Dict[str, Any]) -> Dict[str, Any]: + """Uploads HF dataset to HF hub. + + Parameters: + input (Dict[str, Any]): Input data containing service information and data to process. + + Returns: + Any: The postprocessed output from the service. + + Todo: + - This method is not audio specific and may be moved out of this class. + """ + service = self.get_service(input["service"]) + output = service.upload_HF_dataset_to_HF_hub(input["data"]) + return output + + @AbstractComponent.get_response_time + @AbstractComponent.schema_validator + def read_local_HF_dataset(self, input: Dict[str, Any]) -> Dict[str, Any]: + """Reads HF dataset from disk. + + Parameters: + input (Dict[str, Any]): Input data containing service information and data to process. + + Returns: + Any: The postprocessed output from the service. + + Todo: + - This method is not audio specific and may be moved out of this class. + """ + service = self.get_service(input["service"]) + output = service.read_local_HF_dataset(input["data"]) + return output + + @AbstractComponent.get_response_time + @AbstractComponent.schema_validator + def read_HF_dataset_from_HF_hub(self, input: Dict[str, Any]) -> Dict[str, Any]: + """Reads HF dataset from HF hub. + + Parameters: + input (Dict[str, Any]): Input data containing service information and data to process. + + Returns: + Any: The postprocessed output from the service. + + Todo: + - This method is not audio specific and may be moved out of this class. + """ + service = self.get_service(input["service"]) + output = service.read_HF_dataset_from_HF_hub(input["data"]) + return output \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/input.json b/src/pipepal/audio/tasks/input_output/schemas/input.json deleted file mode 100644 index 2bc151c2..00000000 --- a/src/pipepal/audio/tasks/input_output/schemas/input.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "ServiceConfiguration", - "description": "A JSON Schema for service configuration including additional properties.", - "type": "object", - "properties": { - "service": { - "type": "object", - "properties": { - "service_name": { - "type": "string" - } - }, - "required": ["service_name"], - "additionalProperties": true - } - }, - "additionalProperties": true -} diff --git a/src/pipepal/audio/tasks/input_output/schemas/output.json b/src/pipepal/audio/tasks/input_output/schemas/output.json deleted file mode 100644 index 893b995b..00000000 --- a/src/pipepal/audio/tasks/input_output/schemas/output.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "AcceptEverything", - "description": "A JSON Schema that accepts everything.", - "type": "object", - "additionalProperties": true -} \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/read_HF_dataset_from_HF_hub/input.json b/src/pipepal/audio/tasks/input_output/schemas/read_HF_dataset_from_HF_hub/input.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/read_HF_dataset_from_HF_hub/input.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/read_HF_dataset_from_HF_hub/output.json b/src/pipepal/audio/tasks/input_output/schemas/read_HF_dataset_from_HF_hub/output.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/read_HF_dataset_from_HF_hub/output.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/read_audios_from_disk/input.json b/src/pipepal/audio/tasks/input_output/schemas/read_audios_from_disk/input.json new file mode 100644 index 00000000..d32850c2 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/read_audios_from_disk/input.json @@ -0,0 +1,36 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "description": "Schema for validating input JSON structure for audio file processing services.", + "type": "object", + "properties": { + "data": { + "type": "object", + "description": "Contains all data necessary for processing, specifically a list of audio files.", + "properties": { + "files": { + "type": "array", + "description": "An array of file paths, each with a supported audio format.", + "items": { + "type": "string", + "description": "A file path ending with a valid audio format extension.", + "pattern": ".*\\.(wav|mp3|aac|ogg|flac|mka)$" + } + } + }, + "required": ["files"] + }, + "service": { + "type": "object", + "description": "Contains information about the service handling the files.", + "properties": { + "service_name": { + "type": "string", + "description": "The name of the service processing the audio files." + } + }, + "required": ["service_name"] + } + }, + "required": ["data", "service"], + "additionalProperties": false +} diff --git a/src/pipepal/audio/tasks/example_task/schemas/output.json b/src/pipepal/audio/tasks/input_output/schemas/read_audios_from_disk/output.json similarity index 100% rename from src/pipepal/audio/tasks/example_task/schemas/output.json rename to src/pipepal/audio/tasks/input_output/schemas/read_audios_from_disk/output.json diff --git a/src/pipepal/audio/tasks/input_output/schemas/read_local_HF_dataset/input.json b/src/pipepal/audio/tasks/input_output/schemas/read_local_HF_dataset/input.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/read_local_HF_dataset/input.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/read_local_HF_dataset/output.json b/src/pipepal/audio/tasks/input_output/schemas/read_local_HF_dataset/output.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/read_local_HF_dataset/output.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/save_HF_dataset_to_disk/input.json b/src/pipepal/audio/tasks/input_output/schemas/save_HF_dataset_to_disk/input.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/save_HF_dataset_to_disk/input.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/save_HF_dataset_to_disk/output.json b/src/pipepal/audio/tasks/input_output/schemas/save_HF_dataset_to_disk/output.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/save_HF_dataset_to_disk/output.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/upload_HF_dataset_to_HF_hub copy 2/input.json b/src/pipepal/audio/tasks/input_output/schemas/upload_HF_dataset_to_HF_hub copy 2/input.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/upload_HF_dataset_to_HF_hub copy 2/input.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/schemas/upload_HF_dataset_to_HF_hub copy 2/output.json b/src/pipepal/audio/tasks/input_output/schemas/upload_HF_dataset_to_HF_hub copy 2/output.json new file mode 100644 index 00000000..f5aad854 --- /dev/null +++ b/src/pipepal/audio/tasks/input_output/schemas/upload_HF_dataset_to_HF_hub copy 2/output.json @@ -0,0 +1,7 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Schema that accepts everything", + "description": "This schema will validate anything you throw at it.", + "type": ["object", "array", "string", "number", "boolean", "null"] + } + \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/services/__init__.py b/src/pipepal/audio/tasks/input_output/services/__init__.py index c2a0044d..e33ce2bf 100644 --- a/src/pipepal/audio/tasks/input_output/services/__init__.py +++ b/src/pipepal/audio/tasks/input_output/services/__init__.py @@ -1,6 +1,5 @@ """This module provides the implementation of IO for videos.""" from .datasets import Service as DatasetsService -from .example_service import Service as ExampleService -__all__ = ['DatasetsService', 'ExampleService'] +__all__ = ['DatasetsService'] diff --git a/src/pipepal/audio/tasks/input_output/services/datasets/service.py b/src/pipepal/audio/tasks/input_output/services/datasets/service.py index 9e644139..a01e5932 100644 --- a/src/pipepal/audio/tasks/input_output/services/datasets/service.py +++ b/src/pipepal/audio/tasks/input_output/services/datasets/service.py @@ -1,7 +1,8 @@ """This module implements an example service for the task.""" +import os from typing import Any, Dict -from datasets import Audio, Dataset +from datasets import Audio, Dataset, load_dataset from ...abstract_service import AbstractService @@ -19,38 +20,117 @@ def __init__(self, configs: Dict[str, Any]) -> None: # noqa: ANN401 """ super().__init__() - def preprocess(self, data: Any) -> Any: # noqa: ANN401 - """Preprocess input data. Implementation can be customized. + def read_audios_from_disk(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Reads audio files from disk and creates a Hugging Face `Dataset` object. - Args: - data: The input data to preprocess. + This function checks if all audio files listed under the 'files' key in the input dictionary exist on disk. + If all files are found, it creates a Dataset object where each file is handled as an audio file. The resulting + dataset is then returned inside a dictionary under the key 'output'. + + Parameters: + data (Dict[str, Any]): A dictionary with a key 'files' which is a list of strings. Each string should be + the file path to an audio file. Returns: - The preprocessed data. + Dict[str, Any]: A dictionary with a single key 'output', which contains the `Dataset` object. The 'audio' + column of this dataset is of type `datasets.Audio`. + + Raises: + ValueError: If the 'files' key is not in the input dictionary. + FileNotFoundError: If any of the audio files listed in the 'files' key do not exist. + + Example: + >>> data = {"files": ["path/to/audio1.wav", "path/to/audio2.wav"]} + >>> output_dataset = self.read_audios_from_disk(data) + >>> print(type(output_dataset["output"])) + + >>> print(output_dataset["output"].column_names) + ['audio'] + """ + # Check if 'files' key exists in the data dictionary + if "files" not in data: + raise ValueError("Input data must contain 'files' key with a list of audio file paths.") + + # Controlling that the input files exist + missing_files = [file for file in data["files"] if not os.path.exists(file)] + if missing_files: + raise FileNotFoundError(f"The following files were not found: {missing_files}") + + # Creating the Dataset object + audio_dataset = Dataset.from_dict({"audio": data["files"]}) + + # Specifying the column type as Audio + audio_dataset = audio_dataset.cast_column("audio", Audio(mono=False)) + + # Wrapping the Dataset object in a dictionary + return {"output": audio_dataset} + + def save_HF_dataset_to_disk(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Saves a Hugging Face `Dataset` object to disk. + + Parameters: + input_obj (Dict[str, Any]): A dictionary with + - a key 'dataset' which is a `Dataset` object, + - a key 'output_path' which is a string representing the path to the output directory. + + Returns: + None + + Todo: + - Add error handling + - Add output format as an optional parameter """ - return super().preprocess(data) + # Use os.makedirs to create the output directory, ignore error if it already exists + os.makedirs(input_obj['output_path'], exist_ok=True) - def process(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Process input data. Custom implementation for ExampleService. + # Saving the Dataset object to disk + input_obj["dataset"].save_to_disk(input_obj["output_path"]) + + return { "output": None } + + def upload_HF_dataset_to_HF_hub(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Uploads a Hugging Face `Dataset` object to the Hugging Face Hub. + + Parameters: + input_obj (Dict[str, Any]): A dictionary with + - a key 'dataset' which is a `Dataset` object, + - a key 'output_uri' which is a string representing the URI to the remote directory. + + Returns: + None + + Todo: + - Add error handling + - Add output format as an optional parameter + - Add token handling for private HF repositories + """ + # Uploading the Dataset object to the Hugging Face Hub + input_obj["dataset"].push_to_hub(input_obj["output_uri"]) + + return { "output": None } + + def read_local_HF_dataset(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Loads a Hugging Face `Dataset` object from a local directory. Args: - data: The input data to process. + input_obj (Dict[str, Any]): A dictionary containing: + - path (str): The file path to the local directory containing the dataset. Returns: - A dictionary containing 'output' key with a sample output. + Dict[str, Any]: A dictionary with a key 'output', containing the loaded `Dataset` object. """ - audio_dataset = Dataset.from_dict( - {"audio": data['files']} - ).cast_column("audio", Audio(mono=False)) - return {"output": audio_dataset} + dataset = Dataset.load_from_disk(input_obj["path"]) + return {"output": dataset} - def postprocess(self, data: Any) -> Any: # noqa: ANN401 - """Postprocess processed data. Implementation can be customized. + def read_HF_dataset_from_HF_hub(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Loads a Hugging Face `Dataset` object from the Hugging Face Hub. Args: - data: The data to postprocess. + input_obj (Dict[str, Any]): A dictionary containing: + - uri (str): The URI to the dataset on the Hugging Face Hub. Returns: - The postprocessed data. + Dict[str, Any]: A dictionary with a key 'output', containing the loaded `Dataset` object. """ - return super().postprocess(data) \ No newline at end of file + dataset = load_dataset(input_obj["uri"]) + return {"output": dataset} \ No newline at end of file diff --git a/src/pipepal/audio/tasks/input_output/services/example_service/__init__.py b/src/pipepal/audio/tasks/input_output/services/example_service/__init__.py deleted file mode 100644 index ff62197e..00000000 --- a/src/pipepal/audio/tasks/input_output/services/example_service/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""This module provides the implementation of ExampleService for IO task.""" - -from .service import Service - -__all__ = ['Service'] diff --git a/src/pipepal/audio/tasks/input_output/services/example_service/service.py b/src/pipepal/audio/tasks/input_output/services/example_service/service.py deleted file mode 100644 index 6e6696b0..00000000 --- a/src/pipepal/audio/tasks/input_output/services/example_service/service.py +++ /dev/null @@ -1,52 +0,0 @@ -"""This module implements an example service for the task.""" - -from typing import Any, Dict - -from ...abstract_service import AbstractService - - -class Service(AbstractService): - """Example service that extends AbstractService.""" - - NAME: str = "ExampleService" - - def __init__(self, configs: Dict[str, Any]) -> None: # noqa: ANN401 - """Initialize the service with given configurations. - - Args: - configs: A dictionary of configurations for the service. - """ - super().__init__() - - def preprocess(self, data: Any) -> Any: # noqa: ANN401 - """Preprocess input data. Implementation can be customized. - - Args: - data: The input data to preprocess. - - Returns: - The preprocessed data. - """ - return super().preprocess(data) - - def process(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Process input data. Custom implementation for ExampleService. - - Args: - data: The input data to process. - - Returns: - A dictionary containing 'output' key with a sample output. - """ - return {"output": "ExampleService output"} - - def postprocess(self, data: Any) -> Any: # noqa: ANN401 - """Postprocess processed data. Implementation can be customized. - - Args: - data: The data to postprocess. - - Returns: - The postprocessed data. - """ - return super().postprocess(data) \ No newline at end of file diff --git a/src/pipepal/audio/tasks/raw_signal_processing/interface.py b/src/pipepal/audio/tasks/raw_signal_processing/interface.py deleted file mode 100644 index 81dc2e64..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/interface.py +++ /dev/null @@ -1,63 +0,0 @@ -"""This module defines an API for the task.""" - -import os -from typing import Any, Dict - -from pipepal.utils.abstract_component import AbstractComponent - -from .services import ExampleService, TorchaudioService - - -class Interface(AbstractComponent): - """A factory class for creating and managing service instances. - - It ensures a single instance per service type based on a unique key. - """ - - _instances: Dict[str, Any] = {} # Class attribute for shared instance cache - - def __init__(self) -> None: - """Initialize the Interface class with the path to the base directory.""" - super().__init__(os.path.dirname(__file__)) - - @classmethod - def get_service(cls, service_data: Dict[str, Any]) -> Any: - """Retrieves or creates a service instance based on the provided service data. - - Parameters: - service_data (Dict[str, Any]): Data required to identify or create the service instance. - - Returns: - Any: An instance of the requested service. - - Raises: - ValueError: If the service name is unsupported. - """ - # Use a composite key to uniquely identify instances - key: str = cls.get_data_uuid(service_data) - - if key not in cls._instances: - if service_data["service_name"] == ExampleService.NAME: - cls._instances[key] = ExampleService(service_data) - elif service_data["service_name"] == TorchaudioService.NAME: - cls._instances[key] = TorchaudioService(service_data) - else: - raise ValueError(f"Unsupported service: {service_data['service_name']}") - return cls._instances[key] - - @AbstractComponent.get_response_time - @AbstractComponent.schema_validator - def run(self, input: Dict[str, Any]) -> Any: - """Processes input through a workflow: preprocessing, processing, and postprocessing. - - Parameters: - input (Dict[str, Any]): Input data containing service information and data to process. - - Returns: - Any: The postprocessed output from the service. - """ - service = self.get_service(input["service"]) - preprocessing_output = service.preprocess(input["data"]) - processing_output = service.process(preprocessing_output) - postprocessing_output = service.postprocess(processing_output) - return postprocessing_output \ No newline at end of file diff --git a/src/pipepal/audio/tasks/raw_signal_processing/schemas/input.json b/src/pipepal/audio/tasks/raw_signal_processing/schemas/input.json deleted file mode 100644 index 2bc151c2..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/schemas/input.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "ServiceConfiguration", - "description": "A JSON Schema for service configuration including additional properties.", - "type": "object", - "properties": { - "service": { - "type": "object", - "properties": { - "service_name": { - "type": "string" - } - }, - "required": ["service_name"], - "additionalProperties": true - } - }, - "additionalProperties": true -} diff --git a/src/pipepal/audio/tasks/raw_signal_processing/schemas/output.json b/src/pipepal/audio/tasks/raw_signal_processing/schemas/output.json deleted file mode 100644 index 893b995b..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/schemas/output.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "AcceptEverything", - "description": "A JSON Schema that accepts everything.", - "type": "object", - "additionalProperties": true -} \ No newline at end of file diff --git a/src/pipepal/audio/tasks/raw_signal_processing/services/__init__.py b/src/pipepal/audio/tasks/raw_signal_processing/services/__init__.py deleted file mode 100644 index a3f404f9..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/services/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -"""This module provides the implementation of IO for videos.""" - -from .example_service import Service as ExampleService -from .torchaudio import Service as TorchaudioService - -__all__ = ['ExampleService', 'TorchaudioService'] diff --git a/src/pipepal/audio/tasks/raw_signal_processing/services/example_service/__init__.py b/src/pipepal/audio/tasks/raw_signal_processing/services/example_service/__init__.py deleted file mode 100644 index ff62197e..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/services/example_service/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""This module provides the implementation of ExampleService for IO task.""" - -from .service import Service - -__all__ = ['Service'] diff --git a/src/pipepal/audio/tasks/raw_signal_processing/services/example_service/service.py b/src/pipepal/audio/tasks/raw_signal_processing/services/example_service/service.py deleted file mode 100644 index 6e6696b0..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/services/example_service/service.py +++ /dev/null @@ -1,52 +0,0 @@ -"""This module implements an example service for the task.""" - -from typing import Any, Dict - -from ...abstract_service import AbstractService - - -class Service(AbstractService): - """Example service that extends AbstractService.""" - - NAME: str = "ExampleService" - - def __init__(self, configs: Dict[str, Any]) -> None: # noqa: ANN401 - """Initialize the service with given configurations. - - Args: - configs: A dictionary of configurations for the service. - """ - super().__init__() - - def preprocess(self, data: Any) -> Any: # noqa: ANN401 - """Preprocess input data. Implementation can be customized. - - Args: - data: The input data to preprocess. - - Returns: - The preprocessed data. - """ - return super().preprocess(data) - - def process(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Process input data. Custom implementation for ExampleService. - - Args: - data: The input data to process. - - Returns: - A dictionary containing 'output' key with a sample output. - """ - return {"output": "ExampleService output"} - - def postprocess(self, data: Any) -> Any: # noqa: ANN401 - """Postprocess processed data. Implementation can be customized. - - Args: - data: The data to postprocess. - - Returns: - The postprocessed data. - """ - return super().postprocess(data) \ No newline at end of file diff --git a/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/__init__.py b/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/__init__.py deleted file mode 100644 index ff62197e..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""This module provides the implementation of ExampleService for IO task.""" - -from .service import Service - -__all__ = ['Service'] diff --git a/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/config.json b/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/config.json deleted file mode 100644 index 9e26dfee..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/config.json +++ /dev/null @@ -1 +0,0 @@ -{} \ No newline at end of file diff --git a/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/service.py b/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/service.py deleted file mode 100644 index ce94ba7c..00000000 --- a/src/pipepal/audio/tasks/raw_signal_processing/services/torchaudio/service.py +++ /dev/null @@ -1,241 +0,0 @@ -"""This module implements an example service for the task.""" - -from datasets import Dataset -import numpy as np -import functools -import torch -from typing import Any, Dict, List - -from ...abstract_service import AbstractService - - -class Service(AbstractService): - """torchaudio service that extends AbstractService.""" - - NAME: str = "torchaudio" - - def __init__(self, configs: Dict[str, Any]) -> None: # noqa: ANN401 - """Initialize the service with given configurations. - - Args: - configs: A dictionary of configurations for the service. - """ - super().__init__() - - def preprocess(self, data: Any) -> Any: # noqa: ANN401 - """Preprocess input data. Implementation can be customized. - - Args: - data: The input data to preprocess. - - Returns: - The preprocessed data. - """ - return super().preprocess(data) - - def process(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Process input data. Custom implementation for ExampleService. - - Args: - data: The input data to process. - - Returns: - A dictionary containing 'output' key with a sample output. - """ - new_dataset = data["dataset"] - if "channeling" in data: - new_dataset = filter_dataset_channels(new_dataset, - channels_to_keep=data['channeling']['channels_to_keep']) - print("SHSHSJSJSJS") - input("sjsk") - - return {"output": new_dataset} - - def postprocess(self, data: Any) -> Any: # noqa: ANN401 - """Postprocess processed data. Implementation can be customized. - - Args: - data: The data to postprocess. - - Returns: - The postprocessed data. - """ - return super().postprocess(data) - - -def filter_dataset_channels(dataset: Dataset, channels_to_keep: List[int]) -> Dataset: - """Applies channel filtering to all audio objects in a specified column of a Hugging Face Dataset. - - Parameters: - - dataset (Dataset): The Hugging Face Dataset to process. Assumes the dataset has a column - containing audio objects with 'array' and 'sampling_rate'. - - channels_to_keep (List[int]): A list of channel indices to keep in the filtered audio objects. - - Returns: - - Dataset: A new Hugging Face Dataset with filtered audio objects in the specified column. - """ - - def filter_audio_channels(batch): - """Filters specified channels from audio objects in a batch and returns modified audio objects. - - Parameters: - - batch: A batch from the dataset containing 'audio' objects with 'array' and 'sampling_rate'. - - Returns: - - The batch with filtered audio arrays. - """ - print("channels_to_keep is: ", channels_to_keep) - - - # Extract arrays and sampling_rates - arrays = [item['array'] for item in batch['audio']] - sampling_rates = [item['sampling_rate'] for item in batch['audio']] - - # Ensure arrays are PyTorch tensors for efficient processing - processed_arrays = [] - for array in arrays: - array = torch.tensor(array) if not isinstance(array, torch.Tensor) else array - if array.ndim == 1: - array = array.unsqueeze(0) # Ensure there's a channel dimension - filtered_array = array[channels_to_keep] # Filter channels - processed_arrays.append(filtered_array.squeeze().numpy()) # Convert back to NumPy array for consistency - - # Update the 'audio' objects in the batch - for i, item in enumerate(batch['audio']): - item['array'] = processed_arrays[i] - item['sampling_rate'] = sampling_rates[i] - - return batch - - # Apply the channel filtering using the map function with batched processing - return dataset.map( - function=filter_audio_channels, - batched=True, - batch_size=None, # Auto-batch size or specify your own - with_indices=False, # Set to True if you need indices within the mapping function - remove_columns=None # Specify if you want to remove columns post mapping - ) - - -''' -def filter_dataset_channels(dataset: Dataset, channels_to_keep: List[int]) -> Dataset: - """Applies channel filtering to all audio objects in a specified column of a Hugging Face Dataset. - - Parameters: - - dataset (Dataset): The Hugging Face Dataset to process. Assumes the dataset has a column - containing audio objects with 'array' and 'sampling_rate'. - - channels_to_keep (List[int]): A list of channel indices to keep in the filtered audio objects. - - Returns: - - Dataset: A new Hugging Face Dataset with filtered audio objects in the specified column. - """ - - def filter_audio_channels( - dataset_row: Dict[str, torch.Tensor], channels_to_keep: List[int] - ) -> Dict[str, torch.Tensor]: - """Filters specified channels from an audio object and returns a modified audio object. - - Parameters: - - dataset_row (Dict[str, torch.Tensor]): An audio object containing 'array' and 'sampling_rate'. - The 'array' is expected to be a tensor of shape (num_channels, num_frames). - - channels_to_keep (List[int]): A list of channel indices to keep in the output audio object. - - Returns: - - Dict[str, torch.Tensor]: A modified audio object with filtered channels. This object will contain - 'array' with only the specified channels and the same 'sampling_rate' as the input. - - Example: - >>> dataset_row = {"array": torch.randn(2, 16000), "sampling_rate": 16000} - >>> filtered_audio = filter_audio_channels(dataset_row["audio"], [0]) # Keep only the first channel - >>> print(filtered_audio["audio"]["array"].shape) - torch.Size([1, 16000]) - """ - # Extract the array and sampling_rate from the audio object - array, sampling_rate = dataset_row["audio"]["array"], dataset_row["audio"]["sampling_rate"] - - if isinstance(array, np.ndarray): - array = torch.from_numpy(array) - - if array.ndim == 1: - # If array is 1D, treat it as a single channel (mono audio) - array = array.unsqueeze(0) # Add a channel dimension - - # Filter the channels - filtered_array = array[channels_to_keep] - - filtered_array = filtered_array.squeeze(0) - if isinstance(dataset_row["audio"]["array"], np.ndarray): - filtered_array = filtered_array.numpy() - filtered_array = filtered_array.astype(np.float32) - - # Return the filtered audio object - return { - "audio": { - "array": filtered_array, - "sampling_rate": sampling_rate, - "path": dataset_row["audio"]["path"], - } - } - - def apply_filter(row): - row = filter_audio_channels(row, channels_to_keep) - return row - - return dataset.map(apply_filter) -''' - -def convert_dataset_to_mono(dataset, conversion_method="average", channels=[0]): - """Converts all audio files in a Hugging Face dataset from stereo to mono. - - Parameters: - - dataset: The loaded Hugging Face audio dataset. - - conversion_method: Method of conversion ('average' or 'single'). - - channels: Channels to use for 'single' conversion method. - - Returns: - - A new dataset with mono audio. - """ - - def convert_example_to_mono( - audio_data: dict, conversion_method: str = "average", channels: List[int] = [0] - ) -> Dict[str, Any]: - """Converts stereo audio to mono based on the specified conversion method. - - Parameters: - - audio_data: A dictionary containing 'array' (torch.Tensor) with the audio data and 'sampling_rate'. - - conversion_method: 'average' for averaging all channels, 'single' to use specific channels. - - channels: List of channel indices to use for conversion when conversion_method is 'single'. - - Returns: - - A dictionary containing the mono audio data (torch.Tensor) and the sampling rate (int). - """ - print(audio_data) - input("ssjjs") - if not audio_data or "array" not in audio_data or "sampling_rate" not in audio_data: - raise ValueError( - "audio_data must be a dictionary containing 'array' and 'sampling_rate'." - ) - - if conversion_method not in ["average", "single"]: - raise ValueError("conversion_method must be either 'average' or 'single'.") - - if conversion_method == "average": - mono_audio = torch.mean(audio_data["array"], dim=0, keepdim=True) - elif conversion_method == "single": - if channels is None or not all(isinstance(ch, int) for ch in channels): - raise ValueError( - "When conversion_method is 'single', channels must be a list of integer indices." - ) - mono_audio = torch.mean(audio_data["array"][channels], dim=0, keepdim=True) - else: - raise ValueError("Invalid conversion method.") - - return {"array": mono_audio, "sampling_rate": audio_data["sampling_rate"]} - - # Create a partial function with conversion_method and channels pre-specified - convert_func = functools.partial( - convert_example_to_mono, conversion_method=conversion_method, channels=channels - ) - - # Apply the conversion function to each example in the dataset - return dataset.map(lambda example: convert_func(audio_data=example["audio"])) diff --git a/src/pipepal/audio/tasks/speech_emotion_recognition/.gitkeep b/src/pipepal/audio/tasks/speech_emotion_recognition/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/src/pipepal/audio/tasks/speech_enhancement/.gitkeep b/src/pipepal/audio/tasks/speech_enhancement/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/src/pipepal/audio/tasks/speech_to_text/.gitkeep b/src/pipepal/audio/tasks/speech_to_text/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/src/pipepal/audio/tasks/speech_verification/.gitkeep b/src/pipepal/audio/tasks/speech_verification/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/src/pipepal/audio/tasks/text_to_speech/.gitkeep b/src/pipepal/audio/tasks/text_to_speech/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/src/pipepal/audio/tasks/voice_conversion/.gitkeep b/src/pipepal/audio/tasks/voice_conversion/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/src/pipepal/utils/abstract_component.py b/src/pipepal/utils/abstract_component.py index 8542f7fc..6c759e87 100644 --- a/src/pipepal/utils/abstract_component.py +++ b/src/pipepal/utils/abstract_component.py @@ -30,24 +30,27 @@ class AbstractComponent(ABC): def __init__( self, base_dir: str, - input_schema_file: str = "schemas/input.json", - output_schema_file: str = "schemas/output.json", + input_schema_file: str = "schemas/__FUNCTION_NAME_PLACEHOLDER__/input.json", + output_schema_file: str = "schemas/__FUNCTION_NAME_PLACEHOLDER__/output.json", ) -> None: """Initializes the component with paths to input and output JSON schemas.""" - self.input_schema = self.read_json_schema(os.path.join(base_dir, input_schema_file)) - self.output_schema = self.read_json_schema(os.path.join(base_dir, output_schema_file)) + self.base_dir = base_dir + self.base_input_schema = input_schema_file + self.base_output_schema = output_schema_file @staticmethod def schema_validator(func: Callable) -> Callable: """Decorator to validate input and output against schemas.""" - @wraps(func) def wrapper(self: "AbstractComponent", *args: Any, **kwargs: Any) -> Any: + input_schema = self.read_json_schema(os.path.join(self.base_dir, self.base_input_schema.replace("__FUNCTION_NAME_PLACEHOLDER__", func.__name__))) + output_schema = self.read_json_schema(os.path.join(self.base_dir, self.base_output_schema.replace("__FUNCTION_NAME_PLACEHOLDER__", func.__name__))) + # Validate input input_data = kwargs.get("input_data") or (args[0] if args else {}) - if self.input_schema is not None: + if input_schema is not None: try: - validate(instance=input_data, schema=self.input_schema) + validate(instance=input_data, schema=input_schema) except ValidationError as e: raise ValueError(f"Input validation error: {e}") from e @@ -55,9 +58,9 @@ def wrapper(self: "AbstractComponent", *args: Any, **kwargs: Any) -> Any: result = func(self, *args, **kwargs) # Validate output - if self.output_schema is not None: + if output_schema is not None: try: - validate(instance=result, schema=self.output_schema) + validate(instance=result, schema=output_schema) except ValidationError as e: raise ValueError(f"Output validation error: {e}") from e diff --git a/src/pipepal/audio/tasks/data_augmentation/.gitkeep b/src/pipepal/utils/functions.py similarity index 100% rename from src/pipepal/audio/tasks/data_augmentation/.gitkeep rename to src/pipepal/utils/functions.py diff --git a/src/pipepal/video/input_output/__init__.py b/src/pipepal/video/input_output/__init__.py deleted file mode 100644 index ac08086b..00000000 --- a/src/pipepal/video/input_output/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""This module provides the implementation of IO for videos.""" - -from .interface import Interface - -__all__ = ['Interface'] diff --git a/src/pipepal/video/input_output/schemas/output.json b/src/pipepal/video/input_output/schemas/output.json deleted file mode 100644 index 893b995b..00000000 --- a/src/pipepal/video/input_output/schemas/output.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "AcceptEverything", - "description": "A JSON Schema that accepts everything.", - "type": "object", - "additionalProperties": true -} \ No newline at end of file diff --git a/src/pipepal/video/input_output/services/example_service/config.json b/src/pipepal/video/input_output/services/example_service/config.json deleted file mode 100644 index 9e26dfee..00000000 --- a/src/pipepal/video/input_output/services/example_service/config.json +++ /dev/null @@ -1 +0,0 @@ -{} \ No newline at end of file diff --git a/src/pipepal/video/input_output/services/example_service/service.py b/src/pipepal/video/input_output/services/example_service/service.py deleted file mode 100644 index 90d15d7d..00000000 --- a/src/pipepal/video/input_output/services/example_service/service.py +++ /dev/null @@ -1,52 +0,0 @@ -"""This module implements an example service for the task.""" - -from typing import Any, Dict - -from pipepal.utils.abstract_service import AbstractService - - -class Service(AbstractService): - """Example service that extends AbstractService.""" - - NAME: str = "ExampleService" - - def __init__(self, configs: Dict[str, Any]) -> None: # noqa: ANN401 - """Initialize the service with given configurations. - - Args: - configs: A dictionary of configurations for the service. - """ - super().__init__() - - def preprocess(self, data: Any) -> Any: # noqa: ANN401 - """Preprocess input data. Implementation can be customized. - - Args: - data: The input data to preprocess. - - Returns: - The preprocessed data. - """ - return super().preprocess(data) - - def process(self, data: Any) -> Dict[str, Any]: # noqa: ANN401 - """Process input data. Custom implementation for ExampleService. - - Args: - data: The input data to process. - - Returns: - A dictionary containing 'output' key with a sample output. - """ - return {"output": "ExampleService output"} - - def postprocess(self, data: Any) -> Any: # noqa: ANN401 - """Postprocess processed data. Implementation can be customized. - - Args: - data: The data to postprocess. - - Returns: - The postprocessed data. - """ - return super().postprocess(data) \ No newline at end of file diff --git a/src/pipepal/audio/tasks/data_representation/.gitkeep b/src/pipepal/video/pipes/.gitkeep similarity index 100% rename from src/pipepal/audio/tasks/data_representation/.gitkeep rename to src/pipepal/video/pipes/.gitkeep diff --git a/src/pipepal/video/tasks/__init__.py b/src/pipepal/video/tasks/__init__.py new file mode 100644 index 00000000..36ddaacd --- /dev/null +++ b/src/pipepal/video/tasks/__init__.py @@ -0,0 +1,6 @@ +"""This module provides the implementation of pipepal tasks for audio.""" + +from .example_task import Interface as ExampleTask +from .input_output import Interface as IOTask + +__all__ = ['ExampleTask', 'IOTask'] diff --git a/src/pipepal/video/tasks/example_task/__init__.py b/src/pipepal/video/tasks/example_task/__init__.py new file mode 100644 index 00000000..3e5a1759 --- /dev/null +++ b/src/pipepal/video/tasks/example_task/__init__.py @@ -0,0 +1,5 @@ +"""This module provides the implementation of ExampleTask.""" + +from .interface import Interface + +__all__ = ['Interface'] diff --git a/src/pipepal/audio/tasks/raw_signal_processing/abstract_service.py b/src/pipepal/video/tasks/example_task/abstract_service.py similarity index 100% rename from src/pipepal/audio/tasks/raw_signal_processing/abstract_service.py rename to src/pipepal/video/tasks/example_task/abstract_service.py diff --git a/src/pipepal/video/tasks/example_task/interface.py b/src/pipepal/video/tasks/example_task/interface.py new file mode 100644 index 00000000..e2be79e4 --- /dev/null +++ b/src/pipepal/video/tasks/example_task/interface.py @@ -0,0 +1,117 @@ +"""This module defines an API for managing services related to the example task.""" + +import os +from typing import Any, Dict + +from pipepal.utils.abstract_component import AbstractComponent + +from .services import ExampleService + + +class Interface(AbstractComponent): + """A factory class for creating and managing instances of services related to the example task. + + This class facilitates the retrieval and singleton management of service instances based on + a unique identifier derived from the service data. The main functionality is provided through + the `run` method which processes input through a comprehensive workflow including + preprocessing, processing, and postprocessing phases, using the specified service instance. + + Attributes: + _instances (Dict[str, Any]): A class-level dictionary that caches service instances + to ensure they are singleton per type, indexed by a unique key. + + Examples: + >>> exampleTask = Interface() + >>> example_response = exampleTask.run({ + ... "service": { + ... "service_name": "ExampleService", + ... "model_checkpoint": "model.ckpt", + ... "model_version": "1.0", + ... }, + ... "data": { + ... "hello": "world" + ... } + ... }) + >>> print(example_response) + The output from the ExampleService after processing the input data. + """ + + _instances: Dict[str, Any] = {} # Cache to store unique service instances + + def __init__(self) -> None: + """Initialize the Interface class with the path to the directory where this file is located.""" # noqa: E501 + super().__init__(os.path.dirname(__file__)) + + @classmethod + def get_service(cls, service_data: Dict[str, Any]) -> Any: # noqa: ANN401 + """Retrieves or creates a service instance based on the provided service data. + + This method ensures that each service type, identified by a composite key + (including the service name, model checkpoint, and version), has only one instance. + + Parameters: + service_data (Dict[str, Any]): A dictionary containing the service configuration, + which must include 'service_name', and may include 'model_checkpoint' and 'model_version' + for specific service setups. + + Returns: + Any: An instance of the requested service. + + Raises: + ValueError: If the 'service_name' in service_data is unsupported or not recognized. + + Examples: + >>> service_data = { + ... "service_name": "ExampleService", + ... "model_checkpoint": "model.ckpt", + ... "model_version": "1.0" + ... } + >>> service = Interface.get_service(service_data) + >>> print(service) + Instance of ExampleService configured with model checkpoint 'model.ckpt' and version 1.0 + """ + key: str = f"{service_data.get('service_name')}|{service_data.get('model_checkpoint')}|{service_data.get('model_version')}" # noqa: E501 + + if key not in cls._instances: + if service_data["service_name"] == ExampleService.NAME: + cls._instances[key] = ExampleService(service_data) + else: + raise ValueError(f"Unsupported service: {service_data['service_name']}") + return cls._instances[key] + + @AbstractComponent.get_response_time + @AbstractComponent.schema_validator + def run(self, input_obj: Dict[str, Any]) -> Any: # noqa: ANN401 + """Processes input through a workflow of preprocessing, processing, and postprocessing. + + This method uses a service instance, which is fetched based on the service details provided + in 'input', to run the given data through the service's workflow. This includes preprocessing + the data, processing it according to the service's logic, and then postprocessing results. + + Parameters: + input_obj (Dict[str, Any]): A dictionary containing: + - 'service': A dictionary with the service configuration. + - 'data': The data to be processed by the service. + + Returns: + Any: The output of the service after the workflow has been applied to the input. + + Examples: + >>> input_data = { + ... "service": { + ... "service_name": "ExampleService", + ... "model_checkpoint": "model.ckpt", + ... "model_version": "1.0" + ... }, + ... "data": {"hello": "world"} + ... } + >>> exampleTask = Interface() + >>> output = exampleTask.run(input_data) + >>> print(output) + The processed data. + """ + service = self.get_service(input_obj["service"]) + preprocessing_output = service.preprocess(input_obj["data"]) + processing_output = service.process(preprocessing_output) + postprocessing_output = service.postprocess(processing_output) + return postprocessing_output \ No newline at end of file diff --git a/src/pipepal/video/input_output/schemas/input.json b/src/pipepal/video/tasks/example_task/schemas/run/input.json similarity index 100% rename from src/pipepal/video/input_output/schemas/input.json rename to src/pipepal/video/tasks/example_task/schemas/run/input.json diff --git a/src/pipepal/video/tasks/example_task/schemas/run/output.json b/src/pipepal/video/tasks/example_task/schemas/run/output.json new file mode 100644 index 00000000..23d695a7 --- /dev/null +++ b/src/pipepal/video/tasks/example_task/schemas/run/output.json @@ -0,0 +1,13 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "properties": { + "output": { + "type": "string", + "description": "The output from the ExampleService", + "pattern": "^ExampleService output$" + } + }, + "required": ["output"], + "additionalProperties": true + } diff --git a/src/pipepal/video/input_output/services/__init__.py b/src/pipepal/video/tasks/example_task/services/__init__.py similarity index 57% rename from src/pipepal/video/input_output/services/__init__.py rename to src/pipepal/video/tasks/example_task/services/__init__.py index 4c69094d..2948c46c 100644 --- a/src/pipepal/video/input_output/services/__init__.py +++ b/src/pipepal/video/tasks/example_task/services/__init__.py @@ -1,4 +1,4 @@ -"""This module provides the implementation of IO for videos.""" +"""This module provides the implementation of ExampleTask.""" from .example_service import Service as ExampleService diff --git a/src/pipepal/video/input_output/services/example_service/__init__.py b/src/pipepal/video/tasks/example_task/services/example_service/__init__.py similarity index 87% rename from src/pipepal/video/input_output/services/example_service/__init__.py rename to src/pipepal/video/tasks/example_task/services/example_service/__init__.py index ff62197e..e06a14f0 100644 --- a/src/pipepal/video/input_output/services/example_service/__init__.py +++ b/src/pipepal/video/tasks/example_task/services/example_service/__init__.py @@ -1,4 +1,4 @@ -"""This module provides the implementation of ExampleService for IO task.""" +"""This module provides the implementation of ExampleService for ExampleTask.""" from .service import Service diff --git a/src/pipepal/audio/tasks/input_output/services/example_service/config.json b/src/pipepal/video/tasks/example_task/services/example_service/config.json similarity index 100% rename from src/pipepal/audio/tasks/input_output/services/example_service/config.json rename to src/pipepal/video/tasks/example_task/services/example_service/config.json diff --git a/src/pipepal/video/tasks/example_task/services/example_service/service.py b/src/pipepal/video/tasks/example_task/services/example_service/service.py new file mode 100644 index 00000000..fbcc61eb --- /dev/null +++ b/src/pipepal/video/tasks/example_task/services/example_service/service.py @@ -0,0 +1,110 @@ +"""This module implements an example service class which extends the AbstractService. + +The service is designed to demonstrate a typical implementation setup where +preprocessing, processing, and postprocessing steps are defined to handle data +in a manner specific to the service's requirements. + +Example: + Demonstrate how to use the ExampleService: + + >>> from path.to.this.module import Service + >>> input_obj = { + "data": { + "hello": "world" + }, + "service": { + "service_name": "ExampleService", + "model_checkpoint": "model.ckpt", + "model_version": "1.0", + } + } + >>> service = Service() + >>> preprocessing_output = service.preprocess(input_obj["data"]) + >>> processing_output = service.process(preprocessing_output) + >>> postprocessing_output = service.postprocess(processing_output) + >>> print(postprocessing_output) + {'output': 'ExampleService output'} + +Attributes: + NAME (str): A class attribute which gives the service a name, used internally. +""" + +import os +from typing import Any, Dict + +from ...abstract_service import AbstractService + + +class Service(AbstractService): + """Example service that extends AbstractService to demonstrate custom processing steps. + + This service class exemplifies a basic structure of a service in a system designed + for processing data through computational steps: preprocess, process, and postprocess. + + Attributes: + NAME (str): The public name of the service, intended for identification in registries. + """ + + NAME: str = "ExampleService" + + def __init__(self, configs: Dict[str, Any]) -> None: + """Initialize the service class with the path to the base directory. + + The initialization involves setting up the base directory and potentially other + configurations necessary for the service's operations. + + Args: + configs (Dict[str, Any]): The configs dictionary for the service. + """ + super().__init__() + + def preprocess(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Preprocess the input data to fit the requirements of this service's processing step. + + Args: + data (Dict[str, Any]): The input data to preprocess, expected to be in dictionary format. + + Returns: + Dict[str, Any]: The preprocessed data, adjusted according to the service's needs. This + implementation simply passes the data through without modification. + + Example: + >>> service.preprocess({"hello": "world"}) + {'hello': 'world'} + """ + return data + + def process(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Process the input data to produce a service-specific output. + + This method is the core of the service where the main data manipulation happens. The current + implementation outputs a placeholder dictionary for demonstration purposes. + + Args: + data (Dict[str, Any]): The preprocessed data ready for processing. + + Returns: + Dict[str, Any]: A dictionary containing 'output' key with a string value representing + the result of data processing. + + Example: + >>> service.process({"hello": "preprocessed world"}) + {'output': 'ExampleService output'} + """ + return {"output": "ExampleService output"} + + def postprocess(self, data: Dict[str, Any]) -> Dict[str, Any]: + """Postprocess the processed data to format the output as required by downstream services or storage solutions. + + Args: + data (Dict[str, Any]): The data to postprocess after it has been processed. Typically involves + final adjustments before sending the data to the next step or storing it. + + Returns: + Dict[str, Any]: The postprocessed data, which in this case is the same as the input data. + + Example: + >>> service.postprocess({'output': 'ExampleService output'}) + {'output': 'ExampleService output'} + """ + return data diff --git a/src/pipepal/audio/tasks/raw_signal_processing/__init__.py b/src/pipepal/video/tasks/input_output/__init__.py similarity index 100% rename from src/pipepal/audio/tasks/raw_signal_processing/__init__.py rename to src/pipepal/video/tasks/input_output/__init__.py diff --git a/src/pipepal/video/tasks/input_output/abstract_service.py b/src/pipepal/video/tasks/input_output/abstract_service.py new file mode 100644 index 00000000..d6c3e2f8 --- /dev/null +++ b/src/pipepal/video/tasks/input_output/abstract_service.py @@ -0,0 +1,39 @@ +"""This module defines an abstract service for the video IO task.""" + +from abc import ABC, abstractmethod +from typing import Any, Dict + + +class AbstractService(ABC): + """Abstract base class for video I/O services. + + This class serves as a template for defining services that handle + the input/output operations related to video files, specifically + focusing on extracting audio components from video data. + + Methods defined here outline the expected interface for such services, + including the extraction of audio from video. Implementations should + provide specific logic to handle various video formats and ensure the + integrity and quality of the extracted audio. + + """ + + @abstractmethod + def extract_audios_from_videos(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Extracts the audio track from the video data provided in the input object. + + This method should be implemented by subclasses to extract audio data from + a video file encapsulated within `input_obj`. The specifics of `input_obj` + (such as format and contents) should be clearly defined in subclass implementations. + + Parameters: + input_obj (Dict[str, Any]): A dictionary containing the list of video files + and metadata necessary for the extraction process. The expected format + of this dictionary needs to be defined by the concrete subclass. + + Returns: + Dict[str, Any]: A dictionary containing the list of extracted audio data and + potentially additional metadata pertaining to the audio track. + + """ + pass # implementation of this method is required by subclasses diff --git a/src/pipepal/video/input_output/interface.py b/src/pipepal/video/tasks/input_output/interface.py similarity index 63% rename from src/pipepal/video/input_output/interface.py rename to src/pipepal/video/tasks/input_output/interface.py index a8592565..efa18fa4 100644 --- a/src/pipepal/video/input_output/interface.py +++ b/src/pipepal/video/tasks/input_output/interface.py @@ -5,7 +5,7 @@ from pipepal.utils.abstract_component import AbstractComponent -from .services import ExampleService +from .services import FfmpegService class Interface(AbstractComponent): @@ -37,25 +37,15 @@ def get_service(cls, service_data: Dict[str, Any]) -> Any: key: str = cls.get_data_uuid(service_data) if key not in cls._instances: - if service_data["service_name"] == ExampleService.NAME: - cls._instances[key] = ExampleService(service_data) + if service_data["service_name"] == FfmpegService.NAME: + cls._instances[key] = FfmpegService(service_data) else: raise ValueError(f"Unsupported service: {service_data['service_name']}") return cls._instances[key] @AbstractComponent.get_response_time @AbstractComponent.schema_validator - def run(self, input: Dict[str, Any]) -> Any: - """Processes input through a workflow: preprocessing, processing, and postprocessing. - - Parameters: - input (Dict[str, Any]): Input data containing service information and data to process. - - Returns: - Any: The postprocessed output from the service. - """ - service = self.get_service(input["service"]) - preprocessing_output = service.preprocess(input["data"]) - processing_output = service.process(preprocessing_output) - postprocessing_output = service.postprocess(processing_output) - return postprocessing_output \ No newline at end of file + def extract_audios_from_videos(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + service = self.get_service(input_obj["service"]) + output = service.extract_audios_from_videos(input_obj["data"]) + return output \ No newline at end of file diff --git a/src/pipepal/video/tasks/input_output/schemas/extract_audios_from_videos/input.json b/src/pipepal/video/tasks/input_output/schemas/extract_audios_from_videos/input.json new file mode 100644 index 00000000..a3fdcc55 --- /dev/null +++ b/src/pipepal/video/tasks/input_output/schemas/extract_audios_from_videos/input.json @@ -0,0 +1,53 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "AudioExtractionServiceConfiguration", + "description": "A JSON Schema for configuring a service to extract audio from video files.", + "type": "object", + "properties": { + "data": { + "type": "object", + "properties": { + "files": { + "type": "array", + "items": { + "type": "string", + "pattern": ".*\\.(mp4|mov|wmv|webm|mkv|m4v|flv|avi)$" + }, + "description": "List of paths to video files with extensions .mp4, .mov, .wmv, .webm, .mkv, .m4v, .flv, or .avi. [TODO] In the future we may want to guarantee support more extensions." + }, + "audio_format": { + "type": "string", + "enum": ["wav", "mp3", "aac", "ogg", "flac", "mka"], + "description": "The format of the extracted audio file" + }, + "audio_codec": { + "type": "string", + "enum": ["pcm_s16le", "libmp3lame", "aac", "libvorbis", "flac", "copy"], + "description": "The codec used to encode the extracted audio" + }, + "output_folder": { + "type": "string", + "pattern": "^(\\/|\\w:\\\\|\\.\\.\\/|\\.\\/).+", + "description": "The output folder where the extracted audio files will be saved" + } + }, + "required": ["files", "audio_format", "audio_codec", "output_folder"], + "additionalProperties": false + }, + "service": { + "type": "object", + "properties": { + "service_name": { + "type": "string", + "default": "ffmpeg", + "description": "The name of the service used for extracting audio" + } + }, + "required": ["service_name"], + "additionalProperties": false + } + }, + "required": ["data", "service"], + "additionalProperties": false + } + \ No newline at end of file diff --git a/src/pipepal/video/tasks/input_output/schemas/extract_audios_from_videos/output.json b/src/pipepal/video/tasks/input_output/schemas/extract_audios_from_videos/output.json new file mode 100644 index 00000000..fe283998 --- /dev/null +++ b/src/pipepal/video/tasks/input_output/schemas/extract_audios_from_videos/output.json @@ -0,0 +1,14 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "properties": { + "output": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": ["output"], + "additionalProperties": false +} diff --git a/src/pipepal/video/tasks/input_output/services/__init__.py b/src/pipepal/video/tasks/input_output/services/__init__.py new file mode 100644 index 00000000..6943f703 --- /dev/null +++ b/src/pipepal/video/tasks/input_output/services/__init__.py @@ -0,0 +1,5 @@ +"""This module provides the implementation of IO for videos.""" + +from .ffmpeg import Service as FfmpegService + +__all__ = ['FfmpegService'] diff --git a/src/pipepal/video/tasks/input_output/services/ffmpeg/__init__.py b/src/pipepal/video/tasks/input_output/services/ffmpeg/__init__.py new file mode 100644 index 00000000..a1c5d0af --- /dev/null +++ b/src/pipepal/video/tasks/input_output/services/ffmpeg/__init__.py @@ -0,0 +1,5 @@ +"""This module provides the implementation of ffmpeg for IO task.""" + +from .service import Service + +__all__ = ['Service'] diff --git a/src/pipepal/audio/tasks/raw_signal_processing/services/example_service/config.json b/src/pipepal/video/tasks/input_output/services/ffmpeg/config.json similarity index 100% rename from src/pipepal/audio/tasks/raw_signal_processing/services/example_service/config.json rename to src/pipepal/video/tasks/input_output/services/ffmpeg/config.json diff --git a/src/pipepal/video/tasks/input_output/services/ffmpeg/service.py b/src/pipepal/video/tasks/input_output/services/ffmpeg/service.py new file mode 100644 index 00000000..b411434a --- /dev/null +++ b/src/pipepal/video/tasks/input_output/services/ffmpeg/service.py @@ -0,0 +1,169 @@ +"""This module implements an example service for the task.""" + +import os +from typing import Any, Dict, List + +import ffmpeg +from ...abstract_service import AbstractService + + +class Service(AbstractService): + """Example service that extends AbstractService.""" + + NAME: str = "ffmpeg" + + def __init__(self, configs: Dict[str, Any]) -> None: # noqa: ANN401 + """Initialize the service with given configurations. + + Args: + configs: A dictionary of configurations for the service. + """ + super().__init__() + + def extract_audios_from_videos(self, input_obj: Dict[str, Any]) -> Dict[str, Any]: + """Extracts audio files from the video files provided in the input dictionary. + + This function processes a list of video files, extracting the audio from each file and saving it + in a specified format and directory. Each audio file's name is derived from its corresponding + video file, preserving the original hierarchy in the output directory. + + Parameters: + input_obj (Dict[str, Any]): A dictionary containing the necessary inputs with the following keys: + - 'files' (List[str]): A list of paths to the video files. + - 'output_folder' (str): The directory where the extracted audio files will be saved. + - 'audio_format' (str): The file extension for the output audio files (e.g., '.mp3', '.wav'). + - 'audio_codec' (str): The codec to use for encoding the audio files (e.g., 'mp3', 'aac'). + + Returns: + Dict[str, Any]: A dictionary containing one key: + - 'audio_files' (List[str]): A list of paths to the extracted audio files. + + Example: + >>> input_obj = { + 'files': ['/path/to/video1.mp4', '/path/to/video2.avi'], + 'output_folder': '/path/to/output', + 'audio_format': '.wav', + 'audio_codec': 'pcm_s16le' + } + >>> output = extract_audios_from_videos(input_obj) + >>> output['audio_files'] + ['/path/to/output/video1.wav', '/path/to/output/video2.wav'] + + Note: + The function assumes that all video files reside under a common root directory. + + Todo: + - Optimize the code for efficiency. + """ + audio_files = [] + + # Use os.makedirs to create the directory, ignore error if it already exists + os.makedirs(input_obj['output_folder'], exist_ok=True) + + # Get the common root directory for all files + common_path = get_common_directory(input_obj['files']) + + # Extract audio from each video + for file_path in input_obj['files']: + base_file_name = os.path.splitext(file_path.replace(common_path, ''))[0] + output_audio_path = os.path.join(input_obj['output_folder'], base_file_name) + f".{input_obj['audio_format']}" + extract_audio_from_video(video_path=file_path, + output_audio_path=output_audio_path, + format=input_obj['audio_format'], + acodec=input_obj['audio_codec']) + audio_files.append(output_audio_path) + + return { + 'output': audio_files + } + + +def get_common_directory(files: List[str]) -> str: + """A function to get the common directory from a list of file paths. + + Parameters: + - files: a list of file paths + + Returns: + - the common directory among the file paths + """ + if len(files) == 1: + # Ensure the single path's directory ends with a separator + common_path = os.path.dirname(files[0]) + else: + # Use commonpath to find the common directory for multiple files + common_path = os.path.commonpath(files) + + # Check if the path ends with the os separator, add if not + if not common_path.endswith(os.sep): + common_path += os.sep + + return common_path + +def extract_audio_from_video(video_path: str, output_audio_path: str, format: str = 'wav', acodec: str = 'pcm_s16le') -> None: + """Extracts all audio channels from a video file and saves it in the specified format and codec. + + This function utilizes the ffmpeg library to extract audio without re-encoding if the specified + codec matches the source's codec. The output is in a format that can be specified (e.g., WAV, MP3), + using the desired codec (e.g., PCM_S16LE for WAV, libmp3lame for MP3). + + Parameters: + video_path (str): Path to the input video file. This should be the complete path to the video + file or a path that the script's context can resolve. + output_audio_path (str): Path where the output audio file will be saved. This should include the + filename and the appropriate file extension based on the format. + format (str): The format of the output audio file (default is 'wav'). Common formats include + 'wav', 'mp3', 'aac', etc. + acodec (str): The audio codec to use for the output file (default is 'pcm_s16le'). Common codecs + include 'pcm_s16le' (for WAV), 'libmp3lame' (for MP3), 'aac' (for AAC), etc. + + Returns: + None: This function does not return any values but will raise an error if the extraction fails. + + Raises: + ValueError: If the video path does not exist, or the specified format/codec is not supported. + ffmpeg.Error: An error occurred during the ffmpeg processing, such as an issue with the input file + or codec compatibility. + + Examples: + # Example 1: Extract audio in WAV format with default codec (PCM_S16LE) + extract_audio_from_video("example.mp4", "output_audio.wav") + + # Example 2: Extract audio in MP3 format using the libmp3lame codec + extract_audio_from_video("example.mp4", "output_audio.mp3", format="mp3", acodec="libmp3lame") + + # Example 3: Extract audio in AAC format with native AAC codec + extract_audio_from_video("example.mp4", "output_audio.aac", format="aac", acodec="aac") + + Todo: + - Dinamically check the supported formats and codecs + """ + # Check if the video file exists + if not os.path.exists(video_path): + raise FileNotFoundError(f"The video file {video_path} does not exist.") + + # Validate format and codec + valid_formats = ['wav', 'mp3', 'aac', 'flac', 'ogg', 'mka'] + valid_codecs = ['pcm_s16le', 'libmp3lame', 'aac', 'flac', 'libvorbis', 'copy'] + + if format not in valid_formats: + raise ValueError(f"Unsupported format: {format}. Supported formats: {valid_formats}") + if acodec not in valid_codecs: + raise ValueError(f"Unsupported codec: {acodec}. Supported codecs: {valid_codecs}") + + try: + # Input stream configuration + input_stream = ffmpeg.input(video_path) + + # Audio extraction configuration + audio_stream = input_stream.audio.output( + output_audio_path, + format=format, + acodec=acodec + ) + + # Execute ffmpeg command + ffmpeg.run(audio_stream, overwrite_output=True) + + except ffmpeg.Error as e: + print("An error occurred while extracting audio:", str(e)) \ No newline at end of file diff --git a/src/tests/my_test.py b/src/tests/my_test.py deleted file mode 100644 index b2706f41..00000000 --- a/src/tests/my_test.py +++ /dev/null @@ -1,43 +0,0 @@ -from pipepal.audio.tasks import IOTask, RawSignalProcessingTask - -# Instantiate your class -io_task = IOTask() - -io_response = io_task.run({ - "data": { - "files": [#"/Users/fabiocat/Documents/git/soup/pipepal/data/02___121##0.wav", - #"/Users/fabiocat/Documents/git/soup/pipepal/data/03___144##0.wav", - #"/Users/fabiocat/Documents/git/soup/pipepal/data/04___80##0.wav", - "/Users/fabiocat/Documents/git/soup/pipepal/data/diarization.wav"] - }, - "service": { - "service_name": "Datasets" - } -}) - -audio_dataset = io_response['output'] - -print(audio_dataset[-1]['audio']['array'].shape) -print(audio_dataset[-1]['audio']['array']) - -rawSignalProcessingTask = RawSignalProcessingTask() - -asp_response = rawSignalProcessingTask.run({ - "data": { - "dataset": audio_dataset, - "channeling": { - "method": "selection", # alternative is "average" - "channels_to_keep": [0] - }, - "resampling": { - "rate": 16000, - } - }, - "service": { - "service_name": "torchaudio" - } -}) -new_audio_dataset = asp_response['output'] -print(new_audio_dataset) -print(new_audio_dataset[-1]['audio']['array'].shape) -print(new_audio_dataset[-1]['audio']['array']) \ No newline at end of file diff --git a/src/tests/test_audio_exampletask_exampleservice.py b/src/tests/test_audio_exampletask_exampleservice.py new file mode 100644 index 00000000..f4040471 --- /dev/null +++ b/src/tests/test_audio_exampletask_exampleservice.py @@ -0,0 +1,74 @@ +"""Tests for the run method of the ExampleService in the audio's ExampleTask.""" + +import pytest + +from pipepal.audio.tasks import ExampleTask + +def test_exampletask_run(): + """Test the run method of ExampleTask. + + This test verifies: + 1. The output type is a dictionary. + 2. The output's 'output' key correctly returns the expected string. + """ + # Instantiate your class + exampleTask = ExampleTask() + + # Call the method you wish to test + output = exampleTask.run({ + "data": { + "hello": "world" + }, + "service": { + "service_name": "ExampleService", + "model_checkpoint": "model.ckpt", + "model_version": "1.0", + } + }) + + # Assert conditions about the output + assert type(output) == dict, "Output should be a dictionary" + expected_output_output = "ExampleService output" + assert output['output'] == expected_output_output, "The output of the run method does not match the expected output" + +def test_exampletask_run_missing_service_fields(): + """Test the run method of ExampleTask. + + This test checks for its handling of missing required service fields. + This test iteratively removes each required service field (service_name, model_checkpoint, + model_version) from the input and checks if a ValueError is raised with the appropriate + error message indicating the missing field. + + The test ensures that: + 1. A ValueError is thrown for missing service fields. + 2. The exception message contains the name of the missing field. + """ + # Instantiate your class + example_task = ExampleTask() + + # Define a list of required service fields + required_service_fields = ["service_name", "model_checkpoint", "model_version"] + + # Iterate over each required field and test by removing each one by one + for missing_field in required_service_fields: + # Create a payload with all necessary fields + payload_with_all_fields = { + "data": { + "hello": "world" + }, + "service": { + "service_name": "my_service", + "model_checkpoint": "model.ckpt", + "model_version": "1.0", + } + } + + # Remove the field being tested + del payload_with_all_fields["service"][missing_field] + + # Use pytest.raises to assert that a ValueError is raised due to the missing field + with pytest.raises(ValueError) as excinfo: + example_task.run(payload_with_all_fields) + + # Assert that the error message contains the name of the missing field + assert missing_field in str(excinfo.value), f"Expected ValueError due to missing '{missing_field}' field" diff --git a/src/tests/test_audio_iotask_datasets.py b/src/tests/test_audio_iotask_datasets.py new file mode 100644 index 00000000..16e42787 --- /dev/null +++ b/src/tests/test_audio_iotask_datasets.py @@ -0,0 +1,203 @@ +"""This module tests the audio's IOTask class.""" + +import shutil + +import pytest +from datasets import Dataset + +from pipepal.audio.tasks import IOTask as AudioIOTask + + +def test_read_audios_from_disk_input_errors(): + """Test the read_audios_from_disk method. + + This test checks if the read_audios_from_disk method raises appropriate errors for invalid inputs. + This tests: + 1. Missing 'service' or 'data' keys in the input dictionary. + 2. Invalid file paths in the 'files' list (non-existent files). + """ + with pytest.raises(ValueError): + # Missing 'service' key + AudioIOTask().read_audios_from_disk({ + "data": { + "files": ["/path/to/audio/file1.wav"] + } + }) + + with pytest.raises(ValueError): + # Missing 'data' key + AudioIOTask().read_audios_from_disk({ + "service": { + "service_name": "Datasets" + } + }) + + with pytest.raises(FileNotFoundError): + # Non-existent file path + AudioIOTask().read_audios_from_disk({ + "data": { + "files": ["/non/existent/path/file1.wav"] + }, + "service": { + "service_name": "Datasets" + } + }) + +def test_read_audios_from_disk_output_type(): + """Test the read_audios_from_disk method to check if the output is of type HF datasets.""" + test_input = { + "data": { + "files": ["../../data_for_testing/audio_48khz_mono_16bits.wav", "../../data_for_testing/audio_48khz_stereo_16bits.wav"] + }, + "service": { + "service_name": "Datasets" + } + } + response = AudioIOTask().read_audios_from_disk(test_input) + assert isinstance(response["output"], Dataset), "The output should be an instance of HF_datasets." + +def test_read_audios_from_disk_output_dimensions(): + """Test the read_audios_from_disk method. + + This test checks if the dimensions of the output HF datasets object match the input list of audio files. + Uses mocker to patch the DatasetsService to avoid actual file I/O and simulate reading files. + """ + test_input = { + "data": { + "files": ["../../data_for_testing/audio_48khz_mono_16bits.wav", "../../data_for_testing/audio_48khz_stereo_16bits.wav"] + }, + "service": { + "service_name": "Datasets" + } + } + response = AudioIOTask().read_audios_from_disk(test_input) + assert len(response["output"]) == len(test_input["data"]["files"]), "The number of items in the output should match the number of input files." + + +def test_save_HF_dataset_to_disk(): + """Test the `save_HF_dataset_to_disk` method for successful execution with valid inputs. + + This test ensures that when given a correctly formatted input dictionary that includes + a valid 'service' and 'data' key, the `save_HF_dataset_to_disk` method completes + without throwing any errors. The test assumes that the input 'service' corresponds + to a service that exists and can process the 'data' provided. It is designed to + validate the method's ability to handle expected inputs correctly. + + Assumptions: + - 'service' is a placeholder for an actual service name expected by the application. + - 'valid_dataset_identifier_or_path' is a placeholder for an actual dataset identifier + or path that the method can process. + + The test will fail if the method throws any exceptions, indicating issues with the + method's error handling or functionality with assumed valid inputs. + """ + # Set up the input dictionary as expected by the method + input_data = { + "service": { + "service_name": "Datasets" + }, + "data": { + "dataset": Dataset.from_dict({"pokemon": ["bulbasaur", "squirtle"], "type": ["grass", "water"]}), + "output_path": "../../data_for_testing/output_dataset" + } + } + + # Test if the function runs without errors with valid inputs + try: + AudioIOTask().save_HF_dataset_to_disk(input_data) + except Exception as e: + pytest.fail(f"Function raised an exception with valid input: {e}") + + # shutil.rmtree("../../data_for_testing/output_dataset") + + +def test_upload_HF_dataset_to_HF_hub(): + """Test the `upload_HF_dataset_to_HF_hub` method for successful execution with valid inputs. + + This test checks that the `upload_HF_dataset_to_HF_hub` method does not produce any + errors when executed with an input dictionary containing correct 'service' and 'data' keys. + This verifies the method's capacity to operate as expected under normal conditions. The + 'service' should be a real service name within the application's context, capable of + processing the provided 'data' (dataset identifier or path). + + Assumptions: + - 'valid_service' should be replaced with a real service name known to the system. + - 'valid_dataset_identifier_or_path' should be an actual path or identifier that the + service can handle. + + If the method throws exceptions with these inputs, the test will fail, highlighting + potential problems in the method's implementation or issues with handling inputs + that are presumed to be correct. + + Todo: + - We may want to set up a lab HF account + """ + # Set up the input dictionary as expected by the method + input_data = { + "service": { + "service_name": "Datasets" + }, + "data": { + "dataset": Dataset.from_dict({"pokemon": ["bulbasaur", "squirtle"], "type": ["grass", "water"]}), + "output_uri": "fabiocat/test" + } + } + + # Test if the function runs without errors with valid inputs + try: + AudioIOTask().upload_HF_dataset_to_HF_hub(input_data) + except Exception as e: + pytest.fail(f"Function raised an exception with valid input: {e}") + + +def test_read_local_HF_dataset(): + """Test the `read_local_HF_dataset` method for successful loading with valid local path input. + + This test ensures that the `read_local_HF_dataset` method can correctly load a dataset from + a specified local path without throwing any exceptions when provided with a valid path. + If the method throws exceptions with these inputs, the test will fail, which would indicate issues + with the method's implementation or the provided path. + """ + # Set up the input dictionary as expected by the method + input_data = { + "service": { + "service_name": "Datasets" + }, + "data": { + "path": '../../data_for_testing/output_dataset' + } + } + + # Test if the function runs without errors with valid inputs + try: + response = AudioIOTask().read_local_HF_dataset(input_data) + assert "output" in response, "The key 'output' was not found in the result dictionary." + assert isinstance(response["output"], Dataset), "The result is not a Dataset object as expected." + except Exception as e: + pytest.fail(f"Function raised an exception with valid input: {e}") + +def test_read_HF_dataset_from_HF_hub(): + """Test the `read_HF_dataset_from_HF_hub` method for successful loading with valid URI input. + + This test checks that the `read_HF_dataset_from_HF_hub` method can correctly load a dataset from + the Hugging Face Hub using a provided URI without errors, assuming the URI points to an accessible dataset. + If the method throws exceptions with these inputs, the test will fail, which would indicate problems + either in the method's implementation or in the accessibility of the dataset at the specified URI. + """ + # Set up the input dictionary as expected by the method + input_data = { + "service": { + "service_name": "Datasets" + }, + "data": { + "uri": "fabiocat/test" # Assuming a valid URI that is accessible + } + } + + # Test if the function runs without errors with valid inputs + try: + response = AudioIOTask().read_HF_dataset_from_HF_hub(input_data) + assert "output" in response, "The key 'output' was not found in the result dictionary." + assert isinstance(response["output"]["train"], Dataset), "The result is not a Dataset object as expected." + except Exception as e: + pytest.fail(f"Function raised an exception with valid input: {e}") \ No newline at end of file diff --git a/src/tests/test_exampletask_exampleservice.py b/src/tests/test_exampletask_exampleservice.py deleted file mode 100644 index c964fdda..00000000 --- a/src/tests/test_exampletask_exampleservice.py +++ /dev/null @@ -1,53 +0,0 @@ -"""Tests for the run method of the ExampleService in ExampleTask.""" - -import pytest - -from pipepal.audio.tasks import ExampleTask - - -def test_exampletask_run(): - # Instantiate your class - exampleTask = ExampleTask() - - # Call the method you wish to test - output = exampleTask.run({ - "data": { - "hello": "world" - }, - "service": { - "service_name": "ExampleService", - "model_checkpoint": "model.ckpt", - "model_version": "1.0", - } - }) - - # Assert conditions about the output - # 1 - assert type(output) == dict, "Output should be a dictionary" - - # 2 - expected_output_output = "ExampleService output" - assert output['output'] == expected_output_output, "The output of the run method does not match the expected output" - - -def test_exampletask_run_missing_service_field(): - # Instantiate your class - exampleTask = ExampleTask() - - # Define a payload with a missing field in the 'service' dictionary - payload_with_missing_field = { - "data": { - "hello": "world" - }, - "service": { - # 'service_name' is required but missing - "model_checkpoint": "model.ckpt", - "model_version": "1.0", - } - } - - # Use pytest.raises to assert that a ValueError is raised due to the missing field - with pytest.raises(ValueError) as excinfo: - exampleTask.run(payload_with_missing_field) - - assert "service_name" in str(excinfo.value), "Expected ValueError due to missing 'service_name' field" diff --git a/src/tests/test_video_exampletask_exampleservice.py b/src/tests/test_video_exampletask_exampleservice.py new file mode 100644 index 00000000..f41214ad --- /dev/null +++ b/src/tests/test_video_exampletask_exampleservice.py @@ -0,0 +1,74 @@ +"""Tests for the run method of the ExampleService in the video's ExampleTask.""" + +import pytest + +from pipepal.video.tasks import ExampleTask + +def test_exampletask_run(): + """Test the run method of ExampleTask. + + This test verifies: + 1. The output type is a dictionary. + 2. The output's 'output' key correctly returns the expected string. + """ + # Instantiate your class + exampleTask = ExampleTask() + + # Call the method you wish to test + output = exampleTask.run({ + "data": { + "hello": "world" + }, + "service": { + "service_name": "ExampleService", + "model_checkpoint": "model.ckpt", + "model_version": "1.0", + } + }) + + # Assert conditions about the output + assert type(output) == dict, "Output should be a dictionary" + expected_output_output = "ExampleService output" + assert output['output'] == expected_output_output, "The output of the run method does not match the expected output" + +def test_exampletask_run_missing_service_fields(): + """Test the run method of ExampleTask. + + This test checks for its handling of missing required service fields. + This test iteratively removes each required service field (service_name, model_checkpoint, + model_version) from the input and checks if a ValueError is raised with the appropriate + error message indicating the missing field. + + The test ensures that: + 1. A ValueError is thrown for missing service fields. + 2. The exception message contains the name of the missing field. + """ + # Instantiate your class + example_task = ExampleTask() + + # Define a list of required service fields + required_service_fields = ["service_name", "model_checkpoint", "model_version"] + + # Iterate over each required field and test by removing each one by one + for missing_field in required_service_fields: + # Create a payload with all necessary fields + payload_with_all_fields = { + "data": { + "hello": "world" + }, + "service": { + "service_name": "my_service", + "model_checkpoint": "model.ckpt", + "model_version": "1.0", + } + } + + # Remove the field being tested + del payload_with_all_fields["service"][missing_field] + + # Use pytest.raises to assert that a ValueError is raised due to the missing field + with pytest.raises(ValueError) as excinfo: + example_task.run(payload_with_all_fields) + + # Assert that the error message contains the name of the missing field + assert missing_field in str(excinfo.value), f"Expected ValueError due to missing '{missing_field}' field" diff --git a/src/tests/test_video_iotask_datasets.py b/src/tests/test_video_iotask_datasets.py new file mode 100644 index 00000000..dc1c764e --- /dev/null +++ b/src/tests/test_video_iotask_datasets.py @@ -0,0 +1,93 @@ +"""This module tests the video's IOTask class.""" + +import os + +import pytest + +from pipepal.video.tasks import IOTask as VideoIOTask + + +def test_extract_audios_from_videos_input_errors(): + """Test the extract_audios_from_videos method. + + This test checks if the extract_audios_from_videos method raises appropriate errors for invalid inputs. + This tests: + 1. Missing 'service' or 'data' keys in the input dictionary. + 2. Invalid file paths in the 'files' list (non-existent files). + """ + with pytest.raises(ValueError): + # Missing 'service' key + VideoIOTask().extract_audios_from_videos({ + "data": { + "files": ["/path/to/audio/file1.mp4"], + "output_folder": "/path/to/output/audios", + "audio_format": "wav", + "audio_codec": "pcm_s16le" + } + }) + + with pytest.raises(ValueError): + # Missing 'data' key + VideoIOTask().extract_audios_from_videos({ + "service": { + "service_name": "ffmpeg" + } + }) + + with pytest.raises(FileNotFoundError): + # Non-existent file path + VideoIOTask().extract_audios_from_videos({ + "data": { + "files": ["/non/existent/path/file1.mp4"], + "output_folder": "../../data_for_testing", + "audio_format": "wav", + "audio_codec": "pcm_s16le" + }, + "service": { + "service_name": "ffmpeg" + } + }) + +def test_extract_audios_from_videos_output_type(): + """Test the extract_audios_from_videos method to check if the output is of type list of strings.""" + test_input = { + "data": { + "files": ["../../data_for_testing/video_48khz_stereo_16bits.mp4"], + "output_folder": "../../data_for_testing", + "audio_format": "wav", + "audio_codec": "pcm_s16le" + }, + "service": { + "service_name": "ffmpeg" + } + } + response = VideoIOTask().extract_audios_from_videos(test_input) + assert isinstance(response["output"], list), "Output should be a list" + assert all(isinstance(item, str) for item in response["output"]), "All items in output list should be strings" + + # Clean up + for audio_file in response["output"]: + os.remove(audio_file) + +def test_read_audios_from_disk_output_dimensions(): + """Test the read_audios_from_disk method. + + This test checks if the dimensions of the output list of audio files match the input list of video files. + """ + test_input = { + "data": { + "files": ["../../data_for_testing/video_48khz_stereo_16bits.mp4"], + "output_folder": "../../data_for_testing", + "audio_format": "wav", + "audio_codec": "pcm_s16le" + }, + "service": { + "service_name": "ffmpeg" + } + } + response = VideoIOTask().extract_audios_from_videos(test_input) + assert len(response["output"]) == len(test_input["data"]["files"]), "The number of items in the output should match the number of input files." + + # Clean up + for audio_file in response["output"]: + os.remove(audio_file) diff --git a/src/pipepal/audio/tasks/speaker_diarization/.gitkeep b/tutorials/.gitkeep similarity index 100% rename from src/pipepal/audio/tasks/speaker_diarization/.gitkeep rename to tutorials/.gitkeep diff --git a/tutorials/tutorial.ipynb b/tutorials/tutorial.ipynb new file mode 100644 index 00000000..5c8bfcae --- /dev/null +++ b/tutorials/tutorial.ipynb @@ -0,0 +1,230 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Tutorial" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook will show various snippets of code for varios tasks using the audio and video functionalities provided by the `pipepal` package. \n", + "\n", + "Please note that all file paths and configurations used here are placeholders and should be replaced with actual user-specific paths and settings." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Importing necessary classes from the pipepal package\n", + "from pipepal.audio.tasks import ExampleTask, IOTask as AudioIOTask\n", + "from pipepal.video.tasks import IOTask as VideoIOTask" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example Audio Task" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Example Task\n", + "# ------------\n", + "# This section demonstrates the usage of the ExampleTask which is a basic template task.\n", + "\n", + "# Creating an instance of ExampleTask\n", + "example_task = ExampleTask()\n", + "\n", + "# Running the example task with sample data and service configurations\n", + "example_response = example_task.run({\n", + " \"data\": {\n", + " \"hello\": \"world\"\n", + " },\n", + " \"service\": {\n", + " \"service_name\": \"ExampleService\",\n", + " \"model_checkpoint\": \"model.ckpt\",\n", + " \"model_version\": \"1.0\",\n", + " }\n", + "})\n", + "\n", + "# Output the response from the example task\n", + "print(\"Example Task Response:\", example_response)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Video IO Task" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Video IO Task\n", + "# -------------\n", + "# This section illustrates how to extract audio tracks from video files using VideoIOTask.\n", + "\n", + "# Creating an instance of VideoIOTask\n", + "video_io_task = VideoIOTask()\n", + "\n", + "# Extracting audio from video\n", + "video_io_response = video_io_task.extract_audios_from_videos({\n", + " \"data\": {\n", + " \"files\": [\"/path/to/your/video_file.mp4\"],\n", + " \"audio_format\": \"wav\",\n", + " \"audio_codec\": \"pcm_s16le\",\n", + " \"output_folder\": \"/path/to/output/audios\"\n", + " },\n", + " \"service\": {\n", + " \"service_name\": \"ffmpeg\"\n", + " }\n", + "})\n", + "\n", + "# Output the response from the video IO task\n", + "print(\"Video IO Task Response:\", video_io_response)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Audio IO Task" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Audio IO Task\n", + "# -------------\n", + "# This section shows how to read audio files from disk using AudioIOTask.\n", + "\n", + "# Creating an instance of AudioIOTask\n", + "audio_io_task = AudioIOTask()\n", + "\n", + "# Reading audios from disk\n", + "audio_io_response = audio_io_task.read_audios_from_disk({\n", + " \"data\": {\n", + " \"files\": [\n", + " \"/path/to/audio/file1.wav\",\n", + " \"/path/to/audio/file2.wav\",\n", + " \"/path/to/audio/file3.wav\"\n", + " ]\n", + " },\n", + " \"service\": {\n", + " \"service_name\": \"Datasets\"\n", + " }\n", + "})\n", + "\n", + "# Accessing the output from the response which is a list of audio data\n", + "audio_dataset = audio_io_response['output']\n", + "\n", + "# Displaying information about the first and last audio files in the dataset\n", + "print(\"First audio shape:\", audio_dataset[0]['audio']['array'].shape)\n", + "print(\"First audio array:\", audio_dataset[0]['audio']['array'])\n", + "print(\"Last audio shape:\", audio_dataset[-1]['audio']['array'].shape)\n", + "print(\"Last audio array:\", audio_dataset[-1]['audio']['array'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Raw Signal Processing Task" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\"\"\" # Not working #\n", + "# Raw Signal Processing Task\n", + "# --------------------------\n", + "# This section demonstrates the use of RawSignalProcessingTask to perform operations like channel selection and resampling.\n", + "\n", + "from pipepal.audio.tasks import RawSignalProcessingTask\n", + "\n", + "# Instantiate your class\n", + "raw_signal_processing_task = RawSignalProcessingTask()\n", + "\n", + "# Running the raw signal processing task with the previously created audio dataset\n", + "asp_response = raw_signal_processing_task.run({\n", + " \"data\": {\n", + " \"dataset\": audio_dataset,\n", + " \"channeling\": {\n", + " \"method\": \"selection\", # alternative is \"average\"\n", + " \"channels_to_keep\": [0]\n", + " },\n", + " \"resampling\": {\n", + " \"rate\": 16000,\n", + " }\n", + " },\n", + " \"service\": {\n", + " \"service_name\": \"torchaudio\"\n", + " }\n", + "})\n", + "\n", + "# Accessing the new audio dataset from the response\n", + "new_audio_dataset = asp_response['output']\n", + "\n", + "# Printing details about the processed audio dataset\n", + "print(\"Processed audio dataset:\", new_audio_dataset)\n", + "print(\"Shape of last audio array in processed dataset:\", new_audio_dataset[-1]['audio']['array'].shape)\n", + "print(\"Last audio array in processed dataset:\", new_audio_dataset[-1]['audio']['array'])\n", + "\n", + "# Optionally, pushing the new audio dataset to a hub\n", + "# Uncomment the following line to perform this action\n", + "# new_audio_dataset.push_to_hub(\"your_hub_repository_name\")\n", + "\"\"\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "pipepal-dH98wMGu-py3.10", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.10" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}