Extended domains (social)

Extend social domains (#70)

* Update process_batch.py

* Update process_batch.py

* Update setup.py

* Update spotting.py

* Update spotting.py

Minor preprocessing & case handling fix

Text content won't be lowercased anymore, better keywords extract

v2.5.13

Update setup.py

New Optimized data pipeline (new tag, lower RAM, lighter image)

Optimized data pipeline (new ZS model, better RAM usage)

Update lab_initialization.py
Update pre_install.py
Update zero_shot.py
Update tag.py
Update Dockerfile
Update models.py
Removes a large zero shot model used until now, for a smaller, more efficient, with similar accuracy model.
We now are able to add it to the batch processing data pipeline and remove it from inter-item processing, accelerating the process & lot, reducing RAM usage over time.

The image will now be 1.2Gb smaller or so, and will have very fast item processing time, with all models loaded in RAM beforehand.

Minor domain fix

Batch processing optimization (#66)

* Update lab_initialization.py
* Update pre_install.py
* Update zero_shot.py
* Update tag.py
* Update Dockerfile
* Update models.py


Removes a large zero shot model used until now, for a smaller, more efficient, with similar accuracy model.
We now are able to add it to the batch processing data pipeline and remove it from inter-item processing, accelerating the process & lot, reducing RAM usage over time.

The image will now be 1.2Gb smaller or so, and will have very fast item processing time, with all models loaded in RAM beforehand.

Added raw/translated content

We added the raw & translated content to the IPFS uploads.
This provide more content to the Network.

Improved translation modules + Dockerfile minor fix Latest

Improved translation modules + Dockerfile minor fix
Latest Argostranslate updated to version 1.9.6, with CTranslate2 indirectly bumped to v4.

Real Offline Version (Dockerfile) + More resilience

New Features and Improvements

• Enhanced Offline Support: We've made significant strides in enhancing the offline capabilities of our workers. By setting TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1, and HF_HUB_OFFLINE=1, we ensure that our dependencies on external services like Hugging Face are minimized. This update includes caching for fasttext-langdetect/lid.176.bin, Hugging Face-based models, and Argos Translate models, ensuring that our workers can operate completely offline without any hiccups.

• Increased Resilience: Our workers are now more resilient than ever. They no longer depend heavily on third-party services, which means temporary outages or HTTP errors from platforms like GitHub will not disrupt our operations as before. This marks a significant step towards our goal of decentralization, reducing our dependency on external services and enhancing our system's robustness.

• WtpSplit Version 1.3.0: The WtpSplit tool has been upgraded to version 1.3.0. This update brings improvements in language detection accuracy, making it easier and more reliable to determine the language from text. This enhancement is part of our continuous effort to improve the functionality and performance of our tools.

Thank you for your continued support and feedback, which help us make [Your Project Name] better with each release.

Minor keyword fix + Network Tx gas cap decrease

• Less gas per tx, more tx per block, less time to wait for spotData tx inclusions on Exorde skale chain
• Minor fix on extract_keywords.py, which was leading to divisions by zero

Resilience fix - HuggingFace Models Offline mode + keyword capped at 50 chars max (instead of nocap)

• HuggingFace Models Offline mode
  -> the client will initialize itself and download all models, then set itself to offline mode, to never have any interaction with huggingface
  -> goal = no deps on HuggingFace servers

• minor fix on keywords
  they are now capped at 50 chars max, instead of no limit before