Improved robustness of submodules import/update + minor fixes

v2.3.3

Update module_loader.py

Improved robustness of submodules import/update

• better exception handling for submodules import
• self update minor fix
• a bit more logs

Minor fixes on data

v2.3.1

Update data submodule to latest commit

New data/scraper submodule architecture, 2 new data sources

Update at runtime of remote modules
Scrapers are now in their own repos, and we have a tree of repositories with references
Improve self-client update
Add two new sources
Codebase now ready for open source PR & bounty program

Keywords extraction + self update fix

Fix on keywords -> more keywords, less noise in it
Fix on self-update

Minor Chrome driver & install path fix Latest Latest

Minor fix on chrome driver / chromium executable path, removed autoinstaller causing troubles. Some logs fixed

Optional login-based scraping + Dynamic Keywords with Caching + Improved logging & newness logic in scrapers, Selenium+Chromium in docker image

This 2.1 marks a more stable base for the Exorde client.

1. Optional Twitter Authentication: We have added the ability to authenticate with your Twitter account, providing access to more latest tweets for the Exorde Network: the behavior is minimal (logs in, slowly scrapes 1 page at a time, slowly. No posting, no other features)
2. Selenium and Chromedriver Integration: We have integrated Selenium and Chromedriver into the Docker build to enhance web scraping capabilities. This ensures reliable and efficient data retrieval.
3. Slow, Gentle, and Healthy Scraping: We have implemented a Twitter scraping technique that prioritizes website health and stability. By collecting websites in a slow and gentle manner, we reduce the risk of disruptions and promote responsible scraping.
4. Improved Filtering for Reddit, Twitter, and 4chan: We have enhanced the filtering mechanisms for these platforms, enabling better selection and prioritization of fresh and relevant content.
5. Enhanced Log Visibility: The logging system has been improved to provide more detailed and informative logs, allowing for easier monitoring and troubleshooting. Now shows Worker address, less noise from snscrape & other libs.
6. Adjustable Logging Level: You now have the ability to adjust the logging level according to your preferences, tailoring the amount of information displayed in the logs.
7. Cached Keywords Management: A new feature has been added to manage cached keywords more efficiently. This allows for better tracking and updating of relevant keywords without spamming the source endpoint.

This new release requires (for all features to work) to have the container image rebuilt, as the docker image now embeds selenium, chromium and chrome driver.

V2 minor fixes - self update

Update get_latest_tag.py

changed exorde to exorde-client when calling the api

Exorde Client v2, base Spotting image

A new stable version of the Exorde Client is out !

Complete refactor, asyncio & aiohttp based. Modular architecture: seperation of exorde-protocol core systems & external scrapers
Modular architecture for scrapers in the data folder -> pluging for scrapers!
Added new analysis, dynamic parameters at runtime, self update mechanisms
Strong data typing & verification at runtime
Spotting + analytics (NLP models for translation, sentiment, emotion, metadata)
Fully self initializable (self faucet), autonomous.

Author: @6r17, @FCuzol, @MathiasExorde