Merge pull request #142 from ARGOeu/devel

Version 1.0.5
ARGOeu · Mar 13, 2023 · ec222c3 · ec222c3
2 parents 291ee05 + 6cffa9c
commit ec222c3
Show file tree

Hide file tree

Showing 43 changed files with 32,549 additions and 915 deletions.
diff --git a/CITATION.cff b/CITATION.cff
@@ -0,0 +1,20 @@
+# This CITATION.cff file was generated with cffinit.
+# Visit https://bit.ly/cffinit to generate yours today!
+
+cff-version: 1.2.0
+title: EOSC Recommender System Metrics
+message: A framework for evaluating EOSC Recommender System. Use of additional diagnostic metrics and visualizations offering deeper and sometimes surprising insights about the models performance.
+type: software
+authors:
+  - given-names: Kostas
+    family-names: Kagkelidis
+    email: kaggis@admin.grnet.gr
+    affiliation: GRNET S.A.
+  - given-names: Nikolaos
+    family-names: Triantafyllis
+    email: ntriantafyl@admin.grnet.gr
+    affiliation: GRNET S.A.
+  - given-names: Themis
+    family-names: Zamani
+    email: themis@admin.grnet.gr
+    affiliation: GRNET S.A.
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
@@ -0,0 +1,73 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment,
+we as contributors and maintainers pledge to making participation
+in our project and our community a harassment-free experience for everyone,
+regardless of age, body size, disability, ethnicity, gender identity and
+expression, level of experience, nationality, personal appearance, race,
+religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behaviour that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behaviour by participants include:
+
+* The use of sexualised language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behaviour and are expected to take appropriate and fair corrective action
+in response to any instances of unacceptable behaviour.
+
+Project maintainers have the right and responsibility to remove, edit,
+or reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily
+or permanently any contributor for other behaviours that they deem
+inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community.
+Examples of representing a project or community include using an official
+e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behaviour may be
+reported by contacting the CESSDA Main Office at support AT cessda DOT eu.
+The project team will review and investigate all complaints, and will respond
+in a way that it deems appropriate to the circumstances.
+The project team is obligated to maintain confidentiality with regard to the
+reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct
+in good faith may face temporary or permanent repercussions as determined
+by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,40 @@
+# Contributing to ARGOEU Open Source Software
+
+Thank you for considering contributing to ARGOEU Open Source Software.
+
+Please note that we welcome all forms of user feedback and are generally open
+to many forms of collaboration,
+but please read this document carefully before proceeding.
+
+## User feedback
+
+### Reporting problems
+
+We welcome all feedback you may have when using any of our tools and services.
+
+If you encounter a problem, please use the "Report a problem" button you find
+at the lower right corner of the respective tool or service
+and choose "Bug" as the value of the Components field.
+
+### Suggesting Enhancements
+
+To suggest new functionalities and possible future developments,
+please use the "Report a problem" button you find at the lower right corner of
+the respective tool or service
+and choose "New Feature" or "Improvement" as the value of the Components field.
+
+## Contributing to the development
+
+Development of all ARGOEU tools and services is aligned with our roadmap and
+any functional requirements we have.
+
+External contributions can be accepted, if they provide functionality ARGOEU
+deems appropriate and are of acceptable quality.
+Specific requirements and criteria may apply to specific tools.
+
+If you want to contribute to ARGOEU tools and services,
+please contact argoeu [AT] grnet.gr before proceeding.
+
+### Practical matters
+
+ARGOEU reserves the right to accept or reject any contribution.
diff --git a/Jenkinsfile b/Jenkinsfile
@@ -1,9 +1,9 @@
 pipeline {
     agent { 
-        docker { 
-            image 'node:buster' 
+            docker { 
+                image 'node:buster' 
+            }
         }
-    }
     options {
         checkoutToSubdirectory('eosc-recommender-metrics')
         newContainerPerStage()
@@ -14,8 +14,30 @@ pipeline {
         GH_EMAIL = '<argo@grnet.gr>'
    }
     stages {
-
+         stage('Python syntax style checks') {
+            agent {
+                docker {
+                    image 'argo.registry:5000/python3'
+                    args '-u jenkins:jenkins'
+                }
+            }
+            steps {
+                echo 'Checking Python syntax style with flake8'
+                sh """
+                pipenv --python 3
+                pipenv run pip install flake8
+                pipenv run python -m flake8 ./
+                pipenv --rm
+                """
+            }
+            post {
+                always {
+                    cleanWs()
+                }
+            }
+        }
         stage ('Deploy Docs') {
+
             when {
                 anyOf {
                     branch 'master'

diff --git a/README.md b/README.md
@@ -1,30 +1,33 @@
 # eosc-recommender-metrics
+
 A framework for counting the recommender metrics
 
-# Preprocessor v.0.2
+## Preprocessor v.0.2
+
 <p align="center">
 <a href="https://github.com/nikosT/eosc-recommender-metrics/blob/master/docs/Preprocessor.png">
 <img src="https://github.com/nikosT/eosc-recommender-metrics/blob/master/docs/Preprocessor.png" width="70%"/>
 </a>
 </p>
 
-# RS metrics v.0.2
+## RS metrics v.0.2
+
 <p align="center">
 <a href="https://github.com/nikosT/eosc-recommender-metrics/blob/master/docs/RSmetrics.png">
 <img src="https://github.com/nikosT/eosc-recommender-metrics/blob/master/docs/RSmetrics.png" width="70%"/>
 </a>
 </p>
 
+## Dependencies
 
-
-
-# Dependencies
-1. Install Conda from here: https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html. Tested on conda v 4.10.3.
+1. Install Conda from [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html). Tested on conda v 4.10.3.
 2. Run from terminal: `conda env create -f environment.yml`
 3. Run from terminal: `conda activate rsmetrics`
 4. Run from terminal: `chmod +x ./preprocessor.py ./preprocessor_common.py ./rsmetrics.py`
 
-# Usage
+## Usage
+
+### Usage of the Batch System
 5. Configure `./preprocessor_common.py`, `./preprocessor.py` and `./rsmetrics.py` by editting the `config.yaml` or providing another with `-c`.
 6. Run from terminal: `./preprocessor_common.py` in order to gather `users` and `resources` and store them in the `Datastore`:
 ```bash
@@ -46,10 +49,16 @@ A framework for counting the recommender metrics
 ./rsmetrics.py -p athena # same procedure as the first one for 'athena' provider
 ```
 
-## Reporting
+### Usage of the Streaming System
+10. Run from terminal `./rs-stream.py` in order to listen to the stream for new data, process them, and store them in the `Datastore`, concerning that particular provider:
+```bash
+./rs-stream.py -a username:password -q host:port -t user_actions -d ""mongodb://localhost:27017/datastore"" -p provider_name
+```
 
-The reporting script generates an evalutation report in html format automatically served from a spawed localserver (default: http://localhost:8080)
-and automatically opens the default browser to present the report. 
+### Reporting
+
+The reporting script generates an evalutation report in html format automatically served from a spawed localserver [default: localhost:8080](htpp://localhost:8080)
+and automatically opens the default browser to present the report.
 
 To execute the script issue:
 ```
@@ -62,7 +71,7 @@ The script will automatically look for evaulation result files in the default fo
 
 The `report.py` script can be used with the `--input` parameter: a path to a folder that the results from the evaluation process have been generated (default folder:`./data`). The report script can also take an `--output` parameter: a path to an output folder where the generated report will be served automatically.
 
-_Note:_ the script copies to the output folder all the necessary files such as `pre_metrics.json`, `metrics.json` as well as `report.html.prototype` renamed to `index.html` 
+_Note:_ the script copies to the output folder all the necessary files such as `pre_metrics.json`, `metrics.json` as well as `report.html.prototype` renamed to `index.html`
 
 ```
 usage: report.py [-h] [-i STRING] [-o STRING] [-a STRING] [-p STRING]
@@ -81,9 +90,9 @@ optional arguments:
                         Port to bind and serve the report
 ```
 
-## Utilities
+### Utilities
 
-### Get service catalog script (./get_service_catalog.py)
+#### Get service catalog script (./get_service_catalog.py)
 
 This script contacts EOSC Marketplace catalog and generates a csv with a list of all available services, their name, id and url
 
@@ -93,12 +102,12 @@ chmod u+x ./get_service_catalog.py
 ./get_service_catalog.py
 ```
 
-#### Serve Evaluation Reports as a Service
+##### Serve Evaluation Reports as a Service
 
 The `webservice` folder hosts a simple webservice implemented in Flask framework which can be used to host the report results.
 
 __Note__: Please make sure you work in a virtual environment and you have already downloaded the required dependencies by issuing
-`pip install -r requirements.txt` 
+`pip install -r requirements.txt`
 
 The webservice application serves two endpoints
  - `/` : This is the frontend webpage that displays the Report Results in a UI
@@ -122,3 +131,4 @@ You can override this by editing the `.env` file inside the `/webservice` folder
 
 _Tested with python 3.9_
 
+
diff --git a/config.yaml b/config.yaml
@@ -21,10 +21,4 @@ service:
     # and also in metrics calculations where service is considered
     published: true
 
-    # which origin to use to retrieve Resources
-    # two options available:
-    # - 'source': use the Connector
-    # - 'page_map': use the EOSC Marketplace
-    from: 'page_map' # or 'source'
-
 
diff --git a/environment.yml b/environment.yml
@@ -27,6 +27,7 @@ dependencies:
     - certifi==2022.12.7
     - charset-normalizer==2.0.12
     - click==8.1.3
+    - docopt==0.6.2
     - flask==2.1.2
     - flask-pymongo==2.3.0
     - idna==3.3
@@ -40,7 +41,7 @@ dependencies:
     - pandas==1.4.2
     - pyarrow==10.0.1
     - pymongo==4.1.0
-    - pymongoarrow==0.6.2
+    - pymongoarrow==0.7.0
     - python-dateutil==2.8.2
     - python-dotenv==0.20.0
     - pytz==2022.1
@@ -49,6 +50,9 @@ dependencies:
     - scipy==1.8.0
     - six==1.16.0
     - soupsieve==2.3.2
+    - stomp-py==8.1.0
     - urllib3==1.26.9
-    - werkzeug==2.1.2
+    - websocket-client==1.4.2
+    - werkzeug==2.2.3
     - zipp==3.8.0
+    - flake8==6.0.0