From 32b2dc328b522f37e794e1d632d4d1420e312cde Mon Sep 17 00:00:00 2001 From: a-ghanem Date: Wed, 5 May 2021 11:51:00 +0200 Subject: [PATCH 01/21] Enhanced README.md Added detailed description of the purpose and corrected how-to instructions --- README.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c91e2f3..8f6bfe6 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,12 @@ Docker Swarm deployment of LocalEGA. Please, refer to the project documentation for getting more comprehensive information: https://neic-sda.readthedocs.io/ +## Purpose + +The `docker-template.yml` file can be used in two ways: +1. Deploy LocalEGA locally as containers which can be used used for CI/CD testing in Github (the CEGA BROKER service is not included). +2. One can trim it down to a Public stack or a Private stack depending on what needs to be deployed. + ## Development ### Pre-requisites @@ -13,9 +19,18 @@ Docker Swarm deployment of LocalEGA. Please, refer to the project documentation ### How-to -`make bootstrap deploy` (CEGA-related env-vars should be set manually, e.g. `CEGA_CONNECTION`) +Run: +```bash +> make bootstrap deploy +``` -Cleaning up: `make rm purge`. +**IMPORTANT** +CEGA-related env-vars should be set manually, e.g. `CEGA_USERNAME`, `CEGA_PASSWORD`, `BROKER_HOST`, `BROKER_PORT`, `BROKER_USERNAME`, `BROKER_PASSWORD`, `CEGA_MQ_CONNECTION`, `BROKER_VALIDATE`, `BROKER_VHOST`, and `EXCHANGE`. + +Clean: +```bash +> make rm clean +``` ## Production From 9ae07132eafe886622ba580081e7145cf41d7234 Mon Sep 17 00:00:00 2001 From: a-ghanem Date: Wed, 5 May 2021 12:26:36 +0200 Subject: [PATCH 02/21] Added description of required variables --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8f6bfe6..ba88290 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ Run: ``` **IMPORTANT** -CEGA-related env-vars should be set manually, e.g. `CEGA_USERNAME`, `CEGA_PASSWORD`, `BROKER_HOST`, `BROKER_PORT`, `BROKER_USERNAME`, `BROKER_PASSWORD`, `CEGA_MQ_CONNECTION`, `BROKER_VALIDATE`, `BROKER_VHOST`, and `EXCHANGE`. +CEGA-related env-vars should be set manually, e.g. `CEGA_USERNAME`, `CEGA_PASSWORD`, `BROKER_HOST`, `BROKER_PORT`, `BROKER_USERNAME`, `BROKER_PASSWORD`, `CEGA_MQ_CONNECTION`, `BROKER_VALIDATE`, `BROKER_VHOST`, and `EXCHANGE`(required by proxy and interceptor micro services). Clean: ```bash From 89326b99b0aed3ecd336bef21f505ad6a2f318ce Mon Sep 17 00:00:00 2001 From: a-ghanem Date: Thu, 6 May 2021 09:32:00 +0200 Subject: [PATCH 03/21] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ba88290..5d61948 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,9 @@ The `docker-template.yml` file can be used in two ways: - `crypt4gh` (https://github.com/elixir-oslo/crypt4gh) - `j2cli` (https://github.com/kolypto/j2cli) +**IMPORTANT** +CEGA-related env-vars should be included manually in `Makefile` before running the makefile. The variables are `CEGA_USERNAME`, `CEGA_PASSWORD`, `BROKER_HOST`, `BROKER_PORT`, `BROKER_USERNAME`, `BROKER_PASSWORD`, `CEGA_MQ_CONNECTION`, `BROKER_VALIDATE`, `BROKER_VHOST`, and `EXCHANGE`(required by proxy and interceptor micro services). + ### How-to Run: @@ -24,9 +27,6 @@ Run: > make bootstrap deploy ``` -**IMPORTANT** -CEGA-related env-vars should be set manually, e.g. `CEGA_USERNAME`, `CEGA_PASSWORD`, `BROKER_HOST`, `BROKER_PORT`, `BROKER_USERNAME`, `BROKER_PASSWORD`, `CEGA_MQ_CONNECTION`, `BROKER_VALIDATE`, `BROKER_VHOST`, and `EXCHANGE`(required by proxy and interceptor micro services). - Clean: ```bash > make rm clean From 53c38879520748074c7e62f6f1b62c9ead5c4372 Mon Sep 17 00:00:00 2001 From: FBernal-oPs <62608473+FBernal-oPs@users.noreply.github.com> Date: Thu, 6 May 2021 09:44:13 +0200 Subject: [PATCH 04/21] Update README --- README.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5d61948..17e3abe 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,19 @@ The `docker-template.yml` file can be used in two ways: - `j2cli` (https://github.com/kolypto/j2cli) **IMPORTANT** -CEGA-related env-vars should be included manually in `Makefile` before running the makefile. The variables are `CEGA_USERNAME`, `CEGA_PASSWORD`, `BROKER_HOST`, `BROKER_PORT`, `BROKER_USERNAME`, `BROKER_PASSWORD`, `CEGA_MQ_CONNECTION`, `BROKER_VALIDATE`, `BROKER_VHOST`, and `EXCHANGE`(required by proxy and interceptor micro services). + +CEGA-related env-vars should be included manually in `Makefile` before running the makefile. The variables are: + +`CEGA_USERNAME`, +`CEGA_PASSWORD`, +`BROKER_HOST`, +`BROKER_PORT`, +`BROKER_USERNAME`, +`BROKER_PASSWORD`, +`CEGA_MQ_CONNECTION`, +`BROKER_VALIDATE`, +`BROKER_VHOST`, +`EXCHANGE`(required by proxy and interceptor micro services). ### How-to From b84515d312ae683c10402dfb43cd92531e24dadd Mon Sep 17 00:00:00 2001 From: FBernal-oPs <62608473+FBernal-oPs@users.noreply.github.com> Date: Thu, 6 May 2021 09:47:33 +0200 Subject: [PATCH 05/21] Updated README --- README.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 17e3abe..86bb378 100644 --- a/README.md +++ b/README.md @@ -21,15 +21,18 @@ The `docker-template.yml` file can be used in two ways: CEGA-related env-vars should be included manually in `Makefile` before running the makefile. The variables are: -`CEGA_USERNAME`, -`CEGA_PASSWORD`, -`BROKER_HOST`, -`BROKER_PORT`, -`BROKER_USERNAME`, -`BROKER_PASSWORD`, -`CEGA_MQ_CONNECTION`, -`BROKER_VALIDATE`, -`BROKER_VHOST`, +``` +export CEGA_USERNAME= +export CEGA_PASSWORD= +export BROKER_HOST= +export BROKER_PORT= +export BROKER_USERNAME= +export BROKER_PASSWORD= +export CEGA_MQ_CONNECTION= +export BROKER_VALIDATE= +export BROKER_VHOST= +export EXCHANGE= +``` `EXCHANGE`(required by proxy and interceptor micro services). ### How-to From c91ce2860182e32300c5794dcf5fce9b429e13fe Mon Sep 17 00:00:00 2001 From: FBernal-oPs <62608473+FBernal-oPs@users.noreply.github.com> Date: Thu, 6 May 2021 09:49:24 +0200 Subject: [PATCH 06/21] Update README --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 86bb378..fbce81f 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Docker Swarm deployment of LocalEGA. Please, refer to the project documentation ## Purpose The `docker-template.yml` file can be used in two ways: -1. Deploy LocalEGA locally as containers which can be used used for CI/CD testing in Github (the CEGA BROKER service is not included). +1. Deploy LocalEGA locally as containers which can be used for CI/CD testing in Github (the CEGA BROKER service is not included). 2. One can trim it down to a Public stack or a Private stack depending on what needs to be deployed. ## Development @@ -33,7 +33,7 @@ export BROKER_VALIDATE= export BROKER_VHOST= export EXCHANGE= ``` -`EXCHANGE`(required by proxy and interceptor micro services). +all these variables are required by proxy and interceptor micro services. ### How-to From 9b64b486148dd513afcd2a3e6f88991093c43451 Mon Sep 17 00:00:00 2001 From: Joshua Baskaran Date: Wed, 28 Sep 2022 12:44:39 +0200 Subject: [PATCH 07/21] Made not that this is a fork --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c91e2f3..8ad436f 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,8 @@ # LocalEGA-deploy-swarm ![Integration tests](https://github.com/neicnordic/LocalEGA-deploy-swarm/workflows/Integration%20tests/badge.svg) -Docker Swarm deployment of LocalEGA. Please, refer to the project documentation for getting more comprehensive information: https://neic-sda.readthedocs.io/ +Docker Swarm deployment of LocalEGA. This is a fork meant for internal testing purposes. +Please, refer to the project documentation for getting more comprehensive information: https://neic-sda.readthedocs.io/ ## Development From fc95a6d6beb9d55b5201d4dfd63db6f4a0cf1a1a Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Wed, 28 Sep 2022 15:47:12 +0200 Subject: [PATCH 08/21] Added cegamq cont. and conf, fixed 2 missing other configs --- cega.conf | 16 ++++++++++++++ cega.json | 23 ++++++++++++++++++++ cega.plugins | 1 + docker-template.yml | 51 +++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 89 insertions(+), 2 deletions(-) create mode 100644 cega.conf create mode 100644 cega.json create mode 100644 cega.plugins diff --git a/cega.conf b/cega.conf new file mode 100644 index 0000000..75c3e8f --- /dev/null +++ b/cega.conf @@ -0,0 +1,16 @@ +listeners.ssl.default = 5671 +ssl_options.cacertfile = /etc/rabbitmq/ssl/ca.pem +ssl_options.certfile = /etc/rabbitmq/ssl/mq.pem +ssl_options.keyfile = /etc/rabbitmq/ssl/mq-key.pem +ssl_options.verify = verify_none +ssl_options.fail_if_no_peer_cert = false +ssl_options.versions.1 = tlsv1.2 +management.load_definitions = /etc/rabbitmq/conf/cega.json +management.listener.port = 15671 +management.listener.ssl = true +management.listener.ssl_opts.cacertfile = /etc/rabbitmq/ssl/ca.pem +management.listener.ssl_opts.certfile = /etc/rabbitmq/ssl/mq.pem +management.listener.ssl_opts.keyfile = /etc/rabbitmq/ssl/mq-key.pem +default_vhost = lega +disk_free_limit.absolute = 1GB +log.default.level = debug diff --git a/cega.json b/cega.json new file mode 100644 index 0000000..01f4af9 --- /dev/null +++ b/cega.json @@ -0,0 +1,23 @@ +{"rabbit_version":"3.8", + "users":[{"name":"test", + "password_hash":"C5ufXbYlww6ZBcEqDUB04YdUptO81s+ozI3Ll5GCHTnv8NAm","hashing_algorithm":"rabbit_password_hashing_sha256","tags":"administrator"}], "vhosts":[{"name":"lega"}], + "permissions":[{"user":"test", "vhost":"lega", "configure":".*", "write":".*", "read":".*"}], + + "parameters":[], "global_parameters":[{"name":"cluster_name", "value":"rabbit@localhost"}], + "policies":[], + "queues":[{"name":"v1.files.inbox", "vhost":"lega", "durable":true, "auto_delete":false, "arguments":{}}, + {"name":"v1.stableIDs", "vhost":"lega", "durable":true, "auto_delete":false, "arguments":{}}, + {"name":"v1.files", "vhost":"lega", "durable":true, "auto_delete":false, "arguments":{}}, + {"name":"v1.files.completed", "vhost":"lega", "durable":true, "auto_delete":false, "arguments":{}}, + {"name":"v1.files.verified", "vhost":"lega", "durable":true, "auto_delete":false, "arguments":{}}, + {"name":"v1.files.error", "vhost":"lega", "durable":true, "auto_delete":false, "arguments":{}}], + "exchanges":[{"name":"localega.v1", "vhost":"lega", "type":"topic", "durable":true, "auto_delete":false, "internal":false, "arguments":{}}], + "bindings":[ + {"source":"localega.v1","vhost":"lega","destination_type":"queue","arguments":{},"destination":"v1.stableIDs","routing_key":"stableIDs"}, + {"source":"localega.v1","vhost":"lega","destination_type":"queue","arguments":{},"destination":"v1.files","routing_key":"files"}, + {"source":"localega.v1","vhost":"lega","destination_type":"queue","arguments":{},"destination":"v1.files.inbox","routing_key":"files.inbox"}, + {"source":"localega.v1","vhost":"lega","destination_type":"queue","arguments":{},"destination":"v1.files.error","routing_key":"files.error"}, + {"source":"localega.v1","vhost":"lega","destination_type":"queue","arguments":{},"destination":"v1.files.verified","routing_key":"files.verified"}, + {"source":"localega.v1","vhost":"lega","destination_type":"queue","arguments":{},"destination":"v1.files.completed","routing_key":"files.completed"}] + +} diff --git a/cega.plugins b/cega.plugins new file mode 100644 index 0000000..7e8d295 --- /dev/null +++ b/cega.plugins @@ -0,0 +1 @@ +[rabbitmq_federation,rabbitmq_federation_management,rabbitmq_management,rabbitmq_shovel,rabbitmq_shovel_management]. \ No newline at end of file diff --git a/docker-template.yml b/docker-template.yml index 8aa5c13..d82f246 100644 --- a/docker-template.yml +++ b/docker-template.yml @@ -155,7 +155,7 @@ services: delay: 5s window: 120s environment: - - POSTGRES_PASSWORD + - POSTGRES_PASSWORD={{POSTGRES_PASSWORD}} secrets: - source: init-mappings-db.sh target: /docker-entrypoint-initdb.d/init-mappings-db.sh @@ -220,7 +220,7 @@ services: verify: image: ghcr.io/neicnordic/sda-pipeline:latest - deploy: + deploy: restart_policy: condition: on-failure delay: 5s @@ -252,6 +252,7 @@ services: - DB_SSLMODE=require - DB_CLIENTCERT=/etc/ega/client.cert - DB_CLIENTKEY=/etc/ega/client-key.cert + - INBOX_LOCATION=/ega/inbox - LOG_LEVEL=debug secrets: - source: rootCA.pem @@ -270,6 +271,7 @@ services: target: /etc/ega/ega.sec volumes: - /tmp/vault:/ega/archive + - /tmp/tsd:/ega/inbox command: "sda-verify" finalize: @@ -399,6 +401,50 @@ services: - /tmp/vault:/ega/archive user: "65534:65534" + + # CEGA mock (RabbitMQ only for now) + # based on sda-auth/.github/intergration/docker-compose.yml + + cegamq: + image: rabbitmq:3.8.16-management-alpine + environment: + - RABBITMQ_CONFIG_FILE=/etc/rabbitmq/conf/cega + - RABBITMQ_ENABLED_PLUGINS_FILE=/etc/rabbitmq/conf/cega.plugins + ports: + - "5672:5671" + - "15672:15672" + - "25672:15672" + secrets: + - source: server.pem + target: /etc/rabbitmq/ssl/mq.pem + uid: '100' + gid: '101' + mode: 0600 + - source: server-key.pem + target: /etc/rabbitmq/ssl/mq-key.pem + uid: '100' + gid: '101' + mode: 0600 + - source: rootCA.pem + target: /etc/rabbitmq/ssl/ca.pem + volumes: + - ./cega.conf:/etc/rabbitmq/conf/cega.conf + - ./cega.json:/etc/rabbitmq/conf/cega.json + - ./cega.plugins:/etc/rabbitmq/conf/cega.plugins + healthcheck: + test: + [ + "CMD", + "bash", + "-c", + "rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms" + ] + interval: 5s + timeout: 20s + retries: 3 + + + secrets: rootCA.pem: external: true @@ -426,3 +472,4 @@ secrets: external: true init-mappings-db.sh: external: true + From 5b72bab5668b9584a23c0f3927c682ef7afe4df0 Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Wed, 28 Sep 2022 16:11:17 +0200 Subject: [PATCH 09/21] Update configs of services to use cegamq container --- Makefile | 33 ++++++++++++++++++++++++++++++++- docker-template.yml | 28 ++++++++++++++++++++++++++-- 2 files changed, 58 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 780d8e0..c732d02 100644 --- a/Makefile +++ b/Makefile @@ -2,6 +2,37 @@ SHELL := /bin/bash -O expand_aliases FILES := localhost+5.pem localhost+5-key.pem localhost+5-client.pem localhost+5-client-key.pem rootCA.pem rootCA.p12 localhost+5.p12 localhost+5-client.p12 localhost+5-client-key.der rootCA-key.pem docker-stack.yml jwt.pub.pem jwt.priv.pem ega.pub.pem ega.sec.pass ega.sec.pem server.pem server-key.pem server.p12 client.pem client-key.pem client-key.der client.p12 init-mappings-db.sh +# Manually entered secrets not available in the repo +#################################################### + +# proxy service CEGA NSS +export CEGA_AUTH_URL=https://nss-test.ega-archive.org/users/ +#export CEGA_USERNAME= +#export CEGA_PASSWORD= + + +# Prefilled configs from the repo +################################# + +#Interceptor CEGA details (using "mock" rabbitmq image) +export CEGA_MQ_CONNECTION=amqps://test:test@cegamq:5671/lega?cacertfile=/etc/ega/ssl/CA.cert + +# Proxy service CEGA details (using "mock" rabbitmq image) +export BROKER_HOST=cegamq +export BROKER_PORT=5671 +export BROKER_USERNAME=test +export BROKER_PASSWORD=test +export BROKER_VHOST=lega + +export BROKER_VALIDATE=false +export EXCHANGE=localega.v1 + + + export CAROOT := $(shell mkcert -CAROOT) export ROOT_CERT_PASSWORD=r00t_cert_passw0rd export TSD_ROOT_CERT_PASSWORD=r00t_cert_passw0rd @@ -39,7 +70,7 @@ bootstrap: init $(FILES) @sudo chmod 777 /tmp/tsd /tmp/vault /tmp/db init: - @-docker swarm init + @-$(DOCKER) swarm init mkcert: @mkcert -install diff --git a/docker-template.yml b/docker-template.yml index d82f246..ec09c4d 100644 --- a/docker-template.yml +++ b/docker-template.yml @@ -94,8 +94,13 @@ services: # Public stack proxy: image: ghcr.io/uio-bmi/localega-tsd-proxy:latest + depends_on: + - mq + - cegamq + - postgres + - tsd ports: - - 443:8080 + - 10443:8080 deploy: restart_policy: condition: on-failure @@ -132,6 +137,10 @@ services: interceptor: image: ghcr.io/uio-bmi/mq-interceptor:latest + depends_on: + - cegamq + - mq + - postgres deploy: restart_policy: condition: on-failure @@ -163,6 +172,9 @@ services: # Private stack ingest: image: ghcr.io/neicnordic/sda-pipeline:latest + depends_on: + - mq + - db deploy: restart_policy: condition: on-failure @@ -219,7 +231,10 @@ services: command: "sda-ingest" verify: - image: ghcr.io/neicnordic/sda-pipeline:latest + image: ghcr.io/neicnordic/sda-pipeline:v0.2.5 + depends_on: + - mq + - db deploy: restart_policy: condition: on-failure @@ -276,6 +291,9 @@ services: finalize: image: ghcr.io/neicnordic/sda-pipeline:latest + depends_on: + - mq + - db deploy: restart_policy: condition: on-failure @@ -322,6 +340,9 @@ services: mapper: image: ghcr.io/neicnordic/sda-pipeline:latest + depends_on: + - mq + - db deploy: restart_policy: condition: on-failure @@ -367,6 +388,9 @@ services: doa: image: neicnordic/sda-doa:release-v1.6.0 + depends_on: + - mq + - db ports: - 80:8080 deploy: From dc68eddeba6df205f50e40842c22286d1e9fed3e Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Wed, 28 Sep 2022 16:17:36 +0200 Subject: [PATCH 10/21] Fixed sudo commands for ubuntu like systems, fixed owneship on directory --- Makefile | 48 +++++++++++++++++++++++++----------------------- 1 file changed, 25 insertions(+), 23 deletions(-) diff --git a/Makefile b/Makefile index c732d02..4aeafef 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,6 @@ SHELL := /bin/bash -O expand_aliases +# sudo docker macro to run docker commands as sudo in Ubuntu systems +DOCKER := sudo docker FILES := localhost+5.pem localhost+5-key.pem localhost+5-client.pem localhost+5-client-key.pem rootCA.pem rootCA.p12 localhost+5.p12 localhost+5-client.p12 localhost+5-client-key.der rootCA-key.pem docker-stack.yml jwt.pub.pem jwt.priv.pem ega.pub.pem ega.sec.pass ega.sec.pem server.pem server-key.pem server.p12 client.pem client-key.pem client-key.der client.p12 init-mappings-db.sh @@ -66,7 +68,7 @@ export POSTGRES_CONNECTION=postgres://postgres:p0stgres_passw0rd@postgres:5432/p bootstrap: init $(FILES) @chmod 644 $(FILES) @mkdir -p /tmp/tsd /tmp/vault /tmp/db - @sudo chown 65534:65534 /tmp/vault + @sudo chown 65534:65534 /tmp/vault /tmp/tsd @sudo chmod 777 /tmp/tsd /tmp/vault /tmp/db init: @@ -96,19 +98,19 @@ localhost+5-client-key.der: localhost+5-client-key.pem jwt.priv.pem: @openssl genpkey -algorithm RSA -out jwt.priv.pem -pkeyopt rsa_keygen_bits:4096 - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ jwt.pub.pem: jwt.priv.pem @openssl rsa -pubout -in jwt.priv.pem -out jwt.pub.pem - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ ega.sec.pass: @printf $(KEY_PASSWORD) > ega.sec.pass - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ ega.sec.pem: @crypt4gh generate -n ega -p $(KEY_PASSWORD) - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ ega.pub.pem: ega.sec.pem @@ -117,44 +119,44 @@ docker-stack.yml: rootCA.pem: mkcert @cp "$(CAROOT)/rootCA.pem" rootCA.pem - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ rootCA-key.pem: mkcert @cp "$(CAROOT)/rootCA-key.pem" rootCA-key.pem @chmod 600 rootCA-key.pem - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ rootCA.p12: rootCA.pem rootCA-key.pem @openssl pkcs12 -export -out rootCA.p12 -in rootCA.pem -inkey rootCA-key.pem -passout pass:${ROOT_CERT_PASSWORD} - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ server.pem: localhost+5.pem @cp localhost+5.pem server.pem - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ server-key.pem: localhost+5-key.pem @cp localhost+5-key.pem server-key.pem - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ server.p12: localhost+5.p12 @cp localhost+5.p12 server.p12 - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ client.pem: localhost+5-client.pem @cp localhost+5-client.pem client.pem - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ client-key.pem: localhost+5-client-key.pem @cp localhost+5-client-key.pem client-key.pem - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ client-key.der: localhost+5-client-key.der @cp localhost+5-client-key.der client-key.der - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ client.p12: localhost+5-client.p12 @cp localhost+5-client.p12 client.p12 - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ define mappings #!/bin/bash @@ -175,22 +177,22 @@ export mappings init-mappings-db.sh: @echo "$$mappings" > init-mappings-db.sh - @docker secret create $@ $@ + @$(DOCKER) secret create $@ $@ deploy: init - @docker stack deploy LEGA -c docker-stack.yml + @$(DOCKER) stack deploy LEGA -c docker-stack.yml ls: - @docker service list + @$(DOCKER) service list rm: - @docker stack rm LEGA + @$(DOCKER) stack rm LEGA @sleep 10 clean: - @rm -rf $(FILES) - @rm -rf /tmp/tsd /tmp/vault /tmp/db - @docker secret rm $(FILES) + @sudo rm -rf $(FILES) + @sudo rm -rf /tmp/tsd /tmp/vault /tmp/db + @$(DOCKER) secret rm $(FILES) test: - @mvn --no-transfer-progress test + @mvn -B test | grep -v 'Download.* http' From 0a582eaa19ac3e31db4d06b53f5b6907283bdd04 Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Thu, 29 Sep 2022 20:35:38 +0200 Subject: [PATCH 11/21] Fixed proxy port and cegamq connection url to new setup --- src/test/java/no/neic/localega/deploy/IngestionTest.java | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/test/java/no/neic/localega/deploy/IngestionTest.java b/src/test/java/no/neic/localega/deploy/IngestionTest.java index 35dfb3e..3cca9a9 100644 --- a/src/test/java/no/neic/localega/deploy/IngestionTest.java +++ b/src/test/java/no/neic/localega/deploy/IngestionTest.java @@ -113,7 +113,7 @@ private void upload() throws IOException, InvalidKeySpecException, NoSuchAlgorit log.info("Visa JWT token: {}", token); String md5Hex = DigestUtils.md5Hex(Files.newInputStream(encFile.toPath())); log.info("Encrypted MD5 checksum: {}", md5Hex); - String uploadURL = String.format("https://localhost/stream/%s?md5=%s", encFile.getName(), md5Hex); + String uploadURL = String.format("https://localhost:10443/stream/%s?md5=%s", encFile.getName(), md5Hex); JsonNode jsonResponse = Unirest .patch(uploadURL) .basicAuth(System.getenv("EGA_BOX_USERNAME"), System.getenv("EGA_BOX_PASSWORD")) @@ -123,7 +123,7 @@ private void upload() throws IOException, InvalidKeySpecException, NoSuchAlgorit .getBody(); String uploadId = jsonResponse.getObject().getString("id"); log.info("Upload ID: {}", uploadId); - String finalizeURL = String.format("https://localhost/stream/%s?uploadId=%s&chunk=end&sha256=%s&fileSize=%s", + String finalizeURL = String.format("https://localhost:10443/stream/%s?uploadId=%s&chunk=end&sha256=%s&fileSize=%s", encFile.getName(), uploadId, encSHA256Checksum, @@ -139,7 +139,8 @@ private void upload() throws IOException, InvalidKeySpecException, NoSuchAlgorit private void ingest() throws IOException, TimeoutException, NoSuchAlgorithmException, KeyManagementException, URISyntaxException { log.info("Publishing ingestion message to CentralEGA..."); - String mqConnectionString = System.getenv("CEGA_MQ_CONNECTION"); + // Hardcoding url to mapped port from cegamq container + String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=/certs/ca.pem"); ConnectionFactory factory = new ConnectionFactory(); factory.setUri(mqConnectionString); Connection connectionFactory = factory.newConnection(); From ee32fc99ec10dff5b6b83ad7bafeaa852bdb515f Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Thu, 29 Sep 2022 21:07:27 +0200 Subject: [PATCH 12/21] Updated tests to new local deployment, with improved function names and comments to explain the process --- .../neic/localega/deploy/IngestionTest.java | 84 +++++++++++++++---- 1 file changed, 70 insertions(+), 14 deletions(-) diff --git a/src/test/java/no/neic/localega/deploy/IngestionTest.java b/src/test/java/no/neic/localega/deploy/IngestionTest.java index 3cca9a9..5503829 100644 --- a/src/test/java/no/neic/localega/deploy/IngestionTest.java +++ b/src/test/java/no/neic/localega/deploy/IngestionTest.java @@ -57,6 +57,7 @@ public class IngestionTest { private File encFile; private String rawSHA256Checksum; private String encSHA256Checksum; + private String rawMD5Checksum; private String stableId; private String datasetId; private String archivePath; @@ -69,10 +70,16 @@ public void setup() throws IOException, GeneralSecurityException { RandomAccessFile randomAccessFile = new RandomAccessFile(rawFile, "rw"); randomAccessFile.setLength(fileSize); randomAccessFile.close(); + byte[] bytes = DigestUtils.sha256(Files.newInputStream(rawFile.toPath())); rawSHA256Checksum = Hex.encodeHexString(bytes); log.info("Raw SHA256 checksum: " + rawSHA256Checksum); + byte[] bytes2 = DigestUtils.md5(Files.newInputStream(rawFile.toPath())); + rawMD5Checksum = Hex.encodeHexString(bytes2); + log.info("Raw MD5 checksum: " + rawMD5Checksum); + + log.info("Generating sender and recipient key-pairs..."); KeyPair senderKeyPair = keyUtils.generateKeyPair(); @@ -95,12 +102,27 @@ public void setup() throws IOException, GeneralSecurityException { public void test() { try { upload(); - Thread.sleep(10000); // wait for triggers to be set up at CEGA - ingest(); - Thread.sleep(10000); // wait for ingestion and verification to be finished - verify(); - map(); - download(); + Thread.sleep(5000); // wait for triggers to be set up at CEGA - Not really needed if using local CEGA container + + triggerIngestMessageFromCEGA(); + Thread.sleep(5000); // Wait for the LEGA ingest and verify services to complete and update DB + + triggerAccessionMessageFromCEGA(); + Thread.sleep(5000); // Wait for LEGA finalize service to complete and update DB + + // Verify that everything is ok so far + verifyAfterFinalizeAndLookUpAccessionID(); + + // Trigger the process further, + // with retrieved information from earlier steps + triggerMappingMessageFromCEGA(); + + // No wait needed here probably? Prev mapping step is pretty quick. + + + // Test and check that what we get out match the original + // inserted data at the top + downloadDatasetAndVerifyResults(); } catch (Throwable t) { log.error(t.getMessage(), t); Assert.fail(); @@ -137,7 +159,7 @@ private void upload() throws IOException, InvalidKeySpecException, NoSuchAlgorit Assert.assertEquals(201, jsonResponse.getObject().getInt("statusCode")); } - private void ingest() throws IOException, TimeoutException, NoSuchAlgorithmException, KeyManagementException, URISyntaxException { + private void triggerIngestMessageFromCEGA() throws IOException, TimeoutException, NoSuchAlgorithmException, KeyManagementException, URISyntaxException { log.info("Publishing ingestion message to CentralEGA..."); // Hardcoding url to mapped port from cegamq container String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=/certs/ca.pem"); @@ -164,9 +186,41 @@ private void ingest() throws IOException, TimeoutException, NoSuchAlgorithmExcep connectionFactory.close(); } + private void triggerAccessionMessageFromCEGA() throws IOException, TimeoutException, NoSuchAlgorithmException, KeyManagementException, URISyntaxException { + log.info("Publishing accession message on behalf of CEGA to CEGA RMQ..."); + // Hardcoding url to mapped port from cegamq container + String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=/certs/ca.pem"); + ConnectionFactory factory = new ConnectionFactory(); + factory.setUri(mqConnectionString); + Connection connectionFactory = factory.newConnection(); + Channel channel = connectionFactory.createChannel(); + AMQP.BasicProperties properties = new AMQP.BasicProperties() + .builder() + .deliveryMode(2) + .contentType("application/json") + .contentEncoding(StandardCharsets.UTF_8.displayName()) + .correlationId(UUID.randomUUID().toString()) + .build(); + + String randomFileAccessionID = "EGAF" + getRandomNumber(11); + + String message = String.format("{\"type\":\"accession\",\"user\":\"%s\",\"filepath\":\"/p11-dummy@elixir-europe.org/files/%s\",\"accession_id\":\"%s\", \"decrypted_checksums\": [ { \"type\":\"sha256\", \"value\": \"%s\" },{\"type\":\"md5\", \"value\": \"%s\"} ] }", System.getenv("EGA_BOX_USERNAME"), encFile.getName(), randomFileAccessionID, rawSHA256Checksum, rawMD5Checksum); + log.info(message); + channel.basicPublish("localega.v1", + "files", + properties, + message.getBytes()); + + channel.close(); + connectionFactory.close(); + } + + + + @SuppressWarnings({"SqlResolve", "SqlNoDataSourceInspection"}) - private void verify() throws SQLException { - log.info("Starting verification..."); + private void verifyAfterFinalizeAndLookUpAccessionID() throws SQLException { + log.info("Starting verification of state after finalize step..."); String dbHost = "localhost"; String port = "5432"; String db = "lega"; @@ -195,12 +249,13 @@ private void verify() throws SQLException { log.info("Verification completed successfully"); } - private void map() throws NoSuchAlgorithmException, KeyManagementException, URISyntaxException, IOException, TimeoutException { + private void triggerMappingMessageFromCEGA() throws NoSuchAlgorithmException, KeyManagementException, URISyntaxException, IOException, TimeoutException { log.info("Mapping file to a dataset..."); datasetId = "EGAD" + getRandomNumber(11); - String mqConnectionString = System.getenv("CEGA_MQ_CONNECTION"); + // Hardcode url to mapped port from container to localhost + String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=/certs/ca.pem"); ConnectionFactory factory = new ConnectionFactory(); factory.setUri(mqConnectionString); @@ -217,7 +272,7 @@ private void map() throws NoSuchAlgorithmException, KeyManagementException, URIS String message = String.format("{\"type\":\"mapping\",\"accession_ids\":[\"%s\"],\"dataset_id\":\"%s\"}", stableId, datasetId); log.info(message); channel.basicPublish("localega.v1", - "mapping", + "files", properties, message.getBytes()); @@ -227,7 +282,7 @@ private void map() throws NoSuchAlgorithmException, KeyManagementException, URIS log.info("Permissions granted successfully"); } - private void download() throws GeneralSecurityException, IOException { + private void downloadDatasetAndVerifyResults() throws GeneralSecurityException, IOException { String token = generateVisaToken(datasetId); log.info("Visa JWT token: {}", token); @@ -237,7 +292,8 @@ private void download() throws GeneralSecurityException, IOException { .asString() .getBody(); Assert.assertEquals(String.format("[\"%s\"]", datasetId).strip(), datasets.strip()); - + + String expected = String.format( "[{\"fileId\":\"%s\",\"datasetId\":\"%s\",\"displayFileName\":\"%s\",\"fileName\":\"%s\",\"fileSize\":10490240,\"unencryptedChecksum\":null,\"unencryptedChecksumType\":null,\"decryptedFileSize\":10485760,\"decryptedFileChecksum\":\"%s\",\"decryptedFileChecksumType\":\"SHA256\",\"fileStatus\":\"READY\"}]\n", stableId, From ba57a14154f34bcc7654c96a233f1846f9dbc6f7 Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Thu, 29 Sep 2022 21:27:04 +0200 Subject: [PATCH 13/21] Fixed the manually entered secrets section/comments --- Makefile | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 4aeafef..1aaa8c9 100644 --- a/Makefile +++ b/Makefile @@ -5,16 +5,17 @@ DOCKER := sudo docker FILES := localhost+5.pem localhost+5-key.pem localhost+5-client.pem localhost+5-client-key.pem rootCA.pem rootCA.p12 localhost+5.p12 localhost+5-client.p12 localhost+5-client-key.der rootCA-key.pem docker-stack.yml jwt.pub.pem jwt.priv.pem ega.pub.pem ega.sec.pass ega.sec.pem server.pem server-key.pem server.p12 client.pem client-key.pem client-key.der client.p12 init-mappings-db.sh # Manually entered secrets not available in the repo +# (Either uncomment and set, or load from separate file) #################################################### -# proxy service CEGA NSS -export CEGA_AUTH_URL=https://nss-test.ega-archive.org/users/ +# proxy service CEGA NSS (norway1 connection details) +#export CEGA_AUTH_URL=https://nss-test.ega-archive.org/users/ #export CEGA_USERNAME= #export CEGA_PASSWORD= +#export EGA_BOX_USERNAME=ega-box-XXXX +#export EGA_BOX_PASSWORD= # Prefilled configs from the repo From 8779bf2865f9ba7185e113a9c60bf84f199647c1 Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Thu, 29 Sep 2022 21:31:52 +0200 Subject: [PATCH 14/21] Update tests.yml Removed loading of secrets to environment variables no longer needed in the local cegamq setup. This setup has hardcoded/matching default config values present in the code instead. --- .github/workflows/tests.yml | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 148e40f..00d8703 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -24,17 +24,9 @@ jobs: ACTIONS_ALLOW_UNSECURE_COMMANDS: true BINDIR: /home/runner/.local/bin/ SLEEP: ${{ secrets.SLEEP }} - CEGA_MQ_CONNECTION: ${{ secrets.CEGA_MQ_CONNECTION }} + CEGA_AUTH_URL: ${{ secrets.CEGA_AUTH_URL }} CEGA_USERNAME: ${{ secrets.CEGA_USERNAME }} CEGA_PASSWORD: ${{ secrets.CEGA_PASSWORD }} - BROKER_HOST: ${{ secrets.BROKER_HOST }} - BROKER_PORT: ${{ secrets.BROKER_PORT }} - BROKER_USERNAME: ${{ secrets.BROKER_USERNAME }} - BROKER_PASSWORD: ${{ secrets.BROKER_PASSWORD }} - BROKER_VHOST: ${{ secrets.BROKER_VHOST }} - BROKER_VALIDATE: ${{ secrets.BROKER_VALIDATE }} - EXCHANGE: ${{ secrets.EXCHANGE }} - CEGA_AUTH_URL: ${{ secrets.CEGA_AUTH_URL }} EGA_BOX_USERNAME: ${{ secrets.EGA_BOX_USERNAME }} EGA_BOX_PASSWORD: ${{ secrets.EGA_BOX_PASSWORD }} From 2277864d228304a71092f7bfa33a1c6a6e9e6c08 Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Fri, 30 Sep 2022 08:47:55 +0200 Subject: [PATCH 15/21] Fixed the path to CA cert for hardcoded cegamq URL parameter to match the cert file generated my Makefile --- src/test/java/no/neic/localega/deploy/IngestionTest.java | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/test/java/no/neic/localega/deploy/IngestionTest.java b/src/test/java/no/neic/localega/deploy/IngestionTest.java index 5503829..a82ac23 100644 --- a/src/test/java/no/neic/localega/deploy/IngestionTest.java +++ b/src/test/java/no/neic/localega/deploy/IngestionTest.java @@ -162,7 +162,7 @@ private void upload() throws IOException, InvalidKeySpecException, NoSuchAlgorit private void triggerIngestMessageFromCEGA() throws IOException, TimeoutException, NoSuchAlgorithmException, KeyManagementException, URISyntaxException { log.info("Publishing ingestion message to CentralEGA..."); // Hardcoding url to mapped port from cegamq container - String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=/certs/ca.pem"); + String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=rootCA.pem"); ConnectionFactory factory = new ConnectionFactory(); factory.setUri(mqConnectionString); Connection connectionFactory = factory.newConnection(); @@ -189,7 +189,7 @@ private void triggerIngestMessageFromCEGA() throws IOException, TimeoutException private void triggerAccessionMessageFromCEGA() throws IOException, TimeoutException, NoSuchAlgorithmException, KeyManagementException, URISyntaxException { log.info("Publishing accession message on behalf of CEGA to CEGA RMQ..."); // Hardcoding url to mapped port from cegamq container - String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=/certs/ca.pem"); + String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=rootCA.pem"); ConnectionFactory factory = new ConnectionFactory(); factory.setUri(mqConnectionString); Connection connectionFactory = factory.newConnection(); @@ -255,7 +255,7 @@ private void triggerMappingMessageFromCEGA() throws NoSuchAlgorithmException, Ke datasetId = "EGAD" + getRandomNumber(11); // Hardcode url to mapped port from container to localhost - String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=/certs/ca.pem"); + String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=rootCA.pem"); ConnectionFactory factory = new ConnectionFactory(); factory.setUri(mqConnectionString); From 867307079971efbb1030c30eef2391731798012d Mon Sep 17 00:00:00 2001 From: Parisa Date: Fri, 30 Sep 2022 14:06:55 +0200 Subject: [PATCH 16/21] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8ad436f..29bad0b 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# LocalEGA-deploy-swarm +# LocalEGA-deploy-swarm-fork ![Integration tests](https://github.com/neicnordic/LocalEGA-deploy-swarm/workflows/Integration%20tests/badge.svg) Docker Swarm deployment of LocalEGA. This is a fork meant for internal testing purposes. From 57ccbd118e00b5bea1c19f19cb8ceaf10498cf5a Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Mon, 3 Oct 2022 08:53:57 +0200 Subject: [PATCH 17/21] Changed tsd container to run as nobody to fix Issue #2, updated verify service to latest again --- docker-template.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docker-template.yml b/docker-template.yml index ec09c4d..eb50602 100644 --- a/docker-template.yml +++ b/docker-template.yml @@ -19,6 +19,8 @@ services: target: /etc/ega/ssl/server.cert volumes: - /tmp/tsd:/tsd/p11/data/durable/apps/ega/ + user: "65534:65534" + db: image: ghcr.io/neicnordic/sda-db:latest @@ -231,7 +233,7 @@ services: command: "sda-ingest" verify: - image: ghcr.io/neicnordic/sda-pipeline:v0.2.5 + image: ghcr.io/neicnordic/sda-pipeline:latest depends_on: - mq - db From a857933599559b0924442eac3a4f1aef85ef1aef Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Mon, 3 Oct 2022 09:05:03 +0200 Subject: [PATCH 18/21] Added mock cega authentication container in docker-template.yml file --- docker-template.yml | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/docker-template.yml b/docker-template.yml index eb50602..0ab64d8 100644 --- a/docker-template.yml +++ b/docker-template.yml @@ -428,8 +428,10 @@ services: user: "65534:65534" - # CEGA mock (RabbitMQ only for now) - # based on sda-auth/.github/intergration/docker-compose.yml + # CEGA mock + # Plain RabbitMQ only for now - cega messages triggered manually by test code + # based on neicnordic/sda-auth/.github/intergration/docker-compose.yml + # CEGA-auth based on neicnordic/sda-auth/dev-server/docker-compose.yml cegamq: image: rabbitmq:3.8.16-management-alpine @@ -469,6 +471,26 @@ services: timeout: 20s retries: 3 + cega-auth: + container_name: cega-auth + image: egarchive/lega-base:release.v0.2.0 + volumes: + - ./cega-users:/cega + command: + [ + "python", + "/cega/cega-mock.py", + "0.0.0.0", + "8443", + "/cega/users.json" + ] + environment: + - LEGA_INSTANCES=dummy + - CEGA_USERS_PASSWORD=dummy + - CEGA_USERS_USER=dummy + ports: + - 8443:8443 + secrets: From 58e9d439a9271ca99039ec4f8cfdfc6f073c6637 Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Mon, 3 Oct 2022 11:09:41 +0200 Subject: [PATCH 19/21] Added config files for mock cega-auth service and changed deployment to use this instead of live CEGA auth connection. --- Makefile | 19 ++++---- cega-users/cega-mock.py | 103 ++++++++++++++++++++++++++++++++++++++++ cega-users/users.json | 7 +++ 3 files changed, 120 insertions(+), 9 deletions(-) create mode 100644 cega-users/cega-mock.py create mode 100644 cega-users/users.json diff --git a/Makefile b/Makefile index 1aaa8c9..6194abb 100644 --- a/Makefile +++ b/Makefile @@ -4,18 +4,19 @@ DOCKER := sudo docker FILES := localhost+5.pem localhost+5-key.pem localhost+5-client.pem localhost+5-client-key.pem rootCA.pem rootCA.p12 localhost+5.p12 localhost+5-client.p12 localhost+5-client-key.der rootCA-key.pem docker-stack.yml jwt.pub.pem jwt.priv.pem ega.pub.pem ega.sec.pass ega.sec.pem server.pem server-key.pem server.p12 client.pem client-key.pem client-key.der client.p12 init-mappings-db.sh -# Manually entered secrets not available in the repo -# (Either uncomment and set, or load from separate file) +# Previously manually entered secrets not available in the repo +# Now with default values in mock containers instead #################################################### -# proxy service CEGA NSS (norway1 connection details) -#export CEGA_AUTH_URL=https://nss-test.ega-archive.org/users/ -#export CEGA_USERNAME= -#export CEGA_PASSWORD= +# Test user in CEGA NSS mock service +# (confusing it's the same as ID and SECRET above).... +export EGA_BOX_USERNAME=dummy +export EGA_BOX_PASSWORD=dummy # Prefilled configs from the repo diff --git a/cega-users/cega-mock.py b/cega-users/cega-mock.py new file mode 100644 index 0000000..b771dff --- /dev/null +++ b/cega-users/cega-mock.py @@ -0,0 +1,103 @@ +import sys +import os +import logging +import asyncio +import json +from base64 import b64decode +from aiohttp import web + +#logging.basicConfig(format='[%(asctime)s][%(levelname)-8s] (L:%(lineno)s) %(message)s', datefmt='%Y-%m-%d %H:%M:%S') +logging.basicConfig(format='[%(levelname)-8s] (L:%(lineno)s) %(message)s') +LOG = logging.getLogger(__name__) +LOG.setLevel(logging.INFO) + +filepath = None +instances = {} +store = None +usernames = {} +uids = {} + +def fetch_user_info(identifier, query): + id_type = query.get('idType', None) + if not id_type: + raise web.HTTPBadRequest(text='Missing or wrong idType') + LOG.info(f'Requesting User {identifier} [type {id_type}]') + if id_type == 'username': + pos = usernames.get(identifier, None) + return store[pos] if pos is not None else None + if id_type == 'uid': + try: + pos = uids.get(int(identifier), None) + return store[pos] if pos is not None else None + except Exception: + return None + raise web.HTTPBadRequest(text='Missing or wrong idType') + +async def user(request): + # Authenticate + auth_header = request.headers.get('AUTHORIZATION') + if not auth_header: + raise web.HTTPUnauthorized(text=f'Protected access\n') + _, token = auth_header.split(None, 1) # Skipping the Basic keyword + LOG.debug(f'Token is {token}') + instance, passwd = b64decode(token).decode().split(':', 1) + LOG.debug(f'I am instance {instance} and the password is {passwd}') + info = instances.get(instance) + if info is None or info != passwd: + raise web.HTTPUnauthorized(text=f'Protected access\n') + + # Reload users list + load_users() + + # Find user + user_info = fetch_user_info(request.match_info['identifier'], request.rel_url.query) + if user_info is None: + raise web.HTTPBadRequest(text=f'No info for that user\n') + return web.json_response({ 'header': { "apiVersion": "v1", + "code": "200", + "service": "users", + "developerMessage": "", + "userMessage": "OK", + "errorCode": "1", + "docLink": "https://ega-archive.org", + "errorStack": "" }, + 'response': { "numTotalResults": 1, + "resultType": "eu.crg.ega.microservice.dto.lega.v1.users.LocalEgaUser", + "result": [ user_info ]} + }) + +def main(): + print("Main is being run") + if len(sys.argv) < 3: + print('Usage: {sys.argv[0] }', file=sys.stderr) + sys.exit(2) + + host = sys.argv[1] + port = sys.argv[2] + + global filepath + filepath = sys.argv[3] + + server = web.Application() + load_users() + + # Registering the routes + server.router.add_get('/lega/v1/legas/users/{identifier}', user, name='user') + + # aaaand... cue music + web.run_app(server, host=host, port=port, shutdown_timeout=0) + + +def load_users(): + # Initialization + global filepath, instances, store, usernames, uids + instances[os.environ[f'CEGA_USERS_USER']] = os.environ[f'CEGA_USERS_PASSWORD'] + with open(filepath, 'rt') as f: + store = json.load(f) + for i, d in enumerate(store): + usernames[d['username']] = i + uids[d['uid']] = i + + +if __name__ == '__main__': + main() diff --git a/cega-users/users.json b/cega-users/users.json new file mode 100644 index 0000000..2b16b72 --- /dev/null +++ b/cega-users/users.json @@ -0,0 +1,7 @@ +[{"username": "dummy", + "uid": 1, + "passwordHash": "$2b$12$1gyKIjBc9/cT0MYkXX24xe1LjEUjNwgL4rEk8fDoO.vDQZzWkqrn.", + "gecos": "dummy user", + "sshPublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDiEcu2czBfbQh6+A3DplO2DRHG4LmdUKLRPX1vFOpC30TZusio0cFcgi8TEQv9TlMwu2ujF/wn/0D2VwXiDGk/Rbeq9jgTLpVbDWg/3ZGxGSqjPV3fzl0NzmaSgF0+IZQaSr6OjGVpTmpk5G4d4qqFg4Shjjm+AwlgmruThHNS9KmdJ7Vru+rQ8LwcjuSWtBdf6JM3bjlw1swQt6776p+wTK51YSKdtFEE5yVZjVwxlcPre7sRiem0XwSCFsu9sAfUTbHNTfwQ8lVXbvgRGu9SoW0wwb0Qele1WXZ8YFF10KLxGKpb1u0NsXSIZrJZhk0nxKb5tGBSnKXquoAvLZfVEKc+AXw1sDSaKvZaDw0/GORoAVSt3LDKYydAlzpMw6am4fgcEzm0vwCieWvSxd9uLY9IxV4sx0n0ZcuG55Le2TaQMnSm5XQ8zHBFYOb9ux8h6TY6JO+HmSjoHXkGhILKg8Y7zpq0PWy7HUvWzQVVfShAMN/N2gZ3a2T7amg17/S0wtgvjxxpNtnLc0HnHjr/mwtBjrN8C4n+IYI13rqZzPPU1wZu5qiacmHmeR15XAktEFKrpuvViJcylksjwyl6aY9psm+dwocON/yA3pdJGA8mPvrnDpPkGpzqvTqqIMxQkgel46jF2B7+lLzq6wQOsb7Ct66CKKppM6kpVVHRWQ==", + "enabled": null +}] \ No newline at end of file From eaf7b5dd41106fb403afe05e4ea60e922738c130 Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Mon, 3 Oct 2022 11:28:44 +0200 Subject: [PATCH 20/21] Updated README docs with CEGA mock service information now available. --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index fbce81f..7515522 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,10 @@ Docker Swarm deployment of LocalEGA. Please, refer to the project documentation ## Purpose -The `docker-template.yml` file can be used in two ways: -1. Deploy LocalEGA locally as containers which can be used for CI/CD testing in Github (the CEGA BROKER service is not included). +The `docker-template.yml` file can be used in several ways: +1. Deploy all the NeIC nordic microservices in use by FEGA Norway locally as containers together with mock services for TSD and CEGA functionality needed. 2. One can trim it down to a Public stack or a Private stack depending on what needs to be deployed. +3. GitHub actions can test nightly if the master branch is building and testing ok (without connection to live CEGA services. For that, please visit the upstream neicnordic/localega-deploy-swarm repo) ## Development @@ -19,7 +20,7 @@ The `docker-template.yml` file can be used in two ways: **IMPORTANT** -CEGA-related env-vars should be included manually in `Makefile` before running the makefile. The variables are: +CEGA-related env-vars _is no more needed to be set_ manually in `Makefile` before running the makefile. They decker-template.yml file already contains default values expected by the micro-services to work. The variables are: ``` export CEGA_USERNAME= From 38ba5960d11b29093c24d5a1c56d8815b9a6f90a Mon Sep 17 00:00:00 2001 From: Kjell Petersen Date: Fri, 19 Jan 2024 14:18:15 +0100 Subject: [PATCH 21/21] Updated end 2 end test to new message conventions/DB entries --- .../neic/localega/deploy/IngestionTest.java | 69 +++++++++++++++---- 1 file changed, 56 insertions(+), 13 deletions(-) diff --git a/src/test/java/no/neic/localega/deploy/IngestionTest.java b/src/test/java/no/neic/localega/deploy/IngestionTest.java index a82ac23..5cf0c9e 100644 --- a/src/test/java/no/neic/localega/deploy/IngestionTest.java +++ b/src/test/java/no/neic/localega/deploy/IngestionTest.java @@ -61,6 +61,7 @@ public class IngestionTest { private String stableId; private String datasetId; private String archivePath; + private String correlationId; @Before public void setup() throws IOException, GeneralSecurityException { @@ -117,8 +118,10 @@ public void test() { // with retrieved information from earlier steps triggerMappingMessageFromCEGA(); - // No wait needed here probably? Prev mapping step is pretty quick. + Thread.sleep(1000); // Wait for LEGA mapper service to store mapping + triggerReleaseMessageFromCEGA(); + Thread.sleep(1000); // Wait for LEGA mapper service to update dataset status // Test and check that what we get out match the original // inserted data at the top @@ -135,6 +138,7 @@ private void upload() throws IOException, InvalidKeySpecException, NoSuchAlgorit log.info("Visa JWT token: {}", token); String md5Hex = DigestUtils.md5Hex(Files.newInputStream(encFile.toPath())); log.info("Encrypted MD5 checksum: {}", md5Hex); + log.info("EGA_BOX_USERNAME: {}",System.getenv("EGA_BOX_USERNAME")); String uploadURL = String.format("https://localhost:10443/stream/%s?md5=%s", encFile.getName(), md5Hex); JsonNode jsonResponse = Unirest .patch(uploadURL) @@ -167,12 +171,14 @@ private void triggerIngestMessageFromCEGA() throws IOException, TimeoutException factory.setUri(mqConnectionString); Connection connectionFactory = factory.newConnection(); Channel channel = connectionFactory.createChannel(); + correlationId = UUID.randomUUID().toString(); + AMQP.BasicProperties properties = new AMQP.BasicProperties() .builder() .deliveryMode(2) .contentType("application/json") .contentEncoding(StandardCharsets.UTF_8.displayName()) - .correlationId(UUID.randomUUID().toString()) + .correlationId(correlationId) .build(); String message = String.format("{\"type\":\"ingest\",\"user\":\"%s\",\"filepath\":\"/p11-dummy@elixir-europe.org/files/%s\"}", System.getenv("EGA_BOX_USERNAME"), encFile.getName()); @@ -199,10 +205,10 @@ private void triggerAccessionMessageFromCEGA() throws IOException, TimeoutExcept .deliveryMode(2) .contentType("application/json") .contentEncoding(StandardCharsets.UTF_8.displayName()) - .correlationId(UUID.randomUUID().toString()) + .correlationId(correlationId) .build(); - String randomFileAccessionID = "EGAF" + getRandomNumber(11); + String randomFileAccessionID = "EGAF5" + getRandomNumber(10); String message = String.format("{\"type\":\"accession\",\"user\":\"%s\",\"filepath\":\"/p11-dummy@elixir-europe.org/files/%s\",\"accession_id\":\"%s\", \"decrypted_checksums\": [ { \"type\":\"sha256\", \"value\": \"%s\" },{\"type\":\"md5\", \"value\": \"%s\"} ] }", System.getenv("EGA_BOX_USERNAME"), encFile.getName(), randomFileAccessionID, rawSHA256Checksum, rawMD5Checksum); log.info(message); @@ -230,25 +236,26 @@ private void verifyAfterFinalizeAndLookUpAccessionID() throws SQLException { props.setProperty("password", System.getenv("DB_LEGA_IN_PASSWORD")); props.setProperty("ssl", "true"); props.setProperty("application_name", "LocalEGA"); - props.setProperty("sslmode", "verify-full"); + props.setProperty("sslmode", "verify-ca"); props.setProperty("sslrootcert", new File("rootCA.pem").getAbsolutePath()); props.setProperty("sslcert", new File("localhost+5-client.pem").getAbsolutePath()); props.setProperty("sslkey", new File("localhost+5-client-key.der").getAbsolutePath()); java.sql.Connection conn = DriverManager.getConnection(url, props); - String sql = "select * from local_ega.files where status = 'READY' AND inbox_path = ?"; + String sql = "select archive_path,stable_id from local_ega.files where status = 'COMPLETED' AND inbox_path = ?"; PreparedStatement statement = conn.prepareStatement(sql); statement.setString(1, "/p11-dummy@elixir-europe.org/files/" + encFile.getName()); ResultSet resultSet = statement.executeQuery(); if (resultSet.wasNull() || !resultSet.next()) { Assert.fail("Verification failed"); } - archivePath = resultSet.getString(8); - stableId = resultSet.getString(16); + archivePath = resultSet.getString(1); + stableId = resultSet.getString(2); log.info("Stable ID: {}", stableId); log.info("Archive path: {}", archivePath); log.info("Verification completed successfully"); } + private void triggerMappingMessageFromCEGA() throws NoSuchAlgorithmException, KeyManagementException, URISyntaxException, IOException, TimeoutException { log.info("Mapping file to a dataset..."); @@ -266,7 +273,7 @@ private void triggerMappingMessageFromCEGA() throws NoSuchAlgorithmException, Ke .deliveryMode(2) .contentType("application/json") .contentEncoding(StandardCharsets.UTF_8.displayName()) - .correlationId(UUID.randomUUID().toString()) + .correlationId(correlationId) .build(); String message = String.format("{\"type\":\"mapping\",\"accession_ids\":[\"%s\"],\"dataset_id\":\"%s\"}", stableId, datasetId); @@ -279,9 +286,42 @@ private void triggerMappingMessageFromCEGA() throws NoSuchAlgorithmException, Ke channel.close(); connectionFactory.close(); - log.info("Permissions granted successfully"); + log.info("Mapping file to datset ID message sent successfully"); } + + private void triggerReleaseMessageFromCEGA() throws NoSuchAlgorithmException, KeyManagementException, URISyntaxException, IOException, TimeoutException { + log.info("Releasing the dataset..."); + + // Hardcode url to mapped port from container to localhost + String mqConnectionString = new String("amqps://test:test@localhost:5672/lega?cacertfile=rootCA.pem"); + ConnectionFactory factory = new ConnectionFactory(); + factory.setUri(mqConnectionString); + + Connection connectionFactory = factory.newConnection(); + Channel channel = connectionFactory.createChannel(); + AMQP.BasicProperties properties = new AMQP.BasicProperties() + .builder() + .deliveryMode(2) + .contentType("application/json") + .contentEncoding(StandardCharsets.UTF_8.displayName()) + .correlationId(correlationId) + .build(); + + String message = String.format("{\"type\":\"release\",\"dataset_id\":\"%s\"}", datasetId); + log.info(message); + channel.basicPublish("localega.v1", + "files", + properties, + message.getBytes()); + + channel.close(); + connectionFactory.close(); + + log.info("Dataset release message sent successfully"); + } + + private void downloadDatasetAndVerifyResults() throws GeneralSecurityException, IOException { String token = generateVisaToken(datasetId); log.info("Visa JWT token: {}", token); @@ -293,6 +333,7 @@ private void downloadDatasetAndVerifyResults() throws GeneralSecurityException, .getBody(); Assert.assertEquals(String.format("[\"%s\"]", datasetId).strip(), datasets.strip()); + //Meta data check String expected = String.format( "[{\"fileId\":\"%s\",\"datasetId\":\"%s\",\"displayFileName\":\"%s\",\"fileName\":\"%s\",\"fileSize\":10490240,\"unencryptedChecksum\":null,\"unencryptedChecksumType\":null,\"decryptedFileSize\":10485760,\"decryptedFileChecksum\":\"%s\",\"decryptedFileChecksumType\":\"SHA256\",\"fileStatus\":\"READY\"}]\n", @@ -317,7 +358,9 @@ private void downloadDatasetAndVerifyResults() throws GeneralSecurityException, .getBody(); String obtainedChecksum = Hex.encodeHexString(DigestUtils.sha256(file)); Assert.assertEquals(rawSHA256Checksum, obtainedChecksum); + + KeyPair recipientKeyPair = keyUtils.generateKeyPair(); StringWriter stringWriter = new StringWriter(); keyUtils.writeCrypt4GHKey(stringWriter, recipientKeyPair.getPublic(), null); @@ -340,13 +383,13 @@ private void downloadDatasetAndVerifyResults() throws GeneralSecurityException, private String generateVisaToken(String resource) throws NoSuchAlgorithmException, InvalidKeySpecException, IOException { RSAPublicKey publicKey = getPublicKey(); RSAPrivateKey privateKey = getPrivateKey(); - byte[] visaHeader = Base64.getEncoder().encode(("{\n" + + byte[] visaHeader = Base64.getUrlEncoder().encode(("{\n" + " \"jku\": \"https://login.elixir-czech.org/oidc/jwk\",\n" + " \"kid\": \"rsa1\",\n" + " \"typ\": \"JWT\",\n" + " \"alg\": \"RS256\"\n" + "}").getBytes()); - byte[] visaPayload = Base64.getEncoder().encode(String.format("{\n" + + byte[] visaPayload = Base64.getUrlEncoder().encode(String.format("{\n" + " \"sub\": \"dummy@elixir-europe.org\",\n" + " \"ga4gh_visa_v1\": {\n" + " \"asserted\": 1583757401,\n" + @@ -361,7 +404,7 @@ private String generateVisaToken(String resource) throws NoSuchAlgorithmExceptio " \"jti\": \"f520d56f-e51a-431c-94e1-2a3f9da8b0c9\"\n" + "}", resource).getBytes()); byte[] visaSignature = Algorithm.RSA256(publicKey, privateKey).sign(visaHeader, visaPayload); - return new String(visaHeader) + "." + new String(visaPayload) + "." + Base64.getEncoder().encodeToString(visaSignature); + return new String(visaHeader) + "." + new String(visaPayload) + "." + Base64.getUrlEncoder().encodeToString(visaSignature); } private RSAPublicKey getPublicKey() throws NoSuchAlgorithmException, InvalidKeySpecException, IOException {