Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable servers #485

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open

Configurable servers #485

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
893d233
allow server addresses to be modified for makefile and dockerfile via…
Zeitsperre Sep 22, 2023
6ebc33e
handle urljoin trip-fall
Zeitsperre Sep 27, 2023
d789d33
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 27, 2023
8bd2e8d
better highlighting
Zeitsperre Sep 27, 2023
f0d552d
use concurrency instead of styfle/cancel-workflow-action, update chec…
Zeitsperre Oct 2, 2023
2323f36
Merge branch 'main' into configurable-servers
Zeitsperre Oct 20, 2023
d9f0b65
allow for more substitution via environment variables
Zeitsperre Oct 20, 2023
55f95a4
Merge branch 'main' into configurable-servers
Zeitsperre Dec 11, 2023
9c482a6
Merge branch 'main' into configurable-servers
Zeitsperre May 13, 2024
036295a
Merge branch 'main' into configurable-servers
Zeitsperre May 21, 2024
1db41b7
Merge branch 'main' into configurable-servers
Zeitsperre Oct 18, 2024
85ccf93
fix docstrings
Zeitsperre Oct 18, 2024
32d0c98
revert non-working docker substitutions,
Zeitsperre Oct 18, 2024
39267a8
fix dev guide formatting
Zeitsperre Oct 18, 2024
7bff84c
pin dependencies
Zeitsperre Oct 18, 2024
f5398bf
unpin psycopg2
Zeitsperre Oct 18, 2024
9c763f7
update docs
Zeitsperre Oct 18, 2024
42f7b8a
address comments
Zeitsperre Oct 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONTRIBUTING.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

Please read the Birdhouse `Developer Guide <https://birdhouse.readthedocs.io/en/latest/dev_guide.html>`_
and the `Raven Documentation <https://pavics-raven.readthedocs.io/en/latest>`_ to get started.
Please read the Birdhouse `Developer Guide <https://birdhouse.readthedocs.io/en/latest/guidelines.html>`_
and the `Raven Documentation <https://pavics-sdi.readthedocs.io/projects/raven/en/latest/>`_ to get started.
10 changes: 7 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,20 @@ RUN mamba env create -n raven -f environment.yml \
# Add the raven conda environment to the path
ENV PATH /opt/conda/envs/raven/bin:$PATH

# Set bind address and port from env variable of parent system
ENV RAVEN_BIND_ADDRESS=${RAVEN_BIND_ADDRESS:-0.0.0.0}
ENV RAVEN_BIND_PORT=${RAVEN_BIND_PORT:-9099}

# Copy raven source code
COPY . /code

# Install raven
RUN pip install . --no-deps

# Start WPS service on port 9099 on 0.0.0.0
EXPOSE 9099
# Start WPS service on port 9099
EXPOSE $RAVEN_BIND_PORT

CMD ["gunicorn", "--bind=0.0.0.0:9099", "raven.wsgi:application"]
CMD ["gunicorn", "--bind=$RAVEN_BIND_ADDRESS:$RAVEN_BIND_PORT", "raven.wsgi:application"]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty sure $<env> format does not work in CMD.
Not sure about EXPOSE.

The workaround is to use an entrypoint shell script, which then uses the variables.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One easy way to confirm env var works in CMD and EXPOSE is to build the docker image and test it in the PAVICS stack.

Copy link

@fmigneault fmigneault Sep 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did that and as I remembered, docker run raises something related to $RAVEN_BIND_ADDRESS:$RAVEN_BIND_PORT being invalid because $RAVEN_BIND_PORT doesn't get replaced by the provided integer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given ENV RAVEN_BIND_PORT=${RAVEN_BIND_PORT:-9099}, if we can make the launching command "see" the env var, it should work. So probably use ENTRYPOINT instead of CMD. If needed, force starting a new shell to "see" the env var with bash -c "gunicorn --bind=$RAVEN_BIND_ADDRESS:$RAVEN_BIND_PORT raven.wsgi:application".

Or research a recipe for how to properly start gunicorn inside a Docker image.

#CMD ["exec raven-wps start -b '0.0.0.0' -c etc/demo.cfg"]

# docker build -t pavics/raven .
Expand Down
18 changes: 11 additions & 7 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,23 @@ APP_NAME := raven-wps

OS := $(shell uname)

WPS_PORT := 9099
WPS_URL := http://0.0.0.0:$(WPS_PORT)
WPS_PORT ?= 9099
WPS_URL ?= http://0.0.0.0:$(WPS_PORT)
Zeitsperre marked this conversation as resolved.
Show resolved Hide resolved

# If WPS_URL is overridden, this should also be overridden to match.
WPS_OUTPUT_URL := http://localhost:$(WPS_PORT)/outputs
WPS_OUTPUT_URL ?= http://localhost:$(WPS_PORT)/outputs

# This will only work on Linux (not macOS/homebrew GDAL)
GDAL_VERSION := $(shell gdal-config --version)
# This will only work for Linux and macOS/homebrew
ifeq ($(OS),"Linux")
GDAL_VERSION := $(shell gdal-config --version)
else ifeq ($(OS),"Darwin")
GDAL_VERSION := $(shell gdalinfo --version | awk '{print $2}' | sed s'/.$//')
endif

# Used in target refresh-notebooks to make it looks like the notebooks have
# been refreshed from the production server below instead of from the local dev
# instance so the notebooks can also be used as tutorial notebooks.
OUTPUT_URL = https://pavics.ouranos.ca/wpsoutputs/raven
OUTPUT_URL ?= https://pavics.ouranos.ca/wpsoutputs/raven

SANITIZE_FILE := https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/raw/master/notebooks/output-sanitize.cfg

Expand All @@ -39,7 +43,7 @@ UNAME_S := $(shell uname -s)
DOWNLOAD_CACHE = /tmp/

# Additional servers used by notebooks
FINCH_WPS_URL = https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps
FINCH_WPS_URL ?= https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps

# To run tests on local servers, use
# make FINCH_WPS_URL=http://localhost:5000 test-notebooks
Expand Down
31 changes: 11 additions & 20 deletions docs/source/dev_guide.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,7 @@ Do the same as above using the ``Makefile``.
Running notebooks tests
-----------------------

Assuming that the ``raven`` conda env has already been created and is up-to-date and
raven-wps has been installed with ``make develop``::
Assuming that the ``raven`` conda env has already been created and is up-to-date and `raven-wps`` has been installed with ``make develop``::
Zeitsperre marked this conversation as resolved.
Show resolved Hide resolved

# start local raven-wps server to test against
$ make start # remember to make stop once done
Expand All @@ -93,20 +92,17 @@ raven-wps has been installed with ``make develop``::
# to test a single notebook (note the .run at the end of the notebook path)
$ make docs/source/notebooks/Subset_climate_data_over_watershed.ipynb.run

The notebooks may also require other WPS services (Finch and Flyingpigeon). By
default these are from the production server but we can point the notebooks to
local servers if needed for development purposes::
The notebooks may also require other WPS services (``Finch``). By default these are from the production server but we can point the notebooks to local servers if needed for development purposes::

# to test all notebooks
$ make FLYINGPIGEON_WPS_URL=http://localhost:8093 FINCH_WPS_URL=http://localhost:5000 test-notebooks
$ make FINCH_WPS_URL=http://localhost:5000 test-notebooks

Or::

# to test a single notebook (note the .run at the end of the notebook path)
$ make FLYINGPIGEON_WPS_URL=http://localhost:8093 FINCH_WPS_URL=http://localhost:5000 docs/source/notebooks/Subset_climate_data_over_watershed.ipynb.run
$ make FINCH_WPS_URL=http://localhost:5000 docs/source/notebooks/Subset_climate_data_over_watershed.ipynb.run

If instead we want to run the notebooks against the production raven-wps server
or any other raven-wps servers::
If instead we want to run the notebooks against the production raven-wps server or any other raven-wps servers::

# to test all notebooks
$ make WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps test-notebooks
Expand All @@ -116,23 +112,21 @@ Or::
# to test juste 1 notebook (note the .run at the end of the notebook path)
$ make WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps docs/source/notebooks/Subset_climate_data_over_watershed.ipynb.run

We can also override all three of the server variables (WPS_URL, FINCH_WPS_URL,
FLYINGPIGEON_WPS_URL) to pick and choose any servers/services from anywhere we want.
We can also override all three of the server variables (``WPS_URL``, ``FINCH_WPS_URL``) to pick and choose any servers/services from anywhere we want.


Starting local Jupyter server to edit/develop notebooks
-------------------------------------------------------

Assuming that the ``raven`` conda env has already been created and is up-to-date and
raven-wps has been installed with ``make develop``::
Assuming that the ``raven`` conda env has already been created and is up-to-date and ``raven-wps`` has been installed with ``make develop``::

# start local raven-wps server to test against
$ make start # remember to make stop once done

# to start local jupyter notebook server listing all current notebooks
$ make notebook # Control-C to terminate once done

# Can also override all three WPS_URL, FINCH_WPS_URL and FLYINGPIGEON_WPS_URL here as well,
# Can also override WPS_URL and FINCH_WPS_URL here as well,
# just like 'make test-notebooks' to be able to pick and choose any servers anywhere we want.

# By overriding these variables at the 'make notebook' step, we will not need to
Expand All @@ -143,12 +137,9 @@ raven-wps has been installed with ``make develop``::
Bulk refresh all notebooks output
---------------------------------

This automated refresh only works for notebooks that passed ``make
test-notebooks`` above. For those that failed, manually starting a local
Jupyter server and refresh them manually.
This automated refresh only works for notebooks that passed ``make test-notebooks`` above. For those that failed, manually starting a local Jupyter server and refresh them manually.

Assuming that the ``raven`` conda env has already been created and is up-to-date and
raven-wps has been installed with ``make develop``::
Assuming that the ``raven`` conda env has already been created and is up-to-date and ``raven-wps`` has been installed with ``make develop``::

# start local raven-wps server to test against
$ make start # remember to make stop once done
Expand All @@ -161,7 +152,7 @@ Or::
# to refresh a single notebook (note the .refresh at the end of the notebook path)
$ make docs/source/notebooks/Assess_probabilistic_flood_risk.ipynb.refresh

# Can also override all three of the server variables (WPS_URL, FINCH_WPS_URL and FLYINGPIGEON_WPS_URL) here as well,
# Can also override the server variables (WPS_URL, FINCH_WPS_URL) here as well,
# just like 'make test-notebooks' to be able to pick and choose any servers/services from anywhere we want.


Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ docs = [
raven-wps = "raven.cli:cli"

[project.urls]
"Homepage" = "https://pavics-raven.readthedocs.io/"
"Homepage" = "https://pavics-sdi.readthedocs.io/projects/raven/en/latest/"
"Source" = "https://github.com/Ouranosinc/raven/"
"Changelog" = "https://pavics-sdi.readthedocs.io/projects/raven/en/latest/changes.html"
"Issue tracker" = "https://github.com/Ouranosinc/raven/issues"
Expand Down
Loading