Merge branch 'file-upload-wizard' of https://github.com/rija/gigadb-w…

…ebsite into rija-file-upload-wizard

This commit merges pull request File upload wizard $346:

1. [gigascience#146: User stories, GitHub issues,  Acceptance tests and dependencies] (rija#146)
1. [gigascience#147: Prototype of client/server file upload and software architecture blueprint] (rija#147)
1. [gigascience#148: Setup upload server and ftp servers in Docker Compose and for CI/CD] (rija#148)
1. [gigascience#151: Build admin function to create restricted access: filesystem directories & security token ] (rija#151)
1. [gigascience#152: Build secure and reliable  upload server based on validated prototype] (rija#152)
1. [gigascience#153: Add remaining time estimate for file uploads] (rija#163)
1. [gigascience#165: Add FTP server interface to the Drop-Off area] (rija#165)

This pull request has the following functionalities:

File Upload Wizard FileDrop administration API and a prototype using it:
* Create or Delete a FileDrop account and filesystem for a specific dataset through a REST API.
* JSon Web Token based authentication for accessing the REST API
* The FileDrop account's filesystem can be interacted with through a TUSd file upload server and an FTP server
* A watcher (based on Linux's inotify) service moves files around the filesystem from the drop-off area to a download area (where ftp links used in mockup page are built from)
* The initial prototype has been updated to use the API and to be optionally deployable from within the project
* the File Upload Wizard is a standalone, full-stack web application built out of the [Yii 2 Advanced template](https://www.yiiframework.com/extension/yiisoft/yii2-app-advanced/doc/guide/2.0/en/start-installation) but still accessed through GigaDB website's URL space and sharing the same Database server and code repository
* Nginx setup was updated to proxy multiple web apps and other services (Tusd). See ``ops/configuration/nginx-conf/sites``
* Nginx setup was updated to make it easy to disable/enable specific webapps (used for feature flagging the prototype). See ``ops/configuration/nginx-conf/enable_sites``
* Significant changes to the CI/CD scripts for deploying multiple webapps and for improved operational safety
* Added [TUSd](https://tusd.io) service as container
* Added [Pure-FTPD](https://pureftpd.org) service as container
* Added a [docker-inotify-command](https://github.com/coppit/docker-inotify-command) service as container

This PR is meant to lay down infrastructure changes needed to implement File Upload Wizard, so no visible business features are delivered here but the prototype was updated to use the API and to be deployable at ``/proto/``.

Both the [GoogleDocs spreadsheet for tasks](https://docs.google.com/spreadsheets/d/1bogVGdvchB790TpMQDN3LL_BvolXWoCni76k1Qi7OiA/edit?pli=1#gid=0) and my [Git project board](https://github.com/users/rija/projects/1) should be up-to-date.

No change to **gigadb** database, but a new database **fuwdb** to support File Upload Wizard web application's functions.

**fuwdb**'s schema is almost entirely built out of Yii2 migrations in ``fuw/app/console/migrations``.

Only the **user** table is created by the Yii2 Advanced Template's [Composer script](https://www.yiiframework.com/extension/yiisoft/yii2-app-advanced/doc/guide/2.0/en/start-installation#installing-using-composer).

No change to GigaDb model classes.
New model layer in the File Upload Wizard webapps divided in 3 categories:
* Backend models (``fuw/app/backend/models/FiledropAccount.php`` and ``fuw/app/backend/models/DockerManager.php``): admin functions, used for admin management of Filedrop accounts
* Frontend models: user facing for future integration between GigaDB UX and FUW's REST API, currently just scaffolded code from Yii2 Advanced template
* Common models: model and config used by backend and frontend. ``fuw/app/common/models/User.php`` is used for the JSon web token (JWT) based authentication to the REST API.

No change to GigaDB controller classes.
There is a new REST controller in File Upload Wizard for creating and deleting filedrop accounts ``fuw/app/backend/controllers/FiledropAccountController.php``.
That controller that corresponds to the ``FiledropAccount.php`` model was automatically scaffoled using `gii`.

However, I've added a custom ``fuw/app/backend/actions/FiledropAccountController/DeleteAction.php`` action so that DELETE command marks the model with a special status instead of deleting it.

N/A

N/A

The test setup for GigaDB was updated to fix inconsistent and fragile approach to loading and restoring database data and to work in context of multi-webapp testing. ([relevant commit](rija@f84d739#diff-ddbd905c83513d6eac66f30e9a493068))

Also Behat acceptance tests now use profiles instead of tags for configuration. See ``behat.yml``
So acceptance are run with a profile as argument. e.g:
```
$ docker-compose run --rm test bin/behat --profile local -v --stop-on-failure
```

The 'cli' profile is for Gitlab CI environment.

The File Upload Wizard has unit tests and functional tests. There's no business functionality completed yet so no acceptance tests has been added yet.

The tests are located in:
* ``fuw/app/backend/tests/``
* ``fuw/app/common/tests/``
* ``fuw/app/frontend/tests/``

[Codeception](https://github.com/Codeception/Codeception) is setup in File Upload Wizard directory as a test runner and test framework for unit, functional and acceptance tests (and it can support Behat features). It is integrated, extensible, up-to-date and maintained and work with the most recent and future version of underlying technology (PHP, PHPunit, Yii2, Selenium).

The config files relevant to Codeception are:
* ``fuw/app/backend/codeception.yml``
* ``fuw/app/common/codeception.yml``
* ``fuw/app/frontend/codeception.yml``
* ``fuw/app/codeception.yml``

More info on testing is included in ``docs/fuw/developer_guide.md``.

The overall flow and technologies haven't changed. The following changes have been made to accomodate CI and CD of a multi-webapps project or to fix structural weakness in the CI/CD process:
* Build manifests (Dockerfile and Production-Dockerfile) should be part of each webapp directory space, but service composition (docker-compose.\*.yml), appliance provisioning (terraform) and system configuraiton (ansible) should stay centralized
* It is dangerous to only have one Terraform state for all environments as a provisioning mistake can take down the wrong environment (e.g: production), so now there is a distinct Terraform state for each environment represented by a separate environment directory in ``ops/infrastructure/envs``
* Previously, provisioning a new environment required duplicating Terraform and Ansible code. Now they are modularized with the bulk of the code (Ansible roles and Terraform modules) stored outside the environment directory because they are environment-agnostic.
* The Ansible playbook and variables and Terraform main script and variables are in the environment-specific directory in ``ops/infrastructure/envs``

Furthermore, the provisioning for GigaDB has been upgraded to Debian 9 Stretch and the database server upgraded to Postgresql 9.6.

Finally, there are updates to Gitlab CI config ``.gitlab-ci.yml``:
* distinct stages to deploy on staging for GigaDB only (**deploy_gigadb**), GigaDB and File Upload Wizard API (**deploy_apps**), or GigaDB with File Upload Wizard API and the prototype (**deploy_proto**)
* It is no longer necessary to have a separate stage for initial deployment of Let's Encrypt certificate.

* The CI/CD docs have been updated in light of changes in provisioning
* There's a new ``docs/fuw`` directory with documentation relating to File Upload Wizard (with the quickstart guide in ``docs/fuw/index.md``)
* I now use [mkdocs](https://www.mkdocs.org) to create a doc server and to build a doc web site easily (configured with ``mkdocs.yml`` at the root of the project)
* There's a ``docs/index.md`` with instructions for using mkdocs and to serve as index page for doc server/website.
* updated ``README.md`` and ``docs/INSTALL.md`` to reflect the change in starting and testing the GigaDB webapp

Bind-mounting of Docker Daemon unix socket inside container is a security vulnerability, so to allow inter-container communication, File Upload Wizard use the ``Docker-PHP`` php library that uses the Docker Daemon API instead. See ``DockerManager.php`` for the current implementation.
(I haven't changed yet the GigaDB acceptance tests that still use the unix socket approach).
However on the Mac, the current version of Docker For Desktop doesn't expose the API on a TCP port, so when working on File Upload Wizard, we need to manually expose the API on a TCP port during development. See ``docs/fuw/index.md``

.gitignore

@@ -1,4 +1,4 @@
-index.php
+/index.php
 logs
 assets
 giga_cache
@@ -46,7 +46,7 @@ phpcs.xml
 bin
 vendor
 composer.lock
-composer.json
+/composer.json
 
 # deployment
 .env
@@ -66,3 +66,4 @@ vault
 # docs
 docs/apidocs/phpdoc-cache-*
 docs/cache
+site/

README.md

@@ -39,6 +39,7 @@ $ cd gigadb-website                         # your cloned git repository for Gig
 $ git checkout develop                      # the branch with the latest code base
 $ cp ops/configuration/variables/env-sample .env    # make sure GITLAB_PRIVATE_TOKEN is set to your personal access token
 $ docker-compose run --rm config            # generate the configuration using variables in .env, GitLab, then exit
+$ docker-compose run --rm less				# generate site.css
 ```
 
 >**Note 1**: A `.secrets` file will be created automatically and populated using 
@@ -52,16 +53,21 @@ you will have to provide your own values for the necessary variables using
 >$ vi .secrets
 >```
 
-**(2)** To start the web application, run the following command:
+**(2)** To start the web application, run the following commands:
+
 ```
-$ docker-compose run --rm webapp            # run composer update, then spin up the web application's services, then exit
+$ docker-compose run --rm gigadb                    # Run composer update, then spin up the web application's services, then exit
+$ docker-compose up -d web 							# Start the web server
 ```
 
-The **webapp** container will run composer update using the `composer.json` 
-generated in the previous step, and will launch three containers named **web**, 
+The **gigadb** container will run composer update using the `composer.json` 
+generated in the previous step, and will launch two containers named
 **application** and **database**, then it will exit. It's ok to run the command 
 repeatedly.
 
+Starting the web container will first enable site configuration connecting nginx to gigadb PHP application server before starting the web server as a deamon.
+
+
 **(3)** Upon success, three services will be started in detached mode.
 
 You can then navigate to the website at:
@@ -87,24 +93,43 @@ database schema, you will need to run Yii migration as below:
 ```
 $ docker-compose run --rm  application ./protected/yiic migrate --interactive=0
 ```
-## Testing
+## Testing GigaDB webapp
+
+To run the unit tests:
+```
+$ docker-compose run --rm test ./bin/phpunit --testsuite unit --bootstrap protected/tests/unit_bootstrap.php --verbose --configuration protected/tests/phpunit.xml --no-coverage
+```
+
+To run the functional tests:
+
+```
+$ docker-compose run --rm test ./bin/phpunit --testsuite functional --bootstrap protected/tests/functional_custom_bootstrap.php --verbose --configuration protected/tests/phpunit.xml --no-coverage
+```
+
+There is a bash shortcut available to run both unit tests and functional tests for GigaDB:
+```
+$ ./tests/unit_functional
+```
+
+To run the acceptance tests:
 
-To run the tests:
 ```
-$ docker-compose run --rm test
+$ docker-compose up -d phantomjs
+$ docker-compose run --rm test bin/behat --profile local -v --stop-on-failure
 ```
 
-This will run all the tests and generate a test coverage report. An headless 
+
 Selenium web browser (currently PhantomJS) will be automatically spun-off into 
 its own container. If an acceptance test fails, it will leave a screenshot under 
 the `./tmp` directory.
 
-To only run unit tests, use the command:
+To run test coverage:
+
 ```
-$ docker-compose run --rm test ./tests/unit_functional
+$ docker-compose run --rm test ./bin/phpunit /var/www/protected/tests --testsuite all --bootstrap protected/tests/functional_custom_bootstrap.php --verbose --configuration protected/tests/phpunit.xml
 ```
 
-## Troubleshooting
+## Troubleshooting (for GigaDB)
 
 To access the services logs, use the command below:
 ```
@@ -230,6 +255,29 @@ $ docker-compose pull
 
 ## Generating the documentation
 
+Install mkdocs. On mac you can use brew:
+
+```
+$ brew install mkdocs
+```
+
+Otherwise you can use Python pip:
+
+```
+pip install mkdocs
+```
+
+To start the server, from this project root directory, run the command:
+
+```
+$ mkdocs serve
+```
+
+the documentation will be available at: (http://127.0.0.1:8000)
+
+
+### PHPDocs
+
 To update the browsable API Docs (PHPDoc), run the command below and then commit 
 the changes:
 ```
@@ -238,4 +286,4 @@ $ docker-compose run --rm test ./docs/make_phpdoc
 
 ## Licensing
 
-Please see the file called [LICENSE](./LICENSE).
+Please see the file called [LICENSE](./LICENSE).

behat.yml

@@ -20,3 +20,14 @@ default:
           selenium2:
             wd_host: http://phantomjs:8910/wd/hub
 
+local:
+  suites:
+    default:
+      filters:
+        tags: "@ok&&~@affiliate-login"
+
+cli:
+  suites:
+    default:
+      filters:
+        tags: "@ok&&~@affiliate-login&&~@javascript"

docker-pure-ftpd/Dockerfile

@@ -0,0 +1,8 @@
+FROM stilliard/pure-ftpd:hardened
+
+# setup group for uploader and downloader
+RUN useradd -g ftpgroup -d /home/uploader -s /dev/null uploader && \
+	useradd -g ftpgroup -d /home/downloader -s /dev/null downloader
+
+ENTRYPOINT ["/run.sh"]
+CMD ["-l", "puredb:/etc/pure-ftpd/pureftpd.pdb", "-E -j -R", "-P", "$PUBLICHOST"]

docs/INSTALL.md

@@ -121,7 +121,7 @@ Resolving deltas: 100% (516/516), done.
 Checking connectivity... done.
 ```
 
-## Getting started in 3 steps
+## Getting started with GigaDB website in 3 steps
 
 **(1)** To setup the web application locally, do the following:
 ```
@@ -130,6 +130,7 @@ $ git checkout develop                              # Currently the only branch
 $ cp ops/configuration/variables/env-sample .env    # Make sure GITLAB_PRIVATE_TOKEN is set to your personal access token and GIGADB_ENV=dev
 # Check .env file to see if the correct GROUP_VARIABLES_URL and PROJECT_VARIABLES_URL are used!!!
 $ docker-compose run --rm config                    # Generate the configuration using variables in .env, GitLab, then exit
+$ docker-compose run --rm less				# generate site.css
 ```
 
 >**Note 1**:
@@ -146,19 +147,24 @@ to provide your own values for the necessary variables using
 >$ vi .secrets
 >```
 
-**(2)** To start the web application, run the following command:
+**(2)** To start the web application, run the following commands:
+
 ```
-$ docker-compose run --rm webapp                    # Run composer update, then spin up the web application's services, then exit
+$ docker-compose run --rm gigadb                    # Run composer update, then spin up the web application's services, then exit
+$ docker-compose up -d web 							# Start the web server
 ```
 
-The **webapp** container will run composer update using the `composer.json` 
-generated in the previous step, and will launch three containers named **web**, 
+The **gigadb** container will run composer update using the `composer.json` 
+generated in the previous step, and will launch two containers named
 **application** and **database**, then it will exit. It's ok to run the command 
 repeatedly.
 
+Starting the web container will first enable site configuration connecting nginx to gigadb PHP application server before starting the web server as a deamon.
+
 **(3)** Upon success, three services will be started in detached mode.
 
 You can then navigate to the website at:
+
 * [http://gigadb.gigasciencejournal.com:9170](http://gigadb.gigasciencejournal.com:9170/)
 
 >**Note**:
@@ -191,21 +197,40 @@ database schema, you will need to run Yii migration as below:
 ```
 $ docker-compose run --rm  application ./protected/yiic migrate --interactive=0
 ```
-## Testing
+## Testing GigaDB webapp
+
+To run the unit tests:
+```
+$ docker-compose run --rm test ./bin/phpunit --testsuite unit --bootstrap protected/tests/unit_bootstrap.php --verbose --configuration protected/tests/phpunit.xml --no-coverage
+```
+
+To run the functional tests:
+
+```
+$ docker-compose run --rm test ./bin/phpunit --testsuite functional --bootstrap protected/tests/functional_custom_bootstrap.php --verbose --configuration protected/tests/phpunit.xml --no-coverage
+```
+
+There is a bash shortcut available to run both unit tests and functional tests for GigaDB:
+```
+$ ./tests/unit_functional
+```
+
+To run the acceptance tests:
 
-To run the tests:
 ```
-$ docker-compose run --rm test
+$ docker-compose up -d phantomjs
+$ docker-compose run --rm test bin/behat --profile local -v --stop-on-failure
 ```
 
-This will run all the tests and generate a test coverage report. An headless 
+
 Selenium web browser (currently PhantomJS) will be automatically spun-off into 
 its own container. If an acceptance test fails, it will leave a screenshot under 
 the `./tmp` directory.
 
-To only run unit tests, use the command:
+To run test coverage:
+
 ```
-$ docker-compose run --rm test ./tests/unit_functional
+$ docker-compose run --rm test ./bin/phpunit /var/www/protected/tests --testsuite all --bootstrap protected/tests/functional_custom_bootstrap.php --verbose --configuration protected/tests/phpunit.xml
 ```
 
 ## Troubleshooting