Restructure for clean installations and centralize configuration (#5)

* add an email so at least example is complete

* Add a redirect: /node-red ==> /node-red/

* Add wrappers for cleaner dashboard installation

* Get things working. Now all configuration is centralized in .env (but still .env and setup are tricky)

* Fix typos in SETUP.md

* Ignore the .env file

* Improve README.md

* Rough touch-up of SETUP.md

* Rough touch-up for TODO.md

.gitignore

@@ -1,2 +1,2 @@
-node-red/data
 *.swp
+.env

README.md

@@ -43,10 +43,10 @@ This can be visualized as below:
 ## Assumptions
 
 * Your host system must have docker-compose 1.9 or later (for which see https://github.com/docker-compose -- be aware that apt-get normally doesn't grab this; if configured at all, it frequently gets an out-of-date version).
-* The environment variable `TTN_DASHBOARD_DATA`, if set, points to the common directory for your data.
-* `${TTN_DASHBOARD_DATA}/var/lib/node-red` will have your local Node-RED data.
-* `${TTN_DASHBOARD_DATA}/var/lib/influxdb` will have your local influxdb data (this is what you should back up)
-* `${TTN_DASHBOARD_DATA}/var/lib/grafana` will have your dashboards
+* The environment variable `TTN_DASHBOARD_DATA`, if set, points to the common directory for your data. If not set, docker-compose will quit at startup. (This is by design!) 
+   * `${TTN_DASHBOARD_DATA}node-red` will have your local Node-RED data.
+   * `${TTN_DASHBOARD_DATA}influxdb` will have your local influxdb data (this is what you should back up)
+   * `${TTN_DASHBOARD_DATA}grafana` will have your dashboards
 
 ## Composition and External Ports
 
@@ -64,16 +64,36 @@ Remember, if your server is running on a cloud platform like Microsoft Azure or
 
 1. Make sure your server has `git`, `docker` and `docker-compose` installed, with the versions mentioned above under **Assumptions**.
 2. Use `git clone` to copy this repository to your host.
-3. Define root URL and passwords in `grafana/.env` and `influxdb/.env`
-4. Optional, but important for consistency:
-  `% export TTN_DASHBOARD_DATA=/full/path/to/directory`
-  This will put all the data file for this instance as subdirectories of the specified path. If you leave this underfined, `docker-compose` will print error messages and then put your data in `/var/lib/node-red`, etc.
+3. Create a `.env` file in your repository with the following settings:
+   1. `TTN_DASHBOARD_DATA=/full/path/to/directory/` 
+   The trailing slash is required!
+   This will put all the data file for this instance as subdirectories of the specified path. If you leave this underfined, `docker-compose` will print error messages and quit.
+   2. `TTN_DASHBOARD_APACHE_FQDN=myhost.example.com` 
+   This sets the name of your resulting server. It tells Apache what it's serving out.  It must be a fully-qualified domain name (FQDN) that resolves to the IP address of the container host.
+   3. `TTN_DASHBOARD_CERTBOT_FQDN=myhost.example.com`
+   This should be the same as `TTN_DASHBOARD_APACHE_FQDN`.
+   4. `TTN_DASHBOARD_CERTBOT_EMAIL=someone@example.com`
+   This sets the contact email for Let's Encyrypt. The script automatically accepts the Let's Encrypt terms of service, and this indicates who is doing the accepting.
+   5. `TTN_DASHBOARD_GRAFANA_ADMIN_PASSWORD=SomethingVerySecretIndeed`
+   This sets the *initial* password for the Grafana `admin` login. You should change this via the Grafana UI after booting the server.
+   6. `TTN_DASHBOARD_INFLUXDB_INITIAL_DATABASE_NAME=demo`
+   Change "demo" to the desired name of the initial database that will be created in InfluxDB. 
+Your `.env` file should look like this:
+   ```sh
+    # setup for this docker-ttn-dashboard instance, used by docker-compose
+    TTN_DASHBOARD_DATA=/full/path/to/directory/
+    TTN_DASHBOARD_APACHE_FQDN=myhost.example.com
+    TTN_DASHBOARD_CERTBOT_FQDN=myhost.example.com
+    TTN_DASHBOARD_GRAFANA_ADMIN_PASSWORD=SomethingVerySecretIndeed
+    TTN_DASHBOARD_INFLUXDB_INITIAL_DATABASE_NAME=demo
+    ### end of file ###
+   ```
 5. `% docker-compose build`
    * If this fails with the message, `ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?`, then probably your user ID is not in the `docker` group. To fix this, `sudo adduser MYUSER docker`, where "MYUSER" is your login ID. Then (**very important**) log out and log back in.
-6. `% docker-compose up`
-   * If this fails (for example, Node-RED dies with a "killed" status), confirm taht you have a directory named `/var/lib/node-red`, that it's owned by root, and that it has typical directory permissions.
-6. Open Node-RED on **https://machine.example.net/node-red/**,  and build a flow that stores data in InfluxDB
-7. Open Grafana on **https://machine.example.net**, and build a dashboard that retrieves data from InfluxDB
+6. `% docker-compose up` or `% docker-comose up -d`
+   * If this fails (for example, Node-RED dies with a "killed" status), check your `.env` file.
+7. Open Grafana on **https://myhost.example.com**, and log in as admin. Change your password. Set up your first data source using InfluxDB. See below for more detailed instructions.
+8. Open Node-RED on **https://myhost.example.com/node-red/**,  and build a flow that stores data in InfluxDB.  **Be sure to add the trailing slash! Otherwise you'll get a 404 from Grafana. We'll fix this soon.**
 
 ## Data Files
 
@@ -83,22 +103,22 @@ Data files are kept in the following locations by default.
 
 Component | Data file location on host | Location in container
 ----------|----------------------------|----------------------
-Node-RED | `${TTN_DASHBOARD_DATA}/var/lib/node-red` | `/data`
-InfluxDB | `${TTN_DASHBOARD_DATA}/var/lib/influxdb`| `/data`
-Grafana | `${TTN_DASHBOARD_DATA}/var/lib/grafana`| `/var/lib/grafana`
+Node-RED | `${TTN_DASHBOARD_DATA}node-red` | `/data`
+InfluxDB | `${TTN_DASHBOARD_DATA}influxdb`| `/data`
+Grafana | `${TTN_DASHBOARD_DATA}grafana`| `/var/lib/grafana`
 
-As shown, you can quickly override the default locations on the **host** (e.g. for testing). You do this by setting the environment variable `TTN_DASHBOARD_DATA` to the **absolute path** to the containing directory prior to calling `docker-compose up`. The above paths are appended to the value of `TTN_DASHBOARD_DATA`. Directories are created as needed. Consider the following example:
+As shown, you can easily change locations on the **host** (e.g. for testing). You do this by setting the environment variable `TTN_DASHBOARD_DATA` to the **absolute path** to the containing directory prior to calling `docker-compose up`. The above paths are appended to the value of `TTN_DASHBOARD_DATA`. Directories are created as needed. Consider the following example:
 ```bash
-% export TTN_DASHBOARD_DATA=/dashboard-data
+% export TTN_DASHBOARD_DATA=/dashboard-data/
 % docker-compose up -d
 ```
 In this case, the data files are created in the following locations:
 
 Component | Data file location
 ----------|-------------------
-Node-RED | `/dashboard-data/var/lib/node-red`
-InfluxDB | `/dashboard-data/var/lib/influxdb`
-Grafana | `/dashboard-data/var/lib/grafana`
+Node-RED | `/dashboard-data/node-red`
+InfluxDB | `/dashboard-data/influxdb`
+Grafana | `/dashboard-data/grafana`
 
 ### Reuse and removal of data files
 Since data files on the host are not removed between runs, as long as you
@@ -107,9 +127,9 @@ don't remove the files between runs, your data will preserved.
 Sometimes this is inconvienient, and you'll want to remove some or all of the
 data. For a variety of reasons, the data files and directories are created owned by root, so you must use the `sudo` command to remove the data files. Here's an example of how to do it:
 ```bash
-% sudo rm -rf ${TTN_DASHBOARD_DATA}/var/lib/node-red
-% sudo rm -rf ${TTN_DASHBOARD_DATA}/var/lib/influxdb
-% sudo rm -rf ${TTN_DASHBOARD_DATA}/var/lib/grafana
+% sudo rm -rf ${TTN_DASHBOARD_DATA}node-red
+% sudo rm -rf ${TTN_DASHBOARD_DATA}influxdb
+% sudo rm -rf ${TTN_DASHBOARD_DATA}grafana
 ```
 
 ## Node-RED and Grafana Examples
@@ -121,7 +141,7 @@ This version requires that you set up Node-RED, the database and the grafana das
 There is one point that is somewhat confusing about the connections from Node-RED and Grafana to InfluxDB. Even though InfluxDB is running on the same host, it is logically running on its own virtual machine (created by docker). Because of this, Node-RED and Grafana cannot use **localhost** when connecting to Grafana. A special name is provided by docker: **influxdb**.  Note that there's no DNS suffix.  If you don't use **influxdb**, Node-RED and Grafana will not be able to connect.
 
 ## Logging in to Grafana
-* On the login screen, the user name is "admin". The password is given by the value of the variable `GF_SECURITY_ADMIN_PASSWORD` in `grafana/.env`. Note that if you change the password in `grafana/.env` after the first time you launch the grafana containder, the admin password does not change. If you somehow lose the previous value of the admin password, it's very hard to recover; easiest is to remove `grafana.db` and start over.
+* On the login screen, the user name is "admin". The initial password is given by the value of the variable `GF_SECURITY_ADMIN_PASSWORD` in `grafana/.env`. Note that if you change the password in `grafana/.env` after the first time you launch the grafana containder, the admin password does not change. If you somehow lose the previous value of the admin password, and you don't have another admin login, it's very hard to recover; easiest is to remove `grafana.db` and start over.
 
 ### Data source settings in Grafana
 * Set the URL (under Http Settings) to `http://influxdb:8086`.
@@ -136,7 +156,6 @@ Although the dashboard is already very useful, it's incomplete.  Please refer to
 2. admin script to show roles and maintain the htpasswd 
 3. add the auto-update cron script -- right now you have to restart in order to get the SSL certs updated. Not a big deal, as the patches-requiring-reboot interval is shorter than the life of the certs, but still, this should be fixed.
 4. Switch to [phusion](https://github.com/phusion/baseimage-docker) for the base image, instead of ubuntu.
-5. add a safety check, so that a manual launch without running the configuration script will cause things to display a useful message and shut down. At that point, we can make `TTN_DASHBOARD_DATA` mandatory, and set it to `/var/lib` by default.
 6. providue suitable intial files for Grafana and NodeRed, assuming Catena-4450 sensor node.
 7. the intial script should prompt for the data base name.
 

SETUP.md

@@ -1,48 +1,50 @@
-1. in `./apache/certbot-config.sh`, change the FQDN to the FQDN of this server.
+1. Make sure that you have the following tools installed on the host server.
 
-2. in `./apache/certbot-config.sh`, add CERTBOT\_EMAIL with the right email address for use with certbot certificate requests.
+   Tool|On Ubuntu|On CentOS 7
+   ----|---------|-----------
+   `htpasswd`|`sudo apt-get install apache2-utils`|`sudo yum install httpd-tools`
 
-3. Be aware that the grafana env admin\_password doesn't take unless it's correct **on the
+2. get a fully-qualified domain name for your server, for which you control DNS. Call this the "FQDN" (fully-qualified domain name).
+
+3. Createa a `.env` file as instructed in README.md. 
+
+4. Be aware that the grafana env admin\_password is ignored **except on the
 first boot**.
 
-4. get a fully-qualified domain name for your server, for which you control DNS. Call this the "FQDN" (fully-qualified domain name).
+5. Follow the instructions from README.md to get grafana working and to get the server up.
 
-5. in `./apache/proxy-*.sh`, change all the FQDNs to the FQDN of this server.
+6. verify that grafana is working at https://{FQDN}/ and https://{FQDN}/grafana
 
-6. using `docker-compose apache run /bin/bash`, 
+7. using `docker-compose apache run /bin/bash`, 
 
-   1. add {APACHE}/etc/apache2/authdata as user www-data
+   1. add Apache's /etc/apache2/authdata as user www-data
    ```sh
 	mkdir /etc/apache2/authdata
 	chown www-data /etc/apache2/authdata
    ```
-   2. add {APACHE}/etc/apache2/authdata/.htpasswd as user www-data
+   2. add Apache's /etc/apache2/authdata/.htpasswd as user www-data
    ```sh
 	touch /etc/apache2/authdata/.htpasswd
-	chown www-data /etc/apache2/.htpasswd
+	chown www-data /etc/apache2/authdata/.htpasswd
    ```
    3. Add user logins for influxdb, queries, node-red. Make `USERS` be a list of login IDs.
    ```sh
+   	export USERS="tmm amy josh"
 	for each USER in $USERS ; do 
 		htpasswd /etc/apache2/authdata/.htpasswd $USER
    >>>>enter password twice
 	done
    ```
-   4. add {APACHE}/etc/apache2/authdata/.htgroup (owned by www-data)
+   4. add Apache's /etc/apache2/authdata/.htgroup (owned by www-data)
    ```sh
+   	# this assumes USERS is still set from previous step.
 	touch /etc/apache2/authdata/.htgroup
 	chown www-data /etc/apache2/authdata/.htgroup
 	echo "node-red: ${USERS}" >>/etc/apache2/authdata/.htgroup
 	echo "admin: ${USERS}" >>/etc/apache2/authdata/.htgroup
-	echo "group: ${USERS}" >>/etc/apache2/authdata/.htgroup
+	echo "query: ${USERS}" >>/etc/apache2/authdata/.htgroup
    ```
 
-7. verify that grafana is working at https://{FQDN}/ and https://{FQDN}/grafana
-
-8. verify that you can log in as https://{FQDN}/node-red/ and
-https://{FQDN}/influxdb/
-
-9. In influxdb UI, change the query URL to https://{FQDN}/influxdb, [x] SSL,
-don't fill in user name here, just press save. Browser will ask for credentials; provide credentials.
-
+8. verify that you can log in as https://{FQDN}/node-red/. 
 
+9. Current versions of influxdb may support an administrative interface at https://{FQDN}/influxdb/. If so, in the influxdb UI, change the query URL to https://{FQDN}/influxdb, [x] SSL. Don't fill in user name here, just press save. Browser will ask for credentials; provide credentials.

TODO.md

@@ -1,21 +1,17 @@
 1. Convert to [phusion](https://github.com/phusion/baseimage-docker); documents are [here](http://phusion.github.io/baseimage-docker/)
 
-2. Prepare a script that queries the user during the setup and sets the grafana admin user name and initial password (seeding grafana/.env, or overriding it). Also set name of influxdb database (change from demo).  And change all the FQDNs.And change the email address in certbot-config.sh.
+2. Prepare a script that queries the user during the setup and sets the `.env` file.
 
-3. don't forget to exclude grafana/.env from the git repo, so there are no passwords at all.
-
-4. The grafana instance had better be customized to remove the admin password (or have a reset step) so if the user changes GRAFANA\_ENV\_ADMIN\_PASSWORD after the image has been launched once, it wil be reset. This might be a maintenance script and/or a makefile so that the system detects edits and does the right thing.
+4. Figure out what to do if the user changes GRAFANA\_ENV\_ADMIN\_PASSWORD after the image has been launched once, it wil be reset. This might be a maintenance script and/or a makefile so that the system detects edits and does the right thing.
 
 4. the script should also get names and roles for access to node-red and influxdb. It then will seed .hgaccess and .htgroup.
 
 5. same script should be able to show user-by-user roles, and adjust them.
 
 6. Add the auto-update cron script.
 
-7. See if there's a way to make the docker-compose print a message and stop if the configuration operation hasn't been done.
-
-8. integrate the other things from SETUP.txt
+7. integrate the other things from SETUP.txt
 
-9. Add scripts to backup and restore the user's data directories. Backup should run offline (unless there's a very good way to backup the datasets from all the servers while they're up). restore must run offline. Scripts should do the necessary to ensure that the servers are in fact stopped.
+8. Add scripts to backup and restore the user's data directories. Backup should run offline (unless there's a very good way to backup the datasets from all the servers while they're up). restore must run offline. Scripts should do the necessary to ensure that the servers are in fact stopped.
 
 10. update the README.md (again)

apache/Dockerfile

@@ -21,7 +21,7 @@ RUN /usr/sbin/a2enmod proxy && \
     /usr/sbin/a2enmod authz_user authz_groupfile proxy_wstunnel
 
 # RUN mkdir -p /root
-COPY setup.sh certbot-config.sh proxy-*.conf /root/
+COPY setup.sh proxy-*.conf /root/
 
 #
 # we need to run cron (for letsencrypt update) and apache. Try doing it

apache/proxy-grafana.conf

@@ -1,6 +1,6 @@
-	Redirect 301 / https://mcci-ttn-demo.mcci.com/grafana/
-	Redirect 301 /index.html https://mcci-ttn-demo.mcci.com/grafana/
-        Redirect 301 /grafana https://mcci-ttn-demo.mcci.com/grafana/
+	Redirect 301 / https://@{FQDN}/grafana/
+	Redirect 301 /index.html https://@{FQDN}/grafana/
+        Redirect 301 /grafana https://@{FQDN}/grafana/
         <Location "/grafana/" >
                 ProxyPass http://grafana:3000/
                 ProxyPassReverse http://grafana:3000/

apache/proxy-influxdb.conf

@@ -1,4 +1,4 @@
-        Redirect 301 /influxdb https://mcci-ttn-demo.mcci.com/influxdb/
+        Redirect 301 /influxdb https://@{FQDN}/influxdb/
         <Location "/influxdb/" >
 		AuthType Basic
 		AuthName "InfluxDB admin"

apache/proxy-nodered.conf

@@ -5,7 +5,8 @@
         ProxyPass /node-red/ http://node-red:1880/
         ProxyPassReverse /node-red/ http://node-red:1880/
 
-	<Location "/node-red">
+	Redirect 301 /node-red https://@{FQDN}/node-red/
+	<Location "/node-red/">
                 AuthType Basic
                 AuthName "Node-RED"
                 AuthUserFile /etc/apache2/authdata/.htpasswd

apache/setup.sh

@@ -9,32 +9,54 @@ cron || exit 1
 
 # test that we have a proper setup.
 cd $HOME || exit 2
-if [ -f certbot-config.sh ]; then
-	source certbot-config.sh
+
+# test that authentication is set up, and set permissions as needed by us
+if [ ! -d /etc/apache2/authdata ] ; then
+	echo "The authdata directory is not set; refer to docker-compose script"
+	exit 3
+fi
+if [ ! -f /etc/apache2/authdata/.htpasswd ]; then
+	echo ".htpasswd file not found"
+	exit 3
+fi
+if [ ! -f /etc/apache2/authdata/.htgroup ]; then
+	echo ".htgroup file not found"
+	exit 3
 fi
+chown www-data /etc/apache2/authdata /etc/apache2/authdata/.htpasswd /etc/apache2/authdata/.htgroup
+chmod 700 /etc/apache2/authdata
+chmod 600 /etc/apache2/authdata/.htpasswd /etc/apache2/authdata/.htgroup
 
-if [ -z "$CERTBOT_DOMAINS" ]; then
-	echo "You must set CERTBOT_DOMAINS to value to be passed to certbot for --domains" 
+# check that we got the vars we need
+if [ -z "$CERTBOT_DOMAINS" -o "$CERTBOT_DOMAINS" = "." ]; then
+	echo "The docker-compose script must set CERTBOT_DOMAINS to value to be passed to certbot for --domains" 
 	exit 3
 fi
 
-if [ -z "$CERTBOT_EMAIL" ]; then
-	echo "You must set CERTBOT_EMAIL to an email address useful to certbot/letsencrypt for notifications"
+if [ -z "$CERTBOT_EMAIL" -o "$CERTBOT_EMAIL" = "." ]; then
+	echo "The docker-compose script must set CERTBOT_EMAIL to an email address useful to certbot/letsencrypt for notifications"
+	exit 3
+fi
+
+if [ -z "$APACHE_FQDN" -o "$APACHE_FQDN" = "." ]; then
+	echo "The docker-compose script must set APACHE_FQDN to the (single) fully-qualified domain at the top level"
 	exit 3
 fi
 
 # run cerbot to set up apache
-certbot --agree-tos --email "${CERTBOT_EMAIL}" --non-interactive --domains "$CERTBOT_DOMAINS" --apache --agree-tos --rsa-key-size 4096 --redirect || exit 4
+if [ "$CERTBOT_TEST" != "test" ]; then
+    certbot --agree-tos --email "${CERTBOT_EMAIL}" --non-interactive --domains "$CERTBOT_DOMAINS" --apache --agree-tos --rsa-key-size 4096 --redirect || exit 4
 
-# certbot actually launched apache. The simple hack is to stop it; then launch 
-# it again after we've edited the config files.
-/usr/sbin/apache2ctl stop
+    # certbot actually launched apache. The simple hack is to stop it; then launch 
+    # it again after we've edited the config files.
+    /usr/sbin/apache2ctl stop
+fi
 
 # now, add the fields to the virtual host section for https.
 set -- proxy-*.conf
 if [ "$1" != "proxy-*.conf" ] ; then
 	echo "add proxy-specs to configuration from:" "$@"
-	cat "$@" > /tmp/proxyspecs.conf || exit 5
+	sed -e "s/@{FQDN}/${APACHE_FQDN}/g" "$@" > /tmp/proxyspecs.conf || exit 5
 	sed -e '/^ServerName/r/tmp/proxyspecs.conf' /etc/apache2/sites-available/000-default-le-ssl.conf > /tmp/000-default-le-ssl-local.conf || exit 6
 	mv /tmp/000-default-le-ssl-local.conf /etc/apache2/sites-available || exit 7
 	echo "enable the modified site, and disable the ssl defaults"
@@ -43,5 +65,7 @@ if [ "$1" != "proxy-*.conf" ] ; then
 fi
 
 # launch apache
-echo "launch apache"
-exec /usr/sbin/apache2ctl -DFOREGROUND
+if [ "$APACHE_TEST" != "test" ]; then
+    echo "launch apache"
+    exec /usr/sbin/apache2ctl -DFOREGROUND
+fi