-
Notifications
You must be signed in to change notification settings - Fork 1
staging.ckanhosted.com
s084.okserver.org is setup to host multiple CKAN staging sites. For example if you want to setup a staging or demo site for a production CKAN site, for a new feature on a development branch, for an extension, etc. The setup is similar to the *.ckanhosted.com sites on s082, but where all the sites on s082 share a single CKAN codebase so they can all be easily upgraded at once, each site on s084 has its own copy of CKAN so that they can each run different versions, branches, extensions, etc.
Also see ckanhosted.com
There is some initial setup of the website/frontend server and the solr and database server that only needs to be done once (doesn't need to be repeated each time you boot a new site on the server). I'm not entirely clear on the details of this so maybe someone else can fill this section out a little better. We used ckanbuild to do the initial setup. I think the main steps are:
- Run ckanbuild's
dbserver.shscript on the database server to setup postgres, solr, a firewall, etc. - Run ckanbuild's
webserver.shscript on the frontend server to setup apache, nginx, a firewall, etc. - Use ckanbuild to build a Debian package for a CKAN website.
- Install the Debian package on the frontend server. Note that we don't actually want to use the CKAN site contained in this package, we'll follow a different process (below) to setup each new site on the server, but we're installing the package because it does some more webserver setup stuff that we need.
- Use ckanbuild's
dbserver.sh addwebcommand on the database server to add the internal IP address of the frontend server to the firewall, so that the frontend server can access the database server over their internal network.
Currently we do this manually. It would be nice to script all this and we might like to use this with Ansible to do so one day.
We'll walk through the steps to setup a site called pdeu_staging, a staging site for publicdata.eu.
Make a new /usr/lib/ckan directory to contain the site's virtualenv:
sudo mkdir /usr/lib/ckan/pdeu_staging
sudo chown -R okfn /usr/lib/ckan/pdeu_staging
Create a virtualenv in the directory:
virtualenv --no-site-packages /usr/lib/ckan/pdeu_staging
Install CKAN into the virtualenv:
/usr/lib/ckan/pdeu_staging/bin/pip install -e 'git+https://github.com/okfn/ckan.git#egg=ckan'
Install CKAN's Python dependencies into the virtualenv:
/usr/lib/ckan/pdeu_staging/bin/pip install -r /usr/lib/ckan/pdeu_staging/src/ckan/pip-requirements.txt
Create a new /etc/ckan dir with the site's CKAN config files:
cd /etc/ckan
mkdir pdeu_staging
cp apache.wsgi pdeu_staging/
cp production.ini.template pdeu_staging/production.ini
Now edit the config files. In apache.wsgi set the activate_this path to the path in the site's virtualenv, e.g.:
activate_this = os.path.join('/usr/lib/ckan/pdeu_staging/bin/activate_this.py')
In production.ini, set ckan.site_id, ofs.storage_dir, sqlalchemy.url,
ckan.datastore.write_url, ckan.datastore.read_url, ckan.site_title and
ckan.site_url.
Just copy an existing site's file and then edit it:
sudo cp /etc/apache2/sites-available/existing_site /etc/apache2/sites-available/pdeu_staging
Edit the new file and change everything that refers to the old site to refer to the new one, e.g. ServerName, ServerAlias, WSGIScriptAlias, ErrorLog and CustomLog.
ssh into the database server (at the time of writing the database server for all sites on s084 is s085.okserver.org) and run ckanbuild's dbserver createdb command:
/home/okfn/ckanbuild/bin/dbserver.sh createdb pdeu
This will print out a password that you need to copy and paste into the site's production.ini file on s084.
Note that we use "pdeu" not "pdeu_staging" for the postgres database and user name, this is to make it the same as on the production site which makes it easier to load a database dump from the production site into the staging site or vice-versa.
Run the script on s085, e.g.:
/home/okfn/scratch/create-datastore-database/create-datastore-database.sh pdeu
Back on s084, on the production.ini file use the same user as the previous step for ckan.datastore.write_url and the datastore user for ckan.datastore.read_url.
Back on s084:
/usr/lib/ckan/pdeu_staging/bin/paster --plugin=ckan db -c /etc/ckan/pdeu_staging/production.ini init
sudo mkdir /var/lib/ckan/pdeu_staging
sudo chown www-data /var/lib/ckan/pdeu_staging
sudo a2ensite pdeu_staging
sudo service apache2 reload
You should now be able to browse to http://pdeu_staging.staging.ckanhosted.com.
/usr/lib/ckan/pdeu_staging/bin/paster --plugin=ckan sysadmin -c /etc/ckan/pdeu_staging/production.ini add seanh
Now you can customise the site by editing its production.ini file, by activating the virtual environment at /usr/lib/ckan/{SITE} and installing CKAN extensions and other Python packages in it, and by checking out development branches in the git repo at /usr/lib/ckan/{SITE}/src/ckan. You need to restart apache after doing any customisations.
We have a script for loading demo data (gold prices, Malawi aid projects, etc.). This script is checked out at /home/okfn/load_demo_data/src/load_demo_data. See the scripts README for usage.