Merge pull request #2 from coecms/dbcommand

Dbcommand

README.rst

@@ -0,0 +1,147 @@
+|DOI| \_\_\_
+
+ERA5
+====
+
+The era5 python code is an interface to the CDS api to download ERA5
+data from the CDS data server. It uses a modified version of the CDS api
+which stops after a request has been submitted and executed. The target
+download url is saved and downloads are run in parallel by the code
+using the Pool multiprocessing module. As well as managing the downloads
+the code gets all the necessary information on available variables from
+local json configuration files. Before submitting a request the code
+will check that the file is not already available locally by quering a
+sqlite database. After downloading new files it is important to update
+the database to avoid downloading twice the same file. Files are first
+downloaded in a staging area, a quick qc to see if the file is a valid
+netcdf file is run and finally the file is converted to netcdf4 format
+with internal compression. The code default behaviour is to download
+netcdf files, but it's also possible to download grib, however we
+haven't tested the full workflow for this option.
+
+Getting started
+---------------
+
+To run a download::
+
+    era5 download -s surface -y 2018 -m 11 -p 228.128
+
+Download ERA5 variables, if month argument is not passed then the entire year will be downloaded. By default it downloads hourly data in netcdf format.
+
+Options: 
+      * -q, --queue Create json file to add request to queue 
+      * -u, --urgent In conjunction with queue save the request file in a  separate directory that can be then prioritised. 
+      * -s, --stream [surface|wave|pressure|land|cems_fire|agera5|wdfe5] ECMWF stream currently operative analysis surface, pressure levels, wave model, and derived products ERA5 land, CEMS_fire, AGERA5 and WDFE5
+[required] 
+      * -y, --year TEXT year to download [required] 
+      * -m, --month TEXT month/s to download, if not specified all months for year will be downloaded 
+      * -t, --timestep [mon|hr|day] timestep if not specified hr is default
+      * -b, --back Request backwards all years and months as one file, works only for monthly data 
+      * --format [grib|netcdf|tgz|zip] Format output: netcdf default, some formats work only for certain streams
+      * -p, --param TEXT Grib code parameter for selected variable, pass as param.table i.e. 132.128 If none passed then allthe params listed in era5_<stream>_<tstep>.json will be used. 
+      * --help
+Show this message and exit.
+
+To update files when a new month is released, omit param flag::
+
+    era5 download -s surface -y 2019 -m 05 
+
+
+The 'download' sub command will actually request and download the data
+unless you use the 'queue' flag. If you want only to create a request::
+
+    era5 download -s surface -y 2018 -m 11 -p 228.128 -q
+
+This will create a json file which stores the arguments passed::
+
+    era5_request_<timestamp>.json
+    {"update": false, "format": "netcdf", "stream": "surface", "params": ["228.128"], 
+     "year": "2018", "months": ["11"], "timestep": "hr", "back": false}
+
+To execute the request the tool is used with the 'scan' command option::
+
+    era5 scan -f era5_request_<timestamp>.json
+
+To manage the database use 'era5 db' subcommand::
+
+   era5 db -s surface -p u10
+
+for example will add all new surface hourly (tstep is hr by default) u10 files in database. NB that in this case you pass the variable name not the grib code to the -p flag.   
+'db' can also list all variables in a stream:: 
+
+   era5 db -a list -s land -t mon
+
+Unless a variable is specified this will show all variables regularly updated for the stream, how many corrsponding files are on the filesystem and in the databse. These numbers are compared to the expected number of files based on the current date.
+
+Finally to delete records::
+
+   era5 db -a delete -s land -t mon -p ro -y 2019
+
+ This will delete all corresponding records but will list all records to be deleted and ask for confirmation first.
+
+
+Latest updates
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+August 2020 - Added --urgent flag
+              Added support to distribute requests across multiple users accounts
+
+click era5 command code and input files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  cli.py -- main code, includes click commands and functions to buil
+   request and submit it with pool python cli.py --help to check usage
+-  era5_functions.py -- functions used by cli.py, just separated them
+   so it is overall more readable
+-  era5_db.py -- has all the fuctions relating to db operations 
+-  update_json_vars.py -- script that uses the clef db to update
+   era5_vars.json needs clef module
+-  setup.py and setup.cfg - to install module
+
+To configure the tool
+~~~~~~~~~~~~~~~~~~~~~
+
+These files are in era5/data/
+
+-  config.json -- to set configuration:
+   * staging, data, logs directories,
+   * database name and location,
+   * bash commands to download, resume download, qc, compress and concatenate files,
+   * number of threads,
+   * number of resume download attempts,
+   * slow and fast ips
+
+-  era5_pressure_hr.json -- pressure levels stream arguments to build
+   request and list of params to download at hourly temporal resolution
+-  era5_pressure_mon.json -- pressure levels stream arguments to build
+   request and list of params to download at monthly temporal resolution
+-  era5_wave_hr.json -- wave model surface level stream arguments to
+   build request and list of params to download at hourly temporal
+   resolution
+-  era5_wave_mon.json -- wave model surface level stream arguments to
+   build request and list of params to download at monthly temporal
+   resolution
+-  era5_surface_hr.json -- surface level stream arguments to build
+   request and list of params to download at hourly temporal resolution
+-  era5_surface_mon.json -- surface level stream arguments to build
+   request and list of params to download at monthly temporal resolution
+-  era5_land_hr.json -- Land model surface level stream arguments to
+   build request and list of params to download at hourly temporal
+   resolution
+-  era5_vars.json -- Json file with list of grib codes that can be
+   downloaded from CDS and respective variable and cds names
+-  era5_derived.json -- Json file with list of derived products variables
+
+Other files
+~~~~~~~~~~~
+
+(not included in git)
+
+-  era5.sqlite -- sqlite database
+
+Modified cdsapi code
+~~~~~~~~~~~~~~~~~~~~
+
+-  cdsapi: **init**.py **pycache** api.py
+
+.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3549078.svg
+   :target: https://doi.org/10.5281/zenodo.3549078

docs/gettingstarted.rst

@@ -0,0 +1,75 @@
+Getting Started
+===============
+
+ERA5 is an interface to the CDSapi (Copernicus Climate Data Store API). It helps automating and submitting csapi requests. ERA5 is python3 based and uses the click module so it can be run as command line.
+The configuration of the tool and specific information on the variables is provided by json files. 
+
+Once is installed the tool is accessed through the command-line `era5` program. There are
+presently three subcommands:
+
+ * :code:`clef download` to submit a request and download new files, at least one stream has to be specified. If you omit the variable then all the variables listed in the corresponding stream json file will be downloaded. this is useful for regular updates.
+
+ * :code:`clef scan` to execute a download request starting from a json constraints file previously saved using download with the --queue/-q flag.
+
+Examples
+--------
+
+
+download
++++++
+::
+
+    $ era5 download -s pressure -y 2020 -m 02 -p 132.128 
+
+This will put in a request for temperature (whose grib code is 132.128) for February 2020 on pressure levels and at 1hr timesteps.
+The area and grid resolution are defined in the 
+era5/data/era5_pressure_hr.json file
+
+NB that you can change the timestep by using the `-t` flag, `hr` is the default value, other possibilities are `mon` and `day` . Depending on the stream some of these options might not be valid.
+
+Other flags are:
+     * -h/--help
+     * -q/--queue  which saves the constraints to a json file that can be passed later to the `scan` subcommand
+     * -f/--file works only with `scan`
+     * --back works only with monthly data, it allows to download backwards all available data to the year passed with `-y`. 
+
+Scan example::
+     $ era5 scan -f era5_request_file.json
+
+Update the database::
+
+     $ era5 db -s land -t mon -p ro
+
+NB
+     * -a/--action works only for db subcommand to choose what db action to execute: update (default), list, or delete 
+
+
+Installation
+============
+The tool uses python >=3.6 and you need to install the click package.
+
+To setup the tool::
+    $ python setup.py install
+
+The github repository includes a modified version of the cdsapi. 
+For this to work you need to setup a .csdapirc file in your home directory with the api url and your own api key. It should look like::
+
+     url: https://cds.climate.copernicus.eu/api/v2
+     key: #########################::
+     
+
+The instructions are available from the ECMWF confluence.
+NB most of the tool functionalities ar epreserved even if you use the original api. What the modified api does is to stop the api working before it can download the files. Separating the file download from the requests helps managing the downloads as parallel tasks and avoid potential sisue should the connection being interrupted. 
+
+Step2
+-----
+Configure the tool by defining the following settings in era5/data/config.json:
+
+  The following folders:
+ * log  for log outputs
+ * staging  where the files are initially downloaded
+ * netcdf for the compressed files
+ * era5_derived if you are downloading ERA% derived products
+ * And update directories paths and the sqlite db name
+ * the command you want to use to download/compress the files
+