Elasticsearch backed Speedtest probe

This is a simple Bash script "app" that will run Speedtest and upload the measurements into an Elasticsearch instance.

The script is an all-in-one package that will take care of creating the Elasticsearch resources, run the speedtest app and upload its results, as well as install/uninstall itself.

The Speedtest CLI measurements are spooled locally until they can be updated into Elasticsearch. Note that the spool file is stored by default under system's temporary directory, so it's lost on a reboot. If a measurement is rejected by Elasticsearch due to ingesting errors, the measurements are logged into a dedicated file.

The failures are also sent to local logging system.

Elasticsearch is setup to rotate out the old data, so that the destination indices won't grow forever. It will also drop some of Speedtest CLI redundant JSON fields and consolidate the geographical data into appropriate Elasticsearch data types.

Requirements

There are a few dependencies, besides speedtest that need to be available on the system before running this script:

• curl
• cron
• jq
• bc

These are checked on installation only and expected to be available otherwise.

The Elasticsearch server needs to support index lifecycle management.

Installation

0. Edit the config file, esst.conf and change the defines between the <change me> tags in both "Elasticsearch" and "Local" sections.

1. Create the Elasticsearch resources

   This step needs to be done only once, even if probing with Speedtest on multiple hosts. Run:

   ./esst.sh init
   

   This will create the index template, the pipeline that alters the JSON that Speedtest produces, as well as an index lifecycle managment policy to manage the data rotation.

   The opposite of init is drop, which will delete all resources created initially.

2. Install the "app"

   ./esst.sh install
   

   This will install the needed files under /opt/esst directory (by default), as well as register a cron entry in the crontab of executing user. This won't make it available in the PATH. To later uninstall it, list the cron entries (crontab -l) to find its location and run uninstall; below is an example that works with the default configuration:

   /opt/esst/bin/esst uninstall
   

3. Make a test run

   To produce one measurement, invoke it with the probe argument:

   /opt/esst/bin/esst probe
   

Kibana Dashboard

There is one simple dashboard configuration file that can be uploaded into Kibana, dashboard.ndjson. This will install a graph of upload and download measurements, as well as a gauge with the max RTT to the speedtest server, over the data from all probes: