This workflow tool produces Titan2D hazard maps that display the probability of a volcanic flow depth reaching a critical height following a premonitory volcanic eruption event.
Titan2D is a computer model for simulating granular avalanches over digital elevation models (DEMs) of natural terrain. The Titan2D hazard maps are constructed by creating a statistical surrogate model of the Titan2D computer model, requiring numerous executions of the Titan2D computer model and the emulator's uncertainty quantification (UQ) software. See Workflows for Construction of Spatio-Temporal Probabilistic Maps for Volcanic Hazard Assessment for more information.
The Pegasus Workflow Management System (WMS) provides the structured platform for automating and managing these numerous executions, including staging the jobs, distributing the work, submitting the jobs to run in parallel, as well as handling data flow dependencies and overcoming job failures. This tool is designed to follow the Pegasus WMS, Amazon AWS Batch deployment scenario, which, in turn, is based on the Amazon AWS Fetch & Run Example. See the Welcome to Pegasus WMS’s documentation! Deployment Scenarios, Amazon AWS Batch section, for more information.
This tool runs two Docker images, a remotehostimage Docker image and a submithostimage Docker image. The remotehostimage Docker image is the Fetch & Run Docker image and contains software to run Titan2D and the emulator's UQ software, and, also includes the Bash Fetch & Run script and ENTRYPOINT. The submithostimage Docker image contains the software required to implement the workflow, including a Jupyter Notebook as the interface for running the workflow, HTCondor, and the Pegasus WMS.
This tool requires that you complete the following prerequisites for configuring Amazon AWS Batch.
-
Create an IAM account and an IAM user with administrative access. See Create IAM account and administrative user for information on how to do this. Create an access key pair for the IAM user. See Manage access keys for IAM users for information on how to do this.
-
Sign in as the IAM user and create the IAM roles required for Pegasus Amazon AWS Batch. These are an AWS Batch Service IAM Role named AWSBatchServiceRole, an Amazon Elastic Container Service (ECS) Instance Role named ecsInstanceRole, and an IAM Role named batchJobRole. See the Pegasus WMS Deployment Scenarios, Amazon AWS Batch section, for more information.
-
Sign in as the IAM user and create a Virtual Private Cloud (VPC) and security group for your Amazon AWS Region. See Create a VPC and Create a security group for information on how to do this.
-
Install the Amazon AWS CLI on your personal computer. See Getting started with the AWS CLI for more information on how to do this.
AWS Batch comprises four components for a workflow: a compute environment, a job definition, a job queue, and the jobs. Pegasus Amazon AWS Batch provides an interface for creating and managing these AWS Batch components. To do this, Pegasus requires an AWS credential file, an AWS S3 configuration file, and JSON-formatted information catalogs. These files contain fields that reference your AWS authorization credentials. See the Pegasus WMS Deployment Scenarios, Amazon AWS Batch section, for more information.
This tool requires two AWS authorization credentials files: ~/.aws/config and ~/.aws/credentials.
-
Required contents of the ~/.aws/config file:
[default]
account_id = *
region = * -
Required contents of the ~/.aws/credentials file:
[default]
aws_access_key_id = *
aws_secret_access_key = *Where * is replaced with your IAM account and IAM user authorization credentials.
This tool's ./remosthost and ./submithost directories contain templates for the files that Pegasus requires. The ./pegasus-wms-configure.sh Bash script configures these files with your AWS authorization credentials.
- Note: source ./pegasus-wms-configure.sh must be executed before building the remotehostimage and submithostimage Docker images.
Build the remotehostimage Docker image and push the image to the Amazon Elastic Cloud Registry (ECR).
- Create an Amazon AWS Elastic Container Repository (ECR) for your Amazon AWS Region. See Creating an Amazon ECR private repository to store images for information on how to do this. Name the repository: remotehostimage.
- source ./pegasus-wms-configure.sh
- cd ./remotehost
- docker image build -t remotehostimage . 2>&1 | tee build.log
- source ./push-remotehostimage-to-ECR.sh
- source ./pegasus-wms-configure.sh
- cd ./submithost
- docker image build -t submithostimage . 2>&1 | tee build.log
- docker run --privileged --rm -v $PWD/emulator/LOCAL/shared-storage:/home/submithost/emulator/LOCAL/shared-storage -p 9999:8888 submithostimage
-
The emulator.ipynb Jupyter Notebook provides the interface for running the workflow.
-
When the docker run Sending DC_SET_READY message appears, open a web browser and enter the URL localhost:9999/apps/emulator.ipynb and enter the password emulator. This opens the emulator.ipynb Jupyter notebook in Appmode. To open the emulator.ipynb Jupyter notebook in Edit Mode, enter the URL localhost:9999/notebooks/emulator.ipynb, enter the password emulator, and select Kernel / Restart & Run All.
-
Follow the processing steps in emulator.ipynb to set up and run the workflow. Setup includes selecting a volcano, setting the material model and pile parameters for the selected volcano's eruption, setting parameters for the emulator's UQ software, selecting a VPC subnet and security group, as well as running a script that further configures the JSON-formatted information catalogs that Pegasus requires with your defined parameters.
- Log out of the emulator.ipynb Jupyter Notebook and enter Ctrl+C to stop the running Docker container.
-
Results and interim files generated for running a workflow are stored in the mounted ./submithost/emulator/LOCAL/shared-storage directory.
-
Results include the Hazard_Report.pdf file, which contains information on the probability of a volcanic flow depth reaching a critical height at specific locations.