Occurrence Downloads in Airflow

[fmendezh]

Create a simple Airflow DAG that trigger an occurrence downloads and tracks it status.
The DAG must trigger a Spark job that and its settings must be applied dynamically to assign cores, executors and memory accordingly to the amount of data to process.
Once the DAGs finishes its status must be updated in the registry database.

• Implement a simple DAG that triggers an download Spark job.
• Implement a basic mechanism to assign resources to spark jobs based on the amount of records to process by a download.
• Implement a Airflow notifier to update the status of downloads.
• Implement a mechanism to cancel a running Download DAG.
• Implement a service call to get the status of Download DAG.