In this repo, I will be exploring event based workflows using sensors and deferrable operators in Apache Airflow.
I have written a detailed article here to explain my thought process: https://medium.com/@oeonyema/create-event-driven-airflow-pipeline-with-amazon-sqs-ec3a79485923
I have a detailed article to exlain my implementation: https://medium.com/towards-data-engineering/create-event-driven-airflow-pipeline-with-sns-sqs-and-lambda-c1b706a991e5
Read through this Medium article to follow my approach: https://medium.com/towards-data-engineering/externaltasksensor-vs-triggerdagrunoperator-1aaab9a469b7