the.com/airflow
The scheduler that lets you pretend your data pipeline isn't held together by cron and hope.
means A platform for programmatically authoring, scheduling, and monitoring workflows, usually represented as directed acyclic graphs (DAGs) of tasks.
from Born at Airbnb in the early 2010s, where engineers needed a saner way to manage sprawling data jobs than a thicket of cron entries. The name fuses 'air' (from Airbnb) with 'flow,' nodding to how data and tasks move through the pipeline's graph — work flowing from one node to the next, visibly, instead of vanishing into a black box of scheduled scripts.
- origin
- Built at Airbnb in 2014 to tame cron chaos
- core unit
- DAGs, because your tasks must never loop back
- irony
- Famously flaky at handling its own task failures
- mascot energy
- Turns pipelines into flowcharts you still have to debug