Apache Airflow
OPEN SRCPlatform to programmatically author, schedule, and monitor workflows
Robust Scheduling & Monitoring — Built-in scheduler and web UI for visualizing, monitoring, and troubleshooting workflows.
Extensible Operator Library — Rich set of operators to integrate with various services, with easy custom operator creation.
Scalable & Modular Architecture — Uses a message queue to orchestrate an arbitrary number of workers for horizontal scaling.
Dynamic Pipeline Generation — Pipelines are defined as Python code, enabling dynamic generation and easy versioning.
Databricks · AWS · Slack · Kubernetes · Snowflake · Google Cloud +2 more