Workflow Management¶

Description	Workflow Management engines coordinate the steps in a data pipeline for automated execution. The projects in this category focus on the control flow of tasks. There is some overlap between this category and Ingestion. Ingestion projects tend to focus more on the dataflow and manipulation of individual records.
Projects	4
Lines Committed vs. Age Chart (click to view)

Projects¶

Project	Size Score	Trend Score	Byline
Airflow	8.25	8.5	Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Cromwell	5.25	4.25	Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
Luigi	5.0	3.75	Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Snakemake	7.5	6.5	The Snakemake workflow management system is a tool to create reproducible and scalable data analyses