Workflow Management

Description

Workflow Management engines coordinate the steps in a data pipeline for automated execution. The projects in this category focus on the control flow of tasks. There is some overlap between this category and Ingestion. Ingestion projects tend to focus more on the dataflow and manipulation of individual records.

Projects

4

Lines Committed vs. Age Chart (click to view)

Lines Committed vs. Age Chart (click to view)

Projects

Project

Size Score

Trend Score

Byline

Airflow

8.25

8.5

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Cromwell

5.25

4.25

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments

Luigi

5.0

3.75

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Snakemake

7.5

6.5

The Snakemake workflow management system is a tool to create reproducible and scalable data analyses