Ingestion

Description

Data ingestion frameworks preprocess incoming data for insertion into a data warehouse or data lake. These operate on data in batches, in “micro” batches, or on streams of records/events. There is some overlap between this ccategory and Workflow Management. Workflow Management projects tend to focus more on the control flow of a pipeline rathe than on the actual data manipulation.

Projects

4

Lines Committed vs. Age Chart (click to view)

Lines Committed vs. Age Chart (click to view)

Projects

Project

Size Score

Trend Score

Byline

Beam

9.0

7.25

Apache Beam is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends, including Apache Flink, Apache Spark, Google Cloud Dataflow and Hazelcast Jet.

Flink

9.25

7.5

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.

NiFi

6.75

7.5

Apache NiFi supports highly configurable directed graphs of data routing, transformation, and system mediation logic.

Storm

6.25

2.5

Storm is a distributed realtime computation system.