Spark

The most popular big data processing tool and the successor to Hadoop Map-Reduce. It is mostly used for ETL (extract transform load), but also includes facilities for streaming aggregations, graph computations, and machine learning.

Logo

../_images/apache_spark-small.png

Website

http://spark.apache.org

Repository

https://github.com/apache/spark

Byline

A unified analytics engine for large-scale data processing.

License

Apache 2.0

Project age

12 years

Backers

Apache (Governed by), DataBricks (Commercial Product By)

Lastest News (2021-10-13)

3.2.0 We are happy to announce the availability of Spark 3.2.0! Visit the release notes to read about the new features, or download the … more

Size score (1 to 10, higher is better)

9.5

Trend score (1 to 10, higher is better)

7.25

Education Resources

URL

Resource Type

Description

https://spark.apache.org/docs/3.1.1/

Documentation

Official project documentation.

Git Commit Statistics

Statistics computed using Git data through March 31, 2022.

Statistic

Lifetime

Last 12 Months

Commits

95,733

39,070

Lines committed

44,249,462

11,732,238

Unique committers

2,622

333

Core committers

13

24

../_images/apache_spark-monthly-commits.png

Similar Projects

Project

Size Score

Trend Score

Byline

Dask

6.5

5.0

Parallel computing with task scheduling.

HPCC

6.0

8.25

HPCC Systems (High Performance Computing Cluster) is an open source, massive parallel-processing computing platform for big data processing and analytics.

Hadoop

8.5

7.5

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.

Mars

7.25

6.5

Mars is a tensor-based unified framework for large-scale data computation which scales Numpy, Pandas and Scikit-learn.

Ray

8.75

8.75

An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library.