What is Apache Spark? Benefits of using Apache Spark

The science of Big Data (Big Data) develops rapidly and strongly, requiring the support of specialized computing tools. Apache Spark was born as a necessity, providing an effective solution for data analysis and processing. So What is Apache Spark?? How is Apache Spark structured, what are the advantages and disadvantages? Join Mat Bao to find the answer in the following article.

See more:

What is Apache Spark?

1. What is Apache? What is Apache Spark?

Apache, officially known as Apache HTTP Server, is a free and open source web server software. Apache has been developed by the Apache Software Foundation since 1995, accounting for about 46% of the worldwide website market share. Apache helps webmasters put content on the website, supports the establishment of a safe, effective and economical website.

Apache Spark effectively supports data analysis and processing

Apache Spark is an open source data processing framework at scale. Spark allows users to quickly build predictive models and perform computations on multiple computers or datasets at the same time, without extracting experimental computation samples. In 2009, Apache Spark was initially developed by AMPLab, then donated to the Apache Software Foundation in 2013.

2. Apache Spark components

"What is Apache Spark??” and its main components are the most frequently asked questions. An Apache Spark will have 5 components:

Spark Core is considered the foundation and operating conditions for all components in Apache Spark. This component assumes the task of computing, processing in memory (In - memory computing) and referencing data stored in external storage systems.

This is a component that provides a new type of data abstraction (SchemaRDD) to support structured and semi-structured data. Spark SQL performs operations on DataFrames in Java, Python or Scala, through the help of DSL (Domain - specific language) and SQL.

Main components of Apache Spark

Spark Streaming analyzes streams by treating streams as mini batches to perform RDD transformation. Thereby, the code written for batch processing can be reused into stream processing, making lambda architecture development easier. However, in data processing, this creates a certain delay.

Spark MLlib is a machine learning platform with a distributed memory-based architecture. Compared to the version running on Hadoop, Spark Mllib is 9 times faster.

GraphX is a graphics processing platform that provides APIs that represent computations in graphs using the Pregel Api.

3. What is the architecture of Apache Spark like?

After learn "What is Apache Spark??” then the next thing you need to know is its architecture. Apache Spark is composed of two parts, the executors and the drivers. In particular, the driver helps to convert user code into many tasks (tasks), distributed on processing nodes (worker nodes).

Architecture of Apache Spark

The executor will perform tasks and run on the processing nodes that the driver assigns to it. Besides, Spark can also run in standalone cluster mode which only requires JVM and Apache Spark framework on each machine in the cluster. However, using a cluster management tool as an intermediary between the two components will help to allocate resources on demand and be better utilized.

Apache Spark builds instructions for data processing in a directed acyclic graph, or DAG. DAG is Spark's scheduling layer, which determines which tasks are executed on which node and in what sequence.