2. Apache Spark is a unified computing engine and a set of libraries for parallel data
processing on computer clusters. As of this writing, Spark is the most actively
developed open source engine for this task, making it a standard tool for any
developer or data scientist interested in big data. Spark supports multiple widely
used programming languages (Python, Java, Scala, and R), includes libraries for
diverse tasks ranging from SQL to streaming and machine learning, and runs
anywhere from a laptop to a cluster of thousands of servers. This makes it an
easy system to start with and scale-up to big data processing or incredibly large
scale.
3. Apache Spark’s Philosophy Let’s break down our description of Apache Spark—a
unified computing engine and set of libraries for big data—into its key
components: Unified Spark’s key driving goal is to offer a unified platform for
writing big data applications. What do we mean by unified? Spark is designed to
support a wide range of data analytics tasks, ranging from simple data loading and
SQL queries to machine learning and streaming computation, over the same
computing engine and with a consistent set of APIs. The main insight behind this
goal is that real-world data analytics tasks—whether they are interactive analytics
in a tool such as a Jupyter notebook, or traditional software development for
production applications—tend to combine many different processing types and
libraries
