Big Data, a recent phenomenon. Everyone talks about it, but do you really know what Big Data is? Join our four-part series about Big Data and you will get answers to your questions! We will cover Introduction to Big Data and available platforms which we can use to deal with Big Data. And in the end, we are going to give you an insight into the possible future of dealing with Big Data. Spark, Flink, Presto and many others. This is just a sample of frameworks which are used in real companies and we will talk about some of them. In the previous episode of this Big Data series, we talked about the basic information concerning Big Data. This presentation, however, will be much more technical as we will be covering the most popular platforms you can use to deal with Big Data 2.0 Systems and learn about the key differences between these platforms. Let’s go! #CHEDTEB www.chedteb.eu

1. Available
platforms
for
Big Data 2.0
Based on presentations during
Brno Data Week 2018
by prof Sherif Sakr
Created by: Tichý, T. & Luhan, J.
(Feb. 2019)
https://www.chedteb.eu/

2. Dealing with
data diversity
in Big Data 2.0
Due to the fact that data
is very diverse, we have to
find the right platform
which suits our needs.
Big Data 2.0

3. Spark
• Apache Spark is a fast, general engine for large scale data
processing on a computing cluster (new engine for Hadoop)
• Developed initially at UC Berkeley, in 2009, in Scala, it is
currently supported by Databricks
• One of the most active and fastest growing Apache projects
• Committers from Cloudera, Yahoo, Databricks, UC Berkeley,
Intel, Groupon and others
Big Data 2.0: Platforms

4. Spark vs Hadoop
Big Data 2.0: Platforms

5. Spark vs Hadoop
Big Data 2.0: Platforms

6. Flink
• Apache Flink is a distributed in-memory data processing framework
which represents a flexible alternative for the MapReduce
framework that supports both batch and real-time processing.
• Flink has originated from the Stratosphere research project that was
started at the Technical University of Berlin in 2009 before joining
the Apache‘s Incubator in 2014.
• Instead of the map and reduce abstractions, Flink uses a directed
graph approach that leverages in-memory storage for improving the
performance of the runtime execution.
• Has a fast growing community.
Big Data 2.0: Platforms

7. Hadoop vs Spark vs Flink
Big Data 2.0: Platforms

8. Big Data 2.0 Processing
Systems:
SQL-On-Hadoop

9. Apache Hive
• The first system that was introduced to support SQL-on-Hadoop with
familiar relational database concepts such as tables and columns.
• Hive has been widely used in many organizations to manage and
process large volumes of data, such as Facebook, eBay, LinkedIn and
Yahoo!
• It supports an SQL-like declarative language, HiveQL, which represents
a subset of SQL92 and therefore can be easily understood by anyone
who is familiar with SQL.
• Hive queries automatically compile into MapReduce jobs that are run
by using Hadoop.
Big Data 2.0: SQL based platforms

10. Impala
• Open source project, built by Cloudera, to provide a massively
parallel processing SQL query engine that runs natively in
Apache Hadoop.
• By using Impala, the user can query data which is stored in
Hadoop Distributed File System (HDFS).
• It uses the same metadata, SQL syntax (HiveQL) of Apache Hive.
Big Data 2.0: SQL based platforms

11. IBM Big SQL
• The SQL interface for the IBM big data processing platform,
InfoSphere BigInsights
• Big SQL relies on a built-in query optimizer that rewrites the
input query as a local job to help minimize latencies by using
Hadoop dynamic scheduling mechanisms.
• It uses a massively parallel processing SQL engine that is
deployed directly on the physical Hadoop Distributed File
System (HDFS).
Big Data 2.0: SQL based platforms

12. Presto
• Open source distributed SQL query engine, built by Facebook, for
running interactive analytic queries against large scale structured
data sources of sizes of gigabytes up to petabytes.
• Presto allows querying data where it lives, including Hive, NoSQL
databases (e.g., Cassandra, HBase), relational databases or even
proprietary data stores
• A single Presto query can combine data from multiple sources
• Presto has been recently adopted by big companies and
applications such as Netflix and Airbnb
Big Data 2.0: SQL based platforms

13. To be
continued
In the next episode we will focus
on big streams (data that is
constantly generated), how to
analyze and vizualize them.