The document provides an overview and quick reference guide to big data concepts including Hadoop, MapReduce, HDFS, YARN, Spark, Storm, Hive, Pig, HBase and NoSQL databases. It discusses the evolution of Hadoop from versions 1 to 2, and new frameworks like Tez and YARN that allow different types of processing beyond MapReduce. The document also summarizes common big data challenges around skills, integration and analytics.

1. 1
Big Data Hoopla Simplified –
Hadoop, MapReduce, NoSQL …
TDWI Conference – Memphis, TN
Oct 29, 2014
© Talend 2014

2. 2
About the Presenter
Rajan Kanitkar
• Senior Solutions Engineer
• Rajan Kanitkar is a Pre-Sales Consultant with Talend. He
has been active in the broader Data Integration space for
the past 15 years and has experience with several leading
software companies in these areas. His areas of
specialties at Talend include Data Integration (DI), Big
Data (BD), Data Quality (DQ) and Master Data
Management (MDM).
• Contact: rkanitkar@talend.com
© Talend 2014

3. 3
Big Data Ecosystem
© Talend 2014

4. 4
Quick Reference – Big Data
Hadoop: Apache Hadoop is an open-source software framework for storage and large
scale processing of data-sets on clusters of commodity hardware.
Hadoop v1.0 - Original version that focused on HDFS and MapReduce. The
Resource Manager and Job Tracker were one entity.
Hadoop v2.0 – Sometimes called MapReduce 2 (MRv2). Splits out the Resource
Manager and job monitoring into two separate daemons. Also called YARN. This new
architecture allows for other processing engines to be managed/monitored aside from
just the MapReduce engine.
© Talend 2014

5. 5
Quick Reference - Big Data
• Hadoop: the core project
• HDFS: the Hadoop Distributed File System
• MapReduce: the software framework for distributed
processing of large data sets
• Hive: a data warehouse infrastructure that provides data
summarization and a querying language
• Pig: a high-level data-flow language and execution
framework for parallel computation
• HBase: this is the Hadoop database. Use it when you
need random, realtime read/write access to your Big
Data
• And many many more: Sqoop, HCatalog, Zookeeper,
Oozie, Cassandra, MongoDB, etc.
© Talend 2014

6. 6
Hadoop Core – HDFS
Metadata Operations
Name Node Client
Data Node
© Talend 2014
Block Block
Block Block
Data Node
Block Block
Block Block
Data Node
Block Block
Block Block
Data Node
Block Block
Block Block
Read/Write
Control
Replicate

7. 7
Hadoop Core – MapReduce
© Talend 2014
The „Word Count Example“

8. 8
Quick Reference – Data Services
HCatalog: a set of interfaces that open up access to Hive's metastore for tools inside and outside of the
Hadoop grid. Hortonworks donated to Apache. In March 2013, merged with Hive. Enables users with
different processing tools – Pig, MapReduce, and Hive – to more easily read and write data on the
cluster.
Hbase: a non-relational, distributed database modeled after Google’s Big Table. Good at storing sparse
data. Considered a key-value columnar database. Runs on top of HDFS. Useful for random real-time
read/write access.
Hive: a data warehouse infrastructure built on top of Hadoop. Provides data summarization, ad-hoc
query, and analysis of large datasets. Allows to query data using a SQL-like language called HiveQL
(HQL).
Mahout: a library of scalable machine-learning algorithms, implemented on top of Hadoop. Mahout
supports collaborative filtering, clustering, classification and item set mining.
Pig: allows you to write complex MapReduce transformations using a Pig Latin scripting language. Pig
Latin defines a set of transformations such as aggregate, join and sort. Pig translates the Pig Latin script
into MapReduce so that it can be executed within Hadoop.
SQOOP: utility for bulk data import/export between HDFS and structured data stores such as relational
databases.
© Talend 2014

9. 9
Quick Reference – Operational Services
Oozie: Apache workflow scheduler for Hadoop. It allows for coordination between Hadoop jobs. A workflow in
Oozie is defined in what is called a Directed Acyclical Graph (DAG).
Zookeeper: a distributed, highly available coordination service. Allows distributed processes to coordinate with
each other through a shared hierarchical name space of data registers (called znodes). Writing distributed
applications is hard. It’s hard primarily because of partial failure. ZooKeeper gives you a set of tools to build
distributed applications that can safely handle partial failures.
Kerberos : a computer network authentication protocol which provides mutual authentication. The name is based
on the three- headed dog . The three heads of Kerberos are 1) Key Distribution Center (KDC) 2) the client user 3)
the server with the desired service to access. The KDC performs two service functions: Authentication (are you who
you say you are) and the Ticket-Granting (gives you an expiring ticket that give you access to certain resources). A
Kerberos principal is a unique identity to which Kerberos can assign tickets (like a username). A keytab is a file
containing pairs of Kerberos principals and encrypted keys (these are derived from the Kerberos password).
© Talend 2014

10. 10
MapReduce 2.0, YARN, Storm, Spark
• Yarn: Ensures predictable performance & QoS for all apps
• Enables apps to run “IN” Hadoop rather than “ON”
• Streaming with Apache Storm
• Mini-Batch and In-Memory with Apache Spark
© Talend 2014
Applications Run Natively IN Hadoop
YARN (Cluster Resource Management)
HDFS2 (Redundant, Reliable Storage)
BATCH
(MapReduce)
INTERACTIVE
(Tez)
STREAMING
(Storm, Spark)
GRAPH
(Giraph)
NoSQL
(MongoDB)
EVENTS
(Falcon)
ONLINE
(HBase)
OTHER
(Search)
Source: Hortonworks

11. 11
Quick Reference – Hadoop 2.0 Additions
Storm: distributed realtime computation system. A Storm cluster is similar to a Hadoop cluster. On
Hadoop you run "MapReduce jobs". On Storm you run "topologies". Jobs and topologies are very
different -- in that a MapReduce job eventually finishes, but a topology processes messages forever
(or until you kill it). Storm can run on top of YARN.
Spark: parallel computing program which can operate over any Hadoop input source: HDFS,
HBase, Amazon S3, Avro, etc. Holds intermediate results in memory, rather than writing them to
disk; this drastically reduces query return time. Like Hadoop cluster but supports more than just
MapReduce.
Tez: framework which allows for a complex directed-acyclic-graph of tasks for processing data and
is built atop Apache Hadoop YARN. MapReduce is batch-oriented and unsuited for interactive query.
Tez allows Hive and Pig to be used to process interactive queries on petabyte scale. Support for
machine learning.
© Talend 2014

12. 12
Apache Spark
What is Spark?
• Spark Is An In-Memory Cluster Computing Engine that includes an HDFS
compatible in-memory file system.
Hadoop MapReduce
• Batch processing at scale
• Storage: Hadoop HDFS
• Runs on Hadoop
© Talend 2014
VS
Spark
• Batch, interactive, graph and real-time processing
• Storage: – Hadoop HDFS, Amazon S3, Cassandra…
• Runs on many platforms
• Fast in-memory processing up to 100 x faster than MapReduce (M/R)

13. 13
Apache Storm
What Is Storm?
• Storm Is a Cluster Engine Executing Applications Performing Real-time
Analysis of Streaming Data in Motion – Enabling the Internet of Things for
data such as sensor data, aircraft parts data, traffic analysis etc
Storm
• Real-time stream processing at scale
• Storage: None - Data in Motion
• Runs on Hadoop or on its own cluster
• Fast in-memory processing
© Talend 2014
VS
Spark
• Batch, interactive, graph and real-time processing
• Storage: – Hadoop HDFS, Amazon S3, Cassandra…
• Runs on many plaforms
• Fast in-memory processing

14. 14
Quick Reference – Big Data
Vendors: The Apache Hadoop eco-system is a collection of many projects.
Because of the complexities, “for profit” companies have packaged, added,
enhanced and tried to differentiate one another in the Hadoop world. The main
players are:
- Cloudera – CDH – Cloudera Distribution for Hadoop. Current version is CDH
5.2 (includes YARN)
- Hortonworks - HDP – Hortonworks Data Platform. Spun out of Yahoo in
2001. Current version is HDP 2.2 (YARN)
- MapR – M3 (Community), M5 (Enterprise), M7 (adds NoSQL). Apache
Hadoop derivative. Uses NFS instead of HDFS.
- Pivotal - GPHD – Greenplum Hadoop. Spun out of EMC in 2013. Current is
Pivotal HD 2.0 (YARN)
© Talend 2014

15. 15
Quick Reference – NoSQL
NoSQL: A NoSQL database provides a mechanism for storage and
retrieval of data that is modeled in means other than the tabular relations
used in relational databases – document, graph, columnar databases.
Excellent comparison of NoSQL databases by Kristof Kovacs:
http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
Includes a comparison of:
- Cassandra
- MongoDB
- Riak
- Couchbase
- … and many more
© Talend 2014

16. 16
Quick Reference – NoSQL
Document Storage: stores documents that encapsulate and encode data in some
standard format (including XML, YAML, and JSON as well as binary forms like BSON,
PDF and Microsoft Office documents. Different implementations offer different ways of
organizing and/or grouping documents.
Documents are addressed in the database via a unique key that represents that
document. Big feature is the database will offer an API or query language that will allow
retrieval of documents based on their contents.
CouchDB: Apache database that focuses on embracing the web. Uses JSON to store
data, Javascript as its query language using MapReduce, and HTTP for an API. The
HTTP API is a differentiator between CouchDB and Couchbase.
Couchbase: designed to provide key-value or document access. Native JSON support.
Membase + CouchDB = Couchbase. Couchbase architecture includes auto-sharding,
memcache and 100% uptime redundancy over CouchDB alone. Couchbase has free
version but is not open-source.
MongoDB: JSON/BSON style documents with flexible schemas to store data. A
“collection” is a grouping of MongoDB documents. Collections do not enforce document
structures.
© Talend 2014

17. 17
Quick Reference – NoSQL
Column Storage: stores data tables as sections of columns of data rather than as
rows of data. Good for finding or aggregating on large set of similar data. Column
storage serializes all data for one column contiguous on disk (so very quick read of a
column). Organization of your data REALLY matters in columnar storage. No
restriction on number of columns. One row in relational may be many rows in
columnar.
Cassandra: Apache distributed database designed to handle large amounts of data
across many commodity servers, providing high availability with no single point of
failure.
Dynamo: Amazon NoSQL database service. All data stored on solid state drives. Replicated
across three timezones. Integrated with Amazon EMR and S3. Stores “Items” (collection of key-value
© Talend 2014
pairs) given an ID.
Riak: a distributed fault-tolerant key-value database. HTTP/REST API. Can walk links (similar as
graph). Best used for single-site scalability, availability and fault tolerance – places where even
seconds of downtime hurt data collection. Good for point-of-sale or factory control system data
collection.
HBase: non-relational data store on top of Hadoop. Think of column as key and data as value.
Must create the Column family on table create. Look on Advanced tab to create families, then use
when writing data.

18. 18
Big Data Integration Landscape
© Talend 2014

19. 19
Data-Driven Landscape
© Talend 2014
Hadoop & NoSQL
Data Quality
Latency & Velocity
Expanding Data
Volumes
Master Data Consistency
Lack of Talent / Skills
Siloed Data due to
SAAS
No End-2-End meta-data
visibility

20. 20
Macro Trends Revolutionizing
the Integration Market
© Talend 2014
20
The amount of data will grow
50X from 2010 to 2020
64% of enterprises surveyed
indicate that they’re
deploying or planning Big
Data projects
By 2020, 55% of CIOs will
source all their critical apps
in the Cloud
Source: Gartner and Cisco reports

21. 21
The New Data Integration Economics
“Big data is what
happened when the cost
of keeping information
became less than the
cost of throwing it away.”
– Technology Historian George Dyson
© Talend 2014
45x
savings. $1,000/TB
for Hadoop vs
$45,000/TB for
traditional
$600B
revenue shift by
2020 to companies
that use big data
effectively
6x
faster ROI using
big data analytics
tools vs
traditional EDW
600x
active data.
Neustar moved
from storing 1% of
data for 60 days to
100% for one year

22. 22
Existing Infrastructures Under Distress:
Architecturally and Economically
Weblogs
© Talend 2014
Batch to
real-time
Standard
Reports
Ad-hoc
Query Tools
Data Mining
MDD/OLAP
Relational
Systems/ERP
Analytical
Applications
Data
explosion
Need more
active data
Legacy Systems
Transform
External Data
Sources
Metadata
Data Marts
(the data warehouse)

23. 23
Benefits of Hadoop and NoSQL
NoSQL
ERP
DBMS
/EDW
© Talend 2014
Data
explosion
Batch to
Real-Time
Longer
active data
IOT
NoSQL
Web
Logs
Data Marts
(the data warehouse)
Legacy
Systems
Standard
Reports
Ad-hoc
Query Tools
Data
Mining
MDD/OLAP
Analytical
Applications

24. 24
Top Big Data Challenges
© Talend 2014
Source: Gartner - Survey Analysis: Big Data Adoption in 2013 Shows Substance Behind
the Hype - 12 September 2013 - G00255160
“How To”
Challenges

25. 25
Big Data Integration Capabilities
© Talend 2014

26. 26
Top Big Data Challenges
© Talend 2014
Need Solutions that
Address these
Challenges
Source: Gartner - Survey Analysis: Big Data Adoption in 2013 Shows Substance
Behind the Hype - 12 September 2013 - G00255160

27. 27
Convergence, Big Data & Consumerization
• Next-gen integration platforms need to be designed
& architected with big data requirements in mind
© Talend 2014
ETL / ELT
Parallelization
Processing needs to be
distributed & flexible
Big data technologies need
to be integrated seamlessly
with existing integration
investments
RDBMS

28. 28
Big Data Integration Landscape
© Talend 2014

29. 29
“I may say that this is the greatest
factor: the way in which the
expedition is equipped.”
© Talend 2014
Roald Amundsen
race to the south pole, 1911
Source of Roal Amundsen portrait:
Norwegian National Library
© Talend 2014 29

30. 30
Big Data Integration: Ingest – Transform – Deliver
© Talend 2014
iPaaS MDM
HA Govern
Security Meta
Storm Kafka
CXF Camel
STANDARD-IZE
MACHINE
YARN (Cluster Resource Management)
HDFS2 (Redundant, Reliable Storage)
HIVE
BATCH
(MapReduce)
INTERACTIVE
(Tez)
STREAMING
(Storm, Spark)
GRAPH
(Giraph)
NoSQL
(MongoDB)
Events
(Falcon)
ONLINE
(HBase)
OTHER
(Search)
TRANSFORM (Data Refinement)
MAP PROFILE PARSE CLEANSE CDC
LEARNING
MATCH
INGEST
(Ingestion)
SQOOP
FLUME
HDFS API
HBase API
DELIVER
(as an API)
Karaf ActiveMQ

31. 31
Big Data Integration and Processing
© Talend 2014
Analytics Dashboard
LLooaadd ttoo HHDDFFSS
BIG DATA
(Integration)
Federate to
analytics
HDFS Map/Reduce
HADOOP
Data from Various
Source Systems
Hive

32. 32
Important Objectives
• Moving from hand-code to code generation – MapReduce,
Pig, Hive, SQOOP etc. – using a graphical user interface
• Zero footprint on the Hadoop cluster
• Same graphical user interface for both standard data
integration and Big Data integration
© Talend 2014

33. 33
Trying to get from this…
© Talend 2014

34. 34
“pure Hadoop” and MapReduce
© Talend 2014
Visual design in Map Reduce and optimize before
deploying on Hadoop
to this…

35. 35
Native Map/Reduce Jobs
• Create graphical ETL patterns using native Map/Reduce
© Talend 2014
• Reduce the need for big
data coding skills
• Zero pre-installation on
the Hadoop cluster
• Hadoop is the “engine”
for data processing

36. 36
Other Important Objectives
Enables organizations to leverage existing skills such as
Java and other open source languages
A large collaborative community for support
A large number of components for data and applications including big data
and NoSQL
Works directly on Apache Hadoop API
Native support for YARN and Hadoop 2.0 support for better resource
optimization
Software created through open standards and development
processes that eliminates vendor lock-in
Scalability, portability and performance come for “free” due to Hadoop
© Talend 2014
Page 36

37. 37
© Talend 2014
Talend Solution for Big Data Integration

38. 38
Talend’s Solution
© Talend 2014

39. 39
The Value of Talend for Big Data
Leverage In-house Resources
© Talend 2014
- Easy-to-use familiar Eclipse-tools that generate big data code
- 100% standards-based, open source
- Lots of examples with a large collaborative community
Big Data Ready
- Native support for Hadoop, MapReduce, and NoSQL
- 800+ connectors to all data sources
- Built-in data quality, security and governance (Platform for Big Data)
Lower Costs
- A predictable and scalable subscription model
- Based only on users (not CPUs or connectors)
- Free to download, no runtimes to install on your cluster
$

40. 40
Talend’s Value for Big Data
• New frameworks like Spark and Storm are emerging on
Hadoop and can run on other platforms
• Companies want to accelerate big data processing and do
more sophisticated workloads by exploiting in-memory
capabilities via Spark and for analyzing real-time data in
motion via Storm
• Talend can generate Storm applications to analyze and
filter data in real-time as well as use source data filtered
by Storm applications
• Talend can help customers rapidly exploit new Big Data
technologies to reduce time to value while insulating them
from future extensions and advancements
© Talend 2014

41. 41
Thank You For Your Participation
© Talend 2014