Hortonworks is a company that was founded in 2011 to focus on developing and supporting Apache Hadoop for enterprise use. They distribute the Hortonworks Data Platform (HDP), which is the only 100% open source enterprise Hadoop distribution. HDP includes core Hadoop components like HDFS, YARN, MapReduce as well as data services like Hive, Pig, HBase. It also includes operational services to help manage and monitor large Hadoop clusters.

1. © Hortonworks Inc. 2011
Hadoop Introduction
Olivier Renault
Solution Engineer - Hortonworks

2. © Hortonworks Inc. 2011
Hortonworks

3. © Hortonworks Inc. 2011
A Brief History of Apache Hadoop
Page 3
2013
Focus on INNOVATION
2005: Yahoo! creates
team under E14 to
work on Hadoop
Focus on OPERATIONS
2008: Yahoo team extends focus to
operations to support multiple
projects & growing clusters
Yahoo! begins to
Operate at scale
Enterprise
Hadoop
Apache Project
Established
Hortonworks
Data Platform
2004 2008 2010 20122006
STABILITY
2011: Hortonworks created to focus on
“Enterprise Hadoop“. Starts with 24
key Hadoop engineers from Yahoo

4. © Hortonworks Inc. 2011
Hortonworks Snapshot
Page 4
• We distribute the only 100%
Open Source Enterprise
Hadoop Distribution:
Hortonworks Data
Platform
• We engineer, test & certify
HDP for enterprise usage
• We employ the core
architects, builders and
operators of Apache Hadoop
• We drive innovation within
Apache Software
Foundation projects
• We are uniquely positioned
to deliver the highest quality
of Hadoop support
• We enable the ecosystem to
work better with Hadoop
DevelopDevelop Distribute Support
We develop, distribute and support
the ONLY 100% open source
Enterprise Hadoop distribution
Endorsed by Strategic Partners
Headquarters: Palo Alto, CA
Employees: 180+ and growing
Investors: Benchmark, Index, Yahoo

5. © Hortonworks Inc. 2011
Leadership that Starts at the Core
Page 5
• Driving next generation Hadoop
– YARN, MapReduce2, HDFS2, High
Availability, Disaster Recovery
• 420k+ lines authored since 2006
– More than twice nearest contributor
• Deeply integrating w/ecosystem
– Enabling new deployment platforms
– (ex. Windows & Azure, Linux & VMware HA)
– Creating deeply engineered solutions
– (ex. Teradata big data appliance)
• All Apache, NO holdbacks
– 100% of code contributed to Apache

6. © Hortonworks Inc. 2011
OS Cloud VM Appliance
HDP: Enterprise Hadoop Distribution
Page 6
HORTONWORKS
DATA PLATFORM (HDP)
Hortonworks
Data Platform (HDP)
Enterprise Hadoop
• The ONLY 100% open source
and complete distribution
• Enterprise grade, proven and
tested at scale
• Ecosystem endorsed to
ensure interoperability
PLATFORM SERVICES
HADOOP CORE
Enterprise Readiness: HA,
DR, Snapshots, Security, …
Distributed
Storage & ProcessingHDFS YARN (in 2.0)
WEBHDFS MAP REDUCE
DATA
SERVICES
Store,
Process and
Access Data
HCATALOG
HIVEPIG
HBASE
SQOOP
FLUME
OPERATIONAL
SERVICES
Manage &
Operate at
Scale
OOZIE
AMBARI

7. © Hortonworks Inc. 2011
Overview of Hadoop

8. © Hortonworks Inc. 2011
In the Beginning
• It all started when Google needed a way to:
– Do page ranking
– Determine which web sites to provide for searches
Link Link

9. © Hortonworks Inc. 2011
Page Rank Solution - Simplified
• Google engineers developed an internal solution and provided a paper
on it titled:
“MapReduce: Simplified Data Processing on Large
Clusters”
• It described a process something like this:
Map
Map
1. Many tasks look
at links in parts
of the data
2. Mapped results are
shuffled to Reducers
3. Reducers compute
the links into a result
Links to sites A, C, F
Links to sites B, D, E
1 2 3
Map
Reduce
Reduce

10. © Hortonworks Inc. 2011
Words to Websites - Simplified
• From words provide locations
• Provides what to display for a search
– Note: Page rank determines the order
• For example – to find URLs with books on them
books www.barnesandnoble.com www.amazon.com
email www.google.com www.yahoo.com www.facebook.com
finance www.yahoo.com www.google.com
groceries www.walmart.com www.target.com
jeans www.target.com www.amazon.com
K, V
<url, keyword>
<keyword, url>
www.barnesandnoble.com books calendars
www.yahoo.com sports finance email celebrity
www.amazon.com shoes books jeans
www.google.com finance email search
www.microsoft.com operating-system productivity system
Map
Reduce

11. © Hortonworks Inc. 2011
Data Model
• MapReduce works on <key, value> pairs
(Key input, Value input)
(Key intermediate, Value intermediate)
(Key output, Value output)
Map
Reduce
(books, www.barnesandnoble.com www.amazon.com)
(books, www.barnesandnoble.com)
(www.barnesandnoble.com , books calendars)

12. © Hortonworks Inc. 2011
Hadoop Basic Core Architecture
Hadoop Distributed File System (HDFS)
Map
Reduce
Mapper Reducer
Shuffle
MapReduce

13. © Hortonworks Inc. 2011
HDFS & MapReduce
Enterprise Apache Hadoop
Page 13

14. © Hortonworks Inc. 2011
Hortonworks
• Cluster Topology
• HDFS
• MapReduce
Page 14

15. © Hortonworks Inc. 2011
Cluster Topology
Page 15
Master Services Slave Services

16. © Hortonworks Inc. 2011
Hortonworks
• Cluster Topology
• HDFS
• MapReduce
Page 16

17. © Hortonworks Inc. 2011
OS Cloud VM Appliance
HDP: Enterprise Hadoop Distribution
Page 17
HORTONWORKS
DATA PLATFORM (HDP)
Hortonworks
Data Platform (HDP)
Enterprise Hadoop
• The ONLY 100% open source
and complete distribution
• Enterprise grade, proven and
tested at scale
• Ecosystem endorsed to
ensure interoperability
PLATFORM SERVICES
HADOOP CORE
Enterprise Readiness: HA,
DR, Snapshots, Security, …
Distributed
Storage & ProcessingHDFS YARN (in 2.0)
WEBHDFS MAP REDUCE
DATA
SERVICES
Store,
Process and
Access Data
HCATALOG
HIVEPIG
HBASE
SQOOP
FLUME
OPERATIONAL
SERVICES
Manage &
Operate at
Scale
OOZIE
AMBARI

18. © Hortonworks Inc. 2011
HDFS
• Distributed file system designed to run on commodity Hardware.
• Key Assumptions
– Hardware failure is the norm
– Need streaming access to data sets.
– Optimized for high throughput
– Data sets are large
– Append only file system. Write once read many times
– Moving computation is cheaper than moving data
Page 18

19. © Hortonworks Inc. 2011
HDFS: Key Services
• NameNode
– Master service
– Manages the file system namespace and regulates access to files by clients
– Single service across the cluster
• DataNode
– Slave service. Runs on slave nodes
– Manages block read/write for HDFS
– Pings NameNode for instructions
– If heatbest fails, Datanode is removed from the cluster and replicated blocks take
over
• Seconday NameNode
– Merges Namenode’s file system image and edit logs
– Not a failover namenode !!!
Page 19

20. © Hortonworks Inc. 2011
HDFS: File create lifecycle
Page 20
NameNode
RACK1
RACK2
RACK3
FILEFILE
B
1
B
1
B
2
B
2
FILE
HDFS CLIENT
Create
B
1
B
1
B
1
B
1
B
1
B
1
B
2
B
2
B
2
B
2
B
2
B
2
ack
ack
ack
22 11
33
Complete44

21. © Hortonworks Inc. 2011
Hortonworks
• Cluster Topology
• HDFS
• MapReduce
Page 21

22. © Hortonworks Inc. 2011
OS Cloud VM Appliance
HDP: Enterprise Hadoop Distribution
Page 22
HORTONWORKS
DATA PLATFORM (HDP)
Hortonworks
Data Platform (HDP)
Enterprise Hadoop
• The ONLY 100% open source
and complete distribution
• Enterprise grade, proven and
tested at scale
• Ecosystem endorsed to
ensure interoperability
PLATFORM SERVICES
HADOOP CORE
Enterprise Readiness: HA,
DR, Snapshots, Security, …
Distributed
Storage & ProcessingHDFS YARN (in 2.0)
WEBHDFS MAP REDUCE
DATA
SERVICES
Store,
Process and
Access Data
HCATALOG
HIVEPIG
HBASE
SQOOP
FLUME
OPERATIONAL
SERVICES
Manage &
Operate at
Scale
OOZIE
AMBARI

23. © Hortonworks Inc. 2011
MapReduce
• A software framework for developing distributed applications to
process vast amounts of data in-parallel on large clusters
• MapReduce job splits the input data-set into independent chunks which
are processed by the map tasks in a completely parallel manner across
a cluster of nodes
Page 23

24. © Hortonworks Inc. 2011
MapReduce: Key Services
• Job Tracker
– Master service
– Schedule’s jobs’ component task on the task tracker
– Monitors task progress
– Reschedules failed tasks
• Task Tracker
– Spawn’s Job’s task
– Reports progress to the Job Tracker
– Runs on slave nodes, collocated with DataNode service
• Task (Map / Reduce)
– Spawned by Task Tracker
– Executes Map/Reduce task, encapsulating the business logic.
Page 24

25. © Hortonworks Inc. 2012 Page 25
DataNode 1
Mapper
Data is shuffled
across the network
and sorted
Map Phase Shuffle/Sort Reduce Phase
DataNode 2
Mapper
DataNode 3
Mapper
DataNode 1
Reducer
DataNode 2
DataNode 3
Reducer
MapReduce: Job Lifecycle

26. © Hortonworks Inc. 2011
The Key/Value Pairs of MapReduce
<K1, V1>
Mapper Shuffle/Sort<K2, V2>
Reducer
<K2, (V2,V2,V2,V2)>
<K3, V3>
• Map & Reduce operate on (key, value) pairs and output (key, value) pairs• Map & Reduce operate on (key, value) pairs and output (key, value) pairs
• User provides map and reduce functions
• Input Key and Value is determined by InputFormat
• Common InputFormats: TextInputFormat,
KeyValueTextInputFormat,SequenceFileInputFormat
• Common OutputFormats: TextOutputFormat, SequenceFileOutputFormat

27. © Hortonworks Inc. 2011
PIG, HIVE
Enterprise Apache Hadoop
Page 27

28. © Hortonworks Inc. 2011
Hortonworks
• Pig
• Hive
Page 28

29. © Hortonworks Inc. 2011
OS Cloud VM Appliance
HDP: Enterprise Hadoop Distribution
Page 29
HORTONWORKS
DATA PLATFORM (HDP)
Hortonworks
Data Platform (HDP)
Enterprise Hadoop
• The ONLY 100% open source
and complete distribution
• Enterprise grade, proven and
tested at scale
• Ecosystem endorsed to
ensure interoperability
PLATFORM SERVICES
HADOOP CORE
Enterprise Readiness: HA,
DR, Snapshots, Security, …
Distributed
Storage & ProcessingHDFS YARN (in 2.0)
WEBHDFS MAP REDUCE
DATA
SERVICES
Store,
Process and
Access Data
HCATALOG
HIVEPIG
HBASE
SQOOP
FLUME
OPERATIONAL
SERVICES
Manage &
Operate at
Scale
OOZIE
AMBARI

30. © Hortonworks Inc. 2011
Pig
•An engine for executing programs on top of
Hadoop
•It provides a language, Pig Latin, to specify these
programs
Page 30

31. © Hortonworks Inc. 2011
Why use Pig?
• Suppose you have user data in one file, website data in another, and
you need to find the top 5 most visited sites by users aged 18 - 25
Page 31

32. © Hortonworks Inc. 2011
In Map-Reduce
Page 32
170 lines of code, 4 hours to write

33. © Hortonworks Inc. 2011
In Pig Latin
Users = load ‘input/users’ using PigStorage(‘,’) as (name:chararray,
age:int);
Fltrd = filter Users by age >= 18 and age <= 25;
Pages = load ‘input/pages’ using PigStorage(‘,’) as (user:chararray,
url:chararray);
Jnd = join Fltrd by name, Pages by user;
Grpd = group Jnd by url;
Smmd = foreach Grpd generate group,COUNT(Jnd) as clicks;
Srtd = order Smmd by clicks desc;
Top5 = limit Srtd 5;
store Top5 into ‘output/top5sites’ using PigStorage(‘,’);
Page 33
9 lines of code, 15 minutes to write

34. © Hortonworks Inc. 2011
Essence of Pig
• Map-Reduce is too low a level, SQL too high
• Pig-Latin, a language intended to sit between the two
– Provides standard relational transforms (join, sort, etc.)
– Schemas are optional, used when available, can be defined at runtime
– User Defined Functions are first class citizens
Page 34

35. © Hortonworks Inc. 2011
Pig Elements
Page 35
• High-level scripting language
• Requires no metadata or schema
• Statements translated into a series
of MapReduce jobs
Pig Latin
• Interactive shellGrunt
• Shared repository for User Defined
Functions (UDFs)Piggybank

36. © Hortonworks Inc. 2011
Hortonworks
• Pig
• Hive
Page 36

37. © Hortonworks Inc. 2011
OS Cloud VM Appliance
HDP: Enterprise Hadoop Distribution
Page 37
HORTONWORKS
DATA PLATFORM (HDP)
Hortonworks
Data Platform (HDP)
Enterprise Hadoop
• The ONLY 100% open source
and complete distribution
• Enterprise grade, proven and
tested at scale
• Ecosystem endorsed to
ensure interoperability
PLATFORM SERVICES
HADOOP CORE
Enterprise Readiness: HA,
DR, Snapshots, Security, …
Distributed
Storage & ProcessingHDFS YARN (in 2.0)
WEBHDFS MAP REDUCE
DATA
SERVICES
Store,
Process and
Access Data
HCATALOG
HIVEPIG
HBASE
SQOOP
FLUME
OPERATIONAL
SERVICES
Manage &
Operate at
Scale
OOZIE
AMBARI

38. © Hortonworks Inc. 2011
Motivation
• Hadoop as Enterprise Data Warehouse
• Adhoc query support
• Schema information
• Tool for end-users
Page 38
USED EXTENSIVELY FOR ANALYTICS & BUSINESS INTELLIGENCEUSED EXTENSIVELY FOR ANALYTICS & BUSINESS INTELLIGENCE

39. © Hortonworks Inc. 2011
HiveQL Features
• HiveQL is similar to other SQLs
– Uses familiar relational database concepts
(tables, rows, columns and schema)
– Based on the SQL-92 specification
• Treats Big Data as tables
• Converts SQL queries into MapReduce jobs
– User does not need to know MapReduce
• Also supports plugging custom MapReduce scripts into queries
Page 39

40. © Hortonworks Inc. 2011
Performing Queries
• SELECT
• WHERE clause
• UNION ALL and DISTINCT
• GROUP BY and HAVING
• JOIN
• ORDER BY
• LIMIT clause
– Rows returned are chosen at random
• Can use REGEX Column Specification
Page 40
SELECTSELECT '(ds|hr)?+.+' FROM sales;

41. © Hortonworks Inc. 2011
Hive vs Pig
Page 41
Pig and Hive work well togetherPig and Hive work well together
and many businesses use both.
Hive is a good choice:
• when you want to query the data
• when you need an answer to specific
questions
• if you are familiar with SQL
Hive is a good choice:
• when you want to query the data
• when you need an answer to specific
questions
• if you are familiar with SQL
Pig is a good choice:
• for ETL (Extract -> Transform -> Load)
• for preparing data for easier analysis
• when you have a long series of steps to
perform

42. © Hortonworks Inc. 2011
Ambari
Streamlining Hadoop Operations
Page 42

43. © Hortonworks Inc. 2011
Ambari: Install, Manage, Monitor, Tune
Simplify Deployment and Maintenance: Wizard
based install, handles dependency checks,
recommends service mappings
Ensure a Healthy Cluster: Monitor, alert, heat maps
Optimize Performance: Root cause analysis for
cluster tuning - fix problems BEFORE SLAs are
breached
Integrate with your operations: Open APIs,
standard web-tech
43

44. © Hortonworks Inc. 2011
Community-Driven, Enterprise Class
• Productizes over a combined 100
person-years of operational Hadoop
experience
• Stability and Scale: Ops & Dev team
that took Yahoo! From 1000 to 45,000+
nodes
• Fast-paced, open source community
driven innovation and integration Red
Hat, Teradata, HP, Microsoft
Contributions (and more)
44

45. © Hortonworks Inc. 2011
Demonstration
Page 45