The document outlines the architecture and components of the Hortonworks Data Platform, including versioning for various data management tools like Hadoop, Spark, and Hive. It emphasizes the importance of handling data effectively to derive actionable intelligence and insights from both data in motion and at rest. Additionally, it describes the integration of a core Hadoop cluster with various data ingestion and monitoring systems.

1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Modern Application
Architecture using
Data.gov
Devin Pinkston | Solutions Engineer Ian Brooks | Solutions Engineer
Henry Sowell| Technical Director

2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
HORTONWORKS DATA PLATFORM
Hadoop
&YARN
DATA MGMT DATA ACCESS GOVERNANCE & INTEGRATION OPERATIONS SECURITY
HDP 2.2
Dec 2014
HDP 2.2
Dec 2014
2.2.0
2.4.0
2.6.0
2.7.1
HDP 2.3
Oct 2015
2.7.3
HDP 2.6*
1H2017
2.7.1
HDP 2.4
Mar 2016
* HDP 2.6 – Shows current Apache branches being used. Final component version subject to change based on Apache release process.
** Spark 1.6.3+ Spark 2.1 – HDP 2.6 supports both Spark 1.6.3 and Spark 2.1 as GA.
*** Hive 2.1 is GA within HDP 2.6.
**** Apache Solr is available as an add-on product HDP Search.
2.7.3
Sqoop
1.4.4
1.4.5
1.4.4
1.4.6
1.4.6
1.4.6
1.4.6
Druid
0.9.2
Knox
0.4.0
0.5.0
0.6.0
0.11.0
0.6.0
0.9.0
Ranger
0.4.0
0.5.0
0.7.0
0.5.0
0.6.0
Ambari
1.4.4
2.0.0
1.5.1
2.1.0
2.5.0
2.2.1
2.4.0
Kafka
0.8.2
0.8.1
0.10.1.0
0.9.0
0.10.0
Zookeeper
3.4.5
3.4.6
3.4.5
3.4.6
3.4.6
3.4.6
3.4.6
Flume
1.5.2
1.4.0
1.3.1
1.5.2
1.5.2
1.5.2
1.5.2
Solr
4.10.2
4.7.2
5.2.1
5.5.1
****
5.2.1
5.5.1
Slider
0.60.0
0.80.0
0.91.0
0.80.0
0.91.0
Atlas
0.5.0
0.8.0
0.5.0
0.7.0
Accumulo
1.6.1
1.5.1
1.7.0
1.7.0
1.7.0
1.7.0
Phoenix
4.0.0
4.2.0
4.4.0
4.7.0
4.4.0
4.7.0
Storm
0.9.3
0.10.0
0.9.1
1.1.0
0.10.0
1.0.1
Falcon
0.5.0
0.6.0
0.6.1
0.10.0
0.6.1
0.10.0
Tez
0.4.0
0.5.2
0.7.0
0.7.0
0.7.0
0.7.0
Hive
0.12.0
0.13.0
0.14.0
1.2.1
1.2.1+
2.1***
1.2.1
1.2.1+
2.1***
Pig
0.12.0
0.12.1
0.14.0
0.15.0
0.16.0
0.15.0
0.16.0
HDP 2.5
Aug 2016
Oozie
3.3.2
4.1.0
4.0.0
4.2.0
4.2.0
4.2.0
4.2.0
Spark
1.2.1
1.4.1
1.6.3+
2.1**
1.6.0
1.6.2+
2.0**
HBase
0.98.4
0.96.1
0.98.0
1.1.2
1.1.2
1.1.2
1.1.2
Zeppelin
0.7.0
0.6.0
HDP 2.1
April 2014
HDP 2.0
Oct 2013
Ongoing Innovation in Apache

3. 3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
How Do We Handle The Data?
Actionable
Intelligence from
Connected Data
Platforms
Capturing perishable
insights from data in motion
Ensuring rich, historical insights on
data at rest
Necessary for modern data
applications

4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Single Aggregation Site to Hadoop Cluster Architecture
Core Hadoop Cluster
HCatalog: Shared Table & User Defined Metadata for All Workloads
Ambari: Provision, Manage and Monitor Cluster Resources
Stream
Data Access
° ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
° ° ° ° °
° ° ° ° °
1 ° ° ° ° ° ° ° ° ° ° ° ° ° °
HDFS (Hadoop Distributed File System)
YARN (Cluster Resource Management)
Accumulo
Collection Sources
NiFi
NiFi Put Ingest to
Apache Accumulo
NiFi
NiFi
NiFi
Aggregation Site
NiFi
NiFi
NiFi
NiFi
Kafka
Movement Across
Networks
Storm
Content-based
routing/enrichment
Incident
REST API

5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Basic Accumulo Incident Structure
Row <incident_epoch>
Column Family “Incident” “Geo” ”temporal”
Column Qualifier <cat>, <descript>,
<pdDistrict>,
<incidentNum>, <Pdid>
<address>, <x>, <y>,
<location>
<dayOfWeek>, <date>,
<time>
Value <value> <value> <value>