An Increasing number of Apache Spark deployments are in Cloud and hybrid environments. This often means that Spark workloads are ephemeral but the data exists in a durable storage either in cloud and on-prem. The data also moves between cloud storage and on-prem. With this architecture in place, security and governance have become paramount to run Spark workloads across on-prem and cloud. In this keynote, we will walk through several issues and highlight a Spark workload running in an ephemeral cluster with security and governance across Cloud/On-Prem and how the same security and governance is shared with other workloads.

1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Arun C. Murthy
Hortonworks Co-Founder
@acmurthy

2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Data Center Cloud
Modern
Data
Applications

3. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Exceptions that
Drive Predictive
Maintenance
360 View of
Operations,
Equipment Failure
Analytics, Recalls,
etc.
Deep Historical
Analysis
DATA CEN TER
Stream Analytics
Mfg Line
Data
Connected City,
Infrastructure, and
Devices
Machine
Learning
CLOUD Sensors,
Infotainment,
Control Systems
Edge
Analytics
Time –based Route
Optimization
Connected Data Powers Modern Data Applications

4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Prescient Traveler Is a Modern Data Application
“We know that
when we define a
high-threat area in
a given area of the
world, that it is
underpinned by
very specific data
sources. It’s data-
driven, and we can
point to those
sources—if ever
asked—and say,
‘Here’s why.’”
Mike Bishop, Chief
Systems Architect,
Prescient

6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Prescient Harnesses Machine Learning for Traveler Safety Warnings
S ITUATION
Could only produce one
assessment every 3-4 days
Performs risk
management
Uses humans to
identify false positives
Needed efficient way to
store raw data for analytics
49,500 Data
Sources
700% Productivity
Improvement
5 Petabytes of
Data
Hybrid
Architecture
ingested by HDF
into HDP
for geospatial
analysts
stored in HDP
connected EMC
HDF connects
data center to
cloud
ETL
OFFLOAD
Sensor Data
Ingest
DATA
DISCOVERY
Threat
Assessments
SINGLE
VIEW
Global
Threat Map
P REDICTIVE
AN ALYTICS
Threat-Proximity Mobile Alerts
ACTIVE
ARCHIVE
Streaming
Threat Archive
DATA
ENRICHMENT
Provenance
Metadata
“We know that when we define a high-threat area in a given area of the world, that it is underpinned by
very specific data sources. It’s data-driven, and we can point to those sources—if ever asked—and say,
‘Here’s why.’” Mike Bishop, Chief Systems Architect

7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Symantec: Data Science Speeds Time to Cyber Security Protection
From 4 hours
to 2-Seconds
5000x
10s of PBS
Cloud
Flexibility
threat detection
latency
improved
time-to-protection
of historical data for
machine learning
to meet peak
demand for analysis
S ITUATION
3-4 hr processing latencies
to analyze digital threats
Network has +57M attack
sensors in 157 countries
Data streams from 75M
users on 120M devices
Long open windows of
exposure to cyber attacks
DATA
DISCOVERY
Security
Logs
DATA
DISCOVERY
Threat
Detection
DATA
DISCOVERY
Attacker
Identification
PREDICTIVE
ANALYTICS
Unified
Security
PREDICTIVE
ANALYTICS
Protective
Safeguard
ETL
OFFLOAD
Device Data
Ingest
ACTIVE
ARCHIVE
Threat
Archive
ETL
OFFLOAD
Greenplum
Offload
ETL
OFFLOAD
Metadata
Capture
P REDICTIVE
AN ALYTICS
Threat Predictions
“On any given day, we’ll be processing 40 billion messages into our system…It used to be that queues would
back up. We would see times to analysis on the order of 4 hours. On average, we’ve gotten that down to
two or two and a half seconds.” David Lin, Senior Director of Engineering, Symantec Cloud Platform

8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Progressive: Reward Safe Drivers and Improve Traffic Safety
ETL
OFFLOAD
Sensor Data
Ingest
DATA
DISCOVERY
Web Log
Analysis
ACTIVE
ARCHIVE
Individual
Driving Histories
100% in
2-3 Days
+12 Billion
Web App-
Enabled
$2.6 Billion
driving detail captured
from Snapshot, in HDF
miles driven stored
customers see driving
detail and improve safety
in 2014 Premiums
Existing Data Systems Did
Not Scale Efficiently
Usage-Based “Snapshot”
Insurance Program
In-Car Sensor
Captures IoT Data
~7 Days to Transform
Only 25% of UBI Data
S ITUATION
DATA
DISCOVERY
Online Ad
Placement
DATA
DISCOVERY
Claim Notes
Mining
P REDICTIVE
AN ALYTICS
Usage-Based Insurance (UBI)
“We’re looking at datasets that we never dreamed we could look at…It’s joining dots that in the past we
didn’t even know we could join.” Pawan Divakarla, Data & Analytics Business Leader

9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
HDC Shared Services: Secure & Govern all Modern Data Applications
Centralized Management
• Data in Motion or at Rest
• Across Data Center and
Cloud
• Manage end-to-end data
lifecycle
Cloud
Data Center
Security
Governance
Data Lifecycle
Shared Services

10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Demo

11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved

12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Thank You