Spark Summit 2015 talk

4. Client Data
Fluency
Office
Skype
Bing
Modern Data
Capability
Instrumentation & Ingestion
Processing & Storage
Reporting & Analytics
Information Management
Mobile-First Analytics Experience
Experimentation

7. Data Size
Query
Latency

9. Get results inline
in Zeppelin
Need to open the
results in Excel

10. 0 20 40 60 80 100 120 140 160 180 200
Cosmos
SparkSQL
SparkSQL with Cache
Write and Compile Query Submit and Wait in Job Queue Job Run Time

14. Mesos Cluster/HDFS
Job Manager Zookeeper
Job Frontend Web API
Spark Driver Host Pool
Spark Hive Thrift ServerZeppelin Server
Avocado
(Hive Query + Schedule Task)
Rover
(Drag & Drop
BI tool with
Hive Code Gen)
Zeppelin Web UI
MetastoreDB Hive Loader
Cosmos Storage

15. Partition 1
Partition 2
...
Partition n
Export Cosmos
Partition
Partition 1
Partition 2
...
Partition n
Task 2
HDFS.copyFrom
LocalFile
...
Task n
Partition 1
Partition 2
...
Partition n
saveAsParquetFile
Task 2
...
Task n

16. <Database2>
<Table1>
<Database1>
<Partition1>
<Table2>
<Partition2>
MetastoreDB
Hive Thrift Server
Hive Loader
Zeppelin Server
UserQuery
Query

22. Data
Ingest
Services
Clients
Transform
Compute
Transform
Compute
Data
Streams
Data
Sets
Store
Event
Processing
HDFSData
Transportation
Spark
Streaming
Receiver
Analyst
Zeppelin
Notebooks
Avocado
Simple
query
Query
language
“Analyze”
“Debug”
“Mine”
“Glance”
Data
Unified
platform
Intelligence
Interactive
analytics
Data Products
Better
Digital
Experiences
Dual users
“Bing”
“Office”