1) Jaws is a highly scalable and resilient data warehouse explorer that allows submitting Spark SQL queries concurrently and asynchronously through a RESTful API. 2) It provides features like persisted query logs, results pagination, and pluggable storage layers. Queries can be run on Spark SQL contexts configured to use data from HDFS, Cassandra, Parquet files on HDFS or Tachyon. 3) The architecture allows Jaws to scale on standalone, Mesos, or YARN clusters by distributing queries across multiple worker nodes, and supports canceling running queries.

1. Ema Orhian
@emaorhian
Jaws - Data Warehouse with
Spark SQL

2. 2
• Big Data analytics / Machine Learning
• 4+ years exp with Hadoop ecosystem
• 2 years exp with Spark
About me
http://bigdataresearch.io/
• Co-founder of Big Data Research Group
• Provides open source solutions around Big Data
analytics
http://atigeo.com/

3. Agenda
• jaws-spark-sql-rest (Jaws) intro
• Main features
• Architecture
• Scaling
• Resource manager
• Working with Tachyon
• Working with Parquet files
• Configure Spark Sql context
• Demo 3

4. 4
Shared
Spark Sql
Context
Concurrent
queries run
Query history
Page results
Query editor

5. Jaws
• Highly scalable and resilient data warehouse explorer
• Restful alternative to Spark SQL JDBC and not only …
• Support for Spark 0.9.1/Shark thru Spark 1.5
• Support for hive/MR
https://github.com/atigeo/jaws-spark-sql-rest
5

6. Main features
• Submit queries concurrently and asynchronously
• Provides persisted logs, query history, results with paging
• Pluggable persistent layer (Cassandra/HDFS)
• Supports load balancing with query cancelation
• Provides a metadata browser
• In-memory Parquet warehouse with Tachyon
• Configuration file to fine tune Spark context
• Pluggable UI 6

7. Jaws architecture
7

8. Scaling
8
•Standalone mode
•Mesos
•YARN
Fine grained mode
Coarse grained mode

9. 9
Canceling a query

10. 10
Canceling a query

11. Results persistence
• Queries with limited number of results:
‣ Cassandra
‣ HDFS
• Queries with unlimited number of results:
‣ HDFS
‣ Tachyon
11

12. Working with Tachyon
• Persists unlimited results in Tachyon
• Registers tables over Parquet files from Tachyon
12
Tachyon benefits:
★ in memory storage system
★ share data between applications at a memory
speed

13. Working with Parquet files
• Register tables on top of parquet files
13
Parquet
★ columnar format
★ nested data structures
★ supports schema evolution
★ efficient compression
• Files stored on HDFS or Tachyon
• MetaInfo about table stored in Cassandra (feature before Spark
1.3)

14. Configuring Jaws
14
• Cassandra
• HDFS
• Spray
• Application
• Spark
sparkConfiguration {
spark-master=“spark://devbox.local:7077”
/ “mesos://devbox.local:5050”
/ yarn-client
spark-mesos-coarse=false / true
spark-cores-max=100
spark-executor-instances=10
}

15. Demo
15