Companies analyzing big data help achieve important business objectives such as customer retention, real-time in-context marketing, omni-channel marketing productivity, campaign productivity and operational efficiencies. Cloud-based big data architectures create lower risk, lower startup costs and faster time-to-market. This session will examine the key advantages from deploying big data in the cloud, such as the flexibility to auto scale and the ability to experiment with on-demand and hybrid nodes. We will also discuss lessons learned from big data in the cloud, such as how to avoid bottlenecks by building caches or how to design instances to leverage spotting.

1. Scaling Big Data in the Cloud: Lessons
Learned
Big Data Day LA 2015
Minesh Patel, Solutions Architect @ Qubole (mpatel@qubole.com)
Use case presentation by Piero Cinquegrana, MarketShare

2. Agenda
• - Intro
• - Separation of Compute and Storage...advantages of the cloud object store:
• + built-in reliability
• + ephemeral compute
• -> take full advantage of cloud elasticity (including: auto scaling up and down)
• -> infinite scale (kind of)
• -> experiment and react with ease
• + Ephemeral HDFS + EBS
• + Aligned with cloud pricing
• - Dealing with cloud object store performance and quirks
• + parallel reads / writes
• + caching (to reduce I/O)
• + columnar storage formats: ORC / Parquet
• - Spot in AWS
• - managing costs in an elastic model
• - Elastic Spark clusters (I have this in here because Marketshare will be covering their use case after my talk..so it's kind of unatural, but
will make it work out as a transition)
3

3. Qubole Intro
• Big Data as a Service
• Founded: Late 2011
• Self Managed Hadoop, Yarn, Spark, Hive, Presto, Pig
• Optimized for cloud
• Auto-scaling clusters
• Built-in Work Bench (UI / API)
• Runs on AWS, GCE, Azure
• Optimized platform for Analytics and Infrastructure teams
4

4. Cloud Computing Advantages
Key Cloud Advantages:
• Usability
• Elasticity
• Lower Risk / Cost
• Performance
5

5. Cloud Object Store
Separation of Compute and Data
• Store data on the cloud object store
• Out-of-the-box reliability and availability
• Infinite elastic scalability of storage
6

6. Ephemeral Compute
Ephemeral Compute
• Take full advantage of cloud elasticity
• Scale up and down on the fly
• One size does not fit all
• Experiment and react easily
• 0 down time rolling upgrades
7

7. Pricing Model
Alignment with Cloud Pricing
• Storage is priced per unit of storage
• Compute is priced by time slice
8

8. S3 Performance
What about data locality and performance?
• Well suited for Parallel reads -> more data = more nodes and
threads
• Split Computation
– Don’t use the standard HDFS file listing API
– S3 uses a lexicographic order: take advantage
– Example of 2 years of data, partitioned hourly = 17520 directories
if each directory has 6 files = 105,120 files
9

9. S3 Performance
• Caches
– Columnar caches for json
– Data caches
– Meta data caches
• Pre-fetch files from S3
• Zero-Copy writes (directfileoutputcommitter)
10

10. Ephemeral Compute
Auto Scaling Compute Clusters:
• [todo: add graphic]
11

11. Ephemeral Compute
One Size Does not Fit All
• Mixed compute environment for various workloads
• Experiment with ease
• React with 0 down time rolling upgrades
12

12. SSD Instances
New Gen EC2 instance types:
• AWS now has SSD backed instance types
• Provide a better compute / cost ratio
• Downside: SSDs are often too small for big data workloads!
• Solution: Automatically attached EBS Volumes!
13
Map-Reduce HDFS
Local
SSD
Network Drives

13. Spot Instances
Spot Instances
• Variable configuration
• Custom Block Placement
• Reliable Spot utilization
• [TODO: Add graphic]
14

14. Cost
Managing Cost in an Elastic Model
• Cluster Policy: Min/Max
• Reserved & Spot
• Dashboards and Usage Reports!
• [TODO: Add graphic]
15

15. Spark
Spark as a Service
• Qubole provides Spark as a service
• Auto Scaling Spark Clusters
• Self Service Cluster Setup
• Zeppelin based Notebook
16

16. Spark Use Case
Spark Use Case – Marketshare
17