Running Hive at Scale @ Lyft talk from May 2018 Hive User Group Meeting @ Cloudera - https://www.meetup.com/Hive-User-Group-Meeting/events/249641278/

1. Running Hive at Scale @ Lyft
Debugging at scale
Lyft Confidential
Rohit Menon
Sharanya Santhanam
5/8/2018

2. Hive @ Lyft
Use cases
1. ETL/ Batch Processing - Converting Raw event data into meaningful fact & dim tables
2. Batch Querying for reports , experimentation results , trending etc
Scale
• Datawarehouse Size 16 PB
• Approx 130,000 Hive Queries/ Day

3. Hive Architecture
Mode/Looker
Superset
HAProxy
HiveServer2
HDFS/YARN
Airflow
HiveMetaStore
RDS
S3Custom Clients
Project Version
Hive 2.1.1
Hadoop 2.6.0
Tez 0.8.4
Cluster
EC2
EC2
EC2
API

4. Prerequisite for Running @ Scale
• Configuration Management & Deployment
‒ Salt
‒ Amazon Spot Instances/Autoscaling/Downscaling
‒ Multiple Clusters for diff workloads
• Monitoring & Logging
‒ Datadog, Wavefront, Grafana, Kibana, Superset
• Regression & Deployment Verification
‒ Benchto (Presto benchmark tool)
• Performance Analysis Tools
‒ Jemalloc
‒ Flame Graphs
‒ Eclipse MAT/ VisualVM

5. Tracking Down a Native Leak
• HS2 Memory keeps growing up to 60GB+ process memory however the heap usage remains low.
Indication of Native leak.

6. Tracking Down a Native Leak
• JVM instrumented with Jemalloc
• An alternative implementation of malloc
suited for high performance environment
with great profiling abilities.
• Pinpoint the root cause to inflate methods
In zlib package.
• Tougher Task: Tying Native Methods to
JVM methods.

7. Tracking Down a Native Leak
• ZipInputStream / JarInputStream is a typical source of
such memory leaks, for example, when a
Class.getResourceAsStream is not closed after use.
• Short Term Fix
‒ Decompress JARs used by process (pmap analysis)
• Long Term Fix
‒ Heap Dump Analysis with Visual VM
for Inward References to java.util.zip.Inflater
‒ Root Cause - Session state leak on close resources