This document discusses techniques for automatically tuning memory configurations for Scalding jobs submitted to Hadoop clusters. It proposes leveraging prior run counters from a history repository to estimate heap sizes on a per-step basis. This aims to set optimal configurations while avoiding memory over-estimation issues. It acknowledges challenges around fallback handling for new jobs and estimation inaccuracies that could lead to underestimation issues. Overall contributions to the Scalding project are welcomed to further develop automated memory tuning capabilities.

2. ●
●
●
●
●
●

5. ●
●
●
●
●
●

6. Event
Ingest
Service
Service A Other
Data
Service B
Feature
Switch
Service
Interaction Events
config
Realtime Bucketing
Counts
RDBMS
resultsDDG
Mobile
devices
https://blog.twitter.com/engineering/en_us/a/2015/twitter-experimentation-technical-overview.html

7. Event
Ingest
Service
Service A Other
Data
Service B
Feature
Switch
Service
Interaction Events
config
Realtime Bucketing
Counts
RDBMS
resultsDDG
Mobile
devices
https://blog.twitter.com/engineering/en_us/a/2015/twitter-experimentation-technical-overview.html

8. ●
●
○
○
○
●
●
○
○
●
●
●
○
○
○
●
●
●

9. ●
●
●
●

10. ●
●

14. ●
●
●
Word Count in Scalding:
TypedPipe.from(TextLine(inputPath))
.flatMap { line => tokenize(line) }
.groupBy { word => word } // word as key
.size // in each group, get the size
.write(TypedText.tsv[(String,
Long)](outputPath))

15. ● Each Scalding flow has multiple steps in a DAG
● Steps are Hadoop MR jobs
● Users typically configure settings per flow
● Each production flow has a unique ID

16. ● Different steps can have different memory pressures
● myPipe.hashJoin(someRHS) // loads someRHS in memory on
every mapper
● myPipe.sumByKey(...) // performs map-side aggregation
● myPipe.groupBy(...).join(pipe2) // end up caching data in
memory during accumulation
● Within same step data skew can change footprint

17. ● Accommodate different steps in same flow with very different
needs
● Users unsure of the right settings - cargo culting settings
● React to oncall pain at 3am - fix and forget
○ Over estimation - MegaBytes grow
○ Underestimation - GC pain - Millis grow
○ End result is MB*Millis keeps growing

19. Submit jobs to Hadoop Cluster
Scalding Job

20. Scalding Job
Submit jobs to Hadoop Cluster
● Store prior runs and counters - HRaven
● Scalding already does Reducer estimation
● Can we leverage this?

21. ● Get N prior run counters - step, map / reduce phases
● Compute max(committed_heap_bytes)
● Exponential smoothing to derive heap sizes
● Derive container size on YARN boundary
● Set configurations per step!

23. ● How do we fallback in case of failures? Unable to look
up history? Memory estimate inaccurate?
● Jobs change all the time
● Underestimation without OOM?
○ GC_Millis / CPU_Millis
● Scalding PR: #1667 - contributions / suggestions
welcome!

24. io.sort.mb
Map-side
Aggregation
buffers
Competing Settings

25. ● Tuning is hard for users!
● Lots of potential resource usage savings
● Automation is great but needs to be fail-safe