This document discusses auto-scaling in Google Cloud Dataflow. It describes how Dataflow pipelines can automatically adjust parallelism levels based on throughput, backlog growth, and CPU utilization signals. The scaling policy aims to keep pipelines keeping up with input rates and reducing backlogs quickly. The mechanism for changing parallelism involves splitting computation ranges across additional machines or migrating ranges between machines. Future work may include finer-grained range splitting and approximate throughput modeling.

1. Software Engineer
Google
Manuel Fahndrich
Streaming Auto-Scaling in
Google Cloud Dataflow

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/google-cloud-dataflow

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. https://commons.wikimedia.org/wiki/File:Globe_centered_in_the_Atlantic_Ocean_(green_and_grey_globe_scheme).svg
Addictive
Mobile
Game

5. 1,251,965
1,019,341
989,673
151,365
109,903
98,736
Team RankingIndividual Ranking
Sarah
Joe
Milo
Hourly Ranking Daily Ranking

6. An Unbounded Stream of Game Events
9:008:00 14:0013:0012:0011:0010:002:001:00 7:006:005:004:003:00

7. … with unknown delays.
9:008:00 14:0013:0012:0011:0010:008:00
8:00
8:00
8:00

8. The Resource Allocation Problem
time
workload
over-provisioned
resources
time
workload
under-provisioned
resources
resources
resources

9. Matching Resources to Workload
time
workload
auto-tuned
resources
resources

10. Resources = Parallelism
time
workload
auto-tuned
parallelism
parallelism
More generally: VMs (including CPU, RAM, network, IO).

11. Assumptions
Big Data Problem
Embarrassingly Parallel
Scaling VMs ==> Scales Throughput
Horizontal Scaling

12. Agenda
Streaming Dataflow Pipelines
Pipeline Execution
Adjusting Parallelism Automatically
Summary + Future Work
1
2
3
4

13. Streaming Dataflow1

14. 20122002 2004 2006 2008 2010
MapReduce
GFS Big Table
Dremel
Pregel
FlumeJava
Colossus
Spanner
2014
MillWheel
Dataflow
2016
Google’s Data-Related Systems

15. Google Dataflow SDK
Open Source SDK used to construct a Dataflow pipeline.
(Now Incubating as Apache Beam)

16. Computing Team Scores
// Collection of raw log lines
PCollection<String> raw = ...;
// Element-wise transformation into team/score
// pairs
PCollection<KV<String, Integer>> input =
raw.apply(ParDo.of(new ParseFn()))
// Composite transformation containing an
// aggregation
PCollection<KV<String, Integer>> output = input
.apply(Window.into(FixedWindows.of(Minutes(60))))
.apply(Sum.integersPerKey());

17. Google Cloud Dataflow
● Given code in Dataflow (incubating as Apache Beam)
SDK...
● Pipelines can run…
○ On your development machine
○ On the Dataflow Service on Google Cloud Platform
○ On third party environments like Spark or Flink.

18. Cloud Dataflow
A fully-managed cloud service and
programming model for batch and
streaming big data processing.
Google Cloud Dataflow

19. Google Cloud Dataflow
Optimize
Schedule
GCS GCS

20. time
workload
auto-tuned
parallelism
parallelism
Back to the Problem at Hand

21. Signals measuring Workload
Policy making Decisions
Mechanism actuating Change
Auto-Tuning Ingredients

22. Pipeline Execution2

23. S0
S2
S1
Optimized Pipeline = DAG of Stages
raw input
Individual
points
team
points

24. S0
S2
S1
Stage Throughput Measure
raw input
Individual
points
team
points
throughput
throughput
throughput

25. Picture by Alexandre Duret-Lutz, Creative Commons 2.0 Generic

26. S0
S2
S1
Queues of Data Ready for Processing
Queue Size = Backlog

27. vs.
Backlog Growth
Backlog Size

28. Backlog Growth
=
Processing Deficit

29. S1
Derived Signal: Stage Input Rate
throughput
Input Rate = Throughput + Backlog Growth
backlog growth

30. Constant Backlog...
...could be bad

31. Backlog Time =
Backlog Size
Throughput

32. Backlog Time =
Time to get through backlog

33. Bad Backlog = Long Backlog Time

34. Backlog Growth
and
Backlog Time
Inform Upscaling.
What Signals indicate
Downscaling?

35. Low CPU Utilization

36. Throughput
Backlog growth
Backlog time
CPU utilization
Signals Summary

37. Goals:
1. No backlog growth
2. Short backlog time
3. Reasonable CPU utilization
Policy: making Decisions

38. Upscaling Policy: Keeping Up
Given M machines
For a stage, given:
average stage throughput T
average positive backlog growth G of stage
Machines needed for stage to keep up:
(T + G)
T
M’ = M

39. Upscaling Policy: Catching Up
Given M machines
Given R (time to reduce backlog)
For a stage, given:
average backlog time B
Extra machines to remove backlog:
B
R
Extra = M

40. Upscaling Policy: All Stages
Want all stages to:
1. keep up
2. have log backlog time
Pick Maximum over all stages of M’ + Extra

41. Example (signals)
input rate
throughput
backlog
growth
backlog
time
MB/s
seconds

42. Example (signals)
input rate
throughput
backlog
growth
backlog
time
MB/s
seconds

43. Example (signals)
input rate
throughput
backlog
growth
backlog
time
MB/s
seconds

44. Example (signals)
input rate
throughput
backlog
growth
backlog
time
MB/s
seconds

45. Example (policy)
M’
M
Extra R=60s
machines

46. Example (policy)
M’
M
machines
Extra R=60s

47. Example (policy)
M’
M
machines
Extra R=60s

48. Example (policy)
M’
M
machines
Extra R=60s

49. Preconditions for Downscaling
Low backlog time
No backlog growth
Low CPU utilization

50. How far can we
downscale?
Stay tuned...

51. Adjusting Parallelism of a
Running Streaming Pipeline
Mechanism: actuating Change
3

52. S0
S2
S1
Optimized Pipeline = DAG of Stages

53. S0
S2
S1
Optimized Pipeline = DAG of Stages

54. S0
S2
S1
Optimized Pipeline = DAG of Stages
Machine 0

55. Adding Parallelism
Machine 0
S0
S2
S1
S0
S2
S1
S0
S2
S1

56. Adding Parallelism
S0
S2
S1
S0
S2
S1
Machine 0
Machine 1

57. Adding Parallelism = Splitting Key Ranges
S0
S2
S1
S0
S2
S1
Machine 0
Machine 1

58. Migrating a Computation

59. Adding Parallelism = Migrating Computation Ranges
S0
S2
S1
S0
S2
S1
Machine 0
Machine 1

60. Checkpoint and Recovery
~
Computation Migration

61. Key Ranges and Persistence
S0
S2
S1
Machine 0
S0
S2
S1
Machine 1
S0
S2
S1
Machine 3
S0
S2
S1
Machine 2

62. Downscaling from 4 to 2 Machines
S0
S2
S1
Machine 0
S0
S2
S1
Machine 1
S0
S2
S1
Machine 3
S0
S2
S1
Machine 2

63. Downscaling from 4 to 2 Machines
S0
S2
S1
Machine 0
S0
S2
S1
Machine 1
S0
S2
S1
Machine 3
S0
S2
S1
Machine 2

64. Downscaling from 4 to 2 Machines
S0
S2
S1
Machine 0
S0
S2
S1
Machine 1

65. Downscaling from 4 to 2 Machines
S0
S2
S1
Machine 1
S0
S2
S1
Machine 2
Upsizing = Steps in Reverse

66. Granularity of Parallelism
As of March 2016, Google Cloud Dataflow:
• Splits Key Ranges initially Based on Max Machines
• At Max: 1 Logical Persistent Disk per Machine
Each disk has slice of key ranges from all stages
• Only (relatively) even Disk Distributions
• Results in Scaling Quanta

67. Parallelism Disk per Machine
3 N/A
4 15
5 12
6 10
7 8, 9
8 7, 8
9 6, 7
10 6
12 5
15 4
20 3
30 2
60 1
Example Scaling
Quanta:
Max = 60 Machines

68. Goals:
1. No backlog growth
2. Short backlog time
3. Reasonable CPU utilization
Policy: making Decisions

69. Preconditions for Downscaling
Low backlog time
No backlog growth
Low CPU utilization

70. Next lower scaling quanta => M’ machines
Estimate future CPUM’ per machine:
If new CPUM’ < threshold (say 90%),
downscale to M’
Downscaling Policy
M
M’
CPUM’ = CPUM

71. Summary +
Future Work
4

72. Artificial Experiment

73. Auto-Scaling Summary
Signals: throughput, backlog time,
backlog growth, CPU utilization
Policy: keep up, reduce backlog,
use CPUs
Mechanism: split key ranges,
migrate computations

74. • Experiment with non-uniform disk distributions to
address hot ranges
• Dynamically splitting ranges finer than initially done.
• Approximate model of #VM - throughput relation
Future Work

75. Questions?
Further reading on streaming model:
The world beyond batch: Streaming 101
The world beyond batch: Streaming 102

76. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations/
google-cloud-dataflow