1. ReStream
Accelerating Backtesting and Stream Replay
with Serial-Equivalent Parallel Processing
October 6, 2016
Johann Schleier-Smith, Erik T. Krogen, Joseph M. Hellerstein
UC Berkeley
@jssmith @joe_hellerstein

2. Overview
• Motivations for backtesting and stream replay
• Alternatives for scaling throughput
• ReStream and Multi-Version Parallel Streaming (MVPS)
• Evaluation

3. Research Motivation
• Operating Tagged and hi5
social networks
• >300 million users registered
• Millions of daily active users
Practical Pains Curiosity

4. • >10 million active accounts
• >1000 updates/sec
• Must respond to current activity
• Require near-instant decisions
Real-Time
Spam Detection
for Dating Product

5. • Facts recorded in event log
• Real-time stream-processing
• Need to evaluate new ideas quickly,
e.g., simulate model using data of
past 30 days in under 10 minutes
Real-Time
Spam Detection
for Dating Product

6. Replay lets Agile developers ask
powerful tool for creating and
enhancing streaming applications

7. When latency matters…
streaming shines
• Spam detection
• Payment fraud
• Money laundering
• Real-time recommendations
• Ad serving
• Dynamic pricing and inventory
management for e-commerce,
car-services, etc.
• Financial trading
• Industrial monitoring
• IoT applications
• And more

8. Research Motivation
• Operating Tagged and hi5
social networks
• >300 million users registered
• Millions of daily active users
Practical Pains Curiosity
Given a program that processes
an ordered log sequentially
How can we achieve parallel
speedup?

9. Serial-Equivalent Parallel Replay
12345
Ordered log

10. Serial-Equivalent Parallel Replay
12345 Program

11. 1234567890
Serial-Equivalent Parallel Replay
AB
Program

12. 123456789101
Serial-Equivalent Parallel Replay
AB
Program

13. 234567891011121314
Serial-Equivalent Parallel Replay
ABC5 Program
t=4t=5t=9

14. Program*
Program*
Serial-Equivalent Parallel Replay

15. Serial-Equivalent Parallel Replay
13579
246810 A
BCProgram*
Program*

16. Serial-Equivalent Parallel Replay
1357911121315171921235
246810121416182022246 A
BCProgram*
Program*
t=4
t=5t=9

17. Serial-Equivalent Parallel Replay
• Deterministic output
• More restrictive than transaction serializability
• Partition the input between multiple parallel programs
• Obtain same output as from one program

18. Developers’ Accelerated Replay Wish List
• Semantics of sequential operations with mutable state
• Full fine-grained temporal resolution
• Process months in minutes: 10,000x real-time rate
Want serial-equivalent parallel replay

19. Workload Assumptions
• Total order provided by log
• Abundant cloud resources available
• Per-event latency not a concern

20. Possible Solutions

21. Streaming Databases
• StreamBase / Aurora
• Truviso / TelegraphCQ
• Recent startups
- PipelineDB
- RethinkDB
• Query interface derived from SQL
• Set-oriented approach allows
query plan optimization,
parallelism and reordering
• Some programs can be difficult to
express
• Most systems emphasize latency
over replay throughput
Examples
+
–
–
+/–

22. OLTP Databases
• PostgreSQL
• IBM DB2
• MS SQL Server
• Oracle
• SQL interface
• Robust high-performance
implementations
• Need to coordinate parallel
replay programs
• Transactional serializability
gives weaker consistency than
serial-equivalence
Examples
+
–
–
+/–

23. • Hadoop
• Apache Spark Streaming
• Lambda architecture
• Routinely delivers desired log-
processing throughput
• Easy to integrate arbitrary functions
• MapReduce foundation does not lend
itself naturally to sequential processing
• Throughput and program semantics
may be linked
Parallel Big Data Systems
Examples
+
–
–
+

24. Other Systems
• Other streaming: Google MillWheel, Yahoo S4, Apache Storm,
Twitter Heron, Apache Flink, Apache Samza, Walmart MUP8
• Deterministic databases: Calvin, Bohm
• Transactional: VoltDB / S-Store
• Complex Event Processing: Esper, Tibco, JBoss
• Other recent systems: Trill, Naiad, Google Cloud Dataflow

25. ReStream

26. • Consequence of input data
• Suggests opportunity for parallelism
• Can we maintain order when necessary, but not necessarily
otherwise?
Challenge: serial equivalence and parallelism
Observation: causal dependencies are often sparse

27. • Consequence of input data
• Suggests opportunity for parallelism
• Can we maintain order when necessary, but not necessarily
otherwise?
Challenge: serial equivalence and parallelism
Observation: causal dependencies are often sparse

28. Multi-Versioned State
SET(timestamp=10, key=x, value=3)
SET(timestamp=20, key=x, value=5)
GET(timestamp=15, key=x)
x=3 @ t=10
x=5 @ t=20
GET(timestamp=11, key=x)
GET(timestamp=25, key=x)
→ 3
→ 3
→ 5
x=5 @ t=25
SET(timestamp=21, key=x, value=7)x

29. Social Network Anti-Spam Example
sender has sent 2x messages to non-friends as to friends
AND
> 20% of messages sent from IP contain e-mail address
message is spam

30. Social Network Anti-Spam Example
Express program in four pieces
A. Track friendships
B. Track how often user sends to friends / non-friends
C. Track how often ip address sends text containing e-mail
D. For each message, check B and C to label spam

31. Sample Code
{ e: NewFriendshipEvent =>
}
A

32. Sample Code
{ e: NewFriendshipEvent =>
userPair = (e.userIdA, e.userIdB)
friendships.merge(e.timestamp, userPair, _ => true)
}
A

33. { e: NewFriendshipEvent =>
userPair = (e.userIdA, e.userIdB)
friendships .merge(e.timestamp, userPair, _ => true)
}
Sample Code
A

34. { e: NewFriendshipEvent =>
userPair = (e.userIdA, e.userIdB)
friendships .merge(e.timestamp, userPair, _ => true)
}
WRITE
Sample Code
A

35. friendships .merge( timestamp , key , value )
WRITE
Sample Code

36. { e: NewFriendshipEvent =>
userPair = (e.userIdA, e.userIdB)
friendships .merge(e.timestamp, userPair, _ => true)
}
WRITE
Sample Code
{ e: MessageEvent =>
}
42
A
B

37. { e: MessageEvent =>
userPair = (e.senderId, e.recipientId)
if (friendships .get(e.timestamp, userPair)) {
friendMsgs.merge(e.timestamp, e.senderId, _ + 1)
} else {
nonfriendMsgs.merge(e.timestamp, e.senderId, _ + 1)
}
}
Sample Code
43
{ e: NewFriendshipEvent =>
userPair = (e.userIdA, e.userIdB)
friendships .merge(e.timestamp, userPair, _ => true)
}
WRITE
READ
A
B

38. { e: MessageEvent =>
userPair = (e.senderId, e.recipientId)
if (friendships .get( timestamp , key )) {
friendMsgs.merge(e.timestamp, e.senderId, _ + 1)
} else {
nonfriendMsgs.merge(e.timestamp, e.senderId, _ + 1)
}
}
Sample Code
44
{ e: NewFriendshipEvent =>
userPair = (e.userIdA, e.userIdB)
friendships .merge(e.timestamp, userPair, _ => true)
}
WRITE
READ
A
B

39. { e: MessageEvent =>
userPair = (e.senderId, e.recipientId)
if (friendships .get(e.timestamp, userPair)) {
friendMsgs .merge(e.timestamp, e.senderId, _ + 1)
} else {
nonfriendMsgs .merge(e.timestamp, e.senderId, _ + 1)
}
}
Sample Code
45
{ e: NewFriendshipEvent =>
userPair = (e.userIdA, e.userIdB)
friendships .merge(e.timestamp, userPair, _ => true)
}
WRITE
READ
WRITE
A
B

40. A
B
C
D
nonfriendMsgs
R
R
R
R
R
W
W
W
W
W
friendMsgs
friendships
ipEmailMsgs
ipMsgs

41. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
R
R
R
R
R
W
W
W
W
W
Topological
sort

42. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
1
R
Reading from log

43. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123
W
R
Reading from log,
writing shared
state

44. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
12345
R
W
R
Loose coupling

45. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123456789
Loose coupling

46. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123456789
Must respect
dependencies
NO

47. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123456789
Loose coupling
OK

48. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
1234567891011
OK
Loose coupling

49. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
12345678910111213
OK
Out-of-order
processing

50. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
234567891011121314
OK
Out-of-order
processing

51. Multi-version Parallel Streaming
MVPS:

52. A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
A B C D
MVPS

53. 135791112131517192123
246810121416182022
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
A B C D
MVPS

54. Mini-batches for MVPS
1
10
1112131415161718

55. Mini-batches for MVPS
11
20
1
10

56. 1
10
21
30
41
50
61
70
81
90
101 1101 130
11
20
31
40
51
60
71
80
91
100111 12040
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
A B C D
MVPS with mini-batches

57. • Partitioned parallel dataflow
• Input events passed to all operators
• Globally shared multi-versioned state
• Logical timestamps referenced throughout computation
• Analyze DAG of operator potential read-write dependency
• May use mini-batches to amortize coordination
• Serial-equivalent semantics
Multi-Versioned Parallel Streaming (MVPS)

58. Evaluation

59. ReStream Evaluation Aims
• Demonstrate parallel speedup vs. single-thread (COST)
• Compare to alternative systems
• Understand limits to parallelism

60. ReStream Evaluation Workload
• Simulated social network spam detection
• Structure of read-write dependency
graph linked to structure of social
network
• Can tune workload characteristics by
generating different social graphs

61. ReStream Evaluation Workload
• Simulated social network spam detection
• Structure of read-write dependency
graph linked to structure of social
network
• Can tune workload characteristics by
generating different social graphs
uniform degree distribution
skewed degree distribution

62. Scaling Throughput
0
200,000
400,000
600,000
1 2 4 8 16 32
Hosts
Throughput(events/s)
Execution Engine ReStream MVPS on Spark 1−Thread

63. Modeling Performance
• Greater parallel speedup possible when there are fewer read-write
dataflow dependencies
• Track reads and writes of global state, compute critical path length
along chained dependencies

64. uniform degree distributionskewed degree distribution
Parametrized by α

65. W
eb
out-links
Photo
Sharing
Social
N
etw
orks
W
eb
in-links
0
100,000
200,000
300,000
1.5 2.0 2.5 3.0
Hosts 2 4 8 16
Throughput(event/s)
α
Modeling Performance
R2=0.94
Per-host batch size
2,500-40,000 events
(10,000 shown)
Fit gives

66. ReStream Summary
• Serial-equivalent results from parallel replay
• Throughput much greater than real-time rate
• MVPS consistency: Multi-Versioned Parallel Streaming
- Analyze for potential read-write dependencies
- Timestamped multi-versioned state
- Track logical time at runtime
• Also may apply to online stream processing and deterministic databases
@jssmith @joe_hellerstein
This work was supported in part by
AWS Cloud Credits for Research