Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ReStream
Accelerating Backtesting and Stream Replay
with Serial-Equivalent Parallel Processing
October 6, 2016
Johann Schl...
Overview
• Motivations for backtesting and stream replay
• Alternatives for scaling throughput
• ReStream and Multi-Versio...
Research Motivation
• Operating Tagged and hi5
social networks
• >300 million users registered
• Millions of daily active ...
• >10 million active accounts
• >1000 updates/sec
• Must respond to current activity
• Require near-instant decisions
Real...
• Facts recorded in event log
• Real-time stream-processing
• Need to evaluate new ideas quickly,
e.g., simulate model usi...
Replay lets Agile developers ask
powerful tool for creating and
enhancing streaming applications
When latency matters…
streaming shines
• Spam detection
• Payment fraud
• Money laundering
• Real-time recommendations
• A...
Research Motivation
• Operating Tagged and hi5
social networks
• >300 million users registered
• Millions of daily active ...
Serial-Equivalent Parallel Replay
12345
Ordered log
Serial-Equivalent Parallel Replay
12345 Program
1234567890
Serial-Equivalent Parallel Replay
AB
Program
123456789101
Serial-Equivalent Parallel Replay
AB
Program
234567891011121314
Serial-Equivalent Parallel Replay
ABC5 Program
t=4t=5t=9
Program*
Program*
Serial-Equivalent Parallel Replay
Serial-Equivalent Parallel Replay
13579
246810 A
BCProgram*
Program*
Serial-Equivalent Parallel Replay
1357911121315171921235
246810121416182022246 A
BCProgram*
Program*
t=4
t=5t=9
Serial-Equivalent Parallel Replay
• Deterministic output
• More restrictive than transaction serializability
• Partition t...
Developers’ Accelerated Replay Wish List
• Semantics of sequential operations with mutable state
• Full fine-grained tempor...
Workload Assumptions
• Total order provided by log
• Abundant cloud resources available
• Per-event latency not a concern
Possible Solutions
Streaming Databases
• StreamBase / Aurora
• Truviso / TelegraphCQ
• Recent startups
- PipelineDB
- RethinkDB
• Query inter...
OLTP Databases
• PostgreSQL
• IBM DB2
• MS SQL Server
• Oracle
• SQL interface
• Robust high-performance
implementations
•...
• Hadoop
• Apache Spark Streaming
• Lambda architecture
• Routinely delivers desired log-
processing throughput
• Easy to ...
Other Systems
• Other streaming: Google MillWheel, Yahoo S4, Apache Storm,
Twitter Heron, Apache Flink, Apache Samza, Walm...
ReStream
• Consequence of input data
• Suggests opportunity for parallelism
• Can we maintain order when necessary, but not necessa...
• Consequence of input data
• Suggests opportunity for parallelism
• Can we maintain order when necessary, but not necessa...
Multi-Versioned State
SET(timestamp=10,	key=x,	value=3)
SET(timestamp=20,	key=x,	value=5)
GET(timestamp=15,	key=x)
x=3	@	t...
Social Network Anti-Spam Example
sender has sent 2x messages to non-friends as to friends
AND
> 20% of messages sent from ...
Social Network Anti-Spam Example
Express program in four pieces
A. Track friendships
B. Track how often user sends to frie...
Sample Code
{	e:	NewFriendshipEvent	=>	
}
A
Sample Code
{	e:	NewFriendshipEvent	=>	
		userPair	=	(e.userIdA,	e.userIdB)	
		friendships.merge(e.timestamp,	userPair,	_	...
{	e:	NewFriendshipEvent	=>	
		userPair	=	(e.userIdA,	e.userIdB)	
		friendships	.merge(e.timestamp,	userPair,	_	=>	true)	
}...
{	e:	NewFriendshipEvent	=>	
		userPair	=	(e.userIdA,	e.userIdB)	
		friendships	.merge(e.timestamp,	userPair,	_	=>	true)	
}...
friendships	.merge(	timestamp	,				key		,			value		)	
WRITE
Sample Code
{	e:	NewFriendshipEvent	=>	
		userPair	=	(e.userIdA,	e.userIdB)	
		friendships	.merge(e.timestamp,	userPair,	_	=>	true)	
}...
{	e:	MessageEvent	=>	
		userPair	=	(e.senderId,	e.recipientId)	
		if	(friendships	.get(e.timestamp,	userPair))	{	
				frie...
{	e:	MessageEvent	=>	
		userPair	=	(e.senderId,	e.recipientId)	
		if	(friendships	.get(	timestamp	,				key		))	{	
				frie...
{	e:	MessageEvent	=>	
		userPair	=	(e.senderId,	e.recipientId)	
		if	(friendships	.get(e.timestamp,	userPair))	{	
				frie...
A
B
C
D
nonfriendMsgs
R
R
R
R
R
W
W
W
W
W
friendMsgs
friendships
ipEmailMsgs
ipMsgs
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
R
R
R
R
R
W
W
W
W
W
Topological
sort
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
1
R
Reading from log
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123
W
R
Reading from log,
writing shared
state
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
12345
R
W
R
Loose coupling
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123456789
Loose coupling
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123456789
Must respect
dependencies
NO
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
123456789
Loose coupling
OK
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
1234567891011
OK
Loose coupling
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
12345678910111213
OK
Out-of-order
processing
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
234567891011121314
OK
Out-of-order
processing
Multi-version Parallel Streaming
MVPS:
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
A B C D
MVPS
135791112131517192123
246810121416182022
A B C D
nonfriendMsgs
friendMsgs
friendships
ipEmailMsgs
ipMsgs
A B C D
MVPS
Mini-batches for MVPS
1
10
1112131415161718
Mini-batches for MVPS
11
20
1
10
1
10
21
30
41
50
61
70
81
90
101 1101 130
11
20
31
40
51
60
71
80
91
100111 12040
A B C D
nonfriendMsgs
friendMsgs
friends...
• Partitioned parallel dataflow
• Input events passed to all operators
• Globally shared multi-versioned state
• Logical ti...
Evaluation
ReStream Evaluation Aims
• Demonstrate parallel speedup vs. single-thread (COST)
• Compare to alternative systems
• Unders...
ReStream Evaluation Workload
• Simulated social network spam detection
• Structure of read-write dependency
graph linked t...
ReStream Evaluation Workload
• Simulated social network spam detection
• Structure of read-write dependency
graph linked t...
Scaling Throughput
0
200,000
400,000
600,000
1 2 4 8 16 32
Hosts
Throughput(events/s)
Execution Engine ReStream MVPS on Sp...
Modeling Performance
• Greater parallel speedup possible when there are fewer read-write
dataflow dependencies
• Track read...
uniform degree distributionskewed degree distribution
Parametrized by α
W
eb

out-links
Photo

Sharing
Social

N
etw
orks
W
eb

in-links
0
100,000
200,000
300,000
1.5 2.0 2.5 3.0
Hosts 2 4 8 16
...
ReStream Summary
• Serial-equivalent results from parallel replay
• Throughput much greater than real-time rate
• MVPS con...
Upcoming SlideShare
Loading in …5
×

ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent Parallel Processing

523 views

Published on

Real-time predictive applications can demand continuous and agile development, with new models constantly being trained, tested, and then deployed. Training and testing are done by replaying stored event logs, running new models in the context of historical data in a form of backtesting or ``what if?'' analysis. To replay weeks or months of logs while developers wait, we need systems that can stream event logs through prediction logic many times faster than the real-time rate. A challenge with high-speed replay is preserving sequential semantics while harnessing parallel processing power. The crux of the problem lies with causal dependencies inherent in the sequential semantics of log replay.

We introduce an execution engine that produces serial-equivalent output while accelerating throughput with pipelining and distributed parallelism. This is made possible by optimizing for high throughput rather than the traditional stream processing goal of low latency, and by aggressive sharing of versioned state, a technique we term Multi-Versioned Parallel Streaming (MVPS).
In experiments we see that this engine, which we call ReStream, performs as well as batch processing and more than an order of magnitude better than a single-threaded implementation.

Published in: Data & Analytics
  • Be the first to comment

ReStream: Accelerating Backtesting and Stream Replay with Serial-Equivalent Parallel Processing

  1. 1. ReStream Accelerating Backtesting and Stream Replay with Serial-Equivalent Parallel Processing October 6, 2016 Johann Schleier-Smith, Erik T. Krogen, Joseph M. Hellerstein UC Berkeley @jssmith @joe_hellerstein
  2. 2. Overview • Motivations for backtesting and stream replay • Alternatives for scaling throughput • ReStream and Multi-Version Parallel Streaming (MVPS) • Evaluation
  3. 3. Research Motivation • Operating Tagged and hi5 social networks • >300 million users registered • Millions of daily active users Practical Pains Curiosity
  4. 4. • >10 million active accounts • >1000 updates/sec • Must respond to current activity • Require near-instant decisions Real-Time Spam Detection for Dating Product
  5. 5. • Facts recorded in event log • Real-time stream-processing • Need to evaluate new ideas quickly, e.g., simulate model using data of past 30 days in under 10 minutes Real-Time Spam Detection for Dating Product
  6. 6. Replay lets Agile developers ask powerful tool for creating and enhancing streaming applications
  7. 7. When latency matters… streaming shines • Spam detection • Payment fraud • Money laundering • Real-time recommendations • Ad serving • Dynamic pricing and inventory management for e-commerce, car-services, etc. • Financial trading • Industrial monitoring • IoT applications • And more
  8. 8. Research Motivation • Operating Tagged and hi5 social networks • >300 million users registered • Millions of daily active users Practical Pains Curiosity Given a program that processes an ordered log sequentially How can we achieve parallel speedup?
  9. 9. Serial-Equivalent Parallel Replay 12345 Ordered log
  10. 10. Serial-Equivalent Parallel Replay 12345 Program
  11. 11. 1234567890 Serial-Equivalent Parallel Replay AB Program
  12. 12. 123456789101 Serial-Equivalent Parallel Replay AB Program
  13. 13. 234567891011121314 Serial-Equivalent Parallel Replay ABC5 Program t=4t=5t=9
  14. 14. Program* Program* Serial-Equivalent Parallel Replay
  15. 15. Serial-Equivalent Parallel Replay 13579 246810 A BCProgram* Program*
  16. 16. Serial-Equivalent Parallel Replay 1357911121315171921235 246810121416182022246 A BCProgram* Program* t=4 t=5t=9
  17. 17. Serial-Equivalent Parallel Replay • Deterministic output • More restrictive than transaction serializability • Partition the input between multiple parallel programs • Obtain same output as from one program
  18. 18. Developers’ Accelerated Replay Wish List • Semantics of sequential operations with mutable state • Full fine-grained temporal resolution • Process months in minutes: 10,000x real-time rate Want serial-equivalent parallel replay
  19. 19. Workload Assumptions • Total order provided by log • Abundant cloud resources available • Per-event latency not a concern
  20. 20. Possible Solutions
  21. 21. Streaming Databases • StreamBase / Aurora • Truviso / TelegraphCQ • Recent startups - PipelineDB - RethinkDB • Query interface derived from SQL • Set-oriented approach allows query plan optimization, parallelism and reordering • Some programs can be difficult to express • Most systems emphasize latency over replay throughput Examples + – – +/–
  22. 22. OLTP Databases • PostgreSQL • IBM DB2 • MS SQL Server • Oracle • SQL interface • Robust high-performance implementations • Need to coordinate parallel replay programs • Transactional serializability gives weaker consistency than serial-equivalence Examples + – – +/–
  23. 23. • Hadoop • Apache Spark Streaming • Lambda architecture • Routinely delivers desired log- processing throughput • Easy to integrate arbitrary functions • MapReduce foundation does not lend itself naturally to sequential processing • Throughput and program semantics may be linked Parallel Big Data Systems Examples + – – +
  24. 24. Other Systems • Other streaming: Google MillWheel, Yahoo S4, Apache Storm, Twitter Heron, Apache Flink, Apache Samza, Walmart MUP8 • Deterministic databases: Calvin, Bohm • Transactional: VoltDB / S-Store • Complex Event Processing: Esper, Tibco, JBoss • Other recent systems: Trill, Naiad, Google Cloud Dataflow
  25. 25. ReStream
  26. 26. • Consequence of input data • Suggests opportunity for parallelism • Can we maintain order when necessary, but not necessarily otherwise? Challenge: serial equivalence and parallelism Observation: causal dependencies are often sparse
  27. 27. • Consequence of input data • Suggests opportunity for parallelism • Can we maintain order when necessary, but not necessarily otherwise? Challenge: serial equivalence and parallelism Observation: causal dependencies are often sparse
  28. 28. Multi-Versioned State SET(timestamp=10, key=x, value=3) SET(timestamp=20, key=x, value=5) GET(timestamp=15, key=x) x=3 @ t=10 x=5 @ t=20 GET(timestamp=11, key=x) GET(timestamp=25, key=x) → 3 → 3 → 5 x=5 @ t=25 SET(timestamp=21, key=x, value=7)x
  29. 29. Social Network Anti-Spam Example sender has sent 2x messages to non-friends as to friends AND > 20% of messages sent from IP contain e-mail address message is spam
  30. 30. Social Network Anti-Spam Example Express program in four pieces A. Track friendships B. Track how often user sends to friends / non-friends C. Track how often ip address sends text containing e-mail D. For each message, check B and C to label spam
  31. 31. Sample Code { e: NewFriendshipEvent => } A
  32. 32. Sample Code { e: NewFriendshipEvent => userPair = (e.userIdA, e.userIdB) friendships.merge(e.timestamp, userPair, _ => true) } A
  33. 33. { e: NewFriendshipEvent => userPair = (e.userIdA, e.userIdB) friendships .merge(e.timestamp, userPair, _ => true) } Sample Code A
  34. 34. { e: NewFriendshipEvent => userPair = (e.userIdA, e.userIdB) friendships .merge(e.timestamp, userPair, _ => true) } WRITE Sample Code A
  35. 35. friendships .merge( timestamp , key , value ) WRITE Sample Code
  36. 36. { e: NewFriendshipEvent => userPair = (e.userIdA, e.userIdB) friendships .merge(e.timestamp, userPair, _ => true) } WRITE Sample Code { e: MessageEvent => } 42 A B
  37. 37. { e: MessageEvent => userPair = (e.senderId, e.recipientId) if (friendships .get(e.timestamp, userPair)) { friendMsgs.merge(e.timestamp, e.senderId, _ + 1) } else { nonfriendMsgs.merge(e.timestamp, e.senderId, _ + 1) } } Sample Code 43 { e: NewFriendshipEvent => userPair = (e.userIdA, e.userIdB) friendships .merge(e.timestamp, userPair, _ => true) } WRITE READ A B
  38. 38. { e: MessageEvent => userPair = (e.senderId, e.recipientId) if (friendships .get( timestamp , key )) { friendMsgs.merge(e.timestamp, e.senderId, _ + 1) } else { nonfriendMsgs.merge(e.timestamp, e.senderId, _ + 1) } } Sample Code 44 { e: NewFriendshipEvent => userPair = (e.userIdA, e.userIdB) friendships .merge(e.timestamp, userPair, _ => true) } WRITE READ A B
  39. 39. { e: MessageEvent => userPair = (e.senderId, e.recipientId) if (friendships .get(e.timestamp, userPair)) { friendMsgs .merge(e.timestamp, e.senderId, _ + 1) } else { nonfriendMsgs .merge(e.timestamp, e.senderId, _ + 1) } } Sample Code 45 { e: NewFriendshipEvent => userPair = (e.userIdA, e.userIdB) friendships .merge(e.timestamp, userPair, _ => true) } WRITE READ WRITE A B
  40. 40. A B C D nonfriendMsgs R R R R R W W W W W friendMsgs friendships ipEmailMsgs ipMsgs
  41. 41. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs R R R R R W W W W W Topological sort
  42. 42. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 1 R Reading from log
  43. 43. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 123 W R Reading from log, writing shared state
  44. 44. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 12345 R W R Loose coupling
  45. 45. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 123456789 Loose coupling
  46. 46. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 123456789 Must respect dependencies NO
  47. 47. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 123456789 Loose coupling OK
  48. 48. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 1234567891011 OK Loose coupling
  49. 49. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 12345678910111213 OK Out-of-order processing
  50. 50. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs 234567891011121314 OK Out-of-order processing
  51. 51. Multi-version Parallel Streaming MVPS:
  52. 52. A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs A B C D MVPS
  53. 53. 135791112131517192123 246810121416182022 A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs A B C D MVPS
  54. 54. Mini-batches for MVPS 1 10 1112131415161718
  55. 55. Mini-batches for MVPS 11 20 1 10
  56. 56. 1 10 21 30 41 50 61 70 81 90 101 1101 130 11 20 31 40 51 60 71 80 91 100111 12040 A B C D nonfriendMsgs friendMsgs friendships ipEmailMsgs ipMsgs A B C D MVPS with mini-batches
  57. 57. • Partitioned parallel dataflow • Input events passed to all operators • Globally shared multi-versioned state • Logical timestamps referenced throughout computation • Analyze DAG of operator potential read-write dependency • May use mini-batches to amortize coordination • Serial-equivalent semantics Multi-Versioned Parallel Streaming (MVPS)
  58. 58. Evaluation
  59. 59. ReStream Evaluation Aims • Demonstrate parallel speedup vs. single-thread (COST) • Compare to alternative systems • Understand limits to parallelism
  60. 60. ReStream Evaluation Workload • Simulated social network spam detection • Structure of read-write dependency graph linked to structure of social network • Can tune workload characteristics by generating different social graphs
  61. 61. ReStream Evaluation Workload • Simulated social network spam detection • Structure of read-write dependency graph linked to structure of social network • Can tune workload characteristics by generating different social graphs uniform degree distribution skewed degree distribution
  62. 62. Scaling Throughput 0 200,000 400,000 600,000 1 2 4 8 16 32 Hosts Throughput(events/s) Execution Engine ReStream MVPS on Spark 1−Thread
  63. 63. Modeling Performance • Greater parallel speedup possible when there are fewer read-write dataflow dependencies • Track reads and writes of global state, compute critical path length along chained dependencies
  64. 64. uniform degree distributionskewed degree distribution Parametrized by α
  65. 65. W eb
 out-links Photo
 Sharing Social
 N etw orks W eb
 in-links 0 100,000 200,000 300,000 1.5 2.0 2.5 3.0 Hosts 2 4 8 16 Throughput(event/s) α Modeling Performance R2=0.94 Per-host batch size 2,500-40,000 events (10,000 shown) Fit gives
  66. 66. ReStream Summary • Serial-equivalent results from parallel replay • Throughput much greater than real-time rate • MVPS consistency: Multi-Versioned Parallel Streaming - Analyze for potential read-write dependencies - Timestamped multi-versioned state - Track logical time at runtime • Also may apply to online stream processing and deterministic databases @jssmith @joe_hellerstein This work was supported in part by
 AWS Cloud Credits for Research

×