A data highway is a network that collects, aggregates, and delivers large volumes of data between data centers. The presented data highway collects 250 billion events totaling 800 terabytes of input data and delivers 1.2 petabytes of output data to over 30,000 publisher hosts and 60 tenants. It provides high throughput collection and delivery of data across data centers while allowing customization for individual tenant needs. The architecture includes routers to collect and forward data, gateways to aggregate and buffer data, and prisms to deliver data to various endpoints. Challenges discussed include improving throughput, controlling variable input throughput, and ensuring delivery during system recoveries. Solutions demonstrated rate limiting, load balancing, and additive-increase/multiplicative-decrease

1. DATA HIGHWAY
Petabyte Scale Event Collection, Transport & Delivery
PRESENTED BY Nilam Sharma, Huibing Yin⎪ June 13, 2017

2. About Us
Nilam Sharma
Masters in CS
Huibing Yin
Ph.D. in Control
 Alumni of UIUC
 5 years in Yahoo
 Data Highway team
 Interested in distributed systems

3. Agenda
 What is Data Highway?
 Why?
 How?
 Challenge Stories
http://www.publicdomainpictures.net/view-image.php?image=41013

4. https://commons.wikimedia.org/wiki/File:WaterloopCompetitionHyperloopPod.jpg
What is Data Highway?

5. https://commons.wikimedia.org/wiki/File:WaterloopCompetitionHyperloopPod.jpg
A network of tubes transmitting data

6. Tenant1Tenant1
DC2 DC4
What is Data Highway? - Use Case
DC3
Tenant1Tenant1
DC1
Tenant2
A network of tubes transmitting data

7. DC4
DC1
DC2
DC3
Tenant1
Tenant2
Tenant1
What is Data Highway?
Router Gateway
Prism
Prism
GatewayRouter
Data Highway

8. What is Data Highway?
250
billion inbound events

9. 800
terabytes input
What is Data Highway?

10. 1.2
petabytes output
What is Data Highway?

11. 30K
publisher hosts
What is Data Highway?

12. 60+
tenants in production
What is Data Highway?

13. Data Highway
A platform that collects, aggregates and delivers data
250billion events
800terabytes input
1.2petabytes output
30Kpublisher hosts
60+tenants

14. Why Data Highway?
https://flic.kr/p/9u96mQ

15. Why Data Highway?
 Data aggregation and delivery at large scale (30K publishing hosts)
 Multi-tenant
 Customized for Yahoo requirements and technology
 Onboarding of new tenants
● Quick
● Easy
● Economic

16. Why Data Highway?

17. Salient Features
Collection
 Multi datacenter event
collection (including AWS
Regions)
 Flexible options to publish —
web-service and multilingual
API, server plugins
Delivery
 Multi data-center delivery with
business continuity support
 Batched HDFS file delivery
 Low latency streaming
delivery to Storm and Kafka
Why Data Highway?

18. Supporting Features
 Delivery completeness web-service for audits
 Rate limits for tenant isolation
 Adaptive limits to absorb traffic spikes
Why Data Highway?

19. Supporting Features
 Customizable filters per streaming endpoint
 Partition based on schema/time
 Fault tolerance for network and endpoint failures
 Schema Registry
Why Data Highway?

20. Additional Facts
 Data agnostic
 Event size — 1 MB max
 Accepts Avro or any Blob content
 Production H/W Footprint : ~500 hosts
Why Data Highway?

21. Tenant Data Highway
SLA
99% in < 5 minutes, 99.9999% in < 15 minutes
95% in < 1 second, 99.99% in < 5 seconds
Why Data Highway?

22. How? Data Highway Architecture
https://flic.kr/p/GYtbM

23. Technologies
Data Highway Architecture

24. DC4
DC1
DC2
DC3
Tenant1
Tenant2
Tenant1
Data Highway
Data Highway Architecture — Subsystem View

25. Data Highway Architecture — Subsystem View
DC3DC1
Router Gateway Prism
Tenant1
DC4DC2
Tenant2
Tenant1

26. Data Highway Architecture — Simple Single Tenant View
DC1
Tenant
RouterRouterRouter
GatewayGatewayGateway
PrismPrismPrism
DC3

27. Tenant
Customer code
DH API
(C++, Java, Perl)
DH Emitter Daemon Router
Data Highway Architecture — Emitter
Tenant Router
Gateway
Prism
Spool file
Spools only on failure

28. Router
Data Highway Architecture — Router
Tenant Router
Gateway
Prism
Spool
File
Spools only on failure
HTTP
server
Broker
Forwarder
Gateway1Emitter
Gateway2

29. Gateway
HTTP
server
Broker
Forwarder
Spool
File
PrismRouter
Data Highway Architecture — Gateway
Tenant Router
Gateway
Prism
Grid
Delivery
Agent
avro avro

30. Prism
HTTP
server
Broker
Streaming
Delivery
Agent
Spool
File
Gateway
Data Highway Architecture — Prism
Tenant Router
Gateway
Prism
Kafka
Delivery
Agent

31. Gateway
Prism
Data Highway Architecture — Metadata
Tenant Router
Metadata Store
Publisher
Data Flow
Metadata Flow
Counter
Service

32. Data Highway Architecture - Overall
DC4
DC1
DC2
DC3
Router Gateway
Prism
Tenant1
Tenant2
Tenant1
Prism
GatewayRouter
DC4
DC1
DC2
DC3
Tenant1
Tenant2
Tenant1
Router Gateway
Prism
Prism
GatewayRouter
Data Highway

33. Challenges
and what we have managed
to solve so far!!!
https://flic.kr/p/oKBh6a

34. Challenge Category: Improving Throughput

35. Gateway
HTTP
server
Broker
Forwarder
Spool
File
PrismRouter
Improving Throughput
Tenant Router
Gateway
Prism
Grid
Delivery
Agent
avro avro

36. Grid Delivery Agent
Understanding Event Flow
HDD
Gateway
avro
File Spooler
Thread
Thread
File Uploader
Thread
Thread
Spool avro
21
3
5
4
Broker
HTTP
server
Improving Throughput

37. File Uploader
Broker
Multiple threads fight for resources when reading/writing files to disk
File Spooler
Thread
Thread
Thread
Thread
Thread
https://clipartfest.com/
Challenge: Disk Contention
Improving Throughput

38. Multiple threads fight for resources when reading/writing files to disk
Broker
Solution: Disk I/O Separation
File Spooler
Thread
https://clipartfest.com/
https://clipartfest.com/
Improving Throughput: Disk Contention
Thread
Thread
File Uploader
Thread
Thread

39. File Uploader
File Spooler
Challenge: Namespace Explosion
Thread
Thread
Thread
Thread
https://clipartfest.com/
avro avroavro
Improving Throughput
Aggregate events over longer time,
e.g., 5 mins, 10 mins etc.

40. File Uploader
File Spooler
Solution: File Merging?
Thread
Thread
Thread
Thread
avro
avro
avro
avro
Improving Throughput: Namespace Explosion
https://clipartfest.com/

41. File Uploader
File Spooler
Better Solution: Single Thread Writing
Thread
Thread
Thread
Thread
Thread
avro avroavro
Improving Throughput: Namespace Explosion
Saves 10x namespace
https://clipartfest.com/

42. Improving Throughput: Summary
 Improve Disk I/O by disk separation
 Replace HDD with SSD for avro file creation
 Aggregate files over time and threads in File Spooler
Improved Throughput by 2x

43. Challenge Category: Controlling Throughput

44. Controlling Throughput: Challenges
 Heterogeneous publishers
 Traffic spikes
 Unbalanced receiver
Unanticipated
spikes

45. Controlling Throughput: Traffic Spike
DC4
DC1
DC2
DC3
Router Gateway
Prism
Tenant1
Tenant2
Tenant1
Prism
GatewayRouter
429
Solution: Rate Limiting

46. Better Solution: Adaptive Rate Limiting
Planned
Capacity
Tolerated
Traffic
Bytes
Data (bytes)
Controlling Throughput: Traffic Spike
Headroom

47. Controlling Throughput: Traffic Spike
DC4
DC1
DC2
DC3
Router Gateway
Prism
Tenant1
Tenant2
Tenant1
Prism
GatewayRouter
429
429
429
429 / 200 with
delay header
Solution: Rate Limiting In All Nodes

48.  Instant feedback
● Returns 429 if can’t accept more
● Used across all DH nodes
 Early feedback
● Specify delay in response header
● Used in Storm consumers
Receiver
Solution: Rate Limiting Summary
429
200 but
delay 20 ms
Controlling Throughput: Traffic Spike

49. Controlling Throughput
DC4
DC1
DC2
DC3
Router Gateway
Prism
Tenant1
Tenant2
Tenant1
Prism
GatewayRouter
Challenge: Unbalanced Receiver

50. Challenge: Unbalanced Receiver
Spout
Spout
Spout
Round-robin leads to
more connections
and higher latency for
some Spouts
Controlling Throughput
Prism
SDA

51. Solution: Penalty Based Load Balancing
Spout
Spout
Spout
Penalty = # active connections x latency weight
Controlling Throughput: Unbalanced Receiver
Prism
SDA

52. Controlling Throughput
DC4
DC1
DC2
DC3
Router Gateway
Prism
Tenant1
Tenant2
Tenant1
Prism
GatewayRouter
Challenge: Traffic Flood During Recovery
Extra traffic from
broker spool

53. Challenge: Traffic Flood During Recovery
Controlling Throughput
Router
HTTP
server
Broker Forwarder
GatewayEmitter
Spool File
Rate
Controller

54. Rate Controller
Solution: Rate Control
 Additive-Increase Multiplicative-Decrease (AIMD) like rate control
Controller
Success Rate
Estimator (S)
Throttle Rate
Estimator (T)
Request Rate
Limiter
(Token, R)
Can I send?
Yes / No
200
429
Token available?
Yes / No
Update S
Update T
Update R
Increase R by
fix amount
Decrease R:
Multiply
S/(S+T)
Update R
Controlling Throughput: Traffic Flood

55. Solution: Rate Control
Router
HTTP
server
Broker Forwarder
GatewayEmitter
Spool File
Rate
Controller
Controlling Throughput: Traffic Flood
Reduce 429
drastically

56. Controlling Throughput - Summary
 Adaptive rate limit
 Load balancing using penalty
 AIMD like rate control in sender

57. Challenge: More
 Event duplication
 Zero downtime deployments
 Schema Registry
 and more...

58. Questions