The document discusses the architecture and challenges of real-time systems at Twitter, focusing on key issues like monolithic applications and the need for a self-service infrastructure. It outlines goals such as improving speed, efficiency, reliability, and team independence, along with the implementation of various system components like proxies and connection management. Statistics provided indicate significant improvements in traffic handling and deployment efficiency.

1. real-time systems @twitter
@raffi & @a_a
velocity 2012

2. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
T-Bird
T-Flock +
Haplo
Monorail
Darkwing
Flock(s)

4. what are the big problems?
⇢ monolithic application
⇢ lack of self-service infrastructure
⇢ painful to add new services & features

5. what did we want to achieve?
⇢ big infrastructure wins in speed,
efficiency, reliability
⇢ separation of concerns
⇢ team independence

7. stats.timeFuture("request_latency_ms") {
// dispatch to do work
}

8. ts(
AVG,
timelineservice,
audubon.role.timelineservice,
service/client/
woodstar.prod/
getStatusTimeline/request_latency_ms.p50
)

10. TFE
(HTTP Proxy)
Woodstar

11. TFE
(HTTP Proxy)
Monorail Woodstar

12. 100% Monorail TFE
(HTTP Proxy) 0% Woodstar
Monorail Woodstar

13. 0% Monorail TFE
(HTTP Proxy) 100% Woodstar
Monorail Woodstar

16. TFE
(HTTP Proxy)
Woodstar

17. TFE
(HTTP Proxy)
Woodstar
Tweetypie Gizmoduck
Timeline
(Tweet (User
Service
Service) Service)

18. network substrate
⇢ connection management
⇢ protocol codecs
⇢ transient error handling
⇢ service discovery
⇢ observability

19. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

20. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

21. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

22. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

23. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

24. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

25. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

26. TFE
(HTTP Proxy)
Woodstar
Tweetypie Gizmoduck
Timeline
(Tweet (User
Service
Service) Service)

27. ServerBuilder()
.name("ServiceName")
.reportTo(statsReceiver)
.tracer(ZipkinTracer())
.codec(Http())
.maxConcurrentRequests(1000)
.requestTimeout(500.milliseconds)
.build(Service[Request, Response])

30. TFE
(HTTP Proxy)
Monorail Woodstar

31. TFE
(HTTP Proxy)
Macaw Macaw Macaw
Woodstar
+Activity +Search +Logging

33. class EchoLoadTest(service: ParrotThriftService)
extends RecordProcessor {
val client =
new EchoService.ServiceToClient(
service,
new TBinaryProtocol.Factory())
def processLines(
job: ParrotJob,
lines: Seq[String]) {
lines.map(client.echo(_))
}
}

35. TFE
(HTTP Proxy)
Monorail

36. TFE
(HTTP Proxy)
Monorail Woodstar

37. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
T-Bird
T-Flock +
Haplo
Monorail
Darkwing
Flock(s)

38. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
Tweetypie
Monorail T-Bird
Gizmoduck
T-Flock +
Woodstar
Haplo
TFE TLS
Macaw
Darkwing
+Swift
Social
Graph
Service
Macaw
Flock(s)
+Disco
Story
Service

39. where are we?
⇢ team organization that mimics the
software stack
⇢ able to launch massive features in
parallel

40. mentions statuses/show users/show
3000 500 400
2250 375 300
1500 250 200
750 125 100
0 0 0
p50 p95 p999 p50 p95 p999 p50 p95 p999

41. some more statistics
⇢ 45% of traffic on the JVM stack
⇢ we’re a lot faster
⇢ we’re a lot more reliable
⇢ we fix bugs faster
⇢ 12 deploys, yesterday

42. #JoinTheFlock