Introduction to various open source projects by Twitter. Talk given at Vienna Scala User Group on Aug. 27, 2014.

1. The Twitter Stack
Finagle, Ostrich, Zipkin,…
Dominik Gruber, @the_dom
Vienna Scala User Group – Aug. 27, 2014

2. Dominik Gruber • @the_domTwitter Stack
Agenda
• Introduction
• Prominent Adopters
• Finagle
• Ostrich
• Zipkin
• Iago
• Finatra

3. Dominik Gruber • @the_domTwitter Stack
Introduction
• Twitter started to move off Ruby on Rails in 2010
• Currently still on Scala 2.9

4. Dominik Gruber • @the_domTwitter Stack
Prominent Adopters
• Box
• eBay
• Etsy
• Foursquare
• Gigya
• Nest
• Pinterest
• Rdio
• SoundCloud
• Tumblr

5. Dominik Gruber • @the_domTwitter Stack

6. Dominik Gruber • @the_domTwitter Stack
Finagle
“Finagle is an extensible RPC system for the
JVM, used to construct high-concurrency
servers. Finagle implements uniform client
and server APIs for several protocols, and is
designed for high performance and
concurrency. Most of Finagle’s code is
protocol agnostic, simplifying the
implementation of new protocols.”

7. Dominik Gruber • @the_domTwitter Stack
Finagle provides…
• Connection pools
• Failure detection
• Failover strategies
• Load balancers
• Back-pressure techniques

8. Dominik Gruber • @the_domTwitter Stack
Finagle…
• publishes standard statistics, logs, and exception
reports
• supports distributed tracing across protocols
• optionally uses ZooKeeper for cluster management
• supports common sharding strategies

9. Dominik Gruber • @the_domTwitter Stack
Modules (Examples)
• finagle-http
• finagle-memcached
• finagle-mysql
• finagle-protobuf
• finagle-redis
• finagle-thrift

10. Dominik Gruber • @the_domTwitter Stack
Basic abstractions
• Future
• Service
• Filter

11. Dominik Gruber • @the_domTwitter Stack
Ostrich

12. Dominik Gruber • @the_domTwitter Stack
Ostrich
• A stats collector & reporter
• Collect runtime statistics (counters, gauges, metrics,
and labels)
• Report those statistics through a simple web
interface or into log files
• Similar to Metrics
https://github.com/dropwizard/metrics

13. Dominik Gruber • @the_domTwitter Stack

14. Dominik Gruber • @the_domTwitter Stack
Zipkin
“Zipkin is a distributed tracing system that
helps us gather timing data for all the
disparate services at Twitter. It manages both
the collection and lookup of this data through
a Collector and a Query service.”

15. Dominik Gruber • @the_domTwitter Stack
Zipkin

16. Dominik Gruber • @the_domTwitter Stack
Zipkin’s Architecture

17. Dominik Gruber • @the_domTwitter Stack
Zipkin
• “Scribe” to transport traces to Zipkin and Hadoop
• Data stored in either Cassandra, Redis, HBase,
MySQL, PostgreSQL, SQLite, or H2

18. Dominik Gruber • @the_domTwitter Stack
Iago

19. Dominik Gruber • @the_domTwitter Stack
Iago
• A load generator that replays production or synthetic
traffic
• Iago supports arbitrarily high rates of traffic via built-
in support for creating cluster

20. Dominik Gruber • @the_domTwitter Stack
Protocols
• HTTP
• Thrift
• Memcache
• Kestrel (A queueing system from Twitter)
• UDP

21. Dominik Gruber • @the_domTwitter Stack
Finatra

22. Dominik Gruber • @the_domTwitter Stack

23. Dominik Gruber • @the_domTwitter Stack
Features
• Fully Async using Futures
• Mustache Templates
• Detailed Metrics (p99,p999, etc)
• Profile/Debug over HTTP
• SSL/TLS Support
• Deploys with a single jar

24. Dominik Gruber • @the_domTwitter Stack
There’s more…
•
• Summingbird
• https://github.com/twitter

25. Dominik Gruber • @the_domTwitter Stack
Q & A

26. Dominik Gruber • @the_domTwitter Stack
Source
• http://blog.oskarsson.nu/post/40196324612/the-
twitter-stack
https://blog.twitter.com/2011/finagle-a-protocol-
agnostic-rpc-system
• https://www.youtube.com/watch?v=whuT_-cFb5I
https://www.youtube.com/watch?v=QdER-Zl_YCM
• https://github.com/twitter
https://github.com/finagle