Alex Payne, who has worked at Twitter since 2007, discusses the advantages of Scala for backend services, highlighting its fun, fast performance, and strong community support. He explains why Scala's features, such as rich static typing, immutable variables, and concurrency options, make it a compelling choice over other languages like Ruby. Payne also describes various Scala services and libraries developed at Twitter, emphasizing the importance of reusable code and effective testing.

2. The Why and How
of Scala at Twitter
Alex Payne (@al3x)
Twitter Infrastructure

3. Who’s this guy?
•Working at Twitter since
2007, before it was even a
company.
•Spent a couple years on the
API.
•Co-authored Programming
Scala (O’Reilly, 2009).
•Into programming
languages, scaling stuff.

4. Part I:
Why Scala?

5. So, like, Ruby or Scala?
•Both!
•Ruby mostly on the
frontend.
•Scala mostly on the
backend.
•Thrift and data stores to
connect the two.

6. Elevator Pitch
• It’s fun.
• It’s fast.
• It’s runs on a great virtual machine (JVM).
• It’s got an awesome community.
• It can borrow from Java.

7. No, seriously, why?
• A rich static type system that gets out of your
way when you don’t need it, is awesome when
you do.
• Flexible syntax makes it easy to write
readable, maintainable code.
• Traits for cross-cutting modularity.
• Lots of OOP goodness.

8. I’m not sold yet.
• How about the choice between immutable and
mutable variables?
• Powerful functional programming: mapping,
filtering, folding, currying, so much more.
• Lazy values.
• Pattern matching.
• XML literals, and clean syntax for working
with XML documents built right in!

9. What about concurrency?
• You can do Actor concurrency (message
passing), kinda like Erlang.
• Or, use threads.
• Or, use all the java.util.concurrent tools.
• Or, use other JVM concurrency frameworks
(ex: Netty, Apache Mina).

10. Basically,
it’s the future.

12. Part II:
How do you guys
do Scala?

13. We build services.
• Isolated components, can swap them out.
• Talk to the rest of the system via Thrift (for
now, maybe Avro in the future).
• Independently tested for correctness, load.
• Can have custom operational properties.

14. Scala Services at Twitter
• Kestrel: queuing.
• Flock (and Gizzard): social graph store.
• Hawkwind: people search.
• Hosebird: streaming API.
• more in the works all the time...

15. Thrift Example
struct Results {
1: list<i64> people
2: string suggestion
3: i32 processingTime /* milliseconds */
4: list<i32> timings
5: i32 totalResults
}
service NameSearch {
Results find(1: string name, 2: i32 maxResults, 3: bool
wantSuggestion)
Results find_with_ranking(1: string name, 2: i32 maxResults, 3: bool
wantSuggestion, 4: Ranker ranking)
}

16. We build libraries.
• Reusable chunks of code that we can share
between projects.
• We open source them.
• We try to keep the “NIH” to a minimum, but
we also have very particular requirements.

17. Ostrich
• Gather stats from inside your running process.
• Counters, gauges, and timings.
• Share stats many ways: JMX, JSON-over-
HTTP, plain text Telnet-style socket, log files.

18. Ostrich Examples
Stats.time(“find-users”) {
users.find(queryString)
}
Stats.incr(“write-errors”)
Stats.makeGauge("database-lag") {
dbConnection.getLagInMillis()
}

19. Configgy
• Configuration files and logging.
• Flexible file format offers includes, more.
• Inheritance, variable substitution.
• Tunable logging, Scribe support.
• Subscription API: push and validate
configuration changes.
• Ruby support, Emacs mode.

20. Configgy Examples
hostname = “service001”
port = 8080
use_admin_interface = true
log {
filename = “my-service.log”
level = “error”
roll = “hourly”
}

21. Configgy Examples
Configgy.configure(“/etc/my-service.conf”)
val config = Configgy.config
val port = config.getInt(“port”, 8080)
val hostname = config.getString(“hostname”)
log.info(“service started”)
log.debug(“running on port %d”, port)

22. xrayspecs
• A set of extensions to the fantastic Specs BDD
test framework.
• Handles testing concurrency, time, creating
temporary folders.
• The concurrency features have been rolled into
the main distribution of Specs!

23. xrayspecs Examples
“response arrives from server” in {
get(“/example”) must eventually(notBe(null))
}
“time should stop” in {
Time.freeze()
val nowSnapshot = Time.now
Thread.sleep(30)
Time.now mustEqual nowSnapshot
}

24. scala-json
• A cleaned-up version of the official Scala JSON
codec.
• Makes clever use of parser combinators – a
good way to learn about them.
• Heavily tested, fixes a bunch of edge cases.

25. JSON Examples
val myJsonString = """
{"key": "value", "1": 2, "3": 4,
"myList": [5, 6, 7]
"""
Json.parse(myJsonString)
Json.build(List("user_id", 666))

26. Other Twitter Libraries
• Naggati: build protocols for Apache Mina.
• Smile: a memcached client that uses Actors.
• Querulous: a nice database client.
• Jackhammer: a load testing framework
(coming soon).
• probably stuff I'm leaving out...

27. Stuff We Use
• There's great open source code in the Scala
community.
• We try to use and contribute back to third-
party projects.
• Follow @implicit_ly for new Scala releases.
• Also subscribe to Repopular Scala.

28. sbt – the simple build tool
• Scala’s answer to Ant and Maven.
• Sets up new projects.
• Maintains project configuration, build tasks,
and dependencies in pure Scala. Totally open-
ended.
• Interactive console.
• Will run tasks as soon as files in your project
change – automatically compile and run tests!

29. specs
• As aforementioned: great way to do BDD
testing.
• Set up pre- and post-conditions for
individual tests, suites, etc.
• Extremely flexible and extensible.
• Great, responsive maintainer.
• Supports mocking (we like Mockito).

30. The IDE Question
• We've had the best luck with IntelliJ IDEA,
but it's still pretty rough.
• Most of us use a plain text editor like Emacs,
VIM, or TextMate.
• Combined with sbt, it's actually a really nice
workflow!

31. Questions!
Thanks for listening!