The Why and How of Scala at Twitter

  • 32,452 views
Uploaded on

A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, …

A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, both developed at Twitter and otherwise.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • The Why and How of Scala at Twitter
    Are you sure you want to
    Your message goes here
  • Awesome insight, thanks
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
32,452
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
617
Comments
2
Likes
131

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide































Transcript

  • 1. The Why and How of Scala at Twitter Alex Payne (@al3x) Twitter Infrastructure
  • 2. Who’s this guy? •Working at Twitter since 2007, before it was even a company. •Spent a couple years on the API. •Co-authored Programming Scala (O’Reilly, 2009). •Into programming languages, scaling stuff.
  • 3. Part I: Why Scala?
  • 4. So, like, Ruby or Scala? •Both! •Ruby mostly on the frontend. •Scala mostly on the backend. •Thrift and data stores to connect the two.
  • 5. Elevator Pitch • It’s fun. • It’s fast. • It’s runs on a great virtual machine (JVM). • It’s got an awesome community. • It can borrow from Java.
  • 6. No, seriously, why? • A rich static type system that gets out of your way when you don’t need it, is awesome when you do. • Flexible syntax makes it easy to write readable, maintainable code. • Traits for cross-cutting modularity. • Lots of OOP goodness.
  • 7. I’m not sold yet. • How about the choice between immutable and mutable variables? • Powerful functional programming: mapping, filtering, folding, currying, so much more. • Lazy values. • Pattern matching. • XML literals, and clean syntax for working with XML documents built right in!
  • 8. What about concurrency? • You can do Actor concurrency (message passing), kinda like Erlang. • Or, use threads. • Or, use all the java.util.concurrent tools. • Or, use other JVM concurrency frameworks (ex: Netty, Apache Mina).
  • 9. Basically, it’s the future.
  • 10. Part II: How do you guys do Scala?
  • 11. We build services. • Isolated components, can swap them out. • Talk to the rest of the system via Thrift (for now, maybe Avro in the future). • Independently tested for correctness, load. • Can have custom operational properties.
  • 12. Scala Services at Twitter • Kestrel: queuing. • Flock (and Gizzard): social graph store. • Hawkwind: people search. • Hosebird: streaming API. • more in the works all the time...
  • 13. Thrift Example struct Results { 1: list<i64> people 2: string suggestion 3: i32 processingTime /* milliseconds */ 4: list<i32> timings 5: i32 totalResults } service NameSearch { Results find(1: string name, 2: i32 maxResults, 3: bool wantSuggestion) Results find_with_ranking(1: string name, 2: i32 maxResults, 3: bool wantSuggestion, 4: Ranker ranking) }
  • 14. We build libraries. • Reusable chunks of code that we can share between projects. • We open source them. • We try to keep the “NIH” to a minimum, but we also have very particular requirements.
  • 15. Ostrich • Gather stats from inside your running process. • Counters, gauges, and timings. • Share stats many ways: JMX, JSON-over- HTTP, plain text Telnet-style socket, log files.
  • 16. Ostrich Examples Stats.time(“find-users”) { users.find(queryString) } Stats.incr(“write-errors”) Stats.makeGauge("database-lag") { dbConnection.getLagInMillis() }
  • 17. Configgy • Configuration files and logging. • Flexible file format offers includes, more. • Inheritance, variable substitution. • Tunable logging, Scribe support. • Subscription API: push and validate configuration changes. • Ruby support, Emacs mode.
  • 18. Configgy Examples hostname = “service001” port = 8080 use_admin_interface = true log { filename = “my-service.log” level = “error” roll = “hourly” }
  • 19. Configgy Examples Configgy.configure(“/etc/my-service.conf”) val config = Configgy.config val port = config.getInt(“port”, 8080) val hostname = config.getString(“hostname”) log.info(“service started”) log.debug(“running on port %d”, port)
  • 20. xrayspecs • A set of extensions to the fantastic Specs BDD test framework. • Handles testing concurrency, time, creating temporary folders. • The concurrency features have been rolled into the main distribution of Specs!
  • 21. xrayspecs Examples “response arrives from server” in { get(“/example”) must eventually(notBe(null)) } “time should stop” in { Time.freeze() val nowSnapshot = Time.now Thread.sleep(30) Time.now mustEqual nowSnapshot }
  • 22. scala-json • A cleaned-up version of the official Scala JSON codec. • Makes clever use of parser combinators – a good way to learn about them. • Heavily tested, fixes a bunch of edge cases.
  • 23. JSON Examples val myJsonString = """ {"key": "value", "1": 2, "3": 4, "myList": [5, 6, 7] """ Json.parse(myJsonString) Json.build(List("user_id", 666))
  • 24. Other Twitter Libraries • Naggati: build protocols for Apache Mina. • Smile: a memcached client that uses Actors. • Querulous: a nice database client. • Jackhammer: a load testing framework (coming soon). • probably stuff I'm leaving out...
  • 25. Stuff We Use • There's great open source code in the Scala community. • We try to use and contribute back to third- party projects. • Follow @implicit_ly for new Scala releases. • Also subscribe to Repopular Scala.
  • 26. sbt – the simple build tool • Scala’s answer to Ant and Maven. • Sets up new projects. • Maintains project configuration, build tasks, and dependencies in pure Scala. Totally open- ended. • Interactive console. • Will run tasks as soon as files in your project change – automatically compile and run tests!
  • 27. specs • As aforementioned: great way to do BDD testing. • Set up pre- and post-conditions for individual tests, suites, etc. • Extremely flexible and extensible. • Great, responsive maintainer. • Supports mocking (we like Mockito).
  • 28. The IDE Question • We've had the best luck with IntelliJ IDEA, but it's still pretty rough. • Most of us use a plain text editor like Emacs, VIM, or TextMate. • Combined with sbt, it's actually a really nice workflow!
  • 29. Questions! Thanks for listening!