The Why and How of Scala at Twitter
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

The Why and How of Scala at Twitter

  • 34,968 views
Uploaded on

A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and......

A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, both developed at Twitter and otherwise.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • The Why and How of Scala at Twitter
    Are you sure you want to
    Your message goes here
  • Awesome insight, thanks
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
34,968
On Slideshare
34,281
From Embeds
687
Number of Embeds
23

Actions

Shares
Downloads
589
Comments
2
Likes
118

Embeds 687

http://www.scoop.it 384
http://www.slideshare.net 118
http://mahananda 110
http://coderwall.com 17
http://localhost 11
https://twitter.com 11
http://paper.li 6
http://blog.stefandeboey.be 5
http://latenite.stefandeboey.be 5
http://www.iweb34.com 3
http://www.techgig.com 3
https://www.linkedin.com 2
http://latenite.tumblr.com 2
https://twimg0-a.akamaihd.net 1
http://quantlabs.net 1
http://www.linkedin.com 1
http://www.labnol.org 1
http://www.lmodules.com 1
http://mahananda.gslab.com 1
http://twitter.com 1
http://duckduckgo.com 1
http://www.m.techgig.com 1
http://www.techgig.timesjobs.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

Transcript

  • 1. The Why and How of Scala at Twitter Alex Payne (@al3x) Twitter Infrastructure
  • 2. Who’s this guy? •Working at Twitter since 2007, before it was even a company. •Spent a couple years on the API. •Co-authored Programming Scala (O’Reilly, 2009). •Into programming languages, scaling stuff.
  • 3. Part I: Why Scala?
  • 4. So, like, Ruby or Scala? •Both! •Ruby mostly on the frontend. •Scala mostly on the backend. •Thrift and data stores to connect the two.
  • 5. Elevator Pitch • It’s fun. • It’s fast. • It’s runs on a great virtual machine (JVM). • It’s got an awesome community. • It can borrow from Java.
  • 6. No, seriously, why? • A rich static type system that gets out of your way when you don’t need it, is awesome when you do. • Flexible syntax makes it easy to write readable, maintainable code. • Traits for cross-cutting modularity. • Lots of OOP goodness.
  • 7. I’m not sold yet. • How about the choice between immutable and mutable variables? • Powerful functional programming: mapping, filtering, folding, currying, so much more. • Lazy values. • Pattern matching. • XML literals, and clean syntax for working with XML documents built right in!
  • 8. What about concurrency? • You can do Actor concurrency (message passing), kinda like Erlang. • Or, use threads. • Or, use all the java.util.concurrent tools. • Or, use other JVM concurrency frameworks (ex: Netty, Apache Mina).
  • 9. Basically, it’s the future.
  • 10. Part II: How do you guys do Scala?
  • 11. We build services. • Isolated components, can swap them out. • Talk to the rest of the system via Thrift (for now, maybe Avro in the future). • Independently tested for correctness, load. • Can have custom operational properties.
  • 12. Scala Services at Twitter • Kestrel: queuing. • Flock (and Gizzard): social graph store. • Hawkwind: people search. • Hosebird: streaming API. • more in the works all the time...
  • 13. Thrift Example struct Results { 1: list<i64> people 2: string suggestion 3: i32 processingTime /* milliseconds */ 4: list<i32> timings 5: i32 totalResults } service NameSearch { Results find(1: string name, 2: i32 maxResults, 3: bool wantSuggestion) Results find_with_ranking(1: string name, 2: i32 maxResults, 3: bool wantSuggestion, 4: Ranker ranking) }
  • 14. We build libraries. • Reusable chunks of code that we can share between projects. • We open source them. • We try to keep the “NIH” to a minimum, but we also have very particular requirements.
  • 15. Ostrich • Gather stats from inside your running process. • Counters, gauges, and timings. • Share stats many ways: JMX, JSON-over- HTTP, plain text Telnet-style socket, log files.
  • 16. Ostrich Examples Stats.time(“find-users”) { users.find(queryString) } Stats.incr(“write-errors”) Stats.makeGauge("database-lag") { dbConnection.getLagInMillis() }
  • 17. Configgy • Configuration files and logging. • Flexible file format offers includes, more. • Inheritance, variable substitution. • Tunable logging, Scribe support. • Subscription API: push and validate configuration changes. • Ruby support, Emacs mode.
  • 18. Configgy Examples hostname = “service001” port = 8080 use_admin_interface = true log { filename = “my-service.log” level = “error” roll = “hourly” }
  • 19. Configgy Examples Configgy.configure(“/etc/my-service.conf”) val config = Configgy.config val port = config.getInt(“port”, 8080) val hostname = config.getString(“hostname”) log.info(“service started”) log.debug(“running on port %d”, port)
  • 20. xrayspecs • A set of extensions to the fantastic Specs BDD test framework. • Handles testing concurrency, time, creating temporary folders. • The concurrency features have been rolled into the main distribution of Specs!
  • 21. xrayspecs Examples “response arrives from server” in { get(“/example”) must eventually(notBe(null)) } “time should stop” in { Time.freeze() val nowSnapshot = Time.now Thread.sleep(30) Time.now mustEqual nowSnapshot }
  • 22. scala-json • A cleaned-up version of the official Scala JSON codec. • Makes clever use of parser combinators – a good way to learn about them. • Heavily tested, fixes a bunch of edge cases.
  • 23. JSON Examples val myJsonString = """ {"key": "value", "1": 2, "3": 4, "myList": [5, 6, 7] """ Json.parse(myJsonString) Json.build(List("user_id", 666))
  • 24. Other Twitter Libraries • Naggati: build protocols for Apache Mina. • Smile: a memcached client that uses Actors. • Querulous: a nice database client. • Jackhammer: a load testing framework (coming soon). • probably stuff I'm leaving out...
  • 25. Stuff We Use • There's great open source code in the Scala community. • We try to use and contribute back to third- party projects. • Follow @implicit_ly for new Scala releases. • Also subscribe to Repopular Scala.
  • 26. sbt – the simple build tool • Scala’s answer to Ant and Maven. • Sets up new projects. • Maintains project configuration, build tasks, and dependencies in pure Scala. Totally open- ended. • Interactive console. • Will run tasks as soon as files in your project change – automatically compile and run tests!
  • 27. specs • As aforementioned: great way to do BDD testing. • Set up pre- and post-conditions for individual tests, suites, etc. • Extremely flexible and extensible. • Great, responsive maintainer. • Supports mocking (we like Mockito).
  • 28. The IDE Question • We've had the best luck with IntelliJ IDEA, but it's still pretty rough. • Most of us use a plain text editor like Emacs, VIM, or TextMate. • Combined with sbt, it's actually a really nice workflow!
  • 29. Questions! Thanks for listening!