• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
The Why and How of Scala at Twitter
 

The Why and How of Scala at Twitter

on

  • 31,860 views

A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, ...

A presentation at Twitter's official developer conference, Chirp, about why we use the Scala programming language and how we build services in it. Provides a tour of a number of libraries and tools, both developed at Twitter and otherwise.

Statistics

Views

Total Views
31,860
Views on SlideShare
31,223
Embed Views
637

Actions

Likes
112
Downloads
550
Comments
2

22 Embeds 637

http://www.scoop.it 338
http://www.slideshare.net 118
http://mahananda 108
http://coderwall.com 17
http://localhost 11
https://twitter.com 11
http://paper.li 6
http://blog.stefandeboey.be 5
http://latenite.stefandeboey.be 5
http://www.iweb34.com 3
http://www.techgig.com 3
http://latenite.tumblr.com 2
http://www.linkedin.com 1
http://www.labnol.org 1
http://www.lmodules.com 1
http://mahananda.gslab.com 1
http://www.techgig.timesjobs.com 1
http://duckduckgo.com 1
http://www.m.techgig.com 1
https://twimg0-a.akamaihd.net 1
http://twitter.com 1
http://quantlabs.net 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • The Why and How of Scala at Twitter
    Are you sure you want to
    Your message goes here
    Processing…
  • Awesome insight, thanks
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

The Why and How of Scala at Twitter The Why and How of Scala at Twitter Presentation Transcript

  • The Why and How of Scala at Twitter Alex Payne (@al3x) Twitter Infrastructure
  • Who’s this guy? •Working at Twitter since 2007, before it was even a company. •Spent a couple years on the API. •Co-authored Programming Scala (O’Reilly, 2009). •Into programming languages, scaling stuff.
  • Part I: Why Scala?
  • So, like, Ruby or Scala? •Both! •Ruby mostly on the frontend. •Scala mostly on the backend. •Thrift and data stores to connect the two.
  • Elevator Pitch • It’s fun. • It’s fast. • It’s runs on a great virtual machine (JVM). • It’s got an awesome community. • It can borrow from Java.
  • No, seriously, why? • A rich static type system that gets out of your way when you don’t need it, is awesome when you do. • Flexible syntax makes it easy to write readable, maintainable code. • Traits for cross-cutting modularity. • Lots of OOP goodness.
  • I’m not sold yet. • How about the choice between immutable and mutable variables? • Powerful functional programming: mapping, filtering, folding, currying, so much more. • Lazy values. • Pattern matching. • XML literals, and clean syntax for working with XML documents built right in!
  • What about concurrency? • You can do Actor concurrency (message passing), kinda like Erlang. • Or, use threads. • Or, use all the java.util.concurrent tools. • Or, use other JVM concurrency frameworks (ex: Netty, Apache Mina).
  • Basically, it’s the future.
  • Part II: How do you guys do Scala?
  • We build services. • Isolated components, can swap them out. • Talk to the rest of the system via Thrift (for now, maybe Avro in the future). • Independently tested for correctness, load. • Can have custom operational properties.
  • Scala Services at Twitter • Kestrel: queuing. • Flock (and Gizzard): social graph store. • Hawkwind: people search. • Hosebird: streaming API. • more in the works all the time...
  • Thrift Example struct Results { 1: list<i64> people 2: string suggestion 3: i32 processingTime /* milliseconds */ 4: list<i32> timings 5: i32 totalResults } service NameSearch { Results find(1: string name, 2: i32 maxResults, 3: bool wantSuggestion) Results find_with_ranking(1: string name, 2: i32 maxResults, 3: bool wantSuggestion, 4: Ranker ranking) }
  • We build libraries. • Reusable chunks of code that we can share between projects. • We open source them. • We try to keep the “NIH” to a minimum, but we also have very particular requirements.
  • Ostrich • Gather stats from inside your running process. • Counters, gauges, and timings. • Share stats many ways: JMX, JSON-over- HTTP, plain text Telnet-style socket, log files.
  • Ostrich Examples Stats.time(“find-users”) { users.find(queryString) } Stats.incr(“write-errors”) Stats.makeGauge("database-lag") { dbConnection.getLagInMillis() }
  • Configgy • Configuration files and logging. • Flexible file format offers includes, more. • Inheritance, variable substitution. • Tunable logging, Scribe support. • Subscription API: push and validate configuration changes. • Ruby support, Emacs mode.
  • Configgy Examples hostname = “service001” port = 8080 use_admin_interface = true log { filename = “my-service.log” level = “error” roll = “hourly” }
  • Configgy Examples Configgy.configure(“/etc/my-service.conf”) val config = Configgy.config val port = config.getInt(“port”, 8080) val hostname = config.getString(“hostname”) log.info(“service started”) log.debug(“running on port %d”, port)
  • xrayspecs • A set of extensions to the fantastic Specs BDD test framework. • Handles testing concurrency, time, creating temporary folders. • The concurrency features have been rolled into the main distribution of Specs!
  • xrayspecs Examples “response arrives from server” in { get(“/example”) must eventually(notBe(null)) } “time should stop” in { Time.freeze() val nowSnapshot = Time.now Thread.sleep(30) Time.now mustEqual nowSnapshot }
  • scala-json • A cleaned-up version of the official Scala JSON codec. • Makes clever use of parser combinators – a good way to learn about them. • Heavily tested, fixes a bunch of edge cases.
  • JSON Examples val myJsonString = """ {"key": "value", "1": 2, "3": 4, "myList": [5, 6, 7] """ Json.parse(myJsonString) Json.build(List("user_id", 666))
  • Other Twitter Libraries • Naggati: build protocols for Apache Mina. • Smile: a memcached client that uses Actors. • Querulous: a nice database client. • Jackhammer: a load testing framework (coming soon). • probably stuff I'm leaving out...
  • Stuff We Use • There's great open source code in the Scala community. • We try to use and contribute back to third- party projects. • Follow @implicit_ly for new Scala releases. • Also subscribe to Repopular Scala.
  • sbt – the simple build tool • Scala’s answer to Ant and Maven. • Sets up new projects. • Maintains project configuration, build tasks, and dependencies in pure Scala. Totally open- ended. • Interactive console. • Will run tasks as soon as files in your project change – automatically compile and run tests!
  • specs • As aforementioned: great way to do BDD testing. • Set up pre- and post-conditions for individual tests, suites, etc. • Extremely flexible and extensible. • Great, responsive maintainer. • Supports mocking (we like Mockito).
  • The IDE Question • We've had the best luck with IntelliJ IDEA, but it's still pretty rough. • Most of us use a plain text editor like Emacs, VIM, or TextMate. • Combined with sbt, it's actually a really nice workflow!
  • Questions! Thanks for listening!