Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Go at Uber
Prashant Varanasi, Senior Engineer, RPC
April 26, 2016
● Previously at Google — my first production experience with Go
● At Uber, I work on the RPC team
○ Support the Go client ...
● Traditionally used Python and Node.js
○ Node.js is popular in our marketplace
and dispatch services
○ Python is popular ...
● As Uber grew (in features and
engineers), continuous integration
turned into a liability as deploying the
codebase meant...
● In 2014, Aiden Scandella, one of our early
engineers, experimented with Go
○ Deployed a sandbox marketplace for the Uber...
Building up the ecosystem
● No standardized support for Go (build, lint, coverage) in our internal
infrastructure
○ Custom Makefiles integrated with...
go-buildOld Way
Build Infrastructure
From copy+pasted Makefiles and shell scripts to a common submodule
● Before the GO15VENDOREXPERIMENT, we started with godep
○ Vendored dependencies in every repo
○ Even internal dependencie...
● Base libraries used by almost all services: config, metrics, logging, tracing,
RPC
● Utility and extension libraries:
○ ...
● Now have over a hundred services written in Go
● Two years of production experience
○ Integrated runtime metrics (#gorou...
What’s built in Go?
Goal: given a lat/lng pair, find the list of geofences this location falls in
● HTTP/JSON interface
● High throughput, low...
Goal: match riders to drivers, sharding the matching across machines
● Ringpop for sharding the matching by location
● TCh...
● Embeddable application level sharding:
○ Fault detection: provides membership list of alive
nodes using a variation of S...
● Strong forwarding performance, as it was built for Ringpop forwarding
● Built on top of TCP, provides multiplexing and f...
● Strong integration with Thrift
○ Uses net/context
○ Integrates with error values
val, err := client.Get(ctx, key)
if err...
● Often used internally with Python and Node.js
● Uses pprof output to generate the flamegraph
go-torch
Visualization of p...
● Powers the majority of high QPS services at Uber
● Already the most popular language for new services
● Open source our ...
Thanks
Upcoming SlideShare
Loading in …5
×

Go at uber

2,133 views

Published on

Prashant Varanasi on Go at Uber at the NYC Golang Meetup, April 26, 2016

Published in: Engineering
  • Be the first to comment

Go at uber

  1. 1. Go at Uber Prashant Varanasi, Senior Engineer, RPC April 26, 2016
  2. 2. ● Previously at Google — my first production experience with Go ● At Uber, I work on the RPC team ○ Support the Go client library for TChannel, our open source RPC framework ○ Building out the new fleetwide routing and load balancing sidecar ● One of the core reviewers for internal Go libraries ● Maintainer of the internal Go build infrastructure, go-build About me Java-free for 2 years
  3. 3. ● Traditionally used Python and Node.js ○ Node.js is popular in our marketplace and dispatch services ○ Python is popular for business logic as well as data analysis ● Almost all the business logic was behind a monolithic Python API server About Uber From a large monolithic service... The Uber Monolith Users Products Background Checks Trips Cities Vehicles Payments Documents Promos
  4. 4. ● As Uber grew (in features and engineers), continuous integration turned into a liability as deploying the codebase meant deploying everything ● We followed the lead of other hyper- growth companies (i.e., Twitter, Netflix) and broke up the monolith into many microservices. ● This allowed more flexibility in languages About Uber ...to over one thousand microservices Users ProductsTrips Cities Communication between services uses HTTP/JSON or TChannel/Thrift
  5. 5. ● In 2014, Aiden Scandella, one of our early engineers, experimented with Go ○ Deployed a sandbox marketplace for the Uber External API ● By early 2015, Go was used across the company, ○ Data Engineering: A cross-cluster query cache ○ Marketplace: The Geofences service ● Go was selected for the following characteristics: ○ Simple language, high developer productivity ○ Strong concurrency and parallelism built in Introduction of Go Cross-language microservices
  6. 6. Building up the ecosystem
  7. 7. ● No standardized support for Go (build, lint, coverage) in our internal infrastructure ○ Custom Makefiles integrated with go-junit-report and gocov ● Dependency management, used godep ○ Builds passed locally but not on build machines due to missing deps ○ Vendored dependencies hard to manage and review ● Lack of libraries for internal infrastructure ○ E.g., Kafka logging, dynamic configuration, background tasks Making Go a first-class citizen
  8. 8. go-buildOld Way Build Infrastructure From copy+pasted Makefiles and shell scripts to a common submodule
  9. 9. ● Before the GO15VENDOREXPERIMENT, we started with godep ○ Vendored dependencies in every repo ○ Even internal dependencies were vendored as godep restore was flaky ○ Noisy code reviews, hard to update dependencies, ● The glide migration ○ Users can choose to check in their dependencies in the vendor folder ○ Or check in glide.yaml and glide.lock and have dependencies installed before build ○ Use an internal Github mirror to avoid outages affecting internal builds Dependency Management From godep to glide
  10. 10. ● Base libraries used by almost all services: config, metrics, logging, tracing, RPC ● Utility and extension libraries: ○ Transport helpers, JSON + HTTP, TChannel + Thrift ○ Backoff, LRU cache, flags, worker pools ● Libraries for internal infrastructure built up over time ○ Storage (schemaless -- internal key-value store built on MySQL) ○ Translations, AVRO-encoded logging, dynamic config, experiments Libraries for internal infrastructure Monorepo of Go libraries
  11. 11. ● Now have over a hundred services written in Go ● Two years of production experience ○ Integrated runtime metrics (#goroutines, GC stats) ○ Easy profiling of services running in production ○ Investigated and fixed file descriptor leaks in open source libraries ○ Even found a compiler bug that led to stack corruption ● Strong library and infrastructure support ● Lots of resources: documentation, classes, mailing list ● Now the recommended language for new services Experience with Go Writing code is fun again
  12. 12. What’s built in Go?
  13. 13. Goal: given a lat/lng pair, find the list of geofences this location falls in ● HTTP/JSON interface ● High throughput, low latency (P99 < 100ms) ● No infrastructure support, so project handled everything: ○ Config: Used the standard “encoding/json” package ○ Logging: wrapped the standard “log” package, and reported errors to Sentry ○ Metrics: Wrapped open source StatsD client Blog Post Geofences One of the earliest Go services at Uber
  14. 14. Goal: match riders to drivers, sharding the matching across machines ● Ringpop for sharding the matching by location ● TChannel / Thrift for RPC interface ● Uses our internal libraries for: ○ Config (both static, and dynamic configuration) ○ Logging (includes logging to disk, Kafka, and Sentry) ○ Metrics (reported to M3, our internal Metrics infrastructure) ● Much more infrastructure support for Go services Geobase One of the more recent Go services at Uber
  15. 15. ● Embeddable application level sharding: ○ Fault detection: provides membership list of alive nodes using a variation of SWIM (similar to Serf) ○ Consistent hashing: keys are hashed to a node, and failures will not create load imbalances or redistribute every key ○ Forwarding: provides forward of HTTP or TChannel requests ● Used for sharding work, sharded cache, serializing requests to a resource (in-order) Ringpop Scalable, fault-tolerant application-layer sharding. Open source!
  16. 16. ● Strong forwarding performance, as it was built for Ringpop forwarding ● Built on top of TCP, provides multiplexing and framing ○ Supports out of order responses, no head of line blocking ○ Very simple protocol, easy to implement in multiple languages ● Provides high-level RPC features: ○ Timeouts, consistent retry semantics ○ Connection management and smart peer selection ● Integrates with Thrift as a first-class citizen TChannel Intra-datacenter RPC protocol. Open source, available for Go, Java, Node, and Python
  17. 17. ● Strong integration with Thrift ○ Uses net/context ○ Integrates with error values val, err := client.Get(ctx, key) if err != nil { switch err := err.(type) { case *keyvalue.KeyNotFound: // Handle Thrift exception default: // Unknown error } } TChannel + Thrift Intra-datacenter RPC protocol. Open source, available for Go, Java, Node, and Python
  18. 18. ● Often used internally with Python and Node.js ● Uses pprof output to generate the flamegraph go-torch Visualization of profiling output. Open source!
  19. 19. ● Powers the majority of high QPS services at Uber ● Already the most popular language for new services ● Open source our core libraries and tools ■ TChannel, Ringpop, go-torch, yab (beta), zap (beta) ■ Working with open source community on opentracing ● Check out our engineering blog and Github page for more information Go at Uber
  20. 20. Thanks

×