Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Exploring Twitter’s
Finagle technology stack
for microservices
Tomasz Kogut / @almendar
XII Tricity Scala User Group Meetup
About me:
• Software Engineer at
• Working in the internet advertisement
industry
• I try to create distributed, highly pe...
Microservices
questionnaire
Who uses μ-services approach?
Who has a system that would like to break down into smaller pieces?
What we already know?
• Working on huge monolithic projects is hard
• Complexity and interdependencies become
incomprehens...
The plan
1. Create multiple git repositories
2. Create/copy/move code around repos
3. Pour it all with a fair share of
RES...
Let’s go to production!
So what
happened?
Paying for the lunch
• Easy to develop
• Hard to operate
• More fine-grained services boost both
traits
Things to address
• A lot of stuff from “Fallacies of distributed computing"
• Tracing request across different services
•...
• Historically we were a .Net shop
• Think spring-like approach
• Moved to API first approach (although we were
running so...
• We didn’t want to use akka clustering
• Recreating utils e.g. request retry
• Establishing new project = work upfront be...
• Creating json is annoying and automatic derivation from
classes was useless most of the time
• We had to think how to pr...
What is it?
• Protocol agnostic RPC system for JVM
• Based on Netty
• Allows for building servers and clients
• Core platf...
Foundations
• Futures
• Services
• Filters
Future
• NOT the Scala one unfortunately but very close
• Some api differences
- import com.twitter.util.Future
- Future.v...
Service
• Basic building block in Finagle
• Just a function that returns a Future
• Servers implement services to which Fi...
Example local service (1)
import com.twitter.finagle.Service
import com.twitter.util.Future
val lengthService = Service.mk...
import com.twitter.finagle.http
import com.twitter.finagle.Service
import com.twitter.util.Future
import com.twitter.finag...
import com.twitter.finagle.http.Method.Get
import com.twitter.finagle.{Http, Service}
import com.twitter.finagle.http.{Req...
Other protocols client
examples
var memCache = ClientBuilder()
.name("mc")
.codec(Memcached())
.hostConnectionLimit(config...
Services represents
clients and servers
symmetrically
Please notice that:
Filter
• Again a function that returns a Future
• Filters are always attached to Services altering their
behavior
• Used f...
Filter
• Filters just like any other functions
compose
• There is a andThen method that chains
different filters with each...
Local service filter
import com.twitter.finagle.{Filter, Service}
import com.twitter.util.Future
val someStringMetrics = S...
import com.twitter.finagle.{Http,Service, Filter}
import com.twitter.finagle.http.{Request,Response}
import com.twitter.fi...
Stacking filters
recordHandletime andThen
traceRequest andThen
collectJvmStats andThen
parseRequest andThen
logRequest and...
Cool client filters
(or wannabe-filters)
• MonitorFilter - handling exceptions
• StatsFilter - exposing metrics
• RetryFil...
• Unfortunately not all functionality can be
expressed with Filters
• Some things are a bit hairy - e.g. request
cancelati...
There is more in Finagle to
explore:
• Thrift protocol
• Mux - multiplexing RPC
• ZooKeeper support
• Network location nam...
Twitter
Server
Even more goodies
Twitter Server
• A “template” for finagle-based server
• Flags
• Logging
• Metrics
• Admin interface
Flags (1)
• Cmd line arguments passed to the
application
• A remedy to “How do I start this thing
and what are the options...
Flags (2)
val addr = flag("bind", new InetSocketAddress(0), "Bind address")
val durations = flag("alarms", (1.second, 5.se...
Logging
• Twitter server provides a logging trait
• It can be configured with standard flags
• Can be tuned on-the-fly
Admin interface loggers
control
Metrics
• Defining own statistics
• JVM stats
• Processing and network stats
Others
• Linting your server for problems
• Looking at threads
• Profiling
Admin interface
Finatra
What is it?
• Finatra is build atop Finagle and Twitter
Server
• Finatra does all the heavy-lifting that one
needs to do w...
Finatra features
• Dependency injection using Guice
• JSON handling with Jackson
• Mustache support
• Routing
• Integrates...
package pl.tk.finagle.recommendation.controller
import javax.inject.{Inject, Singleton}
import com.twitter.finagle.http.Re...
Unified error reporting
➜ finagle-spike git:(master) ✗ curl -v 127.0.0.1:2001/recommend/32/wsf
* Trying 127.0.0.1...
* Con...
JSON Support
• Build on jackson-module-scala
• Support for case classes (de)serlialization
• Integrated with Joda
• Error ...
Customizing Json
class Server extends HttpServer {
override def jacksonModule = CustomJacksonModule
...
}
object CustomJac...
Validation
import com.twitter.finatra.validation._
import org.joda.time.DateTime
case class GroupRequest(@NotEmpty name: S...
Testing (1)
• Based on ScalaTest
• Feature testing (both black box and white box)
- looking at external interface of our s...
Testing (2)
• Smart JSON diff
• Integration with DI
• Easy mocking
• Embedded http server with our service
last but not
least…
What is it?
• Distributed request tracing
• Based on Google Dapper paper
• Helps getting insight on how the services
inter...
Zipkin
Python pyramid_zipkin
Pyramid Http
(B3)
Http (B3) Kafka | Scribe Yes
py2, py3
support.
Java brave
Jersey,
RestEASY,...
Zipkin Architecture
DEMO APP
Recommendation
ProductsService
UserProfile
1 GetProducts
2 GetPromotedProducts
GetUser
CookieId, Eshop
Thank you!
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
Exploring Twitter's Finagle technology stack for microservices
Upcoming SlideShare
Loading in …5
×

Exploring Twitter's Finagle technology stack for microservices

2,416 views

Published on

this presentation is an intro to the finagle ecosystem

Published in: Software
  • High Paying Jobs On Facebook And Twitter... How? ♥♥♥ http://t.cn/AieXipTS
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • My son was struggling with GCSE revision and being a typical boy his desire to stay inside and study was not too high on the agenda! His maths has always been pretty average, and he has always had very little confidence in his mathematical abilities. Jeevan's programme is excellent. The break down of all mathematical problems has completely changed the way my son approaches his work and he is no longer doubting his ability. I have no doubt that this programme is the best there is and would have complete confidence in recommending it to anyone studying GCSE maths. On a side note, Jeevan is a perfect gentleman. He is very approachable, kind and generous with his advice. An efficient young man who truly cares about what he is offering. If you purchase this programme, you will certainly not be disappointed. ➤➤ http://ishbv.com/jeevan91/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • After many failed attempts, I almost packed in my GCSE maths altogether. But fortunately I didn't, thanks to Jeevan's guide! When I read it, I found out exactly where I was going wrong all this time! I followed his approach and achieved 90% in my next sitting. I was shocked and I thought it was a total fluke so I put his strategy to the test again. This time, I got 100%! Fantastic! If only I came across Jeevan's strategy sooner. Learn more.. ◆◆◆ http://ishbv.com/jeevan91/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Exploring Twitter's Finagle technology stack for microservices

  1. 1. Exploring Twitter’s Finagle technology stack for microservices Tomasz Kogut / @almendar XII Tricity Scala User Group Meetup
  2. 2. About me: • Software Engineer at • Working in the internet advertisement industry • I try to create distributed, highly performant systems that will run on the JVM • Programming in Scala since ~2011
  3. 3. Microservices
  4. 4. questionnaire Who uses μ-services approach? Who has a system that would like to break down into smaller pieces?
  5. 5. What we already know? • Working on huge monolithic projects is hard • Complexity and interdependencies become incomprehensible at some point • We all fell in our guts that breaking it down is a way to go • Single responsibility principle but for services
  6. 6. The plan 1. Create multiple git repositories 2. Create/copy/move code around repos 3. Pour it all with a fair share of REST/HTTP 4. Extract common code/models into libs 5. Run it!
  7. 7. Let’s go to production!
  8. 8. So what happened?
  9. 9. Paying for the lunch • Easy to develop • Hard to operate • More fine-grained services boost both traits
  10. 10. Things to address • A lot of stuff from “Fallacies of distributed computing" • Tracing request across different services • Service discovery and tracing service topology • Message serialization • Distributing load across nodes • Measuring performance and finding bottlenecks is not easy • Interface contracts can introduce hidden coupling • Duplication of effort
  11. 11. • Historically we were a .Net shop • Think spring-like approach • Moved to API first approach (although we were running some kind of μ-service approach to some extend) • Splitting one of the business features into small- chunks using whatever tech stack we want so we went akka/akka-http
  12. 12. • We didn’t want to use akka clustering • Recreating utils e.g. request retry • Establishing new project = work upfront before creating any business value e.g. metrics • Every project had it’s own structure - mental context switching • REST is good but other protocols might be better • Fighting the “spray DSL” and marshallers 😀
  13. 13. • Creating json is annoying and automatic derivation from classes was useless most of the time • We had to think how to properly version our apis and hold backward compatibility • We had situations where multiple services needed to be deployed for the solution to work (hidden monolith) • How do I run this service locally?
  14. 14. What is it? • Protocol agnostic RPC system for JVM • Based on Netty • Allows for building servers and clients • Core platform on which Twitter is build • "Your Server as a Function" by Marius Eriksen
  15. 15. Foundations • Futures • Services • Filters
  16. 16. Future • NOT the Scala one unfortunately but very close • Some api differences - import com.twitter.util.Future - Future.value == Future.successful - Future.exception == Future.failed - Future.collect == Future.sequence • Usual monadic composition with flatMap and map • There is also twitter’s Try, Duration and a bunch of others…
  17. 17. Service • Basic building block in Finagle • Just a function that returns a Future • Servers implement services to which Finagle dispatches incoming requests • Finagle furnishes clients with instances of Service representing either virtual or concrete remote servers abstract class Service[-Req, +Rep] extends (Req => Future[Rep])
  18. 18. Example local service (1) import com.twitter.finagle.Service import com.twitter.util.Future val lengthService = Service.mk[String, Int] { req => Future.value(req.length) } lengthService("Hello TSUG").foreach(println)
  19. 19. import com.twitter.finagle.http import com.twitter.finagle.Service import com.twitter.util.Future import com.twitter.finagle.Http val service = new Service[http.Request, http.Response] { def apply(req: http.Request): Future[http.Response] = Future.value( http.Response(req.version, http.Status.Ok) ) } val server = Http.serve(":8080", service) Example http server (2)
  20. 20. import com.twitter.finagle.http.Method.Get import com.twitter.finagle.{Http, Service} import com.twitter.finagle.http.{Request, Response} import com.twitter.finagle.service.FailFastFactory.FailFast import com.twitter.util.Future val productServiceHttpClient: Service[Request, Response] = Http.client.configured(FailFast(false)).newClient("127.0.0.1:8080", "productService").toService val response: Future[Response] = productServiceHttpClient(Request(Get,"/products/22")) response.foreach{ response => println(response.statusCode) println(response.contentString) } Example http client (3)
  21. 21. Other protocols client examples var memCache = ClientBuilder() .name("mc") .codec(Memcached()) .hostConnectionLimit(config.concurrency()) .hosts(config.hosts()) val mySqlClient = Mysql.client .withCredentials(username(), password()) .withDatabase(dbname()) .newRichClient("%s:%d".format(host().getHostName, host().getPort))
  22. 22. Services represents clients and servers symmetrically Please notice that:
  23. 23. Filter • Again a function that returns a Future • Filters are always attached to Services altering their behavior • Used for application agnostic concerns such as: timeouts, retry policies, service statistics, and authentication abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])
  24. 24. Filter • Filters just like any other functions compose • There is a andThen method that chains different filters with each other and in the end with service
  25. 25. Local service filter import com.twitter.finagle.{Filter, Service} import com.twitter.util.Future val someStringMetrics = Service.mk[String,Int] { req => Future.value(req.length) } val evenMetricsFilter = new Filter[Float, Boolean, String, Int] { override def apply(input: Float, stringMetricsService: Service[String, Int]): Future[Boolean] = { stringMetricsService(input.toString).map(_ % 2 == 0) } } (evenMetricsFilter andThen someStringMetrics)(1.234f)
  26. 26. import com.twitter.finagle.{Http,Service, Filter} import com.twitter.finagle.http.{Request,Response} import com.twitter.finagle.service.TimeoutFilter import com.twitter.util.MockTimer import com.twitter.conversions.time._ val httpClient: Service[Request, Response] = Http.client.newService("twitter.com") val timeoutFilter: Filter[Request, Response, Request, Response] = new TimeoutFilter[Request, Response](30.seconds, new MockTimer) val httpClientWithTimeout: Service[Request, Response] = timeoutFilter andThen httpClient Filter example http client WARNING This is only a showcase, timeouts for http clients are best configured by Http.client object How to implement this?
  27. 27. Stacking filters recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer Fronted webservers configuration at twitter:
  28. 28. Cool client filters (or wannabe-filters) • MonitorFilter - handling exceptions • StatsFilter - exposing metrics • RetryFilter - retying calls with configured budget and a back-off policy • TimeoutFilter - mentioned earlier, for different aspects like idle time, request time, connection time ect. • LoadBalancing (not really a filter) - distributed load across nodes, just pass multiple host values • FailFast (not really a filter) - mark node as dead for a certain period of time
  29. 29. • Unfortunately not all functionality can be expressed with Filters • Some things are a bit hairy - e.g. request cancelation • Beneath implementations is non-trivial but comprehensible • But the good part is that it all works out to the box for free with almost none configuration!
  30. 30. There is more in Finagle to explore: • Thrift protocol • Mux - multiplexing RPC • ZooKeeper support • Network location naming
  31. 31. Twitter Server Even more goodies
  32. 32. Twitter Server • A “template” for finagle-based server • Flags • Logging • Metrics • Admin interface
  33. 33. Flags (1) • Cmd line arguments passed to the application • A remedy to “How do I start this thing and what are the options?” • Smart with type inference and parsing capabilities
  34. 34. Flags (2) val addr = flag("bind", new InetSocketAddress(0), "Bind address") val durations = flag("alarms", (1.second, 5.second), "2 alarm durations") $ java -jar target/myserver-1.0.0-SNAPSHOT.jar -help AdvancedServer -alarm_durations='1.seconds,5.seconds': 2 alarm durations -help='false': Show this help -admin.port=':9990': Admin http server port -bind=':0': Network interface to use -log.level='INFO': Log level -log.output='/dev/stderr': Output file -what='hello': String to return Option for fail fast when missing a flag
  35. 35. Logging • Twitter server provides a logging trait • It can be configured with standard flags • Can be tuned on-the-fly
  36. 36. Admin interface loggers control
  37. 37. Metrics • Defining own statistics • JVM stats • Processing and network stats
  38. 38. Others • Linting your server for problems • Looking at threads • Profiling
  39. 39. Admin interface
  40. 40. Finatra
  41. 41. What is it? • Finatra is build atop Finagle and Twitter Server • Finatra does all the heavy-lifting that one needs to do when working only with Finagle
  42. 42. Finatra features • Dependency injection using Guice • JSON handling with Jackson • Mustache support • Routing • Integrates it all together nicely
  43. 43. package pl.tk.finagle.recommendation.controller import javax.inject.{Inject, Singleton} import com.twitter.finagle.http.Request import com.twitter.finagle.tracing.ClientTracingFilter.TracingFilter import com.twitter.finatra.http.Controller import com.twitter.finatra.request._ import com.twitter.inject.Logging import pl.tk.finagle.recommendation.engine.{RecommendationCmd, RecommendationEngine} case class RecommendationFromCookieRequest(@Inject request: Request, @RouteParam cookieId: Int, @RouteParam eshopId: Int, @Header `x-auth`: String) @Singleton class RecommendationController @Inject() (recommendationEngine: RecommendationEngine) extends Controller with Logging { get("/recommend/:eshop_id/:cookie_id") { r: RecommendationFromCookieRequest => infoResult("Hello") { TracingFilter("RecommendationEngine") andThen recommendationEngine apply RecommendationCmd(r.cookieId, r.eshopId) } } }
  44. 44. Unified error reporting ➜ finagle-spike git:(master) ✗ curl -v 127.0.0.1:2001/recommend/32/wsf * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 2001 (#0) > GET /recommend/32/wsf HTTP/1.1 > Host: 127.0.0.1:2001 > User-Agent: curl/7.43.0 > Accept: */* > < HTTP/1.1 400 Bad Request < Content-Type: application/json; charset=utf-8 < Server: Finatra < Date: Wed, 30 Mar 2016 07:05:53 +00:00 < Content-Length: 79 < * Connection #0 to host 127.0.0.1 left intact {"errors":["cookie_id: 'wsf' is not a valid int","x-auth: header is required"]}%
  45. 45. JSON Support • Build on jackson-module-scala • Support for case classes (de)serlialization • Integrated with Joda • Error accumulation (instead of fail-fast) • Integration with routing • Most of the cases just return object
  46. 46. Customizing Json class Server extends HttpServer { override def jacksonModule = CustomJacksonModule ... } object CustomJacksonModule extends FinatraJacksonModule { override val additionalJacksonModules = Seq( new SimpleModule { addSerializer(LocalDateParser) }) override val serializationInclusion = Include.NON_EMPTY override val propertyNamingStrategy = CamelCasePropertyNamingStrategy override def additionalMapperConfiguration(mapper: ObjectMapper) { mapper.configure(Feature.WRITE_NUMBERS_AS_STRINGS, true) } }
  47. 47. Validation import com.twitter.finatra.validation._ import org.joda.time.DateTime case class GroupRequest(@NotEmpty name: String, description: Option[String], tweetIds: Set[Long], dates: Dates) { @MethodValidation def validateName = { ValidationResult.validate( name.startsWith("grp-"), "name must start with 'grp-'") } } case class Dates(@PastTime start: DateTime, @PastTime end: DateTime) Also checked at routing time
  48. 48. Testing (1) • Based on ScalaTest • Feature testing (both black box and white box) - looking at external interface of our service • Integration tests - only a subset of modules is instantiated and tested • Finatra does not provide anything for unit testing
  49. 49. Testing (2) • Smart JSON diff • Integration with DI • Easy mocking • Embedded http server with our service
  50. 50. last but not least…
  51. 51. What is it? • Distributed request tracing • Based on Google Dapper paper • Helps getting insight on how the services interact
  52. 52. Zipkin Python pyramid_zipkin Pyramid Http (B3) Http (B3) Kafka | Scribe Yes py2, py3 support. Java brave Jersey, RestEASY, JAXRS2, Apache HttpClient, Mysql Http (B3) Http, Kafka, Scribe Yes Java 6 or higher Scala finagle-zipkin Finagle Http (B3), Thrift Scribe Yes Ruby zipkin-tracer Rack Http (B3) Http, Kafka, Scribe Yes lc support. Ruby 2.0 or higher C# ZipkinTracerM OWIN, Http (B3) Http Yes lc support. Not tied to finagle
  53. 53. Zipkin Architecture
  54. 54. DEMO APP Recommendation ProductsService UserProfile 1 GetProducts 2 GetPromotedProducts GetUser CookieId, Eshop
  55. 55. Thank you!

×