Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Breaking the Monolith - Microservice Extraction at SoundCloud

836 views

Published on

Bob Conference 2015
http://bobkonf.de/2015/kischkel.html

Published in: Software
  • Be the first to comment

Breaking the Monolith - Microservice Extraction at SoundCloud

  1. 1. Breaking the Monolith Microservice Extraction at SoundCloud
  2. 2. Soundcloud ● 11 hours uploaded every minute ● 150 million tracks ● 300 million users 2
  3. 3. History ● Rails 2.3 ● MySQL ● S3 3
  4. 4. What happened then? 4
  5. 5. Success! 5
  6. 6. History ● Rails 2.3 ● MySQL ● AWS ● Cassandra ● Hadoop ● SolR ● RabbitMQ https://developers.soundcloud.com/blog/evolution-of-soundclouds-architecture 6
  7. 7. Still not enough ● More servers ● Add caching layer ● Defer long running tasks to workers 7
  8. 8. Still not enough ● Optimize database schema ● Introduce read slaves ● Dedicated databases for some models 8
  9. 9. Monolith 9
  10. 10. Major pain points ● Testing, building and deploying ● Dependency hell ● “I’d rather not touch this” 10
  11. 11. Rails problems I - No service layer 11 <% Category.all.each do |cat| %> <li><%= cat.name %></li> <% end %>
  12. 12. Rails problems I - No service layer ⇒ Tight coupling with storage layer! 12 <% Category.all.each do |cat| %> <li><%= cat.name %></li> <% end %>
  13. 13. Rails problems II - Active Record Magic 13 class User < ActiveRecord::Base validates_length_of :username, :within => 2..64 before_save :encrypt_password, :accept_terms_of_use has_many :comments, :dependent => :destroy # ... end
  14. 14. Rails problems II - Active Record Magic ⇒ Easy to write, hard to maintain 14 class User < ActiveRecord::Base validates_length_of :username, :within => 2..64 before_save :encrypt_password, :accept_terms_of_use has_many :comments, :dependent => :destroy # ... end
  15. 15. Ruby Problems ● GIL ● Native extensions ● Dependency management can be painful 15
  16. 16. Extract features as Services ● Less painful to maintain ● Easy to replace ● Fun to build 17
  17. 17. An Example: Messages ● 200 million messages ● MySQL database on shared host ● Features: ○ embedded sounds ○ email notifications ○ spam detection 18
  18. 18. Migration Requirements ● New schema to support upcoming features ● Dedicated database ● Zero downtime 19
  19. 19. Chapter 1 The Database
  20. 20. Migrating the Schema 21
  21. 21. Migrating the Schema 22
  22. 22. Migrating the Schema Convenient Dependency Management with @Grapes 23 #!/usr/bin/env groovy @Grapes([ @Grab(group='org.yaml', module='snakeyaml', version='1.12'), @Grab(group='mysql', module='mysql-connector-java', version='5.1.24') ]) import groovy.sql.Sql import org.yaml.snakeyaml.Yaml
  23. 23. Strategy ● Import all messages ● Setup cron job to get new messages ● Listen to events for updates 24
  24. 24. Chapter 2 The Application
  25. 25. Creating a new service 26
  26. 26. Convo ● Scala ● Twitter Finagle ● Scalatra Framework 27
  27. 27. Convo architecture 28
  28. 28. Scala ● Functional ● OOP ● Static but inferred typing 29
  29. 29. Scala Joy = Options I Good bye NullPointerException 30 val opt: Option[String] = params.get("id") val id: Int = opt.map(id => id.toInt).getOrElse(10)
  30. 30. Scala Joy = Options II Safe chaining with for comprehensions 31 for { id <- params.get("id") user <- users.lookup(id) count <- counts.forUser(user) } yield count
  31. 31. Scala Joy = Pattern Matching Expressive code with decomposition 32 Urn("soundcloud:users:20") match { case Urn(_, "tracks", _) => None, case Urn(_, "messages", "20") => None, case Urn(_, "users", id) => Some(id) }
  32. 32. Scala Joy = Functional Goodness Function arguments and references 33 delete("/playlist/:urn/likes")(destroy) def destroy(request: Request) = write(request, 200)(repo.deleteLike) def write (request: Request, statusCode: Int) (f: (UserSession, Urn) => Future[Like]) = { // ... }
  33. 33. Futures!
  34. 34. Finagle ● Twitter rpc library on top of Netty ● Support for multiple protocols ● Future composition 35
  35. 35. Futures Instance API (excerpt) 36 class Future[A] { def get(): A def map[B](f: A => B): Future[B] def flatMap[B](f: A => Future[B]]): Future[B] def onSuccess(f : A => Unit): Future[A] }
  36. 36. Futures Object API (excerpt) 37 object Future { def value[A](a: A): Future[A] def exception[A](e: Throwable): Future[A] def collect[A](fs : Seq[Future[A]]): Future[Seq[A]] }
  37. 37. Futures - Examples Multiple transformations - The ugly way 38 service.getUsers().flatMap { users => service.tracksFor(users).flatMap { tracks => asJson(tracks) } }.onSuccess(json => log(s"found $json"))
  38. 38. Futures - Example Multiple transformations - The nice way 39 val response = for { users <- service.getUsers() tracks <- service.tracksFor(users) json <- asJson(tracks) } yield json response.onSuccess(json => log(s"found $json"))
  39. 39. Scala Problems ● Implicit conversions ● Binary compatibility of libraries ● Tooling still not perfect 40
  40. 40. SBT
  41. 41. IntelliJ ● Code inspection ● Debugging ● SBT support 42
  42. 42. Chapter 3 The Cutover
  43. 43. Integrate Service 44
  44. 44. Integration Risks ● Service failure ● Data loss after rolling back ● Data loss caused by stale clients 45
  45. 45. Integration Risks ● Service failure → load testing, A/B testing ● Data loss after rolling back ● Data loss caused by stale clients 46
  46. 46. Integration Risks ● Service failure → load testing, A/B testing ● Data loss after rolling back → prepare scripts, practice ● Data loss caused by stale clients 47
  47. 47. Integration Risks ● Service failure → load testing, A/B testing ● Data loss after rolling back → prepare scripts, practice ● Data loss caused by stale clients → keep migration running 48
  48. 48. Enable Feature 49
  49. 49. Retire Old Database 50
  50. 50. Convo ● 500 million requests per day ● 1000 qps during peak time ● 5 instances 51
  51. 51. Microservice Problems ● Event bus dependency ● Maintenance overhead ● Distributed tracing 52
  52. 52. Microservices → Not a silver bullet 53
  53. 53. Questions?
  54. 54. Images ● Slide 4,7 - Rails Logo http://en.wikipedia.org/wiki/File:Ruby_on_Rails.svg ● Slide 6,51 - Party Cat http://ghostexist.deviantart.com/art/Party-Cat-logo-287986071 ● Silde 7 - MySQL Logo http://blogwifi.fr/?p=9990 ● Slide 7 - Hadoop Cop https://svn.apache.org/repos/asf/hadoop/logos/out_rgb/hadoop-security-logo. jpg ● Slide 10 - Hello, My Name Is: http://commons.wikimedia.org/wiki/File:Hello_my_name_is_sticker. svg ● Slide 14 - Sad Panda: http://www.whatsupyasieve.com/2012/09/17/lockout-blues/sad-panda-2/ ● Slide 16 - Exit Sign: http://logo-kid.com/emergency-exit-sign-left.htm ● Slide 22 - Groovy Logo: http://groovy.codehaus.org/images/groovy-logo-medium.png ● Slide 20, 25, 44 - Book Page: http://daviddiazolivares.deviantart.com/art/Old-Book-Page- 345869530 ● Slide 34 - Back to the future: http://i.huffpost.com/gen/1369403/thumbs/o-BACK-TO-THE-FUTURE- facebook.jpg ● Slide 42 - Tommy Lee Jones: http://persephonemagazine.com/2014/04/friday-news-bites-airline- pranks-gabriel-garcia-marquez-pulitzers-more/film-title-no-country-for-old-men/ ● Slide 55 - That’s all folks: http://www.hd2wallpapers.com/view/thats_all_folks-1280x800.php 56

×