Brendan O'Bra Scala By the Schuykill
•Download as PPTX, PDF•
2 likes•610 views
Brendan O'Bra discusses the journey of migrating a monolithic Java application called Gumby to microservices using Scala and Akka. Originally a single large codebase, Gumby was split into modular microservices composed together into "microliths" to allow incremental development. The microliths helped address challenges of scaling, deployment, and team collaboration. The architecture eventually evolved to true microservices running independently in a Docker swarm. Challenges included understanding Akka clustering, databases, and tooling choices.
More Related Content
Viewers also liked
Viewers also liked (16)
Similar to Brendan O'Bra Scala By the Schuykill
Similar to Brendan O'Bra Scala By the Schuykill (20)
Recently uploaded
Recently uploaded (20)
Brendan O'Bra Scala By the Schuykill
- 1. Akka Services The Journey from one Big Problem to Many Small Problems Brendan O'Bra - Scala By The Schuylkill 2017 1 Brendan O’Bra, Software Developer, Comcast
- 2. One slide about me 20+ years as a coder “Burned out” Java/maven dude before Scala (probably like everyone else in the room) Born , raised and lived in Northern California my entire life (somewhat of a unicorn in that respect) I actually made a living writing Delphi in the 90’s Used to have REALLY long hair Welcome to my stream of conciousness… Brendan O'Bra - Scala By The Schuylkill 20172
- 3. You know that guy who has the dream project ? Everything works The 1st time At massive scale All the awesome buzzwords are there? Brendan O'Bra - Scala By The Schuylkill 20173
- 4. That’s not me… Brendan O'Bra - Scala By The Schuylkill 20174 All Aboard! Toot Toot!
- 5. If there’s a mistake, it’s quite possible I’ve made it… Things like… “I’ll get to that on the next sprint” 6 months later, the interest has more than accrued on that Tech. Debt “I’ll write the tests later” Later never comes. It attacks in production. We have no way to isolate it. Guess it’s time to write the tests “I can’t find a thing to do X, I’d better write one of those” Look harder next time, there’s a thing that does that very well Brendan O'Bra - Scala By The Schuylkill 20175
- 6. Gumby. WTF is that? Gumby is: • Comcast’ Cloud Deployment tool • Something that provides a REST API • Provides “cloud abstraction” – one payload can create machines in many different backends • The thing that led us to Akka ;) Brendan O'Bra - Scala By The Schuylkill 20176
- 7. What ? Another one of these things? 4 years ago , tools were much less mature Comcast needed a tool that would work with all (OpenStack/EC2/Vsphere) cloud types We had libraries/integrations laying around (written in java) that already worked with openstack/vpshere. These libraries provided “scope narrowing” to provide ease of use to our customers, the app developers Brendan O'Bra - Scala By The Schuylkill 20177
- 8. The History Of Gumby Brendan O'Bra - Scala By The Schuylkill 20178 Play! App built in Maven 2014 2015 2016 2017 Spray/Akka built w/SBT :The Monolith is born Oh crap, we need microservices Oh crap, how do we Move to microservices uLIth! Microservices Many, Many releases ELK Cassandra AppDynamics PagerDuty Docker Our Story Begins Here
- 9. Act I A Science Experiment goes Production Brendan O'Bra - Scala By The Schuylkill 20179
- 10. Act I A Science Experiment goes Production Brendan O'Bra - Scala By The Schuylkill 201710
- 11. In the beginning, there was the Gumby Monolith One big pile of (mainly) scala code, with one and only one release artifact (all modules run in one JVM) Brendan O'Bra - Scala By The Schuylkill 201711
- 12. Brendan O'Bra - Scala By The Schuylkill 201712
- 13. And all was good… We made it We deployed it (one gigantic instance per tenant) It did (most) of what it was supposed to do Brendan O'Bra - Scala By The Schuylkill 201713
- 14. Monolithic Gumby Runtime Architecture Brendan O'Bra - Scala By The Schuylkill 201714
- 15. But.. There were only 2 developers – we were slow Customers wanted it to do new stuff + adoption rate was increasing The product was a bit slow too... Coupling was high Blast Radius was large Brendan O'Bra - Scala By The Schuylkill 201715
- 16. And… Each new tenant required a separate instance Config for each instance was maintained by Gumby developers Each new instance needed to have it’s config maintained Upgrades were a real pain It seemed like all we were doing was support Brendan O'Bra - Scala By The Schuylkill 201716
- 17. We needed to scale our team So we useradd new_awesome_dev (s) Everyone started to git push origin feature/awesome_feature Tests! Features! Yes! And then… The collisions and “blast radius” events started. Coupling was killng us. We started to lose our way.... Brendan O'Bra - Scala By The Schuylkill 201717
- 18. We were now here Brendan O'Bra - Scala By The Schuylkill 201718
- 19. But we knew we needed to be here: Brendan O'Bra - Scala By The Schuylkill 201719
- 20. The solution to this quandry was… Microservices Brendan O'Bra - Scala By The Schuylkill 201720
- 21. Brendan O'Bra - Scala By The Schuylkill 201721
- 22. Microservices – It’s all about developers! • Small, loosely coupled applications • Well defined and relatively small in scope • Emphasis on interfaces/APIs as collaboration ”points”. Could be REST, but does not need to be – depends on use case – make it messaging based • “Micro” is a relative term Brendan O'Bra - Scala By The Schuylkill 201722
- 23. MicroServices == Small Changes Allow developers to operate in the small: •Small changes •Small tests •Small interdependency •Small collateral damage to other code •Only interactions with collaborators are via API (ideally) Brendan O'Bra - Scala By The Schuylkill 201723
- 24. But, of course… Brendan O'Bra - Scala By The Schuylkill 201724
- 25. We are DevOps Brendan O'Bra - Scala By The Schuylkill 201725
- 26. What the Devs in DevOps really want is… Brendan O'Bra - Scala By The Schuylkill 201726
- 27. DevOps and Microservices : BFFs Because there are smaller “pieces”… • There are more things to release –And more things to deploy –And more things to configure Brendan O'Bra - Scala By The Schuylkill 201727 AKA:
- 28. For which there is only one cure Brendan O'Bra - Scala By The Schuylkill 201728
- 30. We realized we need to get from... Brendan O'Bra - Scala By The Schuylkill 201730
- 31. Here: Gumby Single Tenant Architecture Brendan O'Bra - Scala By The Schuylkill 201731
- 32. TO here: Gumby Multitenant ”Logical” Architecture Brendan O'Bra - Scala By The Schuylkill 201732
- 33. Look at those arrows… Brendan O'Bra - Scala By The Schuylkill 201733
- 34. Brendan O'Bra - Scala By The Schuylkill 201734
- 35. So, how do we go Monolith -> Microservices, without Greenfielding? Brendan O'Bra - Scala By The Schuylkill 201735
- 36. Brendan O'Bra - Scala By The Schuylkill 201736 Cloud Cloud-Init Auth Datastore Openstack EC2VMWare Azure UI DNS Bakery Spray Job Control (sharded)
- 37. The History Of Gumby Brendan O'Bra - Scala By The Schuylkill 201737 Play! App built in Maven 2014 2015 2016 2017 Spray/Akka built w/SBT :The Monolith is born Oh crap, we need microservices Oh crap, how do we Move to microservices uLIth! Microservices Many, Many releases ELK Cassandra AppDynamics PagerDuty Docker
- 38. And now…. A Message from the Ops side of the brain Brendan O'Bra - Scala By The Schuylkill 201738
- 39. Ops view of managing a monolith Brendan O'Bra - Scala By The Schuylkill 201739
- 40. Ops View of managing microservices Brendan O'Bra - Scala By The Schuylkill 201740
- 41. Make a microlith (uLith) Brendan O'Bra - Scala By The Schuylkill 201741
- 42. The Lith: A means to an end The uLith is all Gumby Microservices composed into one (not so) Microservice – the microservices still use “proper” interfaces to collaborate, but just happen to run in one JVM This allows us to iteratively move from the monolithic codebase to microservices, without having to “green field” a top-bottom refactor of entire product, while simultaneously adding critical new functionality such as multitenancy and self service Several uLiths run in a cluster, providing HA/H-Scalability (work can be distributed across cluster) Brendan O'Bra - Scala By The Schuylkill 201742
- 43. Shatter the Monolith! Brendan O'Bra - Scala By The Schuylkill 201743 Cloud Cloud-Init Auth Datastore Openstack EC2 VMWareAzure UIDNS Bakery Job Control
- 44. Define Microservices based on Identified services Modularize code into modules that encapsulate each microservice: -External messages (messages that are used to to collaborate with *this* service) are identified -Internal messages are left in each module Brendan O'Bra - Scala By The Schuylkill 201744
- 45. Lexicon – the common module Identify and isolate common objects into common message module This is a compromise for sure, as it creates a bottleneck around the common module In addition to having a authoritative “dictionary” of messages, circular dependencies are reduced/eliminated Also could possibly provide a convenient dependency for external integration Brendan O'Bra - Scala By The Schuylkill 201745
- 46. Externalize EXTERNAL dependencies into common dependencies “module” Unifying versions of things such as Spray, Akka, etc. becomes VERY important This is a great place for an SBT Plugin ;) Brendan O'Bra - Scala By The Schuylkill 201746
- 47. Spray nice to together - Use interfaces + system.actorOf(…) to make the services collaborate - val jobHandlerActor = system.actorOf(JobRequestHandlerFactory.props(imageJ obService, machineJobPersistentService, machineJobProxy, imageJobProxy, userAuthActor )) Brendan O'Bra - Scala By The Schuylkill 201747
- 48. Compose them into one thing (using git submodule and a build.sbt) lazy val projectRoot = Project(id = "gumby-ulith", base = file(".")) .settings(ulithCommonSettings: _*) .aggregate( projectAcmLib, lazy val projectAcmLib = Project(id = "gumby-acm-lib", base = file("gumby-acm/gumby-acm-lib")) .settings(ulithCommonSettings: _*) .settings(libraryDependencies += phantomDsl, libraryDependencies += phantomConnectors, libraryDependencies += phantomTestKit) .dependsOn(projectAcmCommon, … Brendan O'Bra - Scala By The Schuylkill 201748
- 49. Main val restServiceProps = RestApiService.props( machineHandlerFactory = machineHandlerFactoryActor, machineJobPersistentService, userAuthService = userAuthActor, machinePersistenceActor, buildMachineJobProps, imageHandlerFactoryActor, jobHandler = jobHandlerActor, imageJobHandler = imageJobHandlerActor, machineJobHandler = machineJobHandlerActor, jobDetailsHandler = machineJobHandlerActor, userMetadataLookup = userMetadataLookup, acmUserHandlerFactoryActor = acmUserHandlerFactoryActor, resourceAuthorizationService = resourceAuthorizationServiceActor, acmTenantHandlerFactoryActor = acmTenantHandlerFactoryActor ) import sslConfiguration._ logger.info(s"loaded ssl config") IO(Http) ! Http.Bind(system.actorOf(restServiceProps, "ulith-rest-service"), interface = "0.0.0.0", port = config.getInt("app.http_port")) Brendan O'Bra - Scala By The Schuylkill 201749
- 50. Brendan O'Bra - Scala By The Schuylkill 201750 Auth Cloud Datastore Cloud-init UI VMWareOpenstackEC2Bakery DNS Azure Common Messages Common Dependencies
- 51. Eh? Seems stupid, right? Brendan O'Bra - Scala By The Schuylkill 201751
- 52. It is (more than) a little silly… But... It creates a pattern in which: Modularity is increased Coupling is DECREASED Cohesion is increased Developers can think in a more “service oriented” fashion without having to take on full Ops burden all at once Individual microservices can be individually peeled of and deployed independently (outside of uLith) All modules can be released independently Change is incremental Brendan O'Bra - Scala By The Schuylkill 201752
- 53. There are a few side effects… Brendan O'Bra - Scala By The Schuylkill 201753
- 54. Brendan O'Bra - Scala By The Schuylkill 201754
- 55. Brendan O'Bra - Scala By The Schuylkill 201755 Swarm Seed Seed Seed ULith ULith ULith ULith ULith ULithULithULithULith ULith
- 56. The History Of Gumby Brendan O'Bra - Scala By The Schuylkill 201756 Play! App built in Maven 2014 2015 2016 2017 Spray/Akka built w/SBT :The Monolith is born Oh crap, we need microservices Oh crap, how do we Move to microservices uLIth! Microservices Many, Many releases ELK Cassandra AppDynamics PagerDuty Docker We resume here
- 57. Next Stop, (real) Microservices Brendan O'Bra - Scala By The Schuylkill 201757 Brendan O'Bra - Scala By The Schuylkill 201757 Cloud Cloud-Init Auth Datastore Openstack EC2 VMWareAzure UIDNS Bakery Microlith
- 58. Brendan O'Bra - Scala By The Schuylkill 201758 Brendan O'Bra - Scala By The Schuylkill 201758 Swarm Seed Seed Seed Auth Cloud Model Cloud-init UI VMWareOpenstackEC2Bakery DNS Azure
- 59. A few development challenges Singleton can become Singleton(s) in Akka Cluster Sharding can be hard to understand for newbs (and not so newbs) There’s a database (Cassandra) behind it – we’re just a bunch of dopey devs, we don’t know how to admin that stuff Akka has it’s own gotchas (closing over sender, type “sensitivity” with messaging,Cluster Startup Order,split brain) Much consternation about serialization choices Brendan O'Bra - Scala By The Schuylkill 201759
- 60. In the end… Cluster early, cluster often Docker ASAP Don’t be afraid of writing SBT plugins Implicit conversions are like atomic weapons – they have a time and a place Don’t fight ivy, because it will win Be strong in your resistance to using exclude(…) in sbt Don’t be a Scala bigot - use Scala to set an example Scala *really* does help with hiring – not JUST attracting people, but attracting the *right* people Practice EVERYTHING. ALL THE TIME. Cassandra. Networky things. Docker at all points in pipeline Consider sbt test git hooks on dev machines Brendan O'Bra - Scala By The Schuylkill 201760
- 61. Tools/Library Shoutouts Phantom – kickass Cassandra DSL/library https://github.com/outworkers/phantom Ammonite – Just like a scala worksheet, but way better https://github.com/lihaoyi/Ammonite Sbt-release – accept it’s patterns, and your releases will be something you won’t really have to care about anymore https://github.com/sbt/sbt-release Sbt-extras – command line abstraction for the myriad of sbt options https://github.com/paulp/sbt-extras Brendan O'Bra - Scala By The Schuylkill 201761
- 62. Contact me! Brendan_obra@comcast.com @brendanobra Brendan O'Bra - Scala By The Schuylkill 201762
- 63. Brendan O'Bra - Scala By The Schuylkill 201763
Editor's Notes
- Our journey to Microservices from a “classic” monolith About the strange interfaces between design patterns and operations
- I took the original “Functional Progamming in Scala” in 2013 Despite that, everyone around me knows more than I do ;)
- And so many more… sometimes with several repeats
- Started in 2013 First production work for Sochi Olymics with Sports app First X1 work in 2014 Large scale use in 2015 Now deploys entire X1 backend infrastructure plus many other projects Will be open source at some point Currently supports “traditional VMs” (i.e. not docker)
- Yes ,the deployment tool space is now somewhat crowded Why not
- - A lot of apps seems to start as Play! Apps even if Spray is a better choice
- A science experiment goes prod
- That’s 67k LOC, and some of them are even good
- One, bigass sbt-assembly jar. Large, opaque.SNAPSHOTs up the wazzoo. Deployed onto ”tradtional VMs”
- - - - - Compilation errors, general brittleness - Whole days could be spent untying wet knots
- Onboarding was a bitch, due to things like cloud configs, network access and image configuration
- And also…. Multitenancy
- Low coupling == agility Well defined BECAUSE they are small in scope Old school “contracts” make understanding who says what easier to determine DNS service? Hello World? Depends on your project Akka is , of course, perfect for microservices
- On the Gumby team: All do development (write tests + write code) All do production support All get to smell the roses All get to smell the shit All have at least 2(Ops,Dev) inner voices present This allows (forces) us to confront operational AND development issues at the same time. This includes deployment, scaling and other “Enterprisey” things
- They are BFFs because Devops (coupled with automation), makes being incremental more approachable – automate once, run many. This means that, not only is it repeatable, it’s also much more exercisable
- Robots!
- One tenant per vm In/our security model NO self service
- One “grid” with common database Multitenant Self service Granular security control
- We had a dependency graph that looks like this….
- There are 13 of these damn things now ;) The interconnections are the story Job Control is Cluster Sharding, backed by akka-persistence-cassandra Each one of those is probably a build configuration
- - Point out management overhear of circled items.
- This archictecture is fairly complex. Each microservice is a JVM, and ultimately a docker container They are all very connected They need to be able to locate and talk to each other
- Form a management perspective, the monolith is much more attractive – it’s one type of thing to manage, even if there are multiple of them
- Dependencies and interconnections expand geometrically Aggregate answers to questions such as “How is my service performing” become much more complex to answer The use of tools such as Logging-As-A-Service become impossible to do without The sheer volume of data and number of nodes demands automation Automation AND Configuration AND CI expand to be one instance PER Microservice We realized that one runtime “thing” had become 11 (not including external dependencies). And we were also putting all customers onto “this thing” So, what do we do?
- Spinal Tap Anyone? Microlith or “ulith”
- OK.. So we have everything modularized. Let’s make it do something
- Take these services
- Modularize based on previous
- We tried to use a module-per-project for interfaces, but the dependency graph soon became circular Bottleneck is that MANY things will now depend on common-messages module Makes it MUCH easier to write tests around common messaging ;) 3 – Because you are almost forced to anyways Being able to depend on hard versions becomes important to common module. This is another great place for an sbt plugin
- In the end, it will all be packaged as one big thing, and needs to pass the “sbt-release” uberjar sniff test This will be painful and annoyingly “iterative”, but will produce a clean dependency graph Yes, this could be construed as heavy handed and “enterprisey”
- -We favored ‘explicit dependency inject’ The factory (implementations are “known” to the “main method” that will use them (i.e. it’s dependent) Scala Props() work nicely for this.
- And you end up with a dependency graph that looks like this
- Submodules can be a pain, and devs have to remember to update them Delayed integration can be a pain
- Intellij frequently pukes on this module layout and goes all ”red crazy”
- Backed by cassandra, which is managed as ”pets” Things like off node logging and monitoring become indispensible
- - A lot of apps seems to start as Play! Apps even if Spray is a better choice
- With each microservice already mostly defined, each one can be peeled off , adding a “main” to each. The “peeled off” standalone microservices can then be incrementally deleted from the uLith, with the final architecture looking like a a “genuine” microsevices architecture
- We ”needed” a singleton for JWT token (decided we did not want to deal with database stuff) We still love Akka tho Cassandra is great, but needs people who know WTF they are doing. Type sensitivity – unhandled message, and being susceptible to dupicate classnames in different packages Serialization w/respect to cluster sharding. What happens with upgrades?
- 1 Akka in one jvm is not the same as akka in >1 JVM. REALLY learn to cluster 2 Embrace the idiosyncracies of the overlay nework. Leverage the dns 3 We were – eventually we developed SBT plugins to build images for different enviroments , and provide correct config 4 Implicit caused confusion (sort of as expected) for new folks – consider your audience 5 We really struggle with using maven/ivy (local) repos 6 it’s much better to understand and harominize your dependency graph. Exclude is the singleton of sbt ;) 7 We work with many Java folks, and a bunch of Go folks as well. 8 Vikas, Alex – it’s about quality, not quantity 9 Even doing nothing, everything has a shelf life 10 this stems from years of ‘mvn clean install’ providing default test. We’ve had SO many broken tests in CI because of this