Atmosphere Conference 2015: What we've learned from writing dozens of polyglot micro services
•Download as PPTX, PDF•
0 likes•222 views
At Wunderlist, they embrace polyglot microservices to reduce deployment fear and increase change ability. They emphasize small, independent, self-documenting services using various programming languages. Over time, they have expanded from a few large monoliths to many smaller services across languages, which has improved flexibility without increasing complexity through clear interfaces. Their goal is to continuously improve systems through experience with challenging architectures.
More Related Content
Similar to Atmosphere Conference 2015: What we've learned from writing dozens of polyglot micro services
Similar to Atmosphere Conference 2015: What we've learned from writing dozens of polyglot micro services (20)
Recently uploaded
Recently uploaded (20)
Atmosphere Conference 2015: What we've learned from writing dozens of polyglot micro services
- 1. How we build systems at Wunderlist Embracing polyglot immutable micro services
- 2. 🔥 At Wunderlist, we build software that can be thrown away
- 3. 😢 As an industry, we are bad at shipping large systems
- 4. 💀 When deployment becomes scary, you deploy less often
- 5. 😤 “Mostly green? Ship it”
- 6. 😱 Inertia leads to scary technology updates
- 7. 🚧 Abstractions and Design pattern soup
- 8. Not all systems exhibit this problem At least non-software systems…
- 9. Homeostasis Trillions of cells in your body, millions die every second
- 10. 🐜 Small == Good
- 11. Our Rules: Reduce coupling. Remove fear of deployment. Make change trivial
- 12. ☕ Write services so small that they can be understood over coffee
- 13. 📣 Write services that document themselves without extensive comments
- 14. 📈 Test what matters to get the code written, then deploy and watch the metrics
- 15. 📦 Services own and encapsulate their data
- 16. REST everywhere Little tiny requests and responses
- 17. Problem: HTTP connection latency over the open Internet
- 18. Solution: Tunnel HTTP over web sockets
- 19. Hahn Our faucet for web sockets Websocket Connection Client Server { "method":"GET", "headers": ["Content-Type: "X-Request-ID: 1234"], "request-id":"123456", "path":"/api/v1/lists/32" } { "response":200, "headers": ["Content-Type: application/json", "X-Request-ID: 1234"] "body":"[{"id":83526310,"title":"Read Later","revision":10}]" } Request: Response:
- 20. ❔ Wait, what about HTTP/2?
- 21. Heterogeneous by default Ruby, Haskell, Scala, Clojure, Go, JavaScript, Java, Lua, Elixir, Erlang, Objective-C, Swift, C#, and more to come…
- 22. Websocket Wunderlist 3.0 Backend A simplified diagram of our micro services at launch last year *not to scale HTTP Business Logic (English) Services Data Access Logic (German) Services Databases Messaging
- 23. Wunderlist Backend Now Nine months later… *still not to scale WebsocketHTTP Business Logic (English) Services Data Access Logic (German) Services Databases Messaging
- 24. 🚅 We use Rails for production quality prototyping
- 25. ✈ We use Go, Scala, and Clojure for when we need to fly
- 26. 🚀 When we need rockets, we’re not afraid to reach for appropriate toolsets and languages
- 27. 💡 Using multiple languages forces us to keep system interfaces simple
- 28. What’s next?
- 30. 💨 Use spot instances
- 31. 🎭 More actors
- 32. 📲 Apply to client development
- 33. ♻ Go full circle
- 34. Closing Thought If something is difficult, do it all the time and get good at it
- 35. 🙏 Thank you
Editor's Notes
- I’m going to open with a challenging statement. Why do we make it? Because it’s our stand for making software that works.
- All sorts of studies show that we’re pretty bad at shipping software. And, one of the reasons why is that the natural progression of software is towards tight coupling. It’s in our nature. We start typing and building. When you’re under deadline or other pressures, you go for expediency. By the time it’s obvious that you need to break things apart, it’s to late.
- When things get large and brittle, deployment gets scary. What was easy to ship becomes hard. And, in a strange self-reinforcing way, the more you avoid deployment, the scarier it gets.
- Testing is great stuff. It’s the right thing to do. But they can become their own brittle beasts, especially if you keep adding to them and don’t take out the tests that aren’t appropriate anymore. We’ve all seen codebases with large test suites where there are some number of tests that are always failing. That somehow becomes acceptable, even if you know that it shouldn’t be.
- In any monolithic system, technology upgrades become expensive and scary. Going from Ruby 1.9 to 2.0, for example. Or versions of Rails.
- Smart developers come along, see this, and then build abstractions on top of it all to hide it and move along. This happens a few times and you get design pattern soup. Even if you’re a proponent of design patterns, you know what I mean.
- Lots of non-software systems aren’t like this. We can drive around in ancient cars on modern roads, for example. And self-driving cars are going to be using those same roads soon.
- Biology gives us an interesting model to work with. Homeostasis is an organism’s ability to self-regulate even in the face of its own self-destructive tendencies. Balance is the key. What software equivalent is there to a cell? That’s what we want.
- Small pieces of granular functionality that can work with others. They can be replenished and replaced and the system continues to move on. That’s the whole point of microservices.
- So, to fight a slide into bad practices—or maybe as a goad into getting somewhat better, Chad, our CTO, established these simple rules during the building of Wunderlist 3.0 backend. Do it. Deploy it. Go go go.
- We use technologies where we can make things so small that tests might even be considered to be optional, certainly after your done developing the code. If we can make services so small that any programmer can read them over coffee, then they can be replaced with something in another language quickly.
- We believe that code should document itself. All of us know how easy it is to mentally filter out comments in code. Heck, syntax highlighting makes it trival. If you have to comment simple code to understand it, you’re doing it wrong.
- This may be controversial to some, but we believe that if you spend more time in tests to make them work than on the system you’re building, there’s a bigger problem.
- We used to have one huge database for everything. Now we have dozens. This means we can change tech from MySQL to Postgres or even to something else for any single service at will with no other services caring. The only thing we lose is foreign key constraints, but we’re ok with that.
- We think micro services work best when the requests between them are small. And we like REST. Of course, this leads to lots of tiny requests going back and forth.
- Inside a data center, making lots of HTTP requests over a fast network has impact, but not enough to care about. Over the open network, however, connection setup latencies really add up and slow down a system that relies on small packets.
- Because of the clients we needed to support, including our web client, needed to be able to use the networking facilities at hand, we turned to HTTP websockets.
- Once a client establishes a connection, each request is made as an HTTP request encapsulated inside JSON. This mechanic lets us use the same client and server logic no matter whether we are using straight HTTP or the websocket connection. The addition of the request-id header lets us tie requests and responses together over the asynchronous websocket connection.
- Smart of you to ask. Well, it’s still on the way. We should probably roll to it sooner than later and we’ll be happy to.
- We’re native on every client: Web, iOS, Mac, Windows 8, Android. And on the back end, we use the tool for the job.
- When we launched 3.0 last summer, we had decomposed our single large Rails app into a fleet of small apps—most of the Rails based. We had some other things going on. Scala for the Websocket server. Clojure in our messaging router. We also had mostly Postgres databases, but a few others.
- As we’ve progressed and our traffic has scaled up, we’ve replaced many of our Rails business logic services with Scala ones. The performance pick up is massive. We’re also converting much of our data access over to Clojure. Here, we’re taking a step-by-step approach, pairing Rails write servers with Clojure based read servers. MySQL is being phased out.
- Does this mean Ruby on Rails is being totally phased out at Wunderlist? Not at all. We use it for production quality prototyping. The first draft of almost everything is in Ruby. Once we’ve sorted out what it should do and we see that Rails isn’t keeping up, we move on.
- We’ve been really impressed with both JVM languages and Go. Scala is powerful. Sometimes too much of a kitchen sink. Clojure has been an excellent fit for message distribution and CRUD operations. And Go, while it’s a bit odd in its way, is amazing for some purposes.
- Does this mean Ruby on Rails is being totally phased out at Wunderlist? Not at all. Our Haskell service rocks. Elixir and Erlang? Sure.
- Finally, we’ve found that our use of multiple languages helps us keep simple interfaces. When you write everything in Ruby, then you use monkey patching and a bunch of other things because you can. Scala to Scala, same story. Ruby to Scala forces simplicity.
- Where are we going from here?
- Right now, we manage balance in the system manually. We can see in our graphs when something is out of balance and we’ve noticed we follow the same actions. So we’ll code them and make rules so that when pressure shows up in one part of the system, balance can be automatically re-established.
- If we are really serious about machines being replaced all the time, then lets use instances that are cheap, but could be shut down. Currently using for workers in an auction based system. Maybe soon rolling to HTTP responders. Inspired by Pintrest’s work on this.
- Some parts of our infrastructure are based on Akka and we’re really happy with the way that actors work. In fact, the most robust parts of our system are implemented as actors. What if we made our basic data types actors? We’re really interested in where things like Project Orleans are going with virtual reliable actors. Really, if you get to the core of what an Actor is and what messaging is and how it relates to how we build our systems, you get to a pretty universal truth somewhere down there.
- Obviously we have lots of freedom on the server side, but how far can we push it on the client side? How about a client whose components only communicated on a message bus? Or an Android client that used separate Services? What if we broke everything down into small messages passed around via NSOperation on Mac and iOS?
- Now that we’ve learned how to run a massively polyglot immutable infrastructure, what happens if we try to take some of those lessons and build up somewhat bigger services? Maybe even a monolith that’s different. Whoever said that micro services were a poor mans implementation of a good Erlang actor system is probably right. Maybe there are other ways to force simplicity that we should look at. Other forcing functions that do what multiple languages do for us now.
- Finally, a lesson from XP: If something is hard, do it often and get good at it.
- Questions????