Microservices are becoming an increasingly popular way to build software systems. Thanks to evangelism from companies like Netflix, Amazon, Gilt, ThoughtWorks and SoundCloud, more organizations are considering whether or not they should adopt this practice. In this talk, I’ll discuss our experiences evolving 500px from a single, monolithic Ruby on Rails application to a series of composable microservices written in Ruby and Go. I’ll talk about the challenges we faced from a business, engineering, QA and operations perspective and how moving to microservices encouraged (or required) change in our organizational structure and culture. In this talk, you’ll learn how a change in how we develop software affected team structures, development environments, testing infrastructure and encouraged us to explore moving to cloud hosting and to move closer to continuous delivery. You’ll also learn about the pitfalls, both expected and unexpected that we experienced along the way. By sharing some of our experiences, I hope to provide some guidance to engineering teams considering whether or not to adopt microservices.

1. Breaking the Monolith
Organizing Your Team to Embrace Microservices
Paul Osman, 500px

3. “Any organization that designs a system
(defined broadly) will produce a design whose
structure is a copy of the organization’s
communication structure.”
— Melvyn Conway, 1967

4. A Story

5. 500px - Made of people

6. 500px Team
• Approx 40 full-time employees
• Approx 20 full-time engineers + 6 designers / ux researchers
• Product Teams
- iOS & Android (3)
- Prime (2)
- 500px Web (4)
- Platform Engineering (Operations + Dev) (7)
- QA / Release Engineering (3)

7. Microservices

8. Our Journey

9. January 2014
• Classic web architecture (LB, App Servers, DB Servers).
• Many data stores, one large monolithic codebase.
• Scaling: all or nothing.
• Cascading failures. Lots of SPOFs.
• Large, unwieldy codebase.
• “Get in line” approach to deployments.

10. Search & Indexing
• Indexing is naturally asynchronous.
• Introduced dependency on RabbitMQ.
• Wrote new indexer in Go. First service and first Go project.
- Huge performance improvement. 20 hours to 20 minutes.
• Search service - HTTP service that translates query strings into ES queries.
• Eliminated major SPOF. ElasticSearch / Search outages no longer affect customers.
• Isolated knowledge of data store (ElasticSearch).

11. Uploads & Resizing
• Perfect for first service to move to EC2.
• Continued in Go, because it was working well for us.
• Performance improvement. Upload times dropped significantly.
- Bulk uploads via Lightroom plugin.
• Eliminated another SPOF. Next step: front-end work to gracefully communicate to user.
• Eliminated write-half of S3 knowledge. Read-half still to be done.

12. Activities & Notifications
• Ongoing.
• Replacing poorly designed service built on top of MongoDB.
• Will eliminate another SPOF. Fix a lot of reliability complaints.
• Get to revisit architecture, centre around data needs (both real-time and ETL workflows).

13. 3 Lessons

14. Lesson # 1 - DevOps

15. 12 Factor Apps *
* http://12factor.net/

16. Monitor Everything

17. On Call Rotation

18. Measure Incident Response
* MTTD
* MTTR

19. Gameday Exercises

20. Automation

21. Lesson # 2 - Service
Design

22. Where to Start

23. 500px in January

24. Search & Indexing

25. Uploads & Resizing

26. Use Circuit Breakers

27. Use Circuit Breakers

28. https://github.com/500px/ruby-service-client

29. Where to Start
• Pick a business capability that is:
• Off the Critical Path
• Can be double-written to easily (i.e. indexing, aggregating, etc)
• Look out for natural integration points between disparate systems — also good candidates

30. Design for Business Capabilities
• Break your application into business capabilities (i.e. search, recommendations, uploads).
• Tempting to design around technology layers (front-end, back-end). Don’t.
• Even if you don’t (yet) have the numbers, treat each business capability as a team.

31. Lesson # 3 - Teams

32. API Facades

33. Team Structure
• Web App Team - Own front-end code, web-bff API facade
• Search Team - Own search infrastructure & service
• Media Team - Own upload service, resizing & converting services, watermarking, etc
• Mobile Team - Own mobile apps, mobile-bff API facade
• Platform Eng - Everything else, operations, devtools, QA / Release engineering

34. Cross Functional Teams

35. Move People Around

36. Thank you!
Paul Osman
@paulosman
paul@500px.com

37. https://500px.com/jobs