The document discusses Ibotta's transition from a Ruby on Rails monolith to a microservices architecture, emphasizing the challenges and advantages encountered during the process. Key points include the motivation for microservices, initial implementation attempts, and the importance of tooling, observability, and inter-service communication. It also addresses pitfalls such as increased complexity and the need for explicit communication, alongside benefits like empowered teams and the ability to scale services independently.

1. Microservices at ibotta - pitfalls and learnings
Maciek Swiech & Matt Reynolds

2. 2
https://commons.wikimedia.org/wiki/The_Who#/media/File:Who_-_1975.jpg
Who Are You?
I really wanna
know...

3. 3
BACKGROUND

4. 44
ibotta is a rewarded
shopping app built here
in Denver.
The backend began life
as a Ruby on Rails
monolith built on AWS
five years ago

5. Base Architecture
5
● At its most fundamental, the Ibotta backend
was comprised of two systems:
○ Receipt processing via OCR
○ A set of API’s to power a mobile
experience

6. Base Architecture
6credit: @benorama

7. Base Architecture
7credit: @benorama

8. Base Architecture - in
8
meijer
$1 apl
$2 oj
total
$3
Receipt
Processor
OCR
RoR Monolith
MySQL

9. Base Architecture - out
9
Offer / Bonus
Matching
Ledgers
RoR Monolith
MySQL
Notifications

10. Base Architecture -
scale?
10
So that’s simple enough…
But how do you normalize data? Per-retailer processors
And then - you start getting ‘clever’.
Optimizations for specific paths, relying on data structured by
circumstance, relying on global state, tribal knowledge grows...

11. Base Architecture -
scale?
11
So our nice lasagna has managed to
turn into spaghetti after all.
And worse, there are now hidden side-
effects and dependencies.
credit: @benorama

12. Motivation For Microservices
12
A growing company leads to:
● Growing customer base
● Growing code base
● Growing developer base
Using microservices we can:
● Build/scale services independently
● Reduce problem space for individual developers
● Develop features faster

13. Initial Attempts
13
We’re an AWS shop, so let’s try an AWS
product!
Enter Elastic Beanstalk
Docker and environment support

14. Initial Attempts
14
👍
Easy to use our existing deploy scripts
Config changes easy
Autoscaling tuning is easy

15. Initial Attempts
15
👎
no ability to do blue-green deploys
deploys had to be done in batches
batch failure -> total rollback
no “for free” sidecar containers (helm)

16. 16
SERVICE COMMUNICATION

17. Mono → Micro
17
So we want to have our lovely ravioli-architecture
THIS WILL BE A LONG, DIFFICULT PROCESS!
YOU WILL NOT BE CHURNING OUT FEATURES!

18. Mono → Micro
18
OK great, you’ve decided to go to an SOA.
This means all your problems are now over!

19. Mono → Micro
19
OK great, you’ve decided to go to an SOA.
This means all your problems are now over!
just kidding…

20. Mono → Micro
20
● Agreeing on transport layers (HTTP? GRPC? SQS? MQ?)
● Agreeing on message format (JSON? Proto? Avro?)
● What does “backwards compatibility” mean?
● How do you manage inter-service dependencies?

21. Mono → Micro
21
function calls → network calls
observability and metrics become even more important

22. Mono → Micro
22
PERSONAL OBSERVATION:
µServices and SOA work because of 2 things:
1. the “secret sauce” of tooling
2. excellent monitoring and metrics

23. the secret sauce...
23

24. the secret sauce...
24

25. the secret sauce...
25

26. 26
BUILDING A PLATFORM

27. Starting Kubernetes
27
● Use Kops & Terraform to create cluster on AWS
● Built Staging and Production cluster
● Experiment and upgrade Staging first
● Use Daemonsets for log aggregation & metrics
● Kiam for Iam roles in pods
● Run dashboard with minimal role & service account
token

28. Helm and Service Provisioning
28
● Helm can be considered a “package manager” for K8s
● Package resource YAML in “charts”
● Template support with values.yaml
● Create standard service charts for services
● Initially wrap Helm commands with scripts
● Now using Harness.io - deploy as a service

29. Service Mesh Experience
29
● Experiments with early versions (0.3&4) of Istio were
promising
● Built tooling for Canary header, % traffic switch, metrics
to Datadog
● Issues with Pilot under production load forced us to
remove
● Now Running Nginx ingress controllers

30. 30
TOOLING

31. Paving the Road to Production
31
we need to help developers become more self
sufficient by working with them to pave a road to
production with tools.

32. Starting New Services
32
Use Yeoman generators to template new services
This includes:
● Example outline service
● Configuration of CI
● Configuration of Deployment requirements

33. Provisioning Infrastructure
33
● We use Terraform to provision AWS resources
● Develop Terraform modules for common use cases
● Base Yeoman generator for the minimum service
● Deployment pipeline process for making changes
● Training “Infrastructure specialists” in dev groups
● Developers write their own service modules

34. Support Libraries
34
Build common libraries in standard languages for:
logging, metrics, security and other shared services
This reduces work for individual services
Promotes a standard approach
Makes it easier to upgrade
Run as internal open source so devs provide PRs

35. 35
PITFALLS & BENEFITS

36. Pitfalls
36
● More moving pieces
● Takes time to build platform, tools
● Takes time to change mindset, learn new approach
● More explicit communication required
● Harder to test across services
● Long term service ownership

37. Pitfalls
37
● Coordinating dependencies vs independent work / deploys
● Ensuring that your events are general-purpose and robust
● “Agile” teams vs. maintaining a high-level picture and system
design
● Team structure vs service ownership

38. 38
THE LIGHT AT THE END OF THE
TUNNEL!

39. Benefits
39
● Teams are empowered to do what they need
● Smaller services scale independently, lower blast radius
● Kubernetes provides a lot of resilience
● Easier to take risks, try new things
● Easier to work in cross functional teams
● Easier to align closer with other groups

40. Benefits
40
● UNIX philosophy - do one thing and do it well
○ no need to shoehorn everything into 1 framework
● Isolation, and encapsulation of business logic
○ pseudo-functional approach
○ system comprised of composable units
● “Easy” canarying of individual components

41. We’re Hiring…
https://ibotta.com/careers
41

42. 42
QUESTIONS

43. 43
MOUNTAIN DIVIDER