Denver Microservices meetup presentation by Maciek Swiech & Matt Reynolds on lessons learned implementing microservices at ibotta

1. Microservices at ibotta - pitfalls and learnings
Maciek Swiech & Matt Reynolds

2. 2
https://commons.wikimedia.org/wiki/The_Who#/media/File:Who_-_1975.jpg
Who Are You?
I really wanna
know...

3. 3
BACKGROUND

4. 44
ibotta is a rewarded
shopping app built here
in Denver.
The backend began life
as a Ruby on Rails
monolith built on AWS
five years ago

5. Base Architecture
5
● At its most fundamental, the Ibotta backend
was comprised of two systems:
○ Receipt processing via OCR
○ A set of API’s to power a mobile
experience

6. Base Architecture
6credit: @benorama

7. Base Architecture
7credit: @benorama

8. Base Architecture - in
8
meijer
$1 apl
$2 oj
total
$3
Receipt
Processor
OCR
RoR Monolith
MySQL

9. Base Architecture - out
9
Offer / Bonus
Matching
Ledgers
RoR Monolith
MySQL
Notifications

10. Base Architecture -
scale?
10
So that’s simple enough…
But how do you normalize data? Per-retailer processors
And then - you start getting ‘clever’.
Optimizations for specific paths, relying on data structured by
circumstance, relying on global state, tribal knowledge grows...

11. Base Architecture -
scale?
11
So our nice lasagna has managed to
turn into spaghetti after all.
And worse, there are now hidden side-
effects and dependencies.
credit: @benorama

12. Motivation For Microservices
12
A growing company leads to:
● Growing customer base
● Growing code base
● Growing developer base
Using microservices we can:
● Build/scale services independently
● Reduce problem space for individual developers
● Develop features faster

13. Initial Attempts
13
We’re an AWS shop, so let’s try an AWS
product!
Enter Elastic Beanstalk
Docker and environment support

14. Initial Attempts
14
👍
Easy to use our existing deploy scripts
Config changes easy
Autoscaling tuning is easy

15. Initial Attempts
15
👎
no ability to do blue-green deploys
deploys had to be done in batches
batch failure -> total rollback
no “for free” sidecar containers (helm)

16. 16
SERVICE COMMUNICATION

17. Mono → Micro
17
So we want to have our lovely ravioli-architecture
THIS WILL BE A LONG, DIFFICULT PROCESS!
YOU WILL NOT BE CHURNING OUT FEATURES!

18. Mono → Micro
18
OK great, you’ve decided to go to an SOA.
This means all your problems are now over!

19. Mono → Micro
19
OK great, you’ve decided to go to an SOA.
This means all your problems are now over!
just kidding…

20. Mono → Micro
20
● Agreeing on transport layers (HTTP? GRPC? SQS? MQ?)
● Agreeing on message format (JSON? Proto? Avro?)
● What does “backwards compatibility” mean?
● How do you manage inter-service dependencies?

21. Mono → Micro
21
function calls → network calls
observability and metrics become even more important

22. Mono → Micro
22
PERSONAL OBSERVATION:
µServices and SOA work because of 2 things:
1. the “secret sauce” of tooling
2. excellent monitoring and metrics

23. the secret sauce...
23

24. the secret sauce...
24

25. the secret sauce...
25

26. 26
BUILDING A PLATFORM

27. Starting Kubernetes
27
● Use Kops & Terraform to create cluster on AWS
● Built Staging and Production cluster
● Experiment and upgrade Staging first
● Use Daemonsets for log aggregation & metrics
● Kiam for Iam roles in pods
● Run dashboard with minimal role & service account
token

28. Helm and Service Provisioning
28
● Helm can be considered a “package manager” for K8s
● Package resource YAML in “charts”
● Template support with values.yaml
● Create standard service charts for services
● Initially wrap Helm commands with scripts
● Now using Harness.io - deploy as a service

29. Service Mesh Experience
29
● Experiments with early versions (0.3&4) of Istio were
promising
● Built tooling for Canary header, % traffic switch, metrics
to Datadog
● Issues with Pilot under production load forced us to
remove
● Now Running Nginx ingress controllers

30. 30
TOOLING

31. Paving the Road to Production
31
we need to help developers become more self
sufficient by working with them to pave a road to
production with tools.

32. Starting New Services
32
Use Yeoman generators to template new services
This includes:
● Example outline service
● Configuration of CI
● Configuration of Deployment requirements

33. Provisioning Infrastructure
33
● We use Terraform to provision AWS resources
● Develop Terraform modules for common use cases
● Base Yeoman generator for the minimum service
● Deployment pipeline process for making changes
● Training “Infrastructure specialists” in dev groups
● Developers write their own service modules

34. Support Libraries
34
Build common libraries in standard languages for:
logging, metrics, security and other shared services
This reduces work for individual services
Promotes a standard approach
Makes it easier to upgrade
Run as internal open source so devs provide PRs

35. 35
PITFALLS & BENEFITS

36. Pitfalls
36
● More moving pieces
● Takes time to build platform, tools
● Takes time to change mindset, learn new approach
● More explicit communication required
● Harder to test across services
● Long term service ownership

37. Pitfalls
37
● Coordinating dependencies vs independent work / deploys
● Ensuring that your events are general-purpose and robust
● “Agile” teams vs. maintaining a high-level picture and system
design
● Team structure vs service ownership

38. 38
THE LIGHT AT THE END OF THE
TUNNEL!

39. Benefits
39
● Teams are empowered to do what they need
● Smaller services scale independently, lower blast radius
● Kubernetes provides a lot of resilience
● Easier to take risks, try new things
● Easier to work in cross functional teams
● Easier to align closer with other groups

40. Benefits
40
● UNIX philosophy - do one thing and do it well
○ no need to shoehorn everything into 1 framework
● Isolation, and encapsulation of business logic
○ pseudo-functional approach
○ system comprised of composable units
● “Easy” canarying of individual components

41. We’re Hiring…
https://ibotta.com/careers
41

42. 42
QUESTIONS

43. 43
MOUNTAIN DIVIDER