Microservices Development at Scale at Mercari

1. Microservices Development at Scale

2. Vishal Banthia
Software Engineer, Microservices Platform
@vbanthia @vbanthia_

3. Microservices at Mercari

4. Microservices at Mercari
WHY

5. Microservices at Mercari
WHY
WHAT

6. Microservices at Mercari
WHY
WHAT
HOW

7. HOW = Microservices Platform Team

8. A microservice at Mercari

9. echo-service
Centralized GKE Cluster Pub/Sub
Spanner
IAM
RBAC
Cloud Build
Container
Registry
...
echo-service
RBAC
foo-service
foo-service
starter
kit

10. echo-service
A Microservice at mercari is generally a GoLang binary

11. echo-service
We use containers (docker) to package our app

12. echo-service
Centralised GKE Cluster to run microservice
Centralized GKE Cluster

13. echo-service
Use cloud build & container registry to build & store container images
Centralized GKE Cluster
Cloud Build
Container
Registry

14. echo-service
echo-service
Each microservice run in its own namespace
Centralized GKE Cluster
Cloud Build
Container
Registry

15. echo-service
echo-service
Each microservice has its own GCP project
Centralized GKE Cluster
Cloud Build
Container
Registry

16. echo-service
echo-service
Microservice team can choose any GCP service depending on their needs
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
Cloud Build
Container
Registry

17. echo-service
echo-service
Service Account keys are stored in kubernetes secret in same namespace
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
Cloud Build
Container
Registry

18. echo-service
echo-service
Use GitHub to manage source code
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
Cloud Build
Container
Registry

19. echo-service
echo-service
User Kubernetes RBAC to control namespace authZ
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
RBAC
Cloud Build
Container
Registry

20. echo-service
echo-service
Use internal-tool called starter-kit based on terraform to manage microservice team
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
RBAC
starter
kit
Cloud Build
Container
Registry

21. echo-service
echo-service
Use Spinnaker for Continuous Delivery
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
RBAC
starter
kit
Cloud Build
Container
Registry

22. echo-service
echo-service
Use Datadog for monitoring
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
RBAC
starter
kit
Cloud Build
Container
Registry

23. echo-service
echo-service
Use Sentry for Error Tracking
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
RBAC
starter
kit
Cloud Build
Container
Registry

24. echo-service
echo-service
Use PagerDuty for on-call
Centralized GKE Cluster Pub/Sub
Spanner
IAM
...
RBAC
starter
kit
Cloud Build
Container
Registry

25. echo-service
Centralized GKE Cluster Pub/Sub
Spanner
IAM
RBAC
Cloud Build
Container
Registry
...
echo-service
RBAC
foo-service
foo-service
Other microservices also has similar stack
starter
kit

26. echo-ser
vice
Centralized GKE
Cluster
Pub/
Sub
Span
ner
IAM
RBA
C
Cloud
Build
Conta
iner
Regist
ry
...
echo-serv
ice
RBA
C
foo-servic
e
foo-servi
ce
starter
kit
https://speakerdeck.com/tcnksm/microservices-on-gke-at-mercari
https://www.slideshare.net/GoogleCloudPlatformJP/microservices-at-mercari

27. Lifecycle of a Microservice

28. Bootstrap Pre-Release Post-Release Death

29. Bootstrap
● echo service
○ GoLang template service for microservice
● starter kit
○ Internal tool based on Terraform to create / manage infrastructure
as code

30. echo service
This creates a new microservice foo-service with features:
● Application logs
● Datadog integration
● Sentry integration
● CI/CD scripts
● ….

31. starter-kit
This creates terraform resource files for:
● GCP Project
● Github Team
● Kubernetes namespace
● GCP service account and Kubernetes Secrets
● ….
https://speakerdeck.com/b4b4r07/terraform-ops-for-microservices

32. Centralized GKE Cluster Pub/Sub
Spanner
IAM
RBAC
...
foo-service
starter-kit will create all these resources
starter
kit

33. Bootstrap Pre-Release

34. Pre-Release
● Spinnaker
○ Continuous Delivery System
● Pull Request Replication Controller
○ Kubernetes Custom Controller to create branch based QA & Dev
environment
● GRPC translator
○ A transparent gPRC to JSON RPC translator for QA

35. spinnaker

36. spinnaker
https://speakerdeck.com/tcnksm/continuous-delivery-for-microservices-with-spinnaker-at-mercari

37. ● Custom kubernetes controller which creates new deployment
whenever a new pull request is created for a microservice
● Help developers in developing, debugging and testing microservice
in cluster itself
● Enables developers to change environment variables, dependent
microservices etc for their pull request
pull request replication controller
https://www.slideshare.net/VishalBanthia1/kubernetes-controller-for-pull-request-based-environment

38. ● microservices at mercari generally uses gRPC for all
inter-microservices communication
● This tool transparently converts gRPC to JSON
● Mostly used by QA to test microservice using tools such as Postman
grpc translator

39. Bootstrap Pre-Release Post-Release

40. ● Microservice Developers themselves handle on-call
for their microservice
Post-Release
● Monitoring ~ Error Tracing ~ On-Call
○ Use Datadog for monitoring
○ Use Sentry for error tracing
○ Use PageDuty for on-call

41. Bootstrap Pre-Release Post-Release Death

42. There can be multiple reasons for death of a microservice:
● Feature is deprecated
● Service has been replaced with a newer one
Starter-Kit
● Use starter-kit to delete all resources for a dead microservice
Death

43. ● Microservices architecture enables you to move faster but it
also add complexity to system. If not done well, it can be hell!
● With proper planning, tooling & best practices microservices
platform team at mercari is providing a smooth path for this
big transition
Conclusion

44. ● Microservices architecture enables you to move faster but it
also add complexity to system. If not done well, it can be hell!
● With proper planning, tooling & best practices microservices
platform team at mercari is providing a smooth path for this
big transition
● Microservices Platform team to automate all the complexity
and let developers focus only on coding part
Conclusion
alias microservice=”end to end ownership”

45. ● “Microservices promote innovation whereas monolith add
technical-debt”
● “Once I started doing microservice, it’s hard to move back”
Some Feedback from developers