This document discusses Go-Jek's approach to reliability at scale. It outlines four iterations of improving reliability: (1) initializing infrastructure with CI/CD, monitoring, and microservices; (2) improving incident handling with reporting, reviews, and defining response processes; (3) enhancing monitoring and alerting with routing, categorization, and embedded SREs; and (4) engineering for failures through techniques like containerization, circuit breakers, and reducing dependencies. The document concludes by noting reliability is iterative and emphasizes allowing mistakes to improve. Go-Jek has grown significantly from 4 products in 2015 to 18 products today while scaling infrastructure from 100 to over 8,000 instances across datacenters.

1. Reliability At Scale
By
Sankalp Singh
Praveen Shukla

2. About Us
@_praveenshukla
@achilles42
Product Engineer
@_sankalps
@singhsankalp
Product Engineer

3. What is Go-Jek?

4. - Transport, logistics, hyperlocal delivery and payments startup in
Indonesia
- 18 Products
- 40 Millions downloads
- 20 Millions Active users
- 300K driver in platform
- Top 50 Innovative Companies on fortune (go-jek @17)
- 1M TPS(internal API call)

5. - Introduction to reliability
- Iteration 1: Initializing Infrastructure.
- Iteration 2: Incident Handling
- Iteration 3: Monitoring and Alerting
- Iteration 4: Engineering for Failures
- Future solutions
Agenda

6. Define “Reliable”

7. “FAILURES”
Systems operates outside specified
parameter.

8. Always a trade-off between
velocity and stability

9. Iteration 1
Initializing Infrastructure

10. In 2015

11. ● CI / CD
● Configuration management
● Monitoring / Alerting
● Logging
● Microservices
Initializing Infrastructure

12. Reliability - Small group responsible for everything.

14. Iteration 2
Human Errors & Incident
Handling

15. ● People doing the deployments without the overall context.
● Downtime due to surprise changes in the infrastructure.
● Release cycle getting bigger.
● Repeating incidents.
● Lack of coordination at the time of incident.
Issues:

16. ● Incident Reporting (RCA)
● Deployment Review Form
● Changelog Feeds
● Defining war room process
Solutions

17. Iteration 3
Monitoring and Alerting

18. Issues
● Alert getting lost
● Pager fatigue
● Not getting alerts to a right person
● Who is responsible to take action on a particular
alert

19. ● Responsibility segregation
■ Define team structure
■ Routing alert and pagers.
■ Embedded SRE
Solutions

20. ● Define team structure
● Routing alert and pagers.
● Embedded SRE
Solutions

21. ● Define team structure
● Routing alert and pagers.
● Embedded SRE
Solutions

22. Smart Alert Router
● Categorizing alerts
● Routing alerts to specific teams

23. Architecture
Smart alert routerKapacitor
VM
Telegraf
Grafana
Alerting
TS Database
visualization
Agent on VM

24. ● Define team structure
● Routing alert and pagers.
● Embedded SRE
Solutions

25. Embedded SRE

26. Iteration 4
Engineering for Failures

27. ● High deployment / rollback time.
● Cascading failures
● Unexpected load.
Issues

28. Multiple Dependencies
R = 99 %

29. Multiple Dependencies
R = 96 %
R = 99 % R = 99 % R = 99 %

30. Multiple Dependencies
R = 88 %
R = 96 % R = 96 % R = 96 %
R = 99 % R = 99 % R = 99 % R = 99 % R = 99 %

31. ● High deployment / rollback time.
● Cascading failures
● Unexpected load.
Issues

32. Queuing Delay
p
Delay -------
1 - p
p = System Utilization

33. ● Containerizing applications
● Async communication across micro services.
● Circuit Breakers
● System Throttling
● Tighter timeouts
● Reduce dependencies.
Solutions

34. ● Introducing failures in infrastructure
● Smart fallbacks
Future Solutions

35. 2015
● 4 Products
● 10+ Microservices
● 100+ Instances
● 50+ Tech People
● 18 Products
● 250+ Microservices
● 8K+ Instances across 3
datacenters
● 350+ Tech People
4 X
25X
80X
7X
2017

36. Reliability is an iterative
process
Dev should develop, deploy and
reliability
Allow people to make mistakes

37. THANKS!
questions?
_praveenshukla
_sankalps

38. Some fun facts:
1. The first 14 months, GO-JEK has done 100 million completed orders.
2. $5 million Year To Date amount of tips given to drivers by customers.
3. Three (3) Million+: Number of Martabak (Indonesian cakes) that have been delivered in the
last year
4. 300k: Number of drivers who've had their lives changed, and monthly income increase 15%
on average
5. 90k+ number of restaurants on GO-FOOD
6. 50+ Number of cities in Indonesia where GO-JEK services are available
7. Our fleet of 300k drivers collectively cover an average of 4.5 million kilometres - that's more
than 7 round trips to the moon - daily.
8. After partnering with GO-FOOD, the SME's have received upto 345% more sales.
9. More than half of the GO-JEK completed bookings are paid with GO-PAY.