This document discusses how 17 Media, a fast-growing live streaming app startup, implements Site Reliability Engineering (SRE) practices to ensure service reliability and enable their engineering team to work quickly. It outlines how 17 Media defines expectations for availability and downtime, sets up monitoring and on-call policies, maximizes development speed while minimizing risk through practices like continuous integration and canary releases, learns from failures through blameless postmortems, and tracks and analyzes outages. The conclusion states that SRE practices have helped ensure the service's availability while allowing the technical team to develop and release faster.

1. Kevin Li, SVP of Engineering
2018.11.07
Solving Reliability Fears
with SRE at 17 Media

2. About 17 Media
#1 in Live Streaming
10 million downloads
across Asia - Japan,
Hong Kong, and Taiwan

3. Why SRE is important at 17 Media
As a fast-growing app startup, SRE practices
are required to enable our product to be
reliable and robust and enable our team to be
agile

4. SRE at 17 Media
Guarantees the service availability
Minimizes the risk to release and
enable our team to iterate fast
Consists of 5 people, while we have
around 70 engineers.

5. Our Practices

6. Define expectations (Error budget)
Make sure reliability and
innovation are balanced
The hardest thing is to get
organizational buy-in

7. Our expectations currently
99.95% around 1 hour per quarter
What's unacceptable:
● Data corruption
What’s acceptable:
● A little more downtime during non-peak hours

8. Monitors, alerts and on-call policy
Latency,
Traffic,
Errors and Exceptions,
Saturation
For every component,
you’ll need to
monitor…
And setup the alerts

9. Monitors, alerts and on-call policy
For every important alert,
System will first call the on-call engineer,
If no reply, call next one…
If no reply, call everyone

10. Maximize dev speed while minimize risk
Implement continuous integration
Automate the release process
Release frequently (everyday) and at morning
Canary release
Shorten the time to deploy
Rollback effortlessly

11. Learn from failures

12. Blameless Postmortem
Post-mortems should be blameless
Gather all facts
Create the timeline
Focus on fixing the process
Actions items to improve the process

13. Postmortem Review Example
Root Cause Analysis
Timeline (Facts)
Action Items

14. Track, classify, and review outages
Classified as P0 for causing more than $X dollars
lost of revenue
Classified as P1 for causing some revenue loss
Classified as P2 for not causing any revenue loss
Aggregate and classify outages
Conduct monthly or quarterly analysis
Regularly share the lessons learned

15. Recap
Define error budget
Monitors, alerts and on-call policy
Maximize dev speed and minimize risk
Learn from failures

16. Conclusion
Adopting SRE practices not only ensures the
availability of our service, but also enables our
tech team to develop and release faster.

17. Questions?