Running a Cloud Platform reliably comes with its own set of challenges. Irrespective of the source of solution used, In-House or Off-The-Shelf, with the Incident Management spread across integrated systems, it is easy to lose sight of the latent issues. Impact Assessment, Communication & Coordination to add in to the mix. Being an SRE is a tough job since the SRE is expected to know almost all aspects of software delivery. The life of an SRE becomes easier & empowered when equipped with the right set of tools. This presentation is about our experience of building a SRE Operations Platform for VMware Managed Cloud on AWS. We'll talk about how we have combined Automation, Monitoring and Incident Management under a single umbrella to drive down MTTR, increase productivity, tools to effectively communicate fleet-wide health to incident managers and customer success engineers and above all, reducing the toil of an SRE.

1. The MTTR Chronicles
Evolution of SRE Self Service Operations Platform
SRE Director: Jason Wik
Senior SRE: Jayan Kuttagupthan
SRE: Shubham Patil
SRECon Asia 14 June 2019

2. Agenda
2
SRE @ VMC on AWS
Square One
βeta
Connecting the Dots
Virtual War Room
Around the Corner
Takeaways

3. 3
SRE @ VMC on AWS

4. 4
Hybrid Cloud Challenge

5. 5
SRE is focused on ensuring services are homogeneous as they scale
Complexity and variability increase risk to the SLAs
SaaS vs Individual Multi Environment Infrastructure at Scale
Storage
SaaS Scales Horizontally
Compute
Networking

6. 6
SaaS Service continues to scale horizontally with the number of customers
Each customer environment scales vertically
SaaS vs Individual Multi Environment Infrastructure at Scale
Each customer environmentis provisioned as a homogenous environment.
Multi VPN
DX
CPU
Bound
Storage
BoundHCX
DraaSAWS DX
HCX
HorizonsMulti-Az Scaled up
Mgt. VMsCustomer
Environment
Each environment quickly deviates as utilization, features, network access, and resource bounds vary significantly.

7. 7
MTTR directly correlates to meeting your SLAs and SLOs
Information and automation is Key
The right information at the specific time
The MTTR Challenge

8. 8
An Extensible, Dynamic, and Collaborative platform
to
reduce MTTR and improve operational efficiency
for
unique and constantly changing environments
North Star
“
”

9. 9
Square One

10. 10
Information silos
High Context Switch
Cross team Collaboration
Time consuming Postmortem & RCA
Longer MTTR == More Impact to the Customer
High Operational Toil
Cumbersome impact assessment
Past learnings not used effectively
Problems Effects
Problems at Hand

11. 11
Automated Remediation

12. 12
Challenges
What data should
be displayed?
How to leverage
this for users
beyond SRE?
What are the
information
sources?
How should the
data be
displayed?

13. 13
βeta: Services under a roof
How to correlate data & make it work @ scale?

14. 14
Connecting the Dots: Single Pane of Glass

15. 15
Cause, Symptom & Action at a single place
Display appropriate data
Intuitive UI : Realtime & Responsive
Integration with BI* & ticketing service
Increase in user adoption. Easier onboarding
Easier correlation & minimal context switch
Data Driven Decisions & Trend Analysis
~20% improvement in MTTR (on initial release !!)
*BI = Business Intelligence
Changes Results
Connecting the Dots ...
How to collaborate better?

16. 16
Virtual War Room

17. 17
Central Place for
Incident Remediation
More focus on
development
One integrated
platform
Effective Collaboration
with Service Owners
Results
Agility, Consistency
and Control @ SCALE

18. 18
Avoid information Silos
Do not reinvent tools
Display what’s necessary
Automate ! Integrate ! Collaborate !
Learn from past incidents
Cap operational work
More Code, Less Toil !
"Build the platform for your Organization!"
Takeaways to Empower your SRE

19. Thank You
jasonwik@outlook.com
jayank.87@gmail.com
theshubhamp@gmail.com