How Expedia Improved Developer Productivity and Reduced MTTR by Over 90%

1Copyright © 2020 OverOps. All rights reserved.
Thank you for joining us!
We will begin in just a few moments
Move Fast. Fix Faster.

How Expedia Improved Developer
Productivity and Reduced MTTR
By Over 90%
September 29, 2020
Move Fast. Fix Faster.

Welcome...
Director of Technology
Gavan McLaughlin
Thanks for joining!
Ravi Vankamamidi
Site Reliability EngineerVP, Solution Engineering
Eric Mizell

Today
We’re
Covering
● Engineering at Expedia
The role of engineering in delivering a
seamless experience to Expedia customers
– especially during COVID-19
● Code Quality and Reliability Challenges
A look into the Expedia CI/CD pipeline and
troubleshooting process
● OverOps & Continuous Reliability at Expedia
Expedia’s enhanced reliability strategy and
how OverOps helped reduce MTTI & MTTR
● Q&A

Engineering at Expedia

About Expedia
Expedia Group is the leading world travel platform with world-leading brands such as Expedia.com,
Hotwire.com and Vrbo.com, among others.
Serving travelers is at the center of Expedia’s business
Conversation Platform:
● Meeting traveler on a channel
of her choice
● Enabling travelers to self-serve
(using Virtual Agent )
● Connecting traveler to a human
agent for complex issues

Conversation Platform enables exchanging of Multi-participant,
multi-channel messages: Social, Chat, Voice, Email, SMS etc.
● Event-driven architecture:
- Loose coupling
- Microservices (Fulﬁllment, Translation, Routing, Bot
runtime, Plug-ins etc.)
- Easy to introduce new technology
● Distributed and scalable: high complexity
Bottom line: Complex system, requires quick troubleshooting
Expedia Group - Conversation Platform

SRE Principles at Expedia
Quality at the Source
Data Driven Decision Making
Continuous Improvement
Shared Responsibility
Protect The Platform

What does an application error mean to
Expedia’s customers during COVID-19?
Booked a non-refundable trip to Europe.

Pandemic hits; travellers can no longer get to their destination.

Pandemic hits; travellers can no longer get to their destination.
What happens to their hotel reservation? What if they can’t
reach customer service?

Code Quality and
Reliability Challenges

Expedia’s Architecture & Tech Stack
Tech stack:
- Microservices ( Java, Kotlin, Python, Node.js )
- Kafka ( Stream processing, KSQL )
- Graphql, Restful services, Eventing
- AWS, NoSQL-DB
- Conﬁguration Manager ( Versioning, Intelligent defaults )
- ReactJS, Redux, Typescript

The CI/CD Pipeline & Troubleshooting Workﬂow
Code Build & Unit Tests Integration Tests Staging Production
Development CI/CD Pipeline
Go/
No Go
Go/
No Go
Go/
No Go
APM & TracingStatic Analysis Testing Log AnalysisMetrics
Release cycle progress →
Visibility→
Code Review

Speed, quality and cost
are constantly at odds.
Top Challenges
Communication is diﬃcult
across engineering teams.
Learning as a large
organization is challenging.
COST QUALITY
SPEED

Continuous Reliability
at Expedia

Prevention is better than the cure
● Issues are detected prior to reaching Production
● All context needed to reproduce is in the error snapshot (variables & code executed)
Minimal developer effort to onboard
● Exceptions are identiﬁed regardless whether they were logged or not
● Integrates automatically to existing log events
“All you can eat” license
● Ever growing in cloud, reduced maintenance/stress
● Not having to worry about exceeding license limits
1
2
3
Why Expedia Chose OverOps?

OverOps Across the Expedia Pipeline
Automated error detection and complete context for every software defect
Code Build & Unit Tests Integration Tests Staging Production
Developers, QA, SRE/Ops
feedbackﬁx
Dynamic Analysis
CI/CD Pipeline
Is it new or critical? Why did it break? Who is responsible?
Development

Getting Started with OverOps
⇨ Installation for each microservice baked into standard images
⇨ Toggleable installation by development teams (controlled in Git)
⇨ 2 week “soak” after installation to calibrate against known defects and create a baseline
⇨ OverOps data collected at each environment as code is executed
⇨ Go/no-go decision tree after E2E tests based on deﬁned “Critical” exceptions
⇨ Joint effort to implement with support from OverOps services team

Development Team A were hit with a production issue
caused by a NullPointerException that took over 4 hours
to resolve.
In Post Mortem, the team realized using OverOps would
have made it vastly easier to detect and resolve the issue.
Post Incident:
The team addresses NullPointerExceptions prior to
reaching production. And triages new production Null
Pointer Errors immediately following release.
Solving a New NullPointerException

Team B were tasked with solving a P1 bug in production.
They identiﬁed the exception that was causing it in their
log aggregation tool.
Leveraged the tiny URL to take them to the OverOps error
snapshot with the related code and variables.
Result:
The team was able to reproduce the error and began
working on hotﬁx < 15 minutes.
P1 Bug in Production

OverOps’ Business Value for Expedia
Quantiﬁable measurement of quality leads
to a better experience by the customer
Prevention of impact & time saved on
troubleshooting translates to more time
spent on innovation and opens opportunities
(IE: deploying a hotﬁx instead of rolling back)
Improvement of quality and reliability
without major impacts to speed of delivery
1
2
3

Demo

We’re Hiring!
https://lifeatexpediagroup.com/jobs
Come along on our journey to bring the world within reach

Questions?
Director of Technology
Gavan McLaughlinRavi Vankamamidi
Site Reliability EngineerVP, Solution Engineering
Eric Mizell

Thanks for attending!
Start a free trial at overops.com
Follow Expedia Group Technology @ExpediaGroupEng

How Expedia Improved Developer Productivity and Reduced MTTR by Over 90%

More Related Content

What's hot

Similar to How Expedia Improved Developer Productivity and Reduced MTTR by Over 90%

More from DevOps.com

Recently uploaded

How Expedia Improved Developer Productivity and Reduced MTTR by Over 90%