I Love APIs 2015 Vidhya Narayanan, Verizon Chris Webster, Verizon https://www.go90.com/learn Vidhya Narayanan, Director of Engineering, and Christopher Webster, Associate Fellow, Verizon, discuss how Verizon created and launched a mobile-first over the top video platform using over 100 microservices. This session at I Love APIs 2015 covered the architecture for deployment and management of microservices, the technologies used to address scale, availability, and security issues and the pitfalls encountered along the way.

1. Go90
Vidhya Narayanan, Director, Verizon
Christopher Webster, Associate Fellow, Verizon
October 2015
A Look inside go90 services architecture

2. Micro
service
is
a…
a. new
taxi
hailing
app
developed
by
a
middle
schooler
in
India
b. catering
service
run
by
bacteria
c. microwave-­‐as-­‐a-­‐service
from
Verizon
d. Complex
applica@on
broken
down
into
separate
services

3. ©2015
Apigee
Corp.
All
Rights
Reserved.
The
views
expressed
in
this
presenta@on
are
those
of
the
presenter,
and
not
necessarily
those
of
Apigee
Corpora@on
or
the
presenter’s
employer.
3

4. go90.com, #go90
go90.com, #go90

5. Go90
—
from
30,000
P
• 100+
micro
services/systems
• 10+
deployments
per
week
• Mul@ple
clients,
large
target
user
base
• High
reliability/availability,
low
latency,
superior
user
experience

6. In
the
beginning

7. Goals
• Must
scale
and
be
highly
available
• Support
mul@ple
video
products
• Meet
security
requirements
• Need
to
use
exis@ng
services

8. Solu@on
• Services
with
contained
domain
logic
• Resource
centric
APIs
that
expose
data
model
• Common
libraries
• Graceful
degrada@on
• Security
readiness

9. Discovery
STB
Content
Ingest
CDN

10. Accounts
Log
Upload
Search
Profile
Image
Discovery
Event
Q
DRM
Analy@cs
Favorites
Cert
Prov
AWS
HSM
Ads
Offers
and
Packages
DAI
backend
STB
Content
Ingest
CDN
Reco
Discovery
Accounts
Favorites
Search
DRM
Profile
Rights
SU
and…a
100
others
Profiles
Profiles
Q

11. Challenges
• Performance
• Uniformity
• Security

12. Evolu7on

13. Added
Goals
• Reduce
development
@me
for
new
client
placorms
• Improve
release
velocity
• SLA
for
call
latency
• Keep
security
simple

14. Evolu@on
1
• Uniform
API

15. Governor
STB
Maestro
Content
Ingest
CDN

16. Governor
Accounts
Log
Upload
Search
Profile
Image
Discovery
Event
Q
DRM
Analy@cs
Favorites
Cert
Prov
AWS
HSM
Ads
Offers
and
Packages
DAI
backend
STB
Maestro
Content
Ingest
CDN
Reco
Discovery
Accounts
Favorites
Search
DRM
Profile
Rights
SU
and…a
100
others
Profiles
Profiles
Q

17. Challenges
• Monolith
• Itera@on
with
fast
changing
requirements

18. Evolu@on
2
• Microservice
enhancements
• More
preprocessing
• Remove
Logic
from
edge
services

19. Scale

20. Our
Approach
Model
Predicted
Load
Develop
and
Execute
Load
Test
Op@mize
/
Scale
for
Load
Test
Validate
Model
against
known
data

21. Our
Approach
• Benchmark
individual
APIs
• Develop
model
• Execute
cycle
on
system
• Verify
model
• Launch
• Use
produc@on
traffic
logs
• Inject
failure
during
load
tests

22. Benchmark
individual
APIs
• Needed
20x
improvement
• Stress
APIs
and
get
to
a
no
crash
milestone

23. Scenario
Runs
• First
run
• Op@mized
app
call
pagern
• Model
itera@on
using
earlier
results
and
expected
users
projec@ons
• Execute
app
scenario
(on
en@re
system)
with
larger
number
of
users,
but
s@ll
10x
less
than
target

24. Validate
Scenarios
• Increased
scenario
to
target
concurrent
users
• Added
think
@me
to
scenario
• Ran
scenario
with
target
users
• Scenario
running
with
load
<
.1
• Chaos
Monkey

25. Summary
• Measure
first,
op@mize
later
• Start
simple
• Dev
v.
DevOps
• Minimize
infrastructure
debt