MLB Advanced Media (MLBAM) provides digital media distribution for MLB. They have evolved from early streaming in 2002 with few devices and low quality to now being a top streaming service. MLBAM has also evolved their "digital turnstile" approach to access by expanding platforms and partners. They have moved to a DevOps model from siloed development, deployment, and operations teams. This involved adopting AWS for infrastructure and New Relic for monitoring to improve collaboration, transparency, and issue response across teams. New Relic helps MLBAM understand performance of their distributed systems and partners to quickly address issues that could impact the fan experience.

1. MLB Advanced Media
Customer Driven DevOps at Work:
Operating the Digital Turnstile
FutureStack ’16 - 09 Aug 2016
Sean Curtis, SVP Software Engineering

2. Agenda
• MLBAM - Brief Intro
• The New Digital Turnstile
• The Evolution of DevOps at MLBAM
• Our Platform
• Our Customers
• NewRelic Helping Power Customer-Driven DevOps

3. MLBAM - Who we are
• Formed in 2000 to centralize all digital media
distribution on the Internet
• Over 1200 employees, predominantly in NY, San
Francisco
• Over half of our staff are in engineering, systems,
and operations
• Operate all digital functions for 30 MLB teams, as
well as MLB.com league portal

4. The Beginning
• 2002 - First sports league to live stream on the
internet.
• 30K viewers
• One Device platform: PC
• 280 Kbps
• 244x182

5. Where we are now…
• 2016 - #1 OTT Sports
Streaming Service
• 400+ Devices
• 4.5 Mbps
• 720p HD

6. The Digital Turnstile
An Evolution

7. Turnstiles: Early Stage
• One ticket source (theatre)
would sell exclusive tickets to
get into a show
• The “bigger” or more popular
the movie, the longer the line
for access.
• Customer interest inversely
correlated to Customer
experience…

8. Friction!

9. Evolution
• Schedule information made widely
available. Customers have greater
choice in location and showing time
• Purchase tickets ahead of time from
the box office
• Multiple forms of payment or billing.
• Purchase tickets online from
multiple sources. Pay ahead of
time.
• Pick your particular showing
and save your seat!
• The system now scales better to
meet customer demand.

10. The Digital Turnstile
video,
events,
data,
media rights
user,
location,
device,
App Store,
etc.

11. DevOps at MLBAM

12. MLBAM DevOps: A Timeline
2001 2013
MLBAM, like many other Dev shops, has
a “silo-ed” approach to
build vs. deploy / test & monitor
2016
MLBAM moves toward
DevOps using tools for
build/deploy,
configuration and
monitoring
2014

13. “Silo”-ed Ownership
• Engineers build and test locally, then check in
artifacts to shared repository. Developers then
document (when possible) through release notes
& run books
• Systems actually does deployment, installing
artifacts into test environments
• Developers/Project Managers then publish release
to QA to coordinate testing. QA reports findings,
eventually signing off on release. Load tests
ensue, etc.
• If all testing passes, release to production by
Systems.
• Operations now owns first-tier customer response.

14. “Silo”-ed Ownership
Dev
Systems/QA
Operations

15. Challenges we faced
• Who actually owns the customer?
• How do we drive transparency (e.g. ubiquitous metrics)
across all tiers?
• Does anybody or everybody have deep knowledge of the
overall architecture?
• How do we properly escalate issues and priority between
tiers?
• What portion of the system has trouble scaling as
business demands more aggressive timelines?

16. A Shift in Architecture
(and more)

17. Change in Thinking
• MLBAM becomes a trusted partner to many large-scale
sports leagues and media brands (just to name a few)

18. Leverage proven solutions
• Spend more time on the things that matter to
your customers.
• Rely on vendors for building / scaling
infrastructure
• Opportunity to reimagine our existing systems
while iterating on scalability, security and
resiliency

19. MLBAM DevOps
• MLBAM also formalizes an internal team, “Core
Engineering” to productize internal development
tracks such as: toolset selection, configuration
management, and infrastructure deployment
• Core Engineering engages with customer-facing
teams to help “on-board” them to these various
products and tools.
• Also helps spread best practices, architectural
reviews, lessons learned across all dev teams

20. Engineers are motivated by…
• Being challenged to learn a new set of skills
• Owning more of the customer facing solution
• Paying down technical debt and making their
products and services more scalable and
resilient
• Building with transparency

21. Systems & Operations folks enjoy…
• Crisp architecture with time-tested methods for scaling
(bonus points for automation!)
• Redundancy - enough said.
• An explicit networking and security model (and someone
who understands ipchains)
• Automated tools for building and configuring
infrastructure. (Bonus points for VCS integration!)
• A clear escalation path and chain-of-custody model for
customer-facing issues

22. QA folks want…
• The ability to spin up testing resources at a
moments notice
• Deeper metrics that describe system
performance as a function of a recent change
• Sure-fire way of ensuring that iterations to code
indeed satisfy the original requirements in all
phases of testing

23. Business leaders want…
• Ubiquitous set of metrics that describe customer
response and / or impact
• If / when issues arise, the fastest path to root
cause and resolution for the customer
• An approach to scaling that aligns with business
growth

24. Our Platform
+

25. Platform: AWS
• Refactor hosted infrastructure (JMS Brokers, etc.) for
SNS, SQS - Improved scaling and resiliency
• Leverage IAM and other AWS products for access
control, security, auditing, white/blacklist, networking,
etc.
• Implement autoscaling of microservices for resource-
level granularity of usage and cost
• Manage TCO per microservice, understand
overprovisioning at the instance and/or product layer

26. Platform: NewRelic
• Leverage full NewRelic platform across all tiers (mobile,
web, service)
• Foster collaboration between operations, systems,
engineering, QA, and business teams by providing
dashboards with ubiquitous metrics
• Establish fine-grained tolerance thresholds per metric
observed to define business-critical incidents
• Integrate incident creation with other internal tools to
identify, route, triage, resolve, and provide postmortem
metrics

27. The New Turnstile
video,
events,
data,
media rights
user,
location,
device,
App Store,
etc.

28. Our Customers
• Similar to the movie ticketing example, MLBAM
has partnered with many third parties to provide
unprecedented selection in how baseball fans
can access our teams and league

29. Turnstile complexity
• Authenticating users (AuthN)
and Authorizing them for
access (AuthZ) become
distributed operations
• In some cases, MLBAM is
record holder, but in many
cases, third parties sell
MLB.tv to their subscribers
(e.g. IOS, Pay-TV
companies)

30. Turnstile complexity
• We have to scale our turnstile
in order to meet peak demand
(i.e. event start time) which
differs dramatically between
events
• Partner response times are not
always equal or predictable
• From time to time, partners will
have outages, both scheduled
and unscheduled

31. How does New Relic help here?
• APM - provides implicit and explicit application
performance monitoring
• Helps describe the overall health of each of our services
• Will create incidents and send them to downstream
alerting/monitoring systems
• Drives more rapid and accurate insight as to root cause
• Provides single “pane of glass” for Ops, Systems, and
Engineers at BAM

32. A brief example
• The Rangers and Penguins are
playing in a late April NHL
matchup that has playoff
implications
• Hockey fans are trying to join the
game live
• If any of our systems, or our
partner systems has a hiccup,
how can we quickly isolate any
issues and take mitigation steps?

33. A Happy Turnstile (Green)

34. Operational Effects
• NewRelic has become a default standard method of
providing simplified views of the state of our infrastructure
• Engineering now first class citizen in operational aspects
of running our services, vs code/document-handoff, and
have a greater understanding because they own
infrastructure as well.
• SysOps now has fullly transparent view of the health of
each service, but more importantly, the relationships
between services and their dependencies
• Overlapping ownership of the customer!

35. Overlapping Ownership
Engineering Systems/QA Operations Business

36. Thank You!