AppSphere 15 - Whipping your Apps into Shape: Team Beachbody shows you a DevOps Performance Workout for Delighting your Users

Whipping Your Apps Into Shape
Michael Lee, VP Technology

•  Introduction
–  What is Beachbody?
–  Our Growing Pain
–  Our Problem
•  Decide.
–  Why We Chose AppDynamics
•  Commit.
–  Governance Plan
–  Low barrier to entry: Make it accessible
and consumable
•  Succeed.
–  Lean Out
–  Explosive Speed
–  Strength Training
–  Our Solution
•  Q&A
What new features would you like to see in AppDynamics or features Beachbody should take a closer look at?
Text the keyword BEACH and your response to 22333

$1.3B Grew exponentially 1998-2015
•  Fitness company with four major lines of business
–  Direct to consumer
–  Network
–  Certifications
–  On Demand
•  Our programs
–  1998 Turbo Jam
–  2003 P90x
–  2007 Team Beachbody
–  2010 Shakeology
–  2014 Beachbody Live
–  2015 Apparel

Our challenge
•  Growth led to sprawl which led to knowledge isolation
•  Architecture was not ready for our increasing data and customer volumes
–  Many key services SaaS hosted, with companies that did not grow with us
•  Bringing applications in house with a team more accustomed to hosted applications
•  Need to deliver user facing, web speed experience with marketing driven traffic spikes across
eCommerce, CMS, digital distribution, and analytics
–  Need to be able to monitor those businesses on more than just technical metrics in real time

Warehouse
Tax
Orders
Accounting
Payroll
Payments
Media
Customer
Service
Procurement
Continuities
Telemarketing
Billing
1998

Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer
Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000

Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer
Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005
Genealogy
Coach
Club
Commission
Content
Coach
Reports
Chat
Affiliate

Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer
Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005 2010
Assets
Genealogy
Revenue
Revenue
Recognition
Cash
Reporting
Accounts
Payable
Coach
Club
Commission
Email
Management
Content
Coach
Reports
Coach
Relations
General
Ledger
Chat
Accounts
Receivable
Replicated
Sites
Affiliate

Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer
Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005 2010 2015
Assets
Genealogy
Order
Reporting
Properties
Marketing
Database
Projects
Revenue
Revenue
Recognition
Video
Workshops
Cash
Reporting
Accounts
Payable
Coach
Club
Commission
Email
Management
Fitness
Tracking
Content
Coach
Reports
Coach
Relations
General
Ledger
Chat
Accounts
Receivable
Replicated
Sites
Affiliate Email
Marketing

Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer
Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005 2010 2015 2016+
Assets
Genealogy
Order
Reporting
Properties
Marketing
Database
Projects
Revenue
Revenue
Recognition
Video
Workshops
Cash
Reporting
Accounts
Payable
Coach
Club
Commission
Email
Management
Fitness
Tracking
Content
Coach
Reports
Coach
Relations
General
Ledger
Chat
Accounts
Receivable
Replicated
Sites
Affiliate Email
Marketing
International

i3c
Media
Quickbooks
WarehouseTelemarketing
Initial Years
Four Resources in IT
Basic Accounting
<3 3rd Party Integrations

i3c
Media
Quickbooks
WarehouseTelemarketing eCommerce
Call Switch
Message
Boards
Customer
Service
eCometry
mybeachbody
Enterprise Service Bus
Growing Pains
3rd Party Partner
Integrations
150 Resources in IT

i3c
Media
Quickbooks
WarehouseTelemarketing eCommerce
Call Switch
Message
Boards
Customer
Service
eCometry
mybeachbody Coach CRM Certification Digital
Enterprise Service Bus
Today
3rd Party Partners Critical
to our business
300+ Resources in IT

Media
Agencies
Channels &
Affiliates
Dist Center
(3PL)
Telemarketing Manufacturer Supplier
B2B
Countries
COOBB in a Box BB Live bb.com
External Service Layer
CRM Email Delivery
Content
Management
MDM
Commerce
Engine
ERP
MLM Engine
Non-
Transactional
Corporate
Systems
Data Asset
Management
Media Mgmt
(Pioneer)
Merchant
Portal
Campaign &
Test
Management
BI Portal C/S Portal
R&D
(Agile PLM)
Payment
Service
Internal Service Layer
Streaming
Players
Club.tbb.com
2018
Micro-service Architecture
API –driven
Ready for International
Exponential Revenue
Growth Target
Partner Development
Centers

Obese Too many tools, platforms, and services.
PCI Compliance, SoX, Security, &
International Growth
Slow MTTR on $150,000/hour. Business
growing and we couldn’t keep up
Weak 3rd Party Integrations were owning us.
We were losing focus on the user.
Our Problem:
No Foundation. No Anchor. No DevOps Culture

Decide.
Choose your program.

Why we chose AppDynamics
End-to-end visibility, from NOC to Engineer to Product Manager
•  We have a team focused on integration with a large enterprise software footprint
–  Wanted instant value without code changes
•  End user monitoring was key to our strategy of providing higher levels of service to our users
–  Bringing monitoring beyond DevOps all the way to Business Value
•  Integrated triage
–  Minimize cultural impact while rolling out DevOps to a company unaccustomed to the level of responsibility
•  Partnership and support
–  Software with Service, not just SaaS

Commit.
Implement the solution.

Drive Engagement
Culture = Values + Execution
•  Training
•  Governance plan
–  Two TAM’s + a project manager coordinating with our internal teams
–  Across Eight Environments
–  One Dev, one Dev/Int, three QA, one Staging, one UAT, one Production
•  Innovation Hackday
–  iPad to the winner of the AppD prize – Release Engineer automated
rollout to our systems
•  DevOps contest
–  7 Teams
–  15 Individuals
•  Beachbody takeover of LA AppDynamics User Group
–  Over 50 People show up

So, what are we monitoring?
What apps? What databases? What sources? What scale?
1,600
250
0
400
800
1200
1600
2000

Succeed.
The “after” photo.

Copyright © 2015 AppDynamics. All rights reserved. 22
Governance & DevOps Culture
Lean Out.
Focus & Anchor on the Core User

Text the keyword
BEACH and
your response to
22333
ADOPT
6. AppDynamics

BB Live Dashboard (Eng/Product)
Business Metrics Tech Metrics
Product Metrics
Responsive
Desktop

Executive Dashboard

Explosive Speed.
Improve Goodwill and Increase Revenue
Leverage Advanced AppDynamics.
DevOps Solutions Must Empower The Delivery Team

Automatic Application Mapping

That’s not all….
+ =
So, what does this really mean?
1.  AppD detects MySQL is down
2.  It clears out the bad bin files
3.  It restarts MySQL
4.  You know it happened, but nobody had to wake up!
Automated Remediation

Automated Remediation

DevOps + AppDynamics = a better night’s sleep
•  Imagine keeping the entire application relationship in your mind
•  How do you when the application slowness you’re seeing is your
app or a lower tier?
•  With dynamic mapping, you can see how the apps are related,
where slowness in one app is propagating up
•  This allows you to scale up the correct tier, focus on indices in
the correct DB, and not waste your time in examining a victim
rather than the perpetrator
•  You can have your NOC call the right team to fix the underlying
problem, rather than waking somebody up to triage a symptom

War Room
•  Beachbody is heavily dependent on consulting and third parties.
•  Driving consensus and a consolidated view of what’s interesting
during triage is key to resolving a problem
•  War room allows for a single channel to bring people on
disparate systems together looking at the same data, talking in
the same place.
•  It’s a great read-only place to share data with third parties you’re
troubleshooting with
–  Show them yours through the built-in widgets
•  Pull in theirs using iFrames

What’s your normal bridge look like?
Who’s
on the
bridge?
What’s
been
done?
Who’s
on the
bridge?
What’s
been
done?
Who’s
on the
bridge?
What’s
been
done?
What’s
been
done?
Who’s
on the
bridge?
Text the keyword BEACH and
your response to 22333

Wouldn’t this be better….?
What’s been
done?
What logs are
pertinent?
What’s our
checklist?
What’ve we looked
at so far?

Strength Training.
Pack the Muscle. Hold Your Partner Accountable.
Trust but Verify.
Measure, Monitor, and Test what you Manage.

End User Monitoring
•  Auto-baselining for:
–  Requests per Minute
–  Errors per Minute
–  Average Response Time per minute
•  Coupled with Health Rules for:
–  One standard deviation from the baseline
–  Replaces [Insert Your Synthetic Monitoring Here] with something that
has fewer false positives
•  Why use synthetics then?
–  For sites with little traffic or very spiky traffic -- as a secondary
complement to EUM, rather than the other way around

Good news about Synthetics
We’ve been using the beta, working closely with AppD, and it’s getting there.
•  The good:
–  Seeing it alongside our existing metrics
–  Using it as complement to EUM on our newer sites
•  What we’re excited about using:
–  Script based testing
•  Vanilla Python environment
–  Automatic retest on failure
•  No more guessing if it’s the node or the test

EUM vs. Synthetics
Test Locations
Reflective of your actual users, and
scales as your users scale
Test Timing Near-real-time based on actual users
Works with low traffic
and 3rd party sites
You need enough consistent traffic to
baseline; must be on your system
False Positives
Less frequent with sites with enough
traffic
Not necessarily reflective of where
your end users are located
Scheduled intervals
Not traffic or hosting dependent
Dependent on whether individual
nodes are working
EUM Synthetics
Complementary strengths and roles

Accountability of 3rd Parties
•  Synthetic monitoring provides a consistent measure of the
performance of all 3rd party services
Identify deviations from normal performance using consistent measurements

Obese Too many tools, platforms, and services.
PCI Compliance, SoX, Security,
International
Slow MTTR on $150,000/hour. Business
growing and we couldn’t keep up
Weak 3rd Party Integrations were owning us.
We were losing focus on the user.
Our Problem:
No Foundation. No Anchor. No DevOps Culture

Our Solution:
Core. Anchor. DevOps Culture.
Lean Out
Focus on the Core User.
Governance & DevOps Culture
Result: Maximized our Investment.
Explosive
Speed
Leverage Advanced AppDynamics.
DevOps Solutions Must Empower The Delivery Team
Result: MTTR down from hours to minutes
Strength
Training
Trust but Verify.
Measure, Monitor, and Test what you Manage.
Result: 3rd Party Vendors/Partners Improved Stability

Shakeology
Governance
& Culture
Dashboards Automated
Remediation &
Automatic App
Mapping
War Room End User
Monitoring
Synthetics

Questions?

Answer a Question.
Win a Program!
1.  How many AppD features are you using?
2.  How many systems are you monitoring?
3.  What have you found works in your dashboard? What gets
your stakeholders engaged?
4.  How are you using APM or DevOps to support a major
business goal that IT has to deliver?
5.  What is your favorite AppD feature and why?

Appendix

The results: un-Photoshopped
A war isn’t won in a single battle.
•  We reached Engineering and Product
–  Drink our own champagne and start looking at our products more deeply
–  This is the dashboard we’ll see on slide 13:
•  https://beachbody.saas.appdynamics.com/controller/#/
location=CDASHBOARD_DETAIL&timeRange=last_12_hours.BEFORE_N
OW.-1.-1.720&mode=MODE_DASHBOARD&dashboard=47
•  We reached executives
–  Show the power of near-real-time analytics vs. 24hrs to upload analytics data
–  This the dashboard we’ll see on slide 14:
•  [Insert link to executive dashboard]
•  We reached Ops
–  Give our Ops team a chance to sleep at night
–  This is the workflow we’ll see on slide 14: Copyright © 2015 AppDynamics. All rights reserved. 46

Our footprint
Server side
Java
PHP
Node
MySQL (Percona)
Applications Databases
More than 1600 nodes, including 200+ databases
Client side
EUM
Synthetics
Oracle

Isolate test where errors were verified

Compare content across geography and time

AppSphere 15 - Whipping your Apps into Shape: Team Beachbody shows you a DevOps Performance Workout for Delighting your Users

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to AppSphere 15 - Whipping your Apps into Shape: Team Beachbody shows you a DevOps Performance Workout for Delighting your Users

Similar to AppSphere 15 - Whipping your Apps into Shape: Team Beachbody shows you a DevOps Performance Workout for Delighting your Users (20)

More from AppDynamics

More from AppDynamics (20)

AppSphere 15 - Whipping your Apps into Shape: Team Beachbody shows you a DevOps Performance Workout for Delighting your Users