Case Study: Dish Network Utilizes CA Application Performance Management to Monitor and Auto Scale Its Applications

World®
’16
Case Study: Dish Network’s SlingTV
Utilizes CA Application Performance
Management to Monitor and Auto Scale
Its Applications
Michael Miller - Principal Analyst & Developer - Dish Network
DO4X80S
DEVOPS – AGILE OPERATIONS

2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type
of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
Dish provides satellite TV service to approximately 13 million subscribers and leads the industry
in state-of-the-art equipment and technology. SlingTV is its wholly owned subsidiary and offers
mobile over the top (OTT) technology to provide Internet-based television to a new audience of
cord cutters. Dish utilizes CA Application Performance Management (CA APM) to monitor
SlingTV and other mission-critical technologies in order to provide an amazing entertainment
experience for its customers. This session will detail how Dish utilizes the capabilities and
flexibility of CA APM to solve production issues while monitoring multiple technologies, how it
utilizes CA APM to feed its big data/log aggregation platforms based on ElasticSearch, LogStash,
Kibana (ELK) and how it automatically scales application monitoring.
Michael
Miller
Dish Network
Principal Analyst
& Developer

About The Speaker
About The Company
Umbrella Monitoring
Integration with External Entities
Auto Scaling & Cloud Management
1
2
3
4
5
AgendaAgenda

Michael Miller
Principal Analyst / Developer
Michael.Miller@dish.com
TAKE BACK TV®
§ 13 years as an IT professional
§ 8 years working with multiple APM
solutions
§ Administered applications running many
languages, including C, Java, .NET, and
Ruby
§ 1 year consulting / subcontracting for CA
professional services
§ 20 years providing free mechanic
services for family, friends, and neighbors
About MeAbout Me

§ Over 13 million subscribers
§ Fortune 200 ranking
§ Industry-leading, award-winning technology
• Nationwide, live streaming “Over The Top” internet-based TV
• Dish’s Hopper – Best of CES 2013
• Dish’s Virtual Joey – Best of CES 2014
• Dish’s SlingTV streaming service – Best of CES 2015
§ Leading Business Innovation
• Spectrum acquisitions
• Best of industry flex pack
Warp Speed IT Environment – THIS IS SPARTA!
TAKE BACK TV®
About the CompanyAbout the Company

Constant Challenges
Ø Integration with 3rd party tools and partners
Ø Zero tolerance application service level expectations
Ø Sudden spikes in traffic
Ø Limitations of cloud provider services
Ø Troubleshooting multiple technologies
Ø Monitoring data from multiple sources
Ø Becoming agile and proactive
Constant Challenges

Umbrella
Monitoring
CA APM feeds data into ELKK
Extra K for Kafka
CA APM Integration points
allow for data export
Integration
Data from external rResources
CA APM EPAgent
Auto-Scaling &
Cloud Mgmt.
Application health metrics
from CA APM to manage cloud
Auto-scale application
instances based on real user
load
Smart load balancers
Keys to Success

Gathering Data – Integration with External
Tools

§ Some apps run unsupported languages
§ Alternative monitoring solutions limited in
abilities and granularity
§ CA APM abilities may be leveraged once the
data is imported into CA APM
§ CA APM EPAgent provides an easy platform
for integrating with external entities
Why Obtain Data From External Entities?

EPAgents
• Execute scripts which gather metrics from external sources
• Publish metrics to the CA APM Introscope environment
Example EPAgent Script:
#!/bin/bash
curl -sk -X GET "https://api.TheDevil.com/v2/applications/$appid/instances.json" -H 'X-Api-Key:sneakySneakySir' >
/opt/EPAgentMonitor/epaplugins/sling/metrics/perInstanceMetrics$appid.json
#Gathering throughput metric reported by the application overview page in TheDevil
throughput=$(cat /opt/EPAgentMonitor/epaplugins/sling/metrics/perInstanceMetrics$appid.json |
/opt/EPAgentMonitor/epaplugins/sling/jq64/jq –r ".application_instances[$x].application_summary.throughput")
#Publishing the data to CA APM Introscope
echo '<metric type="LongCounter" name="TheDevil|'$appname'|'$hn'|Frontend:Throughput" value="'$throughput'" />'
Gathering Data

Auto Scaling & Cloud Management

Use Native Options Provided By A Vendor
• Easy to use and implement
• Limited number of metrics
• Limited granularity
• Very limited action options
Make API Calls To A Provided Solution
• Requires a custom monitoring and/or decision making solution
• Still limited to provided actions
• If only feeding data, granularity may also be more limited
Use A Custom Solution
• More initial work and maintenance overhead
• Near unlimited number of actions
• Better visibility available
• Catered to the enterprise
Common Scaling Options

• CA APM has the ability to execute scripts as part of an alert action.
• This can be used to send commands to external endpoints:
• APIs to trigger scaling events
• Etc..
We Have the Data, Now How Do We Scale?

This alert:
• Evaluates the throughput
• Triggers the scaling script action when
the alert threshold is breached.
• Triggers a Single Time
• Adds App Instances
Scaling Event

Sudden Spike – Hit The Turbo Button!
This can be handled multiple ways, like using a separate action; we chose to use the same
action, but with the “Notify by individual metric” option
Sudden Spike – Hit the Turbo Button!
This alert:
• Evaluates the throughput
• Triggers the scaling script action when
the alert threshold is breached by any
load balancer instance.
• May trigger multiple times
• Adds app Instances for each trigger

A view of the same alert through scaling events:
Did it Actually Scale?
Instances
coming on-line
Instances
coming on-line
Reduction in load per
instance

Success:
• Send back a success value when
new instance passes health
check and has been activated in
load balancing pool
Failure:
• App Instance Scale Attempt Fails
Health Check
• Reattempt for 12 minutes
before moving on to next
Auto-Scaling Success and Failure Metrics

Running A Green Organization

How Do I Alert if I Cant Scale?
• Many correct Answers
• Our answer:
Scaling event scripts publish metrics for load balancer activation failures, and successes, to a
load balanced endpoint with CA APM EPAgent listeners as the pool members
#sending notice of application still failing health check
echo '<metric type="LongCounter" name="autoScale|Prod|MyLB|'$myLB'|'$myLBpool'|'$1':activateInstanceFailedStatusCheck" value="'$runCount'" />' |
perl /opt/autoScaleScripts/send_to_epagent.pl
#resetting value
echo '<metric type="LongCounter" name="autoScale|Prod|MyLB|'$myLB'|'$myLBpool'|'$1':activateInstanceFailedStatusCheck" value=“0" />' | perl
/opt/autoScaleScripts/send_to_epagent.pl
#sending notice of activation success
echo '<metric type="LongCounter" name="autoScale|Prod|MyLB|'$myLB'|'$myLBpool'|'$1':activateInstanceCommandExecute" value="4" />' | perl
#resetting value
echo '<metric type="LongCounter" name="autoScale|Prod|MyLB|'$myLB'|'$myLBpool'|'$1':activateInstanceCommandExecute" value=“0" />' | perl
Example Auto-Scaler Script
>>Click Here for Additional Information
How Do I Alert If I Can’t Scale?

Why Utilize CA APM to Scale?
• Out of the box, AWS auto
scaling is limited to OS metrics,
like CPU and memory
• Resolution OOB is limited to 5
minutes, or 1 minute with
additional costs
A custom solution, like CA APM, is
required to send app specific
metrics to AWS, and to be able to
react to spikes in a more timely
manor
Limited to OS Metrics
Cloud Provider Limitations
• AWS load balancers (LBs)
require a “warm up” period to
handle sudden, large spikes in
traffic
• AWS LBs have limited
application health check
capabilities
Custom solutions must be utilized
to integrate with third party LBs
that do not have these limitations
Load Balancers Require “Warm Up”

Umbrella Monitoring

Consolidation:
§ One tool to monitor everything
§ Single application to evaluate log data, APM metrics, and any
other data being published
Big data and historical analysis:
§ Many aggregation solutions are designed to accommodate
large data storage and historical analytics
Flexibility:
§ Popular aggregation solutions are often more flexible with 3rd
party integrations
Benefits of Umbrella Monitoring

You Bet Your Analysis
CA APM Data in ELK?

CA APM Data in ELK?

Pushing CA APM Data to ELK
CA APM Makes This Process Very Easy
1. Export Data to File
####
introscope.enterprisemanager.flatfile.homedir=/opt/csvFileOutput
introscope.enterprisemanager.flatfile.maxFileLength=10
introscope.enterprisemanager.flatfile.collection1.agentExpression=.*
introscope.enterprisemanager.flatfile.collection1.metricExpression=.*
introscope.enterprisemanager.flatfile.collection1.frequencyinseconds=1
5
Enterprise
Manager
File

2. Use Logstash to publish to Kafka
####
input {
file {
path => ["/opt/csvFileOutput/*.csv"]
sincedb_path => "/opt/theia/.sincedb"
}
}
Enterprise
Manager
File
Kafka

filter {
mutate { add_field => {"instance" => "%{host}"}}
csv {
columns => [
"Domain","AppGroup","Env","AgentName","Resource","MetricName","RecordType","Period","IntendedEndTimestamp","ActualStartTimestamp"
,"ActualEndTimestamp","ValueCount","ValueType","IntegerValue","IntegerMin","IntegerMax","FloatValue","FloatMin","FloatMax","StringValue",
"DateValue" ]
separator => ","
}
mutate {
remove_field => [ "Domain", "message", "path", "IntendedEndTimestamp", "ActualStartTimestamp" ]
}
ruby{
code => "event['timestamphashkey'] = 'key'+event['@timestamp'].to_s"
}}

Some Benefits To Your CA APM Environment
Dashboards
§ Create dashboards without additional user management related to CA APM
Long-term storage
§ By pushing to Kafka, long-term storage may be moved to other systems, like
Elasticsearch or Hadoop
§ Without the need for long-term storage, the CA APM Enterprise Manager Footprint may
be reduced
Accessibility
§ Other tools and teams may utilize CA APM metric data without taxing the CA APM data
with API requests

Dish Network, Sling TV, & CA APM
TAKE BACK TV®
Yes, we are an exciting organization to work for
Yes, our team is truly as amazing as it seems
Any other questions?

Thank you.
Stay connected at communities.ca.com

DevOps – Agile Ops
For more information on DevOps – Agile Ops, please visit:
http://cainc.to/wYXSg6

Case Study: Dish Network Utilizes CA Application Performance Management to Monitor and Auto Scale Its Applications

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Case Study: Dish Network Utilizes CA Application Performance Management to Monitor and Auto Scale Its Applications

Similar to Case Study: Dish Network Utilizes CA Application Performance Management to Monitor and Auto Scale Its Applications (20)

More from CA Technologies

More from CA Technologies (20)

Recently uploaded

Recently uploaded (20)

Case Study: Dish Network Utilizes CA Application Performance Management to Monitor and Auto Scale Its Applications