Testing in production for online applications has evolved into a critical component of successful performance testing strategies. Dan Bartow explains the fundamentals of cloud computing, its application to full-scale performance validation, and the practices and techniques needed to design and execute a successful testing-in-production strategy. Drawing on his experiences, Dan describes the methodology he has used for testing numerous online applications in a production environment with minimal disruption. He explains how to create a performance testing strategy to give your team critical data about how your online application performs and scales. Learn how to create a robust lab-to-production ecosystem that delivers the answers about what will happen when peak traffic hits your site. Take back practical approaches to mitigate the three most common problems—security, test data, and potential live customer impact—that arise when embarking on testing in production.

1. TK
PM Tutorial
4/30/13 1:00PM
Production Performance
Testing in the Cloud
Presented by:
Dan Bartow
SOASTA, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Dan Bartow
At SOASTA Dan Bartow is vice president of product management. Prior to joining SOASTA, Dan was
senior manager of engineering at Intuit where his team was responsible for the speed and stability of
TurboTax Online. During the past decade he has been responsible for the performance of websites for
leading consumer brands. Dan has set multiple industry precedents including launching the world's largest
stateful JBoss cluster and using more than 2,000 cloud computing cores to generate load against a live
website. He is credited with authoring the industry’s first testing in production methodology. A frequent
industry presenter, Dan has spoken at leading testing and cloud computing conferences.

3. Production Performance
Testing in the Cloud
Dan Bartow
SOASTA
Objectives
1.
2.
3.
4.
5.
Explain the fundamentals of cloud computing and its application to fullscale performance validation
Describe how to create a performance testing strategy to give your team
critical data about how your online application performs and scales
Learn how to create a robust lab-to-production ecosystem that delivers
the answers about what will happen when peak traffic hits your
application
Share the best practices and techniques needed to design and execute
a successful testing-in-production strategy
Give practical approaches to mitigate the three most common problems security, test data, and live customer impact - that arise when designing
production test plans
1

4. Agenda
1. Performance Engineering
Methodology
Categorization of Sites by Traffic
Effort vs. Gain
2. Cloud Computing Overview
What is cloud computing?
The killer app for cloud
Does it work for testing?
3. Lab to Launch Ecosystem
4. Production Performance Testing
Security
Test data in production
Potential live customer impact
Discipline and rigor
Most common bottlenecks
5. Case Studies and Discussion
TurboTax Online
Myspace
Green Button
Essential components - lab and live
Evolution of the testing cycle
Designing good test plans
Monitoring
About the Speaker
SOASTA
VP Product Management
CloudTest Evangelist
Intuit
Sr. Manger, Engineering
TurboTax Online and E-com
ATG
Sr. Deployment Engineer
Work
American Airlines, Best Buy, Target, Turbotax Online, QuickenOnline, MySpace,
Dennys, Dominos, Mattel, Hallmark, FAA, US Army, AT&T Wireless, Alcatel,
Newsweek, Oprah, NeimanMarcus, SBC, Plantronics, Kodak, JCrew, Cingular,
Newell Rubbermaid
2

5. Poor performance.
How Deep Should We Go?
3

6. Black Friday | Cyber Monday - 2011
http://www.internetretailer.com/2011/11/26/outage-hits-victoriassecretcom-black-friday
http://www.internetretailer.com/2011/11/28/thanksgiving-weekend-brings-multiple-site-headaches
No Outages in the Top Ten US
Online Retailers!
4

7. In 2011
Wal-Mart
Kroger
Target
Walgreen
The Home Depot
Costco
CVS Caremark
Lowe’s
Best Buy
Sears Holdings
At Least Six of the Top Ten Online
Retailers in the US
Tested Their Apps – In Production
– from the Cloud
http://www.stores.org/2011/Top-100-Retailers
Methodology
SOASTA CloudTest Methodology
Strategy
Implementation
Execution
Measurement
Define
People
Define
Analyze
Integrate
Processes
Design
Adjust
Test
Assess
5

8. Defining Performance
What does it mean?
Capacity?
Page Speed?
Reliability?
Scalability?
Transaction Speed?
Code Execution Speed?
Defining Performance
1. Capacity
2. Customer Experience
3. Reliability
4. Scalability
5. Transaction
6. Code Level
6

9. Performance Matters
Capacity
7

10. Categorizing Sites by Traffic
Categorizing Sites by Traffic
8

11. Categorizing Sites by Traffic
The top 10 websites get 42% of the visitors to the top 100, and 21% of the visitors
to the top 1,000*
The top 100 get 50% of the visitors to the top 1,000. I.e. the top 100 together get
as many visitors as the following 900 websites counted together*
If you want to break into the top 100 websites, you’ll need at least 22 million
visitors per month. If you “only” aim for the top 1,000, you’ll need at least 4.1
million visitors per month*
Gee, that can’t be so hard, can it…?
*SOURCE: http://royal.pingdom.com/2010/07/05/what-it-takes-to-be-a-top-100-websitecharts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+RoyalPingdom+%28Royal+Pingdom%29
Categorizing Sites by Traffic
Top 10 – Mega Sites*
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Google
Facebook
Youtube
Yahoo!
Windows Live
Baidu.com
Wikipedia
Blogger.com
Twitter
QQ.com
*SOURCE: Alexa Top 500 Worldwide - http://www.alexa.com/topsites
9

12. Categorizing Sites by Traffic
Top 10-500 – Massive Sites
MSN
Amazon.com
Wordpress
eBay
LinkedIn
Craigslist
MySpace
Flickr
Apple
Paypal
Reddit
Mashable
Skype
Amazon WS
Yelp
WSJ
Match.com
Walmart
Salesforce
AT&T
*SOURCE: Alexa Top 500 Worldwide - http://www.alexa.com/topsites
Categorizing Sites by Traffic
Top 500 – 2000 XL Sites
Top 2000-10000 Large Sites
Orbitz
Travelocity
NBA
United
AmericanAirlines
Marriott
NHL
Dominos
Avis
Mint
WhiteHouse.gov
*SOURCE: Alexa - http://www.alexa.com/
10

13. The Point?
Where should we focus the performance engineering efforts?
Testing in Production
Most common problems during peak
traffic periods
Bandwidth
Load Balancers
Database Scale
11

14. Production Performance Testing
Anyone can do it - the NYSE did production performance
testing in the eighties
Some enterprises have teams dedicated
to production performance testing
Most Commonly Used
12

15. Typical Approach to Testing
Test in a lab at a smaller scale and then use
.
Extrapolation
Extrapolation
Extrapolation
Extrapolation
Extrapolation
Extrapolation to Prod Breaks Down
Load
Environment
Invisible walls can only be identified out in the production environment and
through testing at scale – as load and environment changes, so do problems
13

16. Thank You, Public Cloud!
Some Cloud Providers and Locations
Amazon EC2 – Arlington, San Francisco, Portland, Dublin, Japan, Singapore
GoGrid – San Francisco
IBM BlueCloud – Toronto, Denver, Raleigh
Microsoft Azure – Chicago, San Antonio, Amsterdam, Dublin, China, Singapore
RackSpace – London, Dallas, Chicago
14

17. Cloud Computing
Q: What is Cloud Computing?
A: Access to affordable infinite compute power. Accessibility, Availability
and Affordability.
SAAS
PAAS
IAAS
A killer application (commonly shortened to killer app), in the jargon of technologists, has
been used to refer to any computer program that is so necessary or desirable that it proves
the core value of some larger technology…*
*source: wikipedia
Lab to Launch Ecosystem
Essential components – lab and live
15

18. Lab to Launch Ecosystem
Evolution of the testing cycle
100,000
10,000
1,500
Lab
Stage
Production
Lab to Launch Ecosystem
Designing good test plans
Heat Map
Scenario Guide
Test Plan Guide
16

19. Lab to Launch Ecosystem
Monitoring
What
Where
How
Production Performance Testing
Security
Test data in production
Potential live customer impact
Tools and techniques
17

20. Production Performance Testing
Discipline and rigor
The Play Book
Production Performance Testing
Most common bottlenecks
18

21. Practice #1
Test accounts in production
Practice #2
Place orders in production
19

22. Practice #3
Test on the live site
Practice #4
Test the top 6-8 critical transaction flows
20

23. Practice #5
Don’t test from the same cloud you’re hosted in
Case Studies
Source: http://www.soasta.com/customers/financial/ss_turbotax/
21

24. Case Studies
Source: http://highscalability.com/blog/2010/3/4/how-myspace-tested-their-live-site-with-1-million-concurrent.html
Case Studies
Source: http://www.greenbutton.net/blog#post349
22

25. Contact Information
Dan Bartow
VP Product Management
CloudTest Evangelist
SOASTA Inc.
Email:
Twitter:
LinkedIn:
Blog:
dan@soasta.com
www.twitter.com/PerfDan
www.linkedin.com/in/danbartow/
www.soasta.com
23