Test environments these days are rarely built out to the same level as production environments, so they can never achieve the full scale and realism that you’d see in “real life.” Plus, testing environments can easily get stale and out-of-date. Additionally, with the faster pace of modern application development and the rise of DevOps automation, the use of large-scale testing environments has been minimized. As a result, Testing in Production has grown in popularity. But what’s the right way to load and performance test an application in production to get the results you need without impacting your live users?

1. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load and Performance
Scott Barber
Chief Technologist
Brad Stoner
Senior Performance Engineer
Featuring:
Testing in Production

2. 2© 2015 Neotys
NEOTYS INTRODUCTION
Since 2005, Neotys has helped over 1,500 customers ensure their
applications' reliability, performance and quality.
900% Revenue Growth
2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014
1500
1200
1000
800
600
400
200
0
Customer Growth
France
USA
UK
Germany
India

3. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Webinar Introduction
The Questions:
• Is load and performance testing in production right for me?
• How can I avoid impacting real users on the app?
• What can I do to filter results to get to root cause?
• When and where should I consider service virtualization?
• How can I deal with security roadblocks when testing?
• Can I leverage existing tests to monitor the app in prod?
The Speakers:
• Scott Barber, CTO PerfTestPlus, Inc.
• Brad Stoner, Senior Performance Engineer, Neotys

4. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Chief Technologist, PerfTestPlus, Inc.
sbarber@perftestplus.com
www.perftestplus.com
@sbarber
Co-Founder: Workshop On Performance & Reliability
www.performance-workshop.org
Co-Author:Author: Contributing Author:
Books: www.perftestplus.com/pubs About me: about.me/scott.barber

5. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Senior Performance Engineer, Neotys
brad.stoner@neotys.com
www.neotys.com
@sandbreak80
Speaker:

6. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load & Performance Testing in Prod
Some know they are doing it…
… and are having positive results.
Many don’t realize they do it
• Monitor & Tune
• Respond to user complaints
• React to downtime
Most wish they did it better…
… probably why you are here.
But…

7. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load & Performance Testing in Prod
Some know they are doing it…
… and are having positive results.
Many don’t realize they do it
• Monitor & Tune
• Respond to user complaints
• React to downtime
Most wish they did it better…
… probably why you are here.
But…

8. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load & Performance Testing Précis

9. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load & Performance Testing Précis

10. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load & Performance Testing Précis

11. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load & Performance Testing Précis
Key Attributes (when most effective):
• Entire Lifecycle (including production)
• Cross-Functional
• Collaborative (including Ops)
• Continuous & Contiguous
Keys to Production Testing
• Appropriate Pre-Production Testing & Preparation
• Bi-Directional Test Results Flow
• Continual Cross-Team, Cross-Phase Collaboration
• Trends, Trends, Trends, and more Trends

12. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod

13. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod

14. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Important Questions:
What are my Prod Testing Goals?
• Find limits?
• Validate assumptions and/or improvements?
• Establish / improve empirical “extrapolation estimates”
How different are Test and Prod Environments?
• Hardware?
• Data?
• Shared services?
Who needs to be involved?
• Operations
• Architect(s)
• DBA(s)
• Product and/or Project management

15. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Important Questions:
What are the top risks to be mitigated (and how)?
• User impact?
• Data integrity?
• 3rd party service disruption / perceived DoS attack?
• Identifying and responding to “leading indicators” before prior
to serious impact?
• Recovery if “it all goes bad”?
How do to get started?
• Establish contiguous, cross-functional, collaborative teams
• Bake performance monitoring/testing into entire lifecycle
• Store results and build trends from *all* phases centrally
• Begin w/ low load, low impact, low risk “confirmatory” tests to
build confidence

16. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Tips: Reducing Risk to Users
• Understand existing prod perf profile *precisely*
• Know perf impact by scenario in test *precisely*
• Start small: collect, analyse and validate
• Validate “read-only” before “crud”
• Off hours
• Isolate servers/data centers where possible
• Ramp slowly
• Proactive “command center” system & real-user/synthetic
monitoring in real-time
• Practice the “kill switch”

17. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Testing the Waters

18. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Tips: Security & Access
• Ops are your new BFFs
• So are DBAs
• Plan the details in painful detail
• Faux accounts, data, tables and/or databases
• Stub, block, virtualize and/or avoid
• Proactive “command center” system & real-user/synthetic
monitoring in real-time
• Recovery if “it all goes bad”?

19. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Plan for the Worst

20. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Tips: Service Virtualization
• 3rd Parties
• Shared internal services
• Known “good” stuff – just to simplify
• Use realistic & variable response times (and data)
• Test the virtualization *a lot* in advance (single- & multi-
user scenarios)

21. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Impact of External Services

22. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Tips: Test Reuse
• Integrate “low-level” tests from dev builds up
• Integrate “mid-level” tests from integration builds up
• Use a sub-set of *the same* end-user tests for testing
*and* monitoring in prod
• Integrate production monitoring “tests” into dev,
integration & test environment builds/tests
• Results & trend re-use is *at least* as valuable as test
reuse.

23. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Know Where to Apply Load

24. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Tips: Determining Root Cause(s)
• Trends, trends, trends
• Have baselines & benchmarks for comparisons
• Know what each scenario impacts in advance
• Be able to “kill” individual scenarios w/in a test w/out
killing whole test
• Data analysis is a team sport
• Tools can help a lot!

25. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Know Why Something Fails

26. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Load and Performance Testing in
Production Summarized
1. Everybody does it.
a. Some don’t realize they are doing it
b. Most want to do it better (and can)
c. Asking better questions and establishing goals is key
2. Focus on reducing risk to users
3. Address security & access in advance, in detail
4. Consider (appropriate levels of) Service Virtualization
5. Share and reuse assets & artifacts where possible
6. Prepare for collaborative “Root Causing”

27. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Questions?

28. © 2015 PerfTestPlus, Inc. All rights reserved.Load & Performance Testing in Prod
Chief Technologist
about.me/scott.barber
E-mail: sbarber@perftestplus.com
Blog: scott-barber.blogspot.com
Web Site: PerfTestPlus.com
Twitter: @sbarber
Senior Performance Engineer
E-mail: brad.stoner@neotys.com
Blog: neotys.com/blog
Web Site: neotys.com
Twitter: @neotys
Contact Info