a presentation


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

a presentation

  1. 1. StickyMinds.com and Better Software magazine presents… Whose Environment is It Anyway? Performance Tuning in a Virtual Environment. Sponsored by Empirix Non-streaming participants should call 1-866-761-8643 International Non-streaming participants should call 1-904-596-2362
  2. 2. Scott Barber, (A.K.A. “The Perf Guy”) [email_address] Chief Technologist, PerfTestPlus, Inc . www.perftestplus.com Vice President & Executive Director, Association for Software Testing www.associationforsoftwaretesting.org Co-Author, Performance Testing Guidance for Web Applications www.amazon.com/gp/product/0735625700 A Heuristic Approach
  3. 3. <ul><li>Developers write code </li></ul><ul><li>Developers release code for test </li></ul><ul><li>Testers test code, often in various environments </li></ul><ul><li>Testers assess system performance in not very production-like environments </li></ul><ul><li>Managers make decisions based on test results </li></ul><ul><li>Some defects get fixed before release </li></ul><ul><li>The team gets surprised by defects in production </li></ul><ul><li>The team puts in lots of overtime to fix the defects </li></ul>Testing in a Non-Virtual World
  4. 4. <ul><li>Wikipedia lists 8 categories of platform virtualization and 4 categories of resource virtualization </li></ul><ul><li>Platform Virtualization </li></ul><ul><ul><li>Software simulates a computer environment on a host machine, usually called “virtual machines” </li></ul></ul><ul><ul><li>Applications run on “virtual machines” as if on a stand-alone hardware platform </li></ul></ul><ul><ul><li>As if stand-alone ≠ Isolated </li></ul></ul><ul><ul><li>Platform virtualization software has overhead </li></ul></ul><ul><li>( http://en.wikipedia.org/wiki/Virtualization ) </li></ul>What is Virtualization?
  5. 5. <ul><li>Resource Virtualization </li></ul><ul><ul><li>Identify a resource and exclusively assign it’s use to a specific application </li></ul></ul><ul><ul><li>Sometimes partitions single resources, sometimes spans multiple resources </li></ul></ul><ul><ul><li>Resources may or may not be co-located on a physical machine </li></ul></ul><ul><ul><li>Exclusively assigned ≠ Isolated </li></ul></ul><ul><ul><li>Resource virtualization has overhead </li></ul></ul><ul><li>( http://en.wikipedia.org/wiki/Virtualization ) </li></ul>What is Virtualization?
  6. 6. <ul><li>Developers write code </li></ul><ul><li>Developers release code for test </li></ul><ul><li>Testers test code, often in various configurations </li></ul><ul><li>Testers assess system performance in not very production-like environments </li></ul><ul><li>Managers make decisions based on test results </li></ul><ul><li>Some defects get fixed before release </li></ul><ul><li>The team gets surprised by defects in production </li></ul><ul><li>The team puts in lots of overtime to fix the defects </li></ul>Testing in a Virtual World
  7. 7. <ul><li>Virtualization, if done well, can: </li></ul><ul><ul><li>Dramatically reduce the overhead involved in managing test environments </li></ul></ul><ul><ul><li>Create opportunities for unique, targeted, ad-hoc environments without additional hardware costs </li></ul></ul><ul><ul><li>Simplify configuration and release management </li></ul></ul><ul><ul><li>Put the test team in control of their test environments </li></ul></ul><ul><li>But what about performance testing? </li></ul>What’s the Difference?
  8. 8. “ With an order of magnitude fewer variables performance testing could be a science, but for now, performance testing is at best a scientific art.” --Scott Barber
  9. 9. <ul><li>Test environments are still test environments </li></ul><ul><li>Virtual test environments are less likely to match virtual production environments </li></ul><ul><li>There are more places for performance issues to hide in virtual environments </li></ul><ul><ul><li>Exclusively assigned ≠ Isolated </li></ul></ul><ul><ul><li>As if stand-alone ≠ Isolated </li></ul></ul><ul><ul><li>Virtualization adds overhead </li></ul></ul><ul><li>Study : “virtualized servers can handle 14% lower [load] than traditional server configurations” ( http://www.sys-con.com/read/503412.htm ) </li></ul>Testing Virtual Performance?!?
  10. 10. “ Linear extrapolation of performance test results is, at best, black magic. Don’t do it. (Unless you learned from Connie Smith, PhD or Daniel Menasce, PhD)” --Scott Barber
  11. 11. <ul><li>If done well, there can be benefits: </li></ul><ul><ul><li>Dramatically reduce the overhead involved in managing test environments </li></ul></ul><ul><ul><li>Easier for performance testing to have it’s own test environment </li></ul></ul><ul><ul><li>Enable collaborative tuning without having to “borrow” hardware resources </li></ul></ul><ul><ul><li>Put the test team in control of their test environments </li></ul></ul><ul><li>Virtualization does not change the core principles of successful performance testing </li></ul>Testing Virtual Performance?!?
  12. 12. “ Only performance testing at the conclusion of system or functional testing is like ordering a diagnostic blood test after the patient is dead.” --Scott Barber
  13. 13. Core Principles Project context is central to successful performance testing. Business, project, system, & user success criteria. Identify system usage, and key metrics; plan & design tests. Install & prepare environments, tools, & resource monitors . Script the performance tests as designed. Run and monitor tests . Validate tests , test data, and results. Analyze the data individually and as a cross-functional team. Consolidate and share results, customized by audience. &quot;Lather, rinse, repeat&quot; as necessary.
  14. 14. Contact Info <ul><li>Scott Barber </li></ul><ul><li>Chief Technologist </li></ul><ul><li>PerfTestPlus, Inc </li></ul><ul><li>E-mail: </li></ul><ul><li>sbarber@perftestplus.com </li></ul>Web Site: www.PerfTestPlus.com
  15. 15. Whose Environment Is It Anyway? Dan Koloski Web BU CTO and Director of Strategy Empirix IMAGE
  16. 16. Empirix – At a Glance <ul><li>Testing & Monitoring Solutions for: </li></ul><ul><li>Privately held, founded 2000 </li></ul><ul><li>380+ employees </li></ul><ul><li>Worldwide presence; HQ in Boston & offices in Japan, UK, Germany and South Korea </li></ul><ul><li>Patents in test, monitoring and VoIP diagnostics methodology </li></ul>Web Applications & SOA Contact Center & CRM Carrier-Class VoIP & Telco
  17. 17. Empirix in the Quality Ecosystem <ul><li>Empirix Offerings Delivered Outcome </li></ul>PRODUCTS MANAGED SERVICES EXPERT SERVICES Onsite Consulting, Training, Mentoring e-TEST suite ™ One Sight ™ e-LoadExpert ™ Do-it-yourself Remote Validation Performance Tuning Supplement Your Team We Do it for You
  18. 18. A few points to add to Scott’s presentation… <ul><li>Scott said… </li></ul><ul><ul><li>Virtualization does not change the core principles of successful performance testing </li></ul></ul><ul><li>Why? </li></ul><ul><ul><li>Hardware is not virtual </li></ul></ul><ul><ul><li>Whose environment is it, anyway? </li></ul></ul><ul><ul><li>Rapid Bottleneck Identification Tuning a Virtualized Context </li></ul></ul>
  19. 19. Hardware is not virtual <ul><li>Virtualization is app/OS or hardware emulation specific </li></ul><ul><li>Underneath it all is a fixed pool of HW resources! </li></ul><ul><li>Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user </li></ul>Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional , http://qazone,empirix.com.
  20. 20. The Best Performance Data in a Virtual Environment <ul><li>Understand the impact on the hardware… </li></ul><ul><ul><li>Requires accurate workload stimulus representing activity on and through all VMs associated with that device </li></ul></ul><ul><ul><li>Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM </li></ul></ul><ul><ul><li>Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC) </li></ul></ul><ul><li>… as seen from the perspective of the user! </li></ul><ul><ul><li>Your users don’t care about Virtualization </li></ul></ul><ul><ul><li>Metrics that matter are end-user perceived performance </li></ul></ul><ul><ul><ul><li>Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance </li></ul></ul></ul>
  21. 21. Rapid Bottleneck Identification Load Testing in a Virtual Environment <ul><li>What is Load testing? </li></ul><ul><ul><li>“ Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements” </li></ul></ul>
  22. 22. What is a Bottleneck? <ul><li>Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element </li></ul><ul><li>On the Web, bottlenecks directly affect performance and scalability </li></ul><ul><li>Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time. </li></ul>
  23. 23. Where are the Bottlenecks? <ul><li>The vast majority of bottlenecks are caused by throughput limitations – not concurrency. </li></ul>Source: Empirix
  24. 24. Sample throughput limitations in virtual environments <ul><li>Disk I/O </li></ul><ul><ul><li>Multiple applications accessing shared disk resources </li></ul></ul><ul><ul><li>Virtual Memory Paging on shared disk resources </li></ul></ul><ul><li>NIC </li></ul><ul><ul><li>Multiple VMs shipping data through the same physical NIC </li></ul></ul><ul><li>CPU </li></ul><ul><ul><li>Processor-intensive (i.e., database) applications will compete for shared CPU time. </li></ul></ul><ul><li>Memory </li></ul><ul><ul><li>Depends on type of VM configuration how much of an issue this can be </li></ul></ul>
  25. 25. Whose Environment Is It Anyway? <ul><li>Virtualization blurs the line between Dev/Test/Ops environments and ownership </li></ul><ul><ul><li>Can exacerbate “silo” problems between Dev/Test/Ops </li></ul></ul><ul><li>However, there is an opportunity… </li></ul><ul><li>… A common set of measurements can un-blur the line </li></ul><ul><ul><li>Test data should inform production SLAs </li></ul></ul><ul><ul><li>Common measurement/metrics can bridge the gap </li></ul></ul><ul><ul><li>Measurement consistency is the cornerstone of ALM </li></ul></ul>
  26. 26. Load Testing Tools DON’T Collaborate <ul><li>Load Testing Tools Apply Load and Make Measurements </li></ul><ul><li>People (and their expertise & experience) Find Bottlenecks & Resolve Them </li></ul><ul><li>Collaboration With Experts Resolves Issues Faster & Cheaper </li></ul><ul><li>Key to Successful Collaboration = Access to People & Access to Measurements </li></ul><ul><li>Tools Should Support Collaboration </li></ul>
  27. 27. e-TEST suite is designed for collaboration <ul><li>Distributed, high-scale load control </li></ul><ul><li>Multi-user Web UI for real-time individual analysis </li></ul>Each user can view results from the same load test as it runs
  28. 28. Summary <ul><li>Hardware is not virtual – virtualization does NOT change core performance test best practice </li></ul><ul><li>However… </li></ul><ul><ul><li>Virtualization introduces new test planning challenges to cover </li></ul></ul><ul><ul><ul><li>Workload/stimulus </li></ul></ul></ul><ul><ul><ul><li>Measurement of impact on end-user </li></ul></ul></ul><ul><li>In a world of scarce time, focus efforts on throughput-related workload to identify most common bottlenecks </li></ul>
  29. 29. Visit QAZone for the slides <ul><li>http://qazone.empirix.com </li></ul><ul><li>Free resource, open to all (Empirix & non-Empirix users) </li></ul><ul><ul><li>Forums (both QA-centric and Product related discussions) </li></ul></ul><ul><ul><li>Knowledge Base </li></ul></ul><ul><ul><li>Resource Center </li></ul></ul><ul><ul><li>Events, Announcements, Community & Industry News </li></ul></ul><ul><li>Allows members to interact with peers, industry experts and a larger Empirix audience </li></ul>
  30. 30. Have a question for the speakers? Ask now. Q & A