Your SlideShare is downloading. ×
a presentation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

a presentation

247
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
247
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. StickyMinds.com and Better Software magazine presents… Whose Environment is It Anyway? Performance Tuning in a Virtual Environment. Sponsored by Empirix Non-streaming participants should call 1-866-761-8643 International Non-streaming participants should call 1-904-596-2362
  • 2. Scott Barber, (A.K.A. “The Perf Guy”) [email_address] Chief Technologist, PerfTestPlus, Inc . www.perftestplus.com Vice President & Executive Director, Association for Software Testing www.associationforsoftwaretesting.org Co-Author, Performance Testing Guidance for Web Applications www.amazon.com/gp/product/0735625700 A Heuristic Approach
  • 3.
    • Developers write code
    • Developers release code for test
    • Testers test code, often in various environments
    • Testers assess system performance in not very production-like environments
    • Managers make decisions based on test results
    • Some defects get fixed before release
    • The team gets surprised by defects in production
    • The team puts in lots of overtime to fix the defects
    Testing in a Non-Virtual World
  • 4.
    • Wikipedia lists 8 categories of platform virtualization and 4 categories of resource virtualization
    • Platform Virtualization
      • Software simulates a computer environment on a host machine, usually called “virtual machines”
      • Applications run on “virtual machines” as if on a stand-alone hardware platform
      • As if stand-alone ≠ Isolated
      • Platform virtualization software has overhead
    • ( http://en.wikipedia.org/wiki/Virtualization )
    What is Virtualization?
  • 5.
    • Resource Virtualization
      • Identify a resource and exclusively assign it’s use to a specific application
      • Sometimes partitions single resources, sometimes spans multiple resources
      • Resources may or may not be co-located on a physical machine
      • Exclusively assigned ≠ Isolated
      • Resource virtualization has overhead
    • ( http://en.wikipedia.org/wiki/Virtualization )
    What is Virtualization?
  • 6.
    • Developers write code
    • Developers release code for test
    • Testers test code, often in various configurations
    • Testers assess system performance in not very production-like environments
    • Managers make decisions based on test results
    • Some defects get fixed before release
    • The team gets surprised by defects in production
    • The team puts in lots of overtime to fix the defects
    Testing in a Virtual World
  • 7.
    • Virtualization, if done well, can:
      • Dramatically reduce the overhead involved in managing test environments
      • Create opportunities for unique, targeted, ad-hoc environments without additional hardware costs
      • Simplify configuration and release management
      • Put the test team in control of their test environments
    • But what about performance testing?
    What’s the Difference?
  • 8. “ With an order of magnitude fewer variables performance testing could be a science, but for now, performance testing is at best a scientific art.” --Scott Barber
  • 9.
    • Test environments are still test environments
    • Virtual test environments are less likely to match virtual production environments
    • There are more places for performance issues to hide in virtual environments
      • Exclusively assigned ≠ Isolated
      • As if stand-alone ≠ Isolated
      • Virtualization adds overhead
    • Study : “virtualized servers can handle 14% lower [load] than traditional server configurations” ( http://www.sys-con.com/read/503412.htm )
    Testing Virtual Performance?!?
  • 10. “ Linear extrapolation of performance test results is, at best, black magic. Don’t do it. (Unless you learned from Connie Smith, PhD or Daniel Menasce, PhD)” --Scott Barber
  • 11.
    • If done well, there can be benefits:
      • Dramatically reduce the overhead involved in managing test environments
      • Easier for performance testing to have it’s own test environment
      • Enable collaborative tuning without having to “borrow” hardware resources
      • Put the test team in control of their test environments
    • Virtualization does not change the core principles of successful performance testing
    Testing Virtual Performance?!?
  • 12. “ Only performance testing at the conclusion of system or functional testing is like ordering a diagnostic blood test after the patient is dead.” --Scott Barber
  • 13. Core Principles Project context is central to successful performance testing. Business, project, system, & user success criteria. Identify system usage, and key metrics; plan & design tests. Install & prepare environments, tools, & resource monitors . Script the performance tests as designed. Run and monitor tests . Validate tests , test data, and results. Analyze the data individually and as a cross-functional team. Consolidate and share results, customized by audience. "Lather, rinse, repeat" as necessary.
  • 14. Contact Info
    • Scott Barber
    • Chief Technologist
    • PerfTestPlus, Inc
    • E-mail:
    • sbarber@perftestplus.com
    Web Site: www.PerfTestPlus.com
  • 15. Whose Environment Is It Anyway? Dan Koloski Web BU CTO and Director of Strategy Empirix IMAGE
  • 16. Empirix – At a Glance
    • Testing & Monitoring Solutions for:
    • Privately held, founded 2000
    • 380+ employees
    • Worldwide presence; HQ in Boston & offices in Japan, UK, Germany and South Korea
    • Patents in test, monitoring and VoIP diagnostics methodology
    Web Applications & SOA Contact Center & CRM Carrier-Class VoIP & Telco
  • 17. Empirix in the Quality Ecosystem
    • Empirix Offerings Delivered Outcome
    PRODUCTS MANAGED SERVICES EXPERT SERVICES Onsite Consulting, Training, Mentoring e-TEST suite ™ One Sight ™ e-LoadExpert ™ Do-it-yourself Remote Validation Performance Tuning Supplement Your Team We Do it for You
  • 18. A few points to add to Scott’s presentation…
    • Scott said…
      • Virtualization does not change the core principles of successful performance testing
    • Why?
      • Hardware is not virtual
      • Whose environment is it, anyway?
      • Rapid Bottleneck Identification Tuning a Virtualized Context
  • 19. Hardware is not virtual
    • Virtualization is app/OS or hardware emulation specific
    • Underneath it all is a fixed pool of HW resources!
    • Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user
    Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional , http://qazone,empirix.com.
  • 20. The Best Performance Data in a Virtual Environment
    • Understand the impact on the hardware…
      • Requires accurate workload stimulus representing activity on and through all VMs associated with that device
      • Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM
      • Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC)
    • … as seen from the perspective of the user!
      • Your users don’t care about Virtualization
      • Metrics that matter are end-user perceived performance
        • Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance
  • 21. Rapid Bottleneck Identification Load Testing in a Virtual Environment
    • What is Load testing?
      • “ Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements”
  • 22. What is a Bottleneck?
    • Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element
    • On the Web, bottlenecks directly affect performance and scalability
    • Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time.
  • 23. Where are the Bottlenecks?
    • The vast majority of bottlenecks are caused by throughput limitations – not concurrency.
    Source: Empirix
  • 24. Sample throughput limitations in virtual environments
    • Disk I/O
      • Multiple applications accessing shared disk resources
      • Virtual Memory Paging on shared disk resources
    • NIC
      • Multiple VMs shipping data through the same physical NIC
    • CPU
      • Processor-intensive (i.e., database) applications will compete for shared CPU time.
    • Memory
      • Depends on type of VM configuration how much of an issue this can be
  • 25. Whose Environment Is It Anyway?
    • Virtualization blurs the line between Dev/Test/Ops environments and ownership
      • Can exacerbate “silo” problems between Dev/Test/Ops
    • However, there is an opportunity…
    • … A common set of measurements can un-blur the line
      • Test data should inform production SLAs
      • Common measurement/metrics can bridge the gap
      • Measurement consistency is the cornerstone of ALM
  • 26. Load Testing Tools DON’T Collaborate
    • Load Testing Tools Apply Load and Make Measurements
    • People (and their expertise & experience) Find Bottlenecks & Resolve Them
    • Collaboration With Experts Resolves Issues Faster & Cheaper
    • Key to Successful Collaboration = Access to People & Access to Measurements
    • Tools Should Support Collaboration
  • 27. e-TEST suite is designed for collaboration
    • Distributed, high-scale load control
    • Multi-user Web UI for real-time individual analysis
    Each user can view results from the same load test as it runs
  • 28. Summary
    • Hardware is not virtual – virtualization does NOT change core performance test best practice
    • However…
      • Virtualization introduces new test planning challenges to cover
        • Workload/stimulus
        • Measurement of impact on end-user
    • In a world of scarce time, focus efforts on throughput-related workload to identify most common bottlenecks
  • 29. Visit QAZone for the slides
    • http://qazone.empirix.com
    • Free resource, open to all (Empirix & non-Empirix users)
      • Forums (both QA-centric and Product related discussions)
      • Knowledge Base
      • Resource Center
      • Events, Announcements, Community & Industry News
    • Allows members to interact with peers, industry experts and a larger Empirix audience
  • 30. Have a question for the speakers? Ask now. Q & A