Magical Performance tuning with Gomez

2,319 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,319
On SlideShare
0
From Embeds
0
Number of Embeds
52
Actions
Shares
0
Downloads
41
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Key themes: load testing helps ensure success of critical web based initiatives by reducing risk when launching promotions, rolling out new apps, upgrading infrastructure and anticipating seasonal spikes in traffic Talk track How much risk can your organization accept when launching new web-based initiatives? Whether it’s rolling out a new website or application, launching a major marketing campaign, upgrading or virtualizing infrastructure, or preparing for seasonal spikes in traffic, load testing helps you reduce risk and ensure success. Load testing ensures that your key web pages and transactions perform under load across all the geographic locations you care about. Problems across the entire web application delivery chain can be found and fixed prior to launch – maximizing revenue and protecting your brand. With today’s on-demand “2.0” solutions, load testing is accessible to more organizations and departments than ever before
  • Last updated or created: Nov ‘09 Changed title Changed script Key themes: State main benefits we deliver to companies Gomez offers a SaaS-based, easy-to-access solution Gomez is a healthy, established, market leader. Talk track Here is a brief overview of Gomez. We help organizations deliver quality experiences to their Web and mobile users. What is a “quality experience?” Optimal performance and availability – You’ll use Gomez to make your web site fast and always available. For example, one customer reduced page load time from 11 seconds to 3 seconds with Gomez. Quality operation across all users, browsers, devices, and geographies -- You’ll use Gomez to ensure you web and mobile applications work for all of your users, across all the browsers and mobile devices, and across all your key geographies. For example, one customer saves 50% in staff fees We also help you resolve any issues that arise with: Rapid issue notification with actionable diagnostics – if something is wrong with your website, Gomez will alert you immediately, along with diagnostics to help you identify and resolve the root cause. Insight into how these issues affect your business (revenue, brand, cost) – a web site’s performance can be the difference between a sale or a lost customer. One Gomez customer has seen a 10% increase in conversions because of Gomez. Some information about the division: SaaS platform – Gomez is on-demand. No installation required. You get quick startup and rapid payback. And it operates on the world’s most comprehensive testing network – over 100K+ global locations and growing organically every day 2,500+ customers worldwide – Any company who relies on their web or mobile application can benefit from Gomez. We work with very large brands, such as Microsoft, MySpace, Google, and Apple – in fact, 12 of the top 20 visited US sites use Gomez. We also work with as many small companies, ones you’ve probably never heard of. We have experience and experience with all of their businesses, and we share them with you. When you become a Gomez customer, you gain the benefit of that insight. Solid business performance -- Gomez is health and established. When it was an independent company, it had five years of consecutive double digit revenue growth, including 44% in 2008. Offices around the globe. We’re recognized as the industry leader – Forrester calls us “THE leader in Web experience management,” and says we’re in the “on fire” category. Gartner calls us a “cool vendor” to watch. And EMA says Gomez is one of the “few companies who gets it.”
  • In the past, fewer machines served up the same information on one network that now is distributed The applications have gotten more complex and they’re very distributed The average web application now includes SIX separate web applications and they can have an impact on your brand. Third parties can be ads or videos, marketing tracking of visitors. Most of these applications can perform very differently under load., Sometimes the only place that all of these third parties come together is in a browser on the end user’s machine. CDNs can have wildly different performance in different geographies. The effectiveness of your CDN has a huge impact on your user experience and how your customers view the performance of your application.
  • Last updated or created: Nov ‘09 Moved firewall to the right; more detail in data center Key themes: Many things can and do go wrong in delivering Web/mobile applications Your customer expects you to control them all – not just what’s in your data center You can control these issues, but you must be aware of them first Talk track What kinds of things can go wrong in this Web application delivery chain ? Unfortunately, there are a long list of possible issues. This slide shows examples of the various issues that can occur at any point along the chain. It’s pretty ugly, because there’s a lot that can go wrong. Some of these can occur inside your firewall, but they won’t show up on system monitoring tools. Have you ever heard of or experienced any of these problems? Are you using any third party or cloud services? Are you using a CDN? Do you know if your application runs on every browser and O/S and mobile device? They happen every day. And the harsh reality is: they change everyday, too. You can get them right one day and something goes wrong the next. Your users expect you to control these issues. If they try to visit your site or run your Web application and it doesn’t work, they hold you responsible. The good news is that all of these issues can be controlled – but first you need to be aware of them. That’s the first step to fixing them and ensuring your user has a positive experience.
  • Last updated or created: Nov ‘09 Moved firewall to the right; more detail in data center Key themes: Delivering Web/mobile apps is complicated and involves many services. Talk track Why is it your data center monitoring tools can be reporting “green” but your users are unhappy? It’s because of something called “the Web applications delivery chain.” To deliver a Web or mobile application in today’s Internet, you must use a complex set of services and layers that are called the “ Web application delivery chain .” These services must all work together to deliver the application to the user. If any one of them has an issue, your user will have a bad experience. First, your application must go through your major corporate ISP. Then, it travels thru the Internet. Increasingly, Web applications today are using third party or cloud services (we’ve seen companies where their home page has over 20 third party providers or services). It’s also very common to use a Content Delivery Network to accelerate the delivery of the Web application. Eventually, your Web or mobile application goes thru a local ISP or a mobile carrier. Ultimately it reaches your end user or customer, where it has to run on their local device – a computer or smart phone – and in their browser. A few years ago it seemed that only Internet Explorer mattered, but now there are a myriad of browsers on the market and, unfortunately for companies delivering Web applications, they all work a differently.
  • Key themes: Gomez covers the globe with the most comprehensive testing network We are where your customers are Talk track This is a visual depiction of our global testing network. You can see where our Backbone and Last Mile testing locations are. Our Last Mile locations literally span the globe and allow you to test and monitor from any significant location in the world. And it’s growing every day. You can use these for a combination of monitoring and load testing. You can’t see the locations for the virtual test bed because it’s virtual – i.e. location independent.
  • All aspects of the user experience delivery chain must be tested Business Impact Customers negatively impacted by Higher response times Time-out errors
  • Magical Performance tuning with Gomez

    1. 1. Are you ready for more Website visitors? Best Practices to realistically load test your applications beyond the firewall Hari Ramachandran, Project Manager, Cognizant Colin Mason - Product Manager, Gome z
    2. 2. Cognizant Testing Services
    3. 3. Need for Positive User Experience <ul><li>Internet users has increased from 360 Million in 2000 to 1.8 billion in 2009. </li></ul><ul><li>Around Twenty Five per cent of the world’s population now use the Internet. North America (74%) has the highest penetration. </li></ul>
    4. 4. User Expectations Not Met at Peak Times Impact the Business <ul><ul><li>67% of online consumers expect consistently good experiences at peak times regardless of website traffic, but their expectations are not met </li></ul></ul><ul><ul><li>Poor web experiences at peak times impacts revenue, brand & loyalty </li></ul></ul><ul><ul><ul><ul><li>78% of consumers have gone to a competitor site due to poor performance at peak times </li></ul></ul></ul></ul><ul><ul><ul><ul><li>After a poor experience </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>42% have discussed it with family, friends, peers or online </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>47% left with a less positive perception of the company </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>88% of online consumers are less likely to return to a site </li></ul></ul></ul></ul></ul>Total Sample (N=1538)
    5. 5. Many of the industry’s largest merchants suffered site slowdowns or other problems in 2009 <ul><li>Non Functional Requirement’s are not properly defined and tested </li></ul><ul><li>Web server/App Server/Middleware/Database server are not properly configured </li></ul><ul><li>Database is not properly tuned </li></ul><ul><li>Application is not designed to scale for huge volumes </li></ul><ul><li>Infrastructure is not properly sized </li></ul><ul><li>3rd party components or services components didn’t scale under load </li></ul>
    6. 6. <ul><li>Make informed business decisions about capacity, third-party components & services, or future expansions plans </li></ul><ul><li>Verify response-time and end-user experience under load </li></ul><ul><li>Find problems and bottlenecks before real end-users are impacted </li></ul><ul><ul><li>Mitigate risk when launching promotions, rolling-out new applications or features, upgrading or virtualizing infrastructure, or anticipating seasonal spikes in web traffic </li></ul></ul><ul><ul><li>Protect online revenue when it matters most, at peak times </li></ul></ul><ul><ul><li>Ensure performance and availability of key transactions across geographies, for expected user load and beyond </li></ul></ul>Why Performance Testing is Critical? <ul><li>Ensure success of critical Web-based business initiatives </li></ul><ul><li>Validate speed, scalability and stability </li></ul><ul><li>Deploy applications, new features & infrastructure with confidence </li></ul>
    7. 7. The Challenge of Delivering Web Applications <ul><li>Response time should be less than 2 seconds </li></ul><ul><li>Application should be available 24*7 </li></ul><ul><li>Application should support X concurrent users </li></ul><ul><li>Infrastructure should be shared </li></ul><ul><li>CDNs, external 3rd party components & services outside the firewall should also meet SLAs </li></ul>The Web Application Delivery Chain Major ISP Local ISP Mobile Carrier Internet Content Delivery Networks 3rd Party/ Cloud Services Browsers and devices Users Users Storage Web Servers App Servers DB Servers Mainframe Load Balancers Mobile Components Network
    8. 8. Performance Testing – Maturity Levels
    9. 9. When to Performance Test in Development Lifecycle Requirement Analysis <ul><ul><li>Gather the Non Functional requirements – Involve all stakeholders </li></ul></ul><ul><ul><li>Convert the business requirements into measurable performance </li></ul></ul>Development <ul><ul><li>Early Performance Testing/ Component Level Performance Testing </li></ul></ul><ul><ul><li>Application Profiling </li></ul></ul>Design <ul><ul><li>Architectural Benchmarking </li></ul></ul><ul><ul><ul><li>Software benchmarking </li></ul></ul></ul><ul><ul><ul><li>Hardware benchmarking </li></ul></ul></ul>Integration & Testing <ul><ul><li>Pre Production Performance Testing – Load Testing, Stress Testing, Spike Testing, Endurance Testing, Volume Testing, Real End User Testing </li></ul></ul>Implementation <ul><ul><li>Application Performance Monitoring </li></ul></ul><ul><ul><li>Performance Benchmarking </li></ul></ul>
    10. 10. Recommendations For Successful Performance Testing <ul><li>Start early in the project lifecycle </li></ul><ul><li>Gather Application Usage Statistics - Workload Analysis, Business critical transactions, User locations , Resource utilization </li></ul><ul><li>Define Performance Testing objectives </li></ul><ul><li>Identify the right tool for Performance testing Tool </li></ul><ul><ul><li>Inside the firewall – HP Loadrunner, Micro focus SilkPerformer, IBM Rational Performance Tester, Microsoft VSTS, Radview Webload, Neotys NeoLoad, OpenSTA, Jmeter </li></ul></ul><ul><ul><li>Outside the firewall – Gomez Reality Load </li></ul></ul><ul><li>Have Production like environment </li></ul><ul><li>Scripting – Parameterization, Correlation, Error validation, Smoke Tests </li></ul><ul><li>Deploy Monitors to collect server side statistics </li></ul><ul><li>Execute Baseline tests, Plan for iterative Performance testing </li></ul>
    11. 11. Performance Testing & Problem Diagnosis Design Phase : Architecture Performance Benchmarking <ul><li>Application prototype was becnhamarked on the planned production environment </li></ul><ul><li>Early identification of design issues and uncovered major performance risk posed by the third-party products </li></ul><ul><li>saving huge cost and effort </li></ul><ul><li>Company: Real Estate and Relocation Services </li></ul><ul><ul><ul><li>Building a new online real-estate transaction management system </li></ul></ul></ul><ul><ul><ul><li>Perform End to End performance Testing </li></ul></ul></ul>Excessive Threads Spawned High Virtual Memory Usage After implementing Thread Pooling & Fix Patches From Third-Party Components
    12. 12. Performance Testing & Problem Diagnosis Development Phase: Early Performance Testing <ul><li>Early identification of critical code-level performance issues resulted in significant cost and effort saving for all stakeholders </li></ul><ul><li>After fixing, the application code was stable and enabled increased scalability and better performance </li></ul>It is seen that the available memory goes down as the number of loaded assemblies increases Number of loaded assemblies keeps increasing throughout the test, although the same transactions are repeatedly executed Heap dump data showing many dynamically loaded assemblies in the heap <ul><li>Memory dumps show that there are many dynamically generated XMLSerializer assemblies present in memory </li></ul><ul><li>Cache the XMLSeraliazer objects after creation, so they are not created every time. This avoids multiple creation of assemblies </li></ul><ul><li>With the fix, the number of assemblies load stabilizes 5 minutes after the test, while without the fix, it grows continuously </li></ul>
    13. 13. Performance Testing & Problem Diagnosis Integration & Testing Phase: Pre-Production Performance Testing <ul><li>Application Server configurations where optimized for optimal settings after iterative test execution cycle. This eliminated high CPU utilization of the systems </li></ul><ul><li>Reduced the CPU utilization of the application server from 95% to around 20% </li></ul><ul><li>Availability of the system was improved from 87% to around 99% </li></ul>Sudden drop in throughput after 70 users indicating a bottleneck. CPU Utilization also peaks indicating the server side bottleneck
    14. 14. <ul><li>Web applications increasingly assembled at browser level </li></ul><ul><ul><li>3rd party performance from ads, videos, shopping cart vary widely under load </li></ul></ul><ul><li>Content Delivery Network configuration and effectiveness impact end-user experience </li></ul>Web Application Delivery In A Web 2.0 World
    15. 15. The Challenge of Ensuring Quality Web Experiences Zone of customer expectation Zone of customer expectation Zone of customer expectation Zone of customer expectation Traditional zone of control The Web Application Delivery Chain <ul><li>Network peering problems </li></ul><ul><li>Outages </li></ul><ul><li>Inconsistent geo performance </li></ul><ul><li>Bad performance under load </li></ul><ul><li>Blocking content delivery </li></ul><ul><li>Incorrect geo-targeted content </li></ul><ul><li>Configuration issues </li></ul><ul><li>Oversubscribed POP </li></ul><ul><li>Poor routing optimization </li></ul><ul><li>Low cache hit rate </li></ul><ul><li>Network peering problems </li></ul><ul><li>Bandwidth throttling </li></ul><ul><li>Inconsistent connectivity </li></ul><ul><li>Configuration errors </li></ul><ul><li>Application design issues </li></ul><ul><li>Code defects </li></ul><ul><li>Insufficient infrastructure </li></ul><ul><li>Poorly performing JavaScript </li></ul><ul><li>Browser/device incompatibility </li></ul><ul><li>Page size too big </li></ul><ul><li>Too many objects </li></ul><ul><li>Low cache hit rate </li></ul><ul><li>Network resource shortage </li></ul><ul><li>Faulty content transcoding </li></ul><ul><li>SMS routing / latency issues </li></ul>Systems management tools: “OK” … user is NOT happy Traditional zone of control
    16. 16. Gomez Load Testing 2.0 : On-Demand Realistic Load Testing <ul><li>High volume load (HTTP, Browser) </li></ul><ul><li>Find infrastructure breaking points </li></ul><ul><li>Define capacity headroom </li></ul><ul><li>Real-world load </li></ul><ul><li>Find user experience breaking points </li></ul><ul><li>Accurately measure response time </li></ul>Major ISP Local ISP Mobile Carrier Internet Content Delivery Networks 3rd Party/ Cloud Services Browsers and devices Users Storage Web Servers App Servers DB Servers Mainframe Load Balancers Mobile Components Network
    17. 17. Gomez Network: The World’s Most Comprehensive Performance and Testing Network Web Performance Management and Load Testing 100+ locations Backbone Cross-Browser Testing 500+ browser/OS combo’s 5,000+ supported devices Virtual Test Bed Web Performance Management and Load Testing 100,000+ locations Gomez Last Mile
    18. 18. <ul><li>Company: Online Retailer </li></ul><ul><ul><ul><li>Several 3rd Parties now involved in serving up key content </li></ul></ul></ul><ul><ul><ul><li>Goal was to validate performance of entire application </li></ul></ul></ul>Load Testing 2.0 Detects 3 rd- party External Errors 2 Major ISP Local ISP Mobile Carrier Internet Content Delivery Networks 3rd Party/ Cloud Services Browsers and devices Users Storage Web Servers App Servers DB Servers Mainframe Load Balancers Mobile Components Network
    19. 19. Load Testing 2.0 Detects 3 rd- party External Errors cont <ul><li>3rd party hardware was insufficient for overall demands on application </li></ul><ul><li>Based on SLAs 3rd party had to improve performance to get paid </li></ul>The load increases throughout the test The transaction rate increases and then falls off as response times climb Errors are seen, all on a 3 rd party object
    20. 20. <ul><ul><ul><li>Company: Regional Online News Source </li></ul></ul></ul><ul><ul><ul><li>Began testing for the 2008 election season </li></ul></ul></ul><ul><ul><ul><li>Goal was to validate overall performance focusing in 2 key regions </li></ul></ul></ul>Load Testing 2.0 Uncovers Regional Response Time Discrepancies under Load - Case Study 3 Major ISP Local ISP Mobile Carrier Internet Content Delivery Networks 3rd Party/ Cloud Services Browsers and devices Users Storage Web Servers App Servers DB Servers Mainframe Load Balancers Mobile Components Network
    21. 21. By traditional Load Test 1.0 standards the test passed Page response times stayed under 4 seconds, outside of one brief blip There was only 1 page error and 11 errors total out of 60000+ transactions Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99% No Performance Issues Detected From Datacenter
    22. 22. Last Mile data showing substantial number of measurements greater than 4 seconds Performance Issues Detected When Using Real User Desktops Key geographies for this customer are New York and Pennsylvania
    23. 23. Load Testing 2.0 Uncovers Poor Response Time for Top Revenue-Generating Regions The response time never met the 4 second average goal. Availability was Less than 99%. Top Revenue-generating geographies were New York and Pennsylvania. By Load Testing 2.0 standards the test failed.
    24. 24. Load Testing 2.0 versus Load Testing 1.0 Capability <ul><li>Load Test 1.0 </li></ul><ul><li>HTTP traffic behind the firewall </li></ul><ul><li>Load Test 2.0 </li></ul><ul><li>Real-Desktops Last Mile </li></ul><ul><li>High Volume Datacenters </li></ul><ul><ul><li>Accuracy of End-User Response Time </li></ul></ul>Incomplete Most Accurate <ul><ul><li>Ability to drive large load volume </li></ul></ul>Limited- requires too much hardware Better - SaaS model <ul><ul><li>Understand CDN Impact </li></ul></ul>No Yes <ul><ul><li>Understand 3 rd Party (ads, feeds, shopping carts, analytics, cloud providers..) </li></ul></ul>No Yes <ul><ul><li>Real-world load to uncover geo discrepancies </li></ul></ul>No Yes <ul><ul><li>Visibility behind the firewall </li></ul></ul>Better Good Pre-production Testing Better Used closer to deployment --to test under realistic conditions Client-Server Testing Yes No
    25. 25. Find Out How Gomez + Cognizant Can Help You <ul><li>Sign Up Now For Your Free Load Test </li></ul><ul><li>http://www.gomez.com/gomez-load-testing-trial/ </li></ul><ul><li>Reality Load XF </li></ul><ul><li>Test from browser to datacenter, with drill down to internal & external application components </li></ul><ul><li>SaaS with no investment or maintenance costs and rapid payback </li></ul><ul><li>Full browser testing across globally distributed geographies </li></ul><ul><li>Cognizant </li></ul><ul><li>Testing Services to complement your Reality Load investment. </li></ul><ul><li>Business and technical side expertise to provide accelerated results. </li></ul><ul><li>Over 500+ dedicated Performance Testing Professionals to deliver services you need from across the globe. </li></ul>

    ×