Your SlideShare is downloading. ×
0
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Virtual Stress-free Testing in the Cloud
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Virtual Stress-free Testing in the Cloud

3,232

Published on

Testing in the cloud

Testing in the cloud

Published in: Technology
1 Comment
3 Likes
Statistics
Notes
  • good and interesting
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
3,232
On Slideshare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Stress test in the CloudCreate AMIs with libraries and dependenciesAdd “computer power” when needed and turn it off to reduce costsLoad test in the CloudGenerate load from one Availability Zone to test on other.Startup a pre-configured TestBox (EC2 instance) in minutesPerformance test in the CloudTest at Global scale - Latency from different parts of the worldStore all instrumentation data on S3, SimpleDB.Web App testingBrowser based Ajax/Selenium testing from different availability zones (US and EU)Create different deployment environments using scripts Usability TestingOn-demand workforceWhat does Testing in the Cloud mean:Automated, Virtual Test Labs that are live only when you need themStress test in the CloudFind the source of latency, Potential Crashes and/or points of Failure. Get Profile information thru logs and instrumentation and measureLoad test in the CloudGenerating load from one Availability zone to other “staging” servers on or off 100 concurrent browsing users that randomly click on links. Then, the load can be increased by 100 users every 10 minutes until the total expected user load of 100,000 users is reached.Performance test in the CloudHow fast the page is loading for a given user in given stateUsability Testing
  • Testing in the CloudInstant Test Labs in MinutesTesting as a ServiceVirtualization/AMIs for Reuse and ReprosWeb Services for AutomationOn-Demand Workforce of Testers (“Elastic QA Staff”)Client and Server Parallelization
  • Pay as you go - Increased utlization
  • The Scheduler plans the plan, spawns the Load Generators, coordinates activities of the Load Generators, and plays traffic cop for all other tests being conducted in LoadStorm. The Load Generators (LG) produce the requests to the target web applications (server). They handle all communications with the server, including capturing returned pages and status codes. The LG builds an extensive record of raw data regarding the test metrics. The Summarizer uses the database of findings from the LG to calculate the metrics and KPIs that LoadStorm makes available through the graphs and tables for analysis.
  • Testing as Background/Daily activity
  • QTRAXNew Music Site for FREE Music Downloads, with 300,000 registered users
  • TESTING CHALLENGEWanted to Test QTRAX.com “staging” sites located in LA-London-HKWanted to Test several different “real world” user scenario’s (Global)Wanted to Test over 3M users hitting web site in a (1) hour periodWanted to Test a “burst” of 200K concurrent users• QTRAX largest previous Load Test was 100 usersQTRAX TEST SETUPSOASTA Team worked with QTRAX to create (20) user scenariosSOASTA Team then provisioned (650) servers located in NJ, UK in 15 minutesQTRAX decided to monitored over a (800) areas of network, system, and applicationsQTRAX TESTStress and Load Test were Performed on the Qtrax Site located in Los Angeles Data CenterIterative Test Process, Lasted over (3) months, with a total of (60) Hours of actual Test TimeRamping up and spiking to 500,000 concurrent users or 2.32 Gbit per sec.Tested over 10M Hits per hour on the QTRAX siteRecored several TB’s of Test analytics and results Data.QTRAX RESULTSAggregated, correlated Test Data displayed LIVE thru real-time DashBoardsProblems were found-fixed-retested, until goal of 500,000 concurrent was hit
  • The instant when average response times increased -- and at what user loadInformation about which application servers were not being balanced properly by the load balancersInformation about which application servers were having connection problemsInformation about which servers (Database, application server, web servers) were hitting CPU limitations at low virtual user levelsInformation about which user scenarios scaled well as the user load increasedMetrics around errors, error rates, and the causes of those errors
  • The instant when average response times increased -- and at what user loadInformation about which application servers were not being balanced properly by the load balancersInformation about which application servers were having connection problemsInformation about which servers (Database, application server, web servers) were hitting CPU limitations at low virtual user levelsInformation about which user scenarios scaled well as the user load increasedMetrics around errors, error rates, and the causes of those errors
  • Because some WeoGeo Market users reported that they were not receiving email notifications, we had a need for User Testing across a variety of email platforms.
  • TestSourcing = CrowdSourcing + Cloud Computing
  • Cloud Computing is changingIn this Era of Tera, Testing for Scale is imperativeTesting as a ServiceOn-DemandCloud TestingVirtualization for ReuseVirtualization for ReprosTest Labs in Minutes
  • Transcript

    • 1. Virtual Stress-Free<br />Testing in the Cloud<br />Jinesh Varia<br />Technology Evangelist<br />Amazon Web Services<br />
    • 2. Cloud Computing Defined<br />“Cloud Computing is large pool of easily usable and accessible virtualized resources that can be dynamically reconfigured to adjust to a variable load and operated on a pay-per-use model”<br />ACM <br />“Cloud Computing is a style of computing where massively scalable IT-related capabilities are provided ‘as a service’ across the Internet to multiple external customers”<br />Gartner<br />
    • 3. Cloud Computing Defined<br />“Cloud Computing is large pool of easily usable and accessible virtualized resources that can be dynamically reconfigured to adjust to a variable load and operated on a pay-per-use model”<br />ACM <br />“Cloud Computing is a style of computing where massively scalable IT-related capabilities are provided ‘as a service’ across the Internet to multiple external customers”<br />Gartner<br />
    • 4. The Cloud<br />
    • 5. Amazon EC2: Compute on Demand<br />
    • 6. Amazon S3<br />The Internet<br />Customer<br />runinstance<br />Amazon EC2<br />
    • 7. Provision servers using command line interfaces<br />runinstances(image-id,<br />minCount,<br />maxCount,<br />instanceType, {m1.small, m1.large, m1.xlarge, c1.medium, and c1.xlarge}<br />keyName,<br />Placement.AvailabilityZone,{us-east-1a, us-east-1b, us-east-1c, eu-west-1a, eu-west-1b}<br /> …)<br />PROMPT> ec2-run-instances ami-235fba4a -k gsg-keypair<br />
    • 8. WebSphere<br />Hibernate<br />Java<br />Linux<br />Amazon<br />Machine<br />Image<br />Amazon Machine Image - AMI<br />PHP<br />Apache<br />Perl<br />Postgress<br />Linux-Ubuntu<br />Ruby<br />Rails<br />MySQL<br />Fedora-6<br />Amazon<br />Machine<br />Image<br />Amazon<br />Machine<br />Image<br />Amazon S3<br />AMI<br />AMI<br />AMI<br />
    • 9. Amazon S3<br />AMI<br />AMI<br />AMI<br />Amazon EC2<br />
    • 10. Bundle Environments with 3 commands<br />PROMPT> ec2-bundle-vol -d /mnt -k <your-private-key> -c <your-x509-cert> -u <userid> -r i386 -p sampleimage<br />PROMPT> ec2-upload-bundle -b <your-s3-bucket>-m /mnt/sampleimage.manifest.xml-a <aws-access-key-id>-s <aws-secret-access-key><br />PROMPT> ec2-register <your-s3-bucket>/sampleimage.manifest.xml<br />
    • 11. Today’s ‘Era of Tera’<br />Uncertainty<br />Uncertainty in Business (“Slashdot/Techcrunched”)<br />Uncertainty in Economy<br />Users and Data flood<br />Millions of Users and PBs of data<br />Latency Matters <br />Performance is now directly related to customer service<br />Global-Scale<br />Spanning Multiple Geographies<br />Diverse Environments <br />Mobile Platforms<br />Middleware on Variety of DB Servers, App Servers<br />
    • 12.
    • 13. Scale: 50 servers to 5000 servers in 3 days<br />Amazon EC2 easily scaled to handle additional trafficPeak of 5000 instances<br />Number of EC2 Instances<br />“Techcrunched”<br />Launch of Facebook modification.<br />Steady state of ~40 instances<br />4/12/2008<br />4/14/2008<br />4/15/2008<br />4/16/2008<br />4/18/2008<br />4/19/2008<br />4/20/2008<br />4/17/2008<br />4/13/2008<br />
    • 14.
    • 15. How to Test in this “Era of Tera”<br />How will you test to see whether your website is spike-proof ?<br />How will you test your website if 50M users are going to hit your website in next 2 hours?<br />How will you test for 750K concurrent users ?<br />How will you test your latency from different parts of the world ?<br />How will you test when you have minimal testing budget in this economy ?<br />How will you test on different environments ?<br />
    • 16. Everything’s Changed, Nothing’s different<br />Stress Testing<br />Load Testing<br />Web Performance Testing<br />Web App (AJAX) testing<br />Usability Testing<br />Unit Testing<br />Regression Tests<br />Integration Tests<br />
    • 17. Everything’s Changed, Nothing’s different<br />
    • 18. Common problems in our world of testing<br />“I cannot reproduce the bug” (environment mismatch)<br />“Its just take too much time to configure the tools”<br />“Site works fine in US, but does not work from EU”<br />“Its too expensive to set up, maintain and update a test lab”<br />“Its takes too much time and efforts to set up a test lab”<br />“Test phase last for only 2 months : Underutilized Test Boxes”<br />
    • 19. On-demand Test Labs<br />Physical Test Labs become out-dated too fast<br />Maintaining Test Labs is pain<br />Configuration<br />Latest patches<br />Test lab when you need, For the duration you need<br />“Need it now”<br />“Need only for 3 month Test cycle”<br />Elastic scale (Grow and Shrink requirements based on pre-defined SLA)<br />Throw-away Test labs (Get a brand new lab every time)<br />No more begging for more servers required<br />
    • 20. Increased Utilization<br />
    • 21. AMIs for Reuse and Repros<br />Virtualization<br />Create test environments dynamically<br />Bundle AMIs<br />With basic dependencies and OS of your choice<br />Share AMIs<br />Share entire environments with dev/prod teams with few clicks<br />
    • 22. Testing as a Service : “Push it to the Cloud”<br />Traditional enterprise solutions are complex<br />Incur High upfront license fees<br />Steep learning curve in Open source tools<br />Testing as a Service<br />Stress, Load, Performance Testing services<br />Pay as you go<br />Meter bandwidth in/out<br />Meter Instance Usage hours<br />Meter CPU<br />Usage-based costing model<br />
    • 23. Concurrent users<br />Requests per minute<br />Average response time<br />Error rate percentage<br />Total page requests per time interval<br />Maximum response time per time interval<br />Pages by error code<br />Pages by response time<br />Analyze Metrics<br /><ul><li>Concurrent users
    • 24. Requests per minute
    • 25. Error rate percentage
    • 26. Total page requests per time interval
    • 27. Maximum response time per time interval
    • 28. Pages by error code
    • 29. Pages by response time
    • 30. Average response time</li></li></ul><li>
    • 31. Make Testing a Background activity<br /><ul><li>Unit testing
    • 32. AJAX Testing
    • 33. Functional Testing
    • 34. Acceptance Testing
    • 35. Usability Testing
    • 36. Cross-Browser Testing
    • 37. Web Services Testing</li></ul>-Integration testing<br />-Lost testing<br />-Stress Testing<br />-Performance Testing<br />-Follow the Sun Tests<br />
    • 38. “Let’s run it again!”<br />Test more and Test Often<br />Iterative process of test-analyze-fix-test<br />Testing is a background activity<br />Real-time results in Dashboards <br />Automation through Web Services<br />Set up test labs on-demand<br />Automated scripts to launch infrastructure you need<br />Cost-effective Automated Testing<br />Infrastructure up only during the build and test time<br />Build run at 2AM on 2 Instances for 2 Hours : Cost of ~$1/Day<br />
    • 39.
    • 40. Test #60650 EC2 instances spawned<br />Served 500K concurrent Users and 10 Million Hits in 1 Hour<br />Test #0<br />GOAL: <br />3M Users/Hits in 1 Hour 200K concurrent Users<br />Local : 100 Concurrent User test<br />Test-Fix-Test Iterative Process Timeline<br />Test #25300 EC2 instances spawned<br />Crash point: 170K concurrent users<br />Served 3M Users in 1 Hour<br />Test #160 EC2 instances spawned<br />Crash point: 500 concurrent users<br />
    • 41. Test #60650 EC2 instances spawned<br />Served 500K concurrent Users and 10 Million Hits in 1 Hour<br />Test #0<br />GOAL: <br />3M Users/Hits in 1 Hour 200K concurrent Users<br />Local : 100 Concurrent User Fail<br />Goal: Exceeded<br />Timeline : 3 Months <br />Actual Testing time: 60 hours<br />Test-Fix-Test Iterative Process Timeline<br />Test #25300 EC2 instances spawned<br />Crash point: 170K concurrent users<br />Served 3M Users in 1 Hour<br />Test #160 EC2 instances spawned<br />Crash point: 500 concurrent users<br />
    • 42.
    • 43. Generations of Testing*<br />* James Whittaker Blog posts on “Future of Testing”<br />
    • 44.
    • 45.
    • 46. User Testing at WeoGeo<br />“We created a 6 question survey focused on our registration and email validation process and offered 2 cents per completed survey. After only 3 batches of 6 surveys (18 total for a whopping 36 cents!) we identified and confirmed problems with AOL, MSN Hotmail, and Yahoo! Mail. Other EC2 users had reported similar problems which quickly led us to a solution”<br />
    • 47. Create actual test scenarios (Selenium)<br />Usability testing<br />Cross browser testing<br />Analyze test results<br />Test Links on the website<br />Create Surveys to rate look and feel, navigation, search features of your website<br />
    • 48. Generations of Testing<br />* James Whittaker Blog posts on “Future of Testing”<br />TestSourcing = CrowdSourcing + CloudComputing<br />
    • 49. A Test problem : New Video Startup<br />Suppose you just launched a new website that embeds videos on other websites...<br />... and you just landed a biz dev deal that will add 200X more load in less than two weeks!<br />You want to know your site can handle 2000 concurrent video streams and the associated AJAX calls in between each clip.<br />What would you do?<br />
    • 50. A Solution<br />The future of software (and testing) requires ultra-tight iteration loops.<br />Cloud computing is poised to be a rocket on the back of agile techniques.<br />Virtualize for consistent state management<br />Crowd-source for quick human intelligence <br />Massiveparallelization using both <br />But first: How quickly can you obtain 2000 Firefox browsers?<br />
    • 51. On-demand load testing service (pay only for what you use)<br />Uses real Firefox browsers (based on Selenium automation technology)<br />Bypasses traditional load testing approach of simulating HTTP traffic<br />Only possible because of cloud computing<br />Runs from EC2 US-East and EU-West Regions<br />
    • 52. 2000 Browsers in 15 Minutes<br />Massive amount of hardware required...<br />... yet available in minutes<br />334 High-CPU Extra Large EC2 Instances<br />2.6TB of RAM<br />2672 CPU Cores<br />Over 550 mbps <br /> throughput<br />
    • 53. Using Amazon EC2<br />15 minutes before a test: prepare hundreds of EC2 instances<br />Each instance runs a Firefox browser and VNC X Server<br />Failures are visually captured as screenshots<br />Data is consolidated in a local EC2 availability zone and uploaded to S3<br />Our costs only occur when we have revenues, so our pricing can be very low<br />
    • 54.
    • 55. Parallel Machines and People<br />BrowserMob is just the tip of the iceberg!<br />Imagine: What if quality could be verified in minutes instead of hours (1000s mins v/s 1 min)<br />The key is parallel execution:<br />Running automated tests in parallel (eg: unit tests, integration tests, browsers, etc)<br />Using the crowd to temporarily increase your QA staff by 100X<br />
    • 56. Cloud Computing is inevitable<br />Testing in the Cloud<br />Instant Test Labs in Minutes<br />Testing as a Service<br />Virtualization/AMI for Reuse and Repros<br />Web Services for Automation<br />On-Demand Workforce of Testers<br />Client and Server Parallelization<br />
    • 57. http://aws.amazon.com<br />/sdb<br />/sqs<br />/ec2<br />Thank You!<br />Jinesh Varia<br />jvaria@amazon.com<br />/s3<br />/support<br />/forums<br />/blog<br />

    ×