Testing in the cloud

1. Virtual Stress-Free Testing in the Cloud Jinesh Varia Technology Evangelist Amazon Web Services

2. Cloud Computing Defined “Cloud Computing is large pool of easily usable and accessible virtualized resources that can be dynamically reconfigured to adjust to a variable load and operated on a pay-per-use model” ACM “Cloud Computing is a style of computing where massively scalable IT-related capabilities are provided ‘as a service’ across the Internet to multiple external customers” Gartner

3. Cloud Computing Defined “Cloud Computing is large pool of easily usable and accessible virtualized resources that can be dynamically reconfigured to adjust to a variable load and operated on a pay-per-use model” ACM “Cloud Computing is a style of computing where massively scalable IT-related capabilities are provided ‘as a service’ across the Internet to multiple external customers” Gartner

4. The Cloud

5. Amazon EC2: Compute on Demand

6. Amazon S3 The Internet Customer runinstance Amazon EC2

7. Provision servers using command line interfaces runinstances(image-id, minCount, maxCount, instanceType, {m1.small, m1.large, m1.xlarge, c1.medium, and c1.xlarge} keyName, Placement.AvailabilityZone,{us-east-1a, us-east-1b, us-east-1c, eu-west-1a, eu-west-1b} …) PROMPT> ec2-run-instances ami-235fba4a -k gsg-keypair

8. WebSphere Hibernate Java Linux Amazon Machine Image Amazon Machine Image - AMI PHP Apache Perl Postgress Linux-Ubuntu Ruby Rails MySQL Fedora-6 Amazon Machine Image Amazon Machine Image Amazon S3 AMI AMI AMI

9. Amazon S3 AMI AMI AMI Amazon EC2

10. Bundle Environments with 3 commands PROMPT> ec2-bundle-vol -d /mnt -k <your-private-key> -c <your-x509-cert> -u <userid> -r i386 -p sampleimage PROMPT> ec2-upload-bundle -b <your-s3-bucket>-m /mnt/sampleimage.manifest.xml-a <aws-access-key-id>-s <aws-secret-access-key> PROMPT> ec2-register <your-s3-bucket>/sampleimage.manifest.xml

11. Today’s ‘Era of Tera’ Uncertainty Uncertainty in Business (“Slashdot/Techcrunched”) Uncertainty in Economy Users and Data flood Millions of Users and PBs of data Latency Matters Performance is now directly related to customer service Global-Scale Spanning Multiple Geographies Diverse Environments Mobile Platforms Middleware on Variety of DB Servers, App Servers

13. Scale: 50 servers to 5000 servers in 3 days Amazon EC2 easily scaled to handle additional trafficPeak of 5000 instances Number of EC2 Instances “Techcrunched” Launch of Facebook modification. Steady state of ~40 instances 4/12/2008 4/14/2008 4/15/2008 4/16/2008 4/18/2008 4/19/2008 4/20/2008 4/17/2008 4/13/2008

15. How to Test in this “Era of Tera” How will you test to see whether your website is spike-proof ? How will you test your website if 50M users are going to hit your website in next 2 hours? How will you test for 750K concurrent users ? How will you test your latency from different parts of the world ? How will you test when you have minimal testing budget in this economy ? How will you test on different environments ?

16. Everything’s Changed, Nothing’s different Stress Testing Load Testing Web Performance Testing Web App (AJAX) testing Usability Testing Unit Testing Regression Tests Integration Tests

17. Everything’s Changed, Nothing’s different

18. Common problems in our world of testing “I cannot reproduce the bug” (environment mismatch) “Its just take too much time to configure the tools” “Site works fine in US, but does not work from EU” “Its too expensive to set up, maintain and update a test lab” “Its takes too much time and efforts to set up a test lab” “Test phase last for only 2 months : Underutilized Test Boxes”

19. On-demand Test Labs Physical Test Labs become out-dated too fast Maintaining Test Labs is pain Configuration Latest patches Test lab when you need, For the duration you need “Need it now” “Need only for 3 month Test cycle” Elastic scale (Grow and Shrink requirements based on pre-defined SLA) Throw-away Test labs (Get a brand new lab every time) No more begging for more servers required

20. Increased Utilization

21. AMIs for Reuse and Repros Virtualization Create test environments dynamically Bundle AMIs With basic dependencies and OS of your choice Share AMIs Share entire environments with dev/prod teams with few clicks

22. Testing as a Service : “Push it to the Cloud” Traditional enterprise solutions are complex Incur High upfront license fees Steep learning curve in Open source tools Testing as a Service Stress, Load, Performance Testing services Pay as you go Meter bandwidth in/out Meter Instance Usage hours Meter CPU Usage-based costing model

24. Requests per minute

25. Error rate percentage

26. Total page requests per time interval

27. Maximum response time per time interval

28. Pages by error code

29. Pages by response time

32. AJAX Testing

33. Functional Testing

34. Acceptance Testing

35. Usability Testing

36. Cross-Browser Testing

37. Web Services Testing-Integration testing -Lost testing -Stress Testing -Performance Testing -Follow the Sun Tests

38. “Let’s run it again!” Test more and Test Often Iterative process of test-analyze-fix-test Testing is a background activity Real-time results in Dashboards Automation through Web Services Set up test labs on-demand Automated scripts to launch infrastructure you need Cost-effective Automated Testing Infrastructure up only during the build and test time Build run at 2AM on 2 Instances for 2 Hours : Cost of ~$1/Day

40. Test #60650 EC2 instances spawned Served 500K concurrent Users and 10 Million Hits in 1 Hour Test #0 GOAL: 3M Users/Hits in 1 Hour 200K concurrent Users Local : 100 Concurrent User test Test-Fix-Test Iterative Process Timeline Test #25300 EC2 instances spawned Crash point: 170K concurrent users Served 3M Users in 1 Hour Test #160 EC2 instances spawned Crash point: 500 concurrent users

41. Test #60650 EC2 instances spawned Served 500K concurrent Users and 10 Million Hits in 1 Hour Test #0 GOAL: 3M Users/Hits in 1 Hour 200K concurrent Users Local : 100 Concurrent User Fail Goal: Exceeded Timeline : 3 Months Actual Testing time: 60 hours Test-Fix-Test Iterative Process Timeline Test #25300 EC2 instances spawned Crash point: 170K concurrent users Served 3M Users in 1 Hour Test #160 EC2 instances spawned Crash point: 500 concurrent users

43. Generations of Testing* * James Whittaker Blog posts on “Future of Testing”

46. User Testing at WeoGeo “We created a 6 question survey focused on our registration and email validation process and offered 2 cents per completed survey. After only 3 batches of 6 surveys (18 total for a whopping 36 cents!) we identified and confirmed problems with AOL, MSN Hotmail, and Yahoo! Mail. Other EC2 users had reported similar problems which quickly led us to a solution”

47. Create actual test scenarios (Selenium) Usability testing Cross browser testing Analyze test results Test Links on the website Create Surveys to rate look and feel, navigation, search features of your website

48. Generations of Testing * James Whittaker Blog posts on “Future of Testing” TestSourcing = CrowdSourcing + CloudComputing

49. A Test problem : New Video Startup Suppose you just launched a new website that embeds videos on other websites... ... and you just landed a biz dev deal that will add 200X more load in less than two weeks! You want to know your site can handle 2000 concurrent video streams and the associated AJAX calls in between each clip. What would you do?

50. A Solution The future of software (and testing) requires ultra-tight iteration loops. Cloud computing is poised to be a rocket on the back of agile techniques. Virtualize for consistent state management Crowd-source for quick human intelligence Massiveparallelization using both But first: How quickly can you obtain 2000 Firefox browsers?

51. On-demand load testing service (pay only for what you use) Uses real Firefox browsers (based on Selenium automation technology) Bypasses traditional load testing approach of simulating HTTP traffic Only possible because of cloud computing Runs from EC2 US-East and EU-West Regions

52. 2000 Browsers in 15 Minutes Massive amount of hardware required... ... yet available in minutes 334 High-CPU Extra Large EC2 Instances 2.6TB of RAM 2672 CPU Cores Over 550 mbps throughput

53. Using Amazon EC2 15 minutes before a test: prepare hundreds of EC2 instances Each instance runs a Firefox browser and VNC X Server Failures are visually captured as screenshots Data is consolidated in a local EC2 availability zone and uploaded to S3 Our costs only occur when we have revenues, so our pricing can be very low

55. Parallel Machines and People BrowserMob is just the tip of the iceberg! Imagine: What if quality could be verified in minutes instead of hours (1000s mins v/s 1 min) The key is parallel execution: Running automated tests in parallel (eg: unit tests, integration tests, browsers, etc) Using the crowd to temporarily increase your QA staff by 100X

56. Cloud Computing is inevitable Testing in the Cloud Instant Test Labs in Minutes Testing as a Service Virtualization/AMI for Reuse and Repros Web Services for Automation On-Demand Workforce of Testers Client and Server Parallelization

57. http://aws.amazon.com /sdb /sqs /ec2 Thank You! Jinesh Varia jvaria@amazon.com /s3 /support /forums /blog