News InternationalHow to look for a service provider with an agile, resilientinfrastructure for your online assets?Paul Byrom, Head Of Quality
Utility ComputingWho is this guyPaul Byrom - Head Of Quality - News International- In charge of Quality Assurance across digital projects at News International.What do we testAll major titles across web and mobile apps such as:The TimesThe SunThe Sunday TimesWhat is Utility Computing- Think about it like how you pay for your electricity based on the consumption you use. Or aswikipedia says computation, storage and services, as a metered service.
Problem & ChallengeProblem Statement @ NI- Our load testing was no longer valid as we could not afford to runlive like load tests.The Challenge- To reduce risk of failure by performing representative load tests atminimal cost, and minimal impact.- To fail fast and provide useful feedback that allows the team toeasily investigate and fix potential bottlenecks caused by load.
Where we were- Very expensive Load Runner. HP wanted 121k to allow us to run 1000 VUs.- Limited virtual users due to cost- A ton of tin (environments) on-site- Performance testing occurred too late to make positive change- Load Runner experts are expensive.- Existing tool cloud model not very flexible or cost effective.- Couldnt run projects in parallel on perf env causing a queue (shared VUs)- Difficult to integrate into a CI system (cmd line support, results in HP format)DEV SI UAT STAGE PERF PROD
Where we are now- Decommissioned old perf env- Jmeter is a free open source tool which is more adaptable to our needs- Unlimited virtual users allowing us to perform live like load tests- Presentation layer enables quick comparison of data for rapid resolution- Automated script creates, runs test, and destroys test hardware for each run in the cloud.- Failing faster as perf testing now occurs on CI, SI, UAT, and STAGECIDEV SI UAT STAGE PRODAWSLoad
The Proof- Using 30 EC2 instances last year, DreamTeam tested with 10000virtual users against the website and achieved 8000 hits per second(600 page views/second).- That would have previously costed £1.2mil to run the test on LoadRunner.- It cost us $17.00 using Jmeter and AWS.- Currently Cost is about £70/mon right now across all products forAWS load injector costs.
Whats Next- Trending as data is now all in one place- Standardised reporting across all products- Continuous automated load testing