Performance Testing in the Cloud
•Download as PPTX, PDF•
7 likes•5,920 views
This Slideshare made up part of Senior Performance Consultant Richard Bishop's presentation at the BCS Special Interest Group in Software Testing (SIGiST) summer conference 2012. Richard introduces cloud computing as a concept in performance testing and describes the pros and cons of the platforms, and an overview of the tools available. Visit Intechnica online - http://www.intechnica.co.uk/ Check out CloudFlex, a tool that allows everyone in your business to manage cloud environments as needed with ease - http://www.cloudflex.co.uk/
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (18)
More from Intechnica
More from Intechnica (9)
Recently uploaded
Recently uploaded (20)
Performance Testing in the Cloud
- 1. Testing in the Cloud An investigation of the cloud platform for performance testing Richard Bishop Senior Performance Consultant - Intechnica
- 2. • • • •
- 3.
- 17. vs.
- 20. “Cloud computing performance varies more than you might think….” “….. the price of consistency likely exceeds what you want to spend” http://www.infoworld.com/d/cloud-computing/face-the-facts-cloud-performance-isnt-always-stable-170066 http://www.twitter.com/DavidLinthicum
- 21. “Cloud computing performance varies more than you might think….” “….. the price of consistency likely exceeds what you want to spend” This is true for testers as well as developers – we need to work around cost uncertainty & variable performance of cloud platforms. Measure the performance and consider provisioning some load from conventional sources too!
- 22. As testers, need to balance realism and repeatability. Realism Repeatability But how will it Did my change really perform? make a difference? Different types of tests require different approaches
- 23. • • • • Intechnica’s tool, based on Facilita Forecast, for flexible load & stress testing from the cloud.
- 24. • • • Intechnica’s tool for provisioning cloud-based environments with simplified interface & startup/shutdown scheduler – Available now
- 25. o o
- 26. • • • • •
- 27. more@intechnica.co.uk http://www.intechnica.co.uk/ @intechnica Contact Web: www.intechnica.co.uk Email: more@intechnica.co.uk Tel: 0845 680 9679 Fax: 0845 2991647 Address: Fourways House, 4th Floor, 57 Hilton Street, Manchester, M1 2EJ
Editor's Notes
- Changed time in IT and working as tester after checking with LinkedIn.Added three years to experience !!At HBoS – presented here 6.5 years ago……. 9th December 2005!As someone involved in testing and quality, I should get my facts right!
- Agenda is split into two partsIntroduction, reasons for conducting our research, choice of platform, methodology etc.Test results, what the results mean, cloud futures (in and out of the test lab), opportunity to discuss
- Definition from: US National Institute of Standards and TechnologyKey points:flexibilityon-demand provisioningNot necessarily about cost reduction, unless used sensibly.Not simplification…..
- Simplification is one misconception about cloud, but there are others. Cloud doesn’t promise to make life easier.Cloud doesn’t reduce system complexity.Won’t reduce costs, unless implemented sensibly and well managed.Key points – backed up by recent articles. e.g. Sunday Telegraph supplementGenerally speaking, NOT cheaper, NOT less complex, NOT faster than what you already have.People who make these assumptions run risk of “falling into the trough in hype cycle”
- Simplification is one misconception about cloud, but there are others. Cloud doesn’t promise to make life easier.Cloud doesn’t reduce system complexity.Won’t reduce costs, unless implemented sensibly and well managed.Key points – backed up by recent articles. e.g. Sunday Telegraph supplementGenerally speaking, NOT cheaper, NOT less complex, NOT faster than what you already have.People who make these assumptions run risk of “falling into the trough in hype cycle”
- Well…..Gartner suggest that all new technologies go through “Hype Cycle” Describes the maturity, adoption and application of new technology.So why bother with cloud?Lots of negative press, easy to dismiss positive articles as hype.Like all technologies, cloud has its limitations.Key is to develop a strategy which exploits benefits and reduce impacts of disadvantages.Everybody is wary of the “trough of disillusionment”.Handouts include hidden slides giving details of hype cycle…..
- Gartner’s Hype Cycle image for July 2011, new one due soon.Cloud Computing, Private Cloud Computing are still at the top of the “hype curve”Past the peak and predicted to be mainstream within 2-5 years.Cloud/Web Platforms (IaaS) is entering the “trough”……. e.g. Sunday Telegraph articleFail to meet inflated expectations, press loses interest, becomes less fashionable.Key to success is avoiding the trough and planning now for the long term benefits that cloud can bring.Latest “Hype cycle” is due out from Gartner in July 2012
- Cloud is a very broad term and it makes sense to sub divide it further.In traditional IT, you manage entire stack. Apps, Data, Middleware, OS, Physical hardware etc…IAAS: responsibility for raw block storage, networking and hardware is outsourced. Crossover point is at OS level.Utility computing model, pay for what you use.PAAS: responsibility for support and upgrades of operating system is also passed over to third party.Similar “pay for what you use” computing model.SAAS: complete application, data, responsibility for management, maintenance etc. are passed to a third party.
- There are many concerns about cloudAs relevant for testers as other cloud users.Unproven tech:reliability, can you trust your core business to the cloud? All eggs in one basket.Service / support: model is immature & learning curve hinders adoption.Lock in: Proprietary platforms, lock in. Some migration tools available.Costs: Should be cheap, but costs are difficult to quantify. Consider bandwidth, uptime, db costs.Security: Biggest concern, but same as hosting your own platforms. Same problem, different perspective.Performance: Key concern, especially against a background of increasing application complexityI won’t address all of these issues, but they all need to be considered before adopting cloud.
- Google search this morning….153m hits on “cloud testing”.Paid ads from “Microsoft, SOASTA, Blazemeter, IBM”What future for cloud testing? – The slide answers this. Test companies increasing cloud offerings.Who’s heard of these tools?Is anybody using them?How do they work “in the cloud”? Either traditional – put load gens in cloud.New wave – Cloud based platforms.Hybrid – bit of both or multiple offerings.
- David Linthicum, writing in Infoworld. @DavidLinthicumTrue for testers as well as application developers.Pros = scalability, low cost, on demandCons = uncertain costs, variable performanceWork around these cons.Manage costs, automate downtime.Measure performance, consider non-cloud tests too, or some load from conventional source.
- David Linthicum, writing in Infoworld. @DavidLinthicumTrue for testers as well as application developers.Pros = scalability, low cost, on demandCons = uncertain costs, variable performanceWork around these cons.Manage costs, automate downtime.Measure performance, consider non-cloud tests too, or some load from conventional source.
- As testers, need to balance realism and repeatability.Realism – need this to answer the “but how will it really perform?” question.Repeatability – need this to answer the “did my change make a difference?” question.Different types of tests require different approaches.e.g. break test vs comparative performance test.Consider time of test, duration of test, running multiple tests (without changing anything).Do more stats work – T-tests, histogram plots, response time distributions.
- “Revalidate” tests in “real” environments or repeat tests ………………….does increase costs.Need to understand your infrastructure and plan ahead. Need to recognise it won’t always work first time.On a learning curve….Used carefully, cloud benefits outweigh the disadvantages (most of the time).e.g. We have clients who do regular small tests for application tuning and infrequent “break” tests where we use large numbers of AWS servers. Costs savings are significant.
- Choosing a test tool ….. This list is valid for cloud or conventional environments.Need to add a weighting to each feature and determine whether worth paying for.Big differences between cloud and conventional test tools that need to be considered.
- Read through slide first….Pros and cons need to be balanced.Need to consider which parts of infrastructure can move to cloud, test tool, test environment, neither or bothNeed to look at test tools capabilities of operating in cloud environment, ease of implementationRequirements for additional monitoring when moving to the cloud, possible requirements for deep dive analysis or monitoring.Give it a try, you might like it Questions