BCS SIGiST - How Fast is the Cloud?
•Download as PPTX, PDF•
0 likes•633 views
An investigation of cloud platform performance. Discussing performance tests used to measure cloud platform performance as well as potential pitfalls when performance testing and hosting test systems in cloud environments.
Recommended
Recommended
More Related Content
Viewers also liked
Similar to BCS SIGiST - How Fast is the Cloud?
Similar to BCS SIGiST - How Fast is the Cloud? (20)
Recently uploaded
Recently uploaded (20)
BCS SIGiST - How Fast is the Cloud?
- 1. How Fast is the Cloud? An investigation of cloud platform performance Richard Bishop Senior Performance Consultant - Intechnica
- 2. • • • •
- 3. • • •
- 5. ≠ simplification
- 9. vs.
- 10.
- 11. • • • •
- 12. • • • • •
- 13. • • • • •
- 14. • • • • • 30 mins •
- 15. 2.00 1.80 1.60 1.40 1.20 1.00 0.80 0.60 0.40 • 0.20 • 0.00 00:00:00 00:03:40 00:05:30 00:07:20 00:14:40 00:16:30 00:18:20 00:22:00 00:23:50 00:27:30 00:29:20 00:33:00 00:34:50 00:36:40 00:01:50 00:09:10 00:11:00 00:12:50 00:20:10 00:25:40 00:31:10 00:38:30 • Disk IO (Performance Test (EC2), SQL, Node #1) Disk IO (Performance Test (Virtual), SQL, Node #1) Disk IO (Performance Test (Physical), SQL, Node #1) • • •
- 16. Results
- 17. vs. Average Response time (s) 2.5 2 1.5 1 0.5 0 AWS (out of Azure (ported Physical (out VMWare (out box) to optimised of box) of box) IaaS 1 PaaSdb) Tin IaaS 2
- 18. • • • • •
- 19. • • •
- 20. • • •
- 21. vs.
- 22. Test 1 Test 2
- 23. Cloud Testing
- 25. “Cloud computing performance varies more than you might think….” “….. the price of consistency likely exceeds what you want to spend” http://www.infoworld.com/d/cloud-computing/face-the-facts-cloud-performance- isnt-always-stable-170066
- 26. Realism Repeatability But how will it Did my change really perform? make a difference?
- 27. • • • • • • •
- 28. • •
- 29. • • • • •
- 30. careers@intechnica.co.uk http://www.intechnica.co.uk/Associates Questions Web: www.intechnica.co.uk Email: more@intechnica.co.uk Tel: 0845 680 9679 Fax: 0845 2991647 Address: Fourways House, 4th Floor, 57 Hilton Street, Manchester, M1 2EJ
Editor's Notes
- Changed time in IT and working as tester after checking with LinkedIn.Added three years to experience !!At HBoS – presented here 6.5 years ago……. 9th December 2005!As someone involved in testing and quality, I should get my facts right!
- Agenda is split into two partsIntroduction, reasons for conducting our research, choice of platform, methodology etc.Test results, what the results mean, cloud futures (in and out of the test lab), opportunity to discuss
- Definition from: US National Institute of Standards and TechnologyKey points:flexibilityon-demand provisioningNot necessarily about cost reduction, unless used sensibly.Not simplification…..
- Simplification is one misconception about cloud, but there are others. Cloud doesn’t promise to make life easier.Cloud doesn’t reduce system complexity.Won’t reduce costs, unless implemented sensibly and well managed.Key points – backed up by recent articles. e.g. Sunday Telegraph supplementGenerally speaking, NOT cheaper, NOT less complex, NOT faster than what you already have.People who make these assumptions run risk of “falling into the trough in hype cycle”
- Well…..Gartner suggest that all new technologies go through “Hype Cycle” Describes the maturity, adoption and application of new technology.So why bother with cloud?Lots of negative press, easy to dismiss positive articles as hype.Like all technologies, cloud has its limitations.Key is to develop a strategy which exploits benefits and reduce impacts of disadvantages.Everybody is wary of the “trough of disillusionment”.Handouts include hidden slides giving details of hype cycle…..
- This is Gartner’s Hype Cycle image for 2010.Cloud Computing, Private Cloud Computing are towards the top of the “hype curve”.Early adopters starting to implement cloud. Some negative publicity, prices start to fallBoth are predicted to be mainstream within 2-5 years.
- Gartner’s Hype Cycle image for July 2011, new one due soon.Cloud Computing, Private Cloud Computing are still at the top of the “hype curve”Past the peak and predicted to be mainstream within 2-5 years.Cloud/Web Platforms (IaaS) is entering the “trough”……. e.g. Sunday Telegraph articleFail to meet inflated expectations, press loses interest, becomes less fashionable.Key to success is avoiding the trough and planning now for the long term benefits that cloud can bring.Latest “Hype cycle” is due out from Gartner in July 2012
- Cloud is a very broad term and it makes sense to sub divide it further.In traditional IT, you manage entire stack. Apps, Data, Middleware, OS, Physical hardware etc…IAAS: responsibility for raw block storage, networking and hardware is outsourced. Crossover point is at OS level.Utility computing model, pay for what you use.PAAS: responsibility for support and upgrades of operating system is also passed over to third party.Similar “pay for what you use” computing model.SAAS: complete application, data, responsibility for management, maintenance etc. are passed to a third party.
- There are many concerns about cloudAs relevant for testers as other cloud users.Unproven tech:reliability, can you trust your core business to the cloud? All eggs in one basket.Service / support: model is immature & learning curve hinders adoption.Lock in: Proprietary platforms, lock in. Some migration tools available.Costs: Should be cheap, but costs are difficult to quantify. Consider bandwidth, uptime, db costs.Security: Biggest concern, but same as hosting your own platforms. Same problem, different perspective.Performance: Key concern, especially against a background of increasing application complexityI won’t address all of these issues, but they all need to be considered before adopting cloud.
- Why conduct our research?High profile failures widely reported - dent people’s confidence in cloud - successes go unnoticed.These sites have one thing in common... both hosted on Amazon EC2.Police UK – failed. Why? GZIP compressionoff, caching off (no cache headers returned by landing page), multiple CSS/JS files. Developers relied on EC2s inherent scalability, disregarded best-practice. - 18m hits/hr = 4k/sec, Decent caches, cache warming/pre-population would certainly have helped.Wikileaks – “Iraq war logs” published. Despite hacking group called “Operation Payback” DDoS (10Gbps traffic directed at Amazon) when leaks published. Throughout the attack, site remained available (until Amazon turned it off for breaching terms of contract). Suggests platform isn’t to blame! - Worthy of investigation - How do they perform in “real world” scenarios?
- IaaS (AWS and VM)PaaS (Azure)Physical hardwareNeeded app suitable for eachNeeded familiar test tool, used physical server
- After identifying target platforms, looked for an application to test.NopCommerce = two-tier ecommerce application.NET / SQL architecturePre-populated with sample dataImportantly, already ported to Azure! ….. Up and running on 4 platforms quicklyInstalled same application and test data on 4 platforms – attempted to choose similar sized/cost options
- Demo < 100 products in the database by default. Fine for functional tests, not for performance. We increased this to 7,500 products, and associated multiple images with each product. Text for product names and product descriptions from a copy of “War and Peace” downloaded from the Project Gutenberg archive. Products were created in bulk using an SQL script which created products, product descriptions, assigned them to random product categories and associated pictures with the products.
- Developed scripts using Forecast.Scripts and scenarioMixture of business processes simulating real user action.Retail site - 85% of users browsing/searching Remainder placing orders.Load profile“Ramp up” to peak load over a 5-minute period, maintain load for 30 minutes and then ramp-down. Load equivalent to 46 users pausing for between 5-10 seconds was simulated.User load equivalent to 20,000 pages per hour
- During our tests we used..Standard Windows PERFMON monitoring for each platformIn-house Intechnica tools to compare / write reports (MetricsWizard and KPIManager)Tools produce reports to compare response times and infrastructure performanceNot break tests, just baseline comparison atconstant, consistent load for each platformStandard tests.Same data – (text from war and peace, random images)Same volume of users for each test, just changed target IP in scriptsSame workload per user (pacing between user actions)
- Have given a flavour of our tests and platform choice, now time for the results.The big question is how do they perform?Before I show results, who thinks cloud is faster? – show of hands.Answer to, “how fast is the cloud compared to physical hardware?” the answer is.Faster……. and …….. slower!
- This study highlights the benefits of developing applications specifically for a platform.The “port” of nopCommerce to Azure specifically exploited the performance benefits of the highly optimised PaaS (SQL Azure back-end database) and produced results that are “Faster than tin”. Simply placing the application on a normal instance of SQL (IaaS, PaaS or physical) doesn’t give the same performance as tuned back end database e.g. Azure or Amazon RDS, because theyaren’t as highly optimised for database performance.
- Analysis - First step -look at the initial TrafficSpike report.Check transaction count, check error rate, ensure tests comparable.Need to ensure like-for-like comparison.Can’t compare tests with very different error rates – skews results.Once we know tests are valid we use our in-house analysis tools to perform more detailed analysis.
- Metrics Wizard.Compares response times, transaction count etc. between tests to identify any changes.Usually for before/after comparison, but used to compare results from different platforms.Chart shows comparison of physical hardware with VMware platform.Red/Green used to show improvements or deteriorations in performance between tests.In this case although we see red and green it’s apparent that difference between platforms was negligible.Physical hardware ave response time = 0.712sVMWare ave response time = 0.690sThis doesn’t tell the whole story, would be good to perform “break tests” to establish headroom.For this study we looked at CPU utilisation to see if we could identify differences between platforms.
- KPI Manager.This tool takes Windows PERFMON logs and compares observed performance with Microsoft best practicee.g. database server performance. IaaS server exceeded 50% CPU utilisation during the test. 50.10% observed.Key observation …. We can see that the database is inefficient from the counters below.Calculate full table scans divided by index searches.We can see lots of full table scans which are inefficient, compared to index searches.This may be an inherent feature of nopCommerce, or may be due to the “artificial data” that we used.Only three columns because Azure stats aren’t available.Azure is presented as PAAS, rather than IAAS.
- We knew our database was struggling.PERFMONshowed the application doing a large number of page lookups/sec. Could reduce by moving images out of the database onto the file system and better caching on the webserver. Too many table scans, analogous to looking through the entire phonebook to find a person’s name line by line, rather than using the index to start your search closer to the name/number that you’re looking for.Latency between EC2 back end to web layer is unknown and may vary depending on location of webserver and dbserver. Hear of people “spinning up” multiple servers and pairing those with “closer” IP addresses in hope of improving performance. This can be done, but Amazon’s advice is to use their optimised databaseSignificantly less latency in physical hardware / vmwareWhat we do know…..For this application, with this test data Azure is best performing.With different applications / test data / use cases, others may perform better.
- We weren’t satisfied with original IaaS “out of box performance” and felt that it misrepresented the platform.The relatively poor response times shown by IaaS initiallydemonstrated that simply moving an application to the cloud isn’t likely to result in acceptable performance. This underlines the fact that applications need to be developed specifically for the environment in which they are to be used, or optimised for the target platform, e.g. the Azure nopCommerce implementation.Further analysis of the application identified an inefficient stored procedures in the nopCommerce SQLdatabase which was used repeatedly. The effects of optimising the single most frequently used stored procedure resulted in a 57% improvement to average response times. (2.011s to 1.276s)We spent a few hours spent on this, it is likely that further improvements would also be possible by optimising other code.
- Previous slides covered platform choice for apps.As well as being aware that test platforms etc. are moving to the cloud….How does the cloud affect us as testers?
- Google search this morning….153m hits on “cloud testing”.Paid ads from “Microsoft, SOASTA, Blazemeter, IBM”What future for cloud testing? – The slide answers this. Test companies increasing cloud offerings.Who’s heard of these tools?Is anybody using them?How do they work “in the cloud”? Either traditional – put load gens in cloud.New wave – Cloud based platforms.Hybrid – bit of both or multiple offerings.
- David Linthicum, writing in Infoworld. @DavidLinthicumTrue for testers as well as application developers.Pros = scalability, low cost, on demandCons = uncertain costs, variable performanceWork around these cons.Manage costs, automate downtime.Measure performance, consider non-cloud tests too, or some load from conventional source.
- As testers, need to balance realism and repeatability.Realism – need this to answer the “but how will it really perform?” question.Repeatability – need this to answer the “did my change make a difference?” question.Different types of tests require different approaches.e.g. break test vs comparative performance test.Consider time of test, duration of test, running multiple tests (without changing anything).Do more stats work – T-tests, histogram plots, response time distributions.
- “Revalidate” tests in “real” environments or repeat tests ………………….does increase costs.Need to understand your infrastructure and plan ahead. Need to recognise it won’t always work first time.On a learning curve….Used carefully, cloud benefits outweigh the disadvantages (most of the time).e.g. We have clients who do regular small tests for application tuning and infrequent “break” tests where we use large numbers of AWS servers. Costs savings are significant.
- Choosing a test tool ….. This list is valid for cloud or conventional environments.Need to add a weighting to each feature and determine whether worth paying for.Big differences between cloud and conventional test tools that need to be considered.
- Read through slide first….Pros and cons need to be balanced.Need to consider which parts of infrastructure can move to cloud, test tool, test environment, neither or bothNeed to look at test tools capabilities of operating in cloud environment, ease of implementationRequirements for additional monitoring when moving to the cloud, possible requirements for deep dive analysis or monitoring.Give it a try, you might like it Questions