Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
•Download as PPTX, PDF•
0 likes•122 views
This document discusses non-functional issues in cloud-based systems. It outlines challenges like performance, security, availability, and legislation compliance. It proposes solutions like testing measures to evaluate these non-functional risks at different stages from selection to production. Key areas for testing include performance, security, manageability, availability, functionality, and compliance with regulations. Continuous testing of functional and non-functional requirements is needed as cloud systems experience continuous changes.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
Similar to Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol (20)
More from Kees Blokland
More from Kees Blokland (8)
Recently uploaded
Recently uploaded (20)
Non-functional Issues in Cloud Based Systems by Kees Blokland and Martin Pol
- 1. Non-Functional Issues in Cloud Based Systems Martin Pol Kees Blokland www.polteq.com
- 2. Agenda • Introduction Cloud computing • Challenges Non-Functional Risks • Solutions Test measures Traditional New “Cloud related” Ways to test Ways to help reduce the risks
- 4. Email Surf Transfer Develop and Test Operate and Manage Store
- 5. Email Surf Transfer Develop and Test Operate and Manage Store storage claim 80% unused redundancy limitations ecological unfriendly
- 6. Email Surf Transfer Develop and Test Operate and Manage Store standard software bandwidth internet technologySOA virtualization
- 7. searching, recording, accounting, paying, writing, reviewing, tracking, calculating, developing, listening, analyzing, transmitting, learning, controlling, purchasing, testing, alarming, changing, updating, deleting, accessing, rejecting, correcting, studying, booking, receiving, tracing, protecting, deciding, managing, teaching, facilitating, identifying, copying, removing, demonstrating, checking, showing, selecting, subscribing, unsubscribing, sharing, mailing, communicating, reading, playing, working, meeting, gambling, shopping, storing, cross checking, retrieving, configuring, sketching, saving, accelerating, enhancing, creating, growing, checking in, checking out, finding out, reaching, denying, talking, designing, making, verifying, measuring Email Surf Transfer Develop and Test Operate and Manage Store
- 9. 9© 2013 Essential characteristics Service models Deployment models USA: National Institute of Standards and Technology http://www.nist.gov https://www.bsi.bund.de/DE/Themen/CloudComputing/Grundlagen/Grundlagen_node.html
- 10. 10© 2013 Essential characteristics • On-demand service • Broad network access • Resource pooling • Rapid elasticity • Measured service USA: National Institute of Standards and Technology http://www.nist.gov https://www.bsi.bund.de/DE/Themen/CloudComputing/Grundlagen/Grundlagen_node.html
- 11. 11© 2013 Service models • No Cloud • Software as a Service • Platform as a Service • Infrastructure as a Service Application Platform Virtualization Hardware CloudOn Premise USA: National Institute of Standards and Technology http://www.nist.gov https://www.bsi.bund.de/DE/Themen/CloudComputing/Grundlagen/Grundlagen_node.html
- 12. 12© 2013 Deployment models • Public • Private • Community • Hybrid USA: National Institute of Standards and Technology http://www.nist.gov https://www.bsi.bund.de/DE/Themen/CloudComputing/Grundlagen/Grundlagen_node.html
- 13. What is “done” in the cloud? Application, Audit Back office, Billing Communications Desktop, Data Email, Environment File transfer Hardware Infrastructure, IT, ID Learning Music Network Platform QA Sourcing, Storage, Software Testing V&V Windows, Workspace AaaS BaaS CaaS DaaS EaaS FaaS HaaS IaaS LaaS MaaS NaaS PaaS QaaS SaaS TaaS VaaS WaaS
- 14. Continuity Privacy Multi platform Legislation Cyber crime Impact organisation Standards 143
- 15. Continuity Privacy Multi platform Legislation Cyber crime Impact organisation StandardsPerformance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks
- 16. Other customers Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks YOUR Operational Profile On Premise In Cloud YOUR Operational Profile PLUS
- 17. Everything over the web Performance Security Availability & Continuity Functionality Manageability Legislation & Regulations Suppliers & Outsourcing Risks On Premise The idea: “it’s safe” In Cloud Home ground for hackers
- 18. Bring Your Own Device Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks In Cloud Endless possibilities On Premise No free choice of device (Bring Your Own Disaster)
- 19. Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks Internet connection lost @ supplier @ user @ other systems ‘Off line” does not work Information is lost
- 20. Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks
- 21. Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks
- 22. Backup and recovery Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks On Premise Taken care of …. In Cloud Who will support me?
- 23. Updates, patches, fixes, … Performance Security Availability & Continuity Functionality Manageability Legislation & Regulations Suppliers & Outsourcing Risks On Premise Planned and controlled In Cloud Do I have a choice?
- 24. Performance Security Availability & Continuity Functionality Maintainability Legislation & regulations Suppliers & Outsourcing Risks
- 25. Where is my data? And is that OK? Performance Security Availability & Continuity Functionality Maintainability Legislation & regulations Suppliers & Outsourcing Risks On Premise In house … In Cloud Somewhere …
- 26. Performance Security Availability & Continuity Functionality Maintainability Legislation & regulations Suppliers & outsourcing Risks
- 27. Performance Security Availability & Continuity Functionality Maintainability Legislation & regulations Suppliers & outsourcing Risks Vendor lock in No agreements Supplier of the supplier of the supplier … Supplier is taken over
- 28. Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks
- 30. Testing! Check Review Monitor Interview Proof of concept TestenProef Intake InterviewProof of concept
- 31. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures TestenProef Intake InterviewProof of concept
- 32. Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures
- 33. Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures
- 34. Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures
- 35. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks From “individual” risks to “individual” test measures Architecture
- 36. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Risks
- 38. Known measures tuned and tweaked New measures developed Use: – Your skills – Heuristics – The Book – Your peers Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures
- 39. Selection Implementation Production Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures
- 40. Selection Implementation Production Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures
- 41. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks
- 42. Selection Criteria Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Completeness Controllable For service For supplier Spec’s and terms References ……
- 43. Proof of Concept Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Dynamic testing More suppliers Time boxing Representative
- 44. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Selection Implementation Production
- 45. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks
- 46. Load Testing Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures YOUR Operational Profile On Premise In Cloud YOUR Operational Profile PLUS ACTUAL MOMENT
- 48. Stress Testing Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures On Premise Yes, you can! In Cloud Definitely NOT!
- 49. Elasticity Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures On Premise Load and stress In Cloud Load and elasticity
- 50. load load test – ‘up’ extend? 200 charged 100 charged no yes path test 99 100 101 boundary values ‘up’ tc 1: use=99, pay 100 tc 2: use=100, pay 100 tc 3: use=101, pay 200 ‘down’ tc1: use=101, pay 200 tc2: use=100, pay 100 tc3: use=99, pay 100 boundary values load test – ‘down’
- 51. load load test – ‘up’ extend? 200 charged 100 charged no yes path test 99 100 101 boundary values ‘up’ tc 1: use=99, pay 100 tc 2: use=100, pay 100 tc 3: use=101, pay 200 ‘down’ tc1: use=101, pay 200 tc2: use=100, pay 100 tc3: use=99, pay 100 boundary values load test – ‘down’ • (Automatic) scaling up or down does not perform as required • At scaling moments functional problems emerge • Insight in use based costs is not sufficient
- 52. • Security at: – Network – Supplier – User • Encryption • Authentication and authorisation • Test logs and audit trails • Security Audits Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures IDaaS Experts Security patch routines
- 53. • Completeness and correctness of specifications and manuals – Supplier – User • Availability of test environments Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Interface specifications Supported platforms Business process specs User manuals
- 54. Manageablity of test environments • Everything in the cloud
- 55. Manageablity of test environments • Link all current environments to the service
- 56. • Role of system architecture • Monitoring and Logging • Guarantees and SLA’s • Test fail-over mechanism • Test online/offline Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures
- 57. Fail-over testing A: disrupted B: active A: active B: inactive A is disrupted B takes over service A: inactive B: active disruptioninAended nochange Aisdisrupted nochange A: active B: disrupted B is disrupted A takes over service Bisdisrupted nochange disruptioninBended nochange A: disrupted B: disrupted
- 58. Online – Offline Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures In Cloud Use case testing Global testing On Premise
- 59. Online – Offline Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures In Cloud Use case testing Global testing On Premise Off line tests focussed on problems: • Work continues, based on out-of-date information, and this information could be changed in the cloud during the offline period • The users are not aware that they are working (partly) online (and are lead to believe differently) • Synchronization conflicts arise because data is changed locally as well as in the cloud
- 60. Online – Offline Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures In Cloud Use case testing Global testing On Premise Off line test cases: • End the connection and check whether the users can see that they are working offline. • Disrupt the connection (for instance, a port or a certain type of IP traffic) and check whether problems arise. • Check whether changes that are made offline find their way to the cloud when online status is regained. • Check whether conflicts between offline and cloud data are handled robustly (which is in fact a functional requirement).
- 61. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks
- 62. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Continuous End-to-End Testing Continuous Change Functionals and Non-Functionals
- 63. Continuity Privacy Multi platform Legislation Cyber crime Impact organisation Standards Check Intake Monitor Interview Proof of concept
- 64. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks Testing starts early: in selection Scope of testing is widened Testing continues in production Testing Cloud Based Systems
- 65. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks Questions? www.polteq.com martin.pol@polteq.com kees.blokland@polteq.com Testing Cloud Based Systems
- 66. Performance Testing Security Testing Manageability Testing Availability & Continuity Testing Functional Testing Migration Testing Testing due to Legislation & Regulations Testing in Production Testing during Selection TestMeasures Performance Security Availability & Continuity Functionality Maintainability Legislation & Regulations Suppliers & Outsourcing Risks Thank you! www.polteq.com martin.pol@polteq.com kees.blokland@polteq.com Testing Cloud Based Systems
Editor's Notes
- the cloud…
- But, where do we come from? Consider a traditional setting, where An organization is housed in a building and everything is contained in that building: I am in the building My computer is in the building The network is in the building The servers are in the building My data is stored in the building The IT maintenance people are in the building. Aha, for email, websurfing and file transfer we leave the building Well, the data is leaving the building, I’m still in it.
- The traditional -all in house- situation though, has a number of issues that become more serious over time Take the datacenter equipment, for instance: on average 80% of it’s capabilities is UNused. Organizations are having more and more problems with the high capital investments in equipment that are required to accommodate peak loads and future growth. And growth there is a lot: such as The constant claim for more Storage And the always growing requirements for Hardware (became obvious to me again when I was surfing the web last night using my ipad1: two years ago top of the bill, but now having more trouble in browsing through typical websites with reasonable speed) Organizations are faced with new requirements too: There is an increasing criticism on power consumption, is your organization green enough? And: the need to support the New World of Work and the Bring You Own Device phenomenon
- Recent technological innovations and developments can cope with most of the issues. For instance: Virtualization: to increase the hardware exploitation and decrease the capital expenses Standard software for commodity services to further cut costs Service Orientation and high bandwidth internet technology that enable the delivery of services at any location using any device
- The new technological possibilities AND the search from organizations to get solutions for there issues give rise to the following trend: IT goes into the cloud!
- Wat is de cloud? Of cloud computing? But what is the Cloud? What characterizes Cloud Computing?
- We use he Cloud Computing definition of the National Institute of standards and technology I will give short explanation of the essential characteristics of cloud computing according to NIST because these give rise to many specific risks that are introduced with cloud computing on-demand services: you can get what you want at the time that you want it without needing help from anyone; example: you can go to a website and have a IT environment in the cloud up and running in minutes Broad network access: a service is available anywhere in the world and with any capacity that is needed Resource pooling: the customers are sharing supplier resources; this provides higher cost efficiency Rapid elasticity: like on demand services: more needed? Via a simple manual action on the internet of even automatically you get more (or less) capacity; example: dropbox Measured services: pay per use: only pay what you use The cloud computing deployment model determines with who the resources are shared. The main focus in this presentation is on Software as a service This is where the cloud supplier provides a full functional service, the customer only needs a device to get access and the cloud service provider is responsible for everything else
- We use he Cloud Computing definition of the National Institute of standards and technology I will give short explanation of the essential characteristics of cloud computing according to NIST because these give rise to many specific risks that are introduced with cloud computing on-demand services: you can get what you want at the time that you want it without needing help from anyone; example: you can go to a website and have a IT environment in the cloud up and running in minutes Broad network access: a service is available anywhere in the world and with any capacity that is needed Resource pooling: the customers are sharing supplier resources; this provides higher cost efficiency Rapid elasticity: like on demand services: more needed? Via a simple manual action on the internet of even automatically you get more (or less) capacity; example: dropbox Measured services: pay per use: only pay what you use The cloud computing deployment model determines with who the resources are shared. The main focus in this presentation is on Software as a service This is where the cloud supplier provides a full functional service, the customer only needs a device to get access and the cloud service provider is responsible for everything else
- Een perfect voorbeeld van een community cloud is bijvoorbeeld EuroCloud, deze non-profit organisatie heeft een specifieke omgeving waar hun eigen community ervaringen, ideeën, rapporten, whitepapers en dergelijke met elkaar uitwisselt. Toegang tot die groepen kan alleen verkregen worden als men lid is.
- The cloud, what does that mean for testing? the answer to this question starts with finding an answer to another question: what are the risks of using cloud computing? There are all kinds of aspects of cloud computing that come with risks We made an inventory and identified 143 risks
- And we grouped them into 7 risk categories
- I give a few examples of cloud related risks, starting with performance risks Traditionally: is the application and it’s environment fast enough considering my expected usage? Usage that is modelled into an operation profile In the cloud there are other users too (remember: one of the essential characteristics is resource pooling) The risk of insufficient performance not only depends on my operational profile but it also depends on the usage of other customers that share the same cloud service resources. If many customers are having a peak load at the same time, this will introduce a high likelihood of performance issues.
- A security example Traditionally: everything is in house, operated and controlled by your own trusted people. In other words: it is safe. In the cloud: the user gets access to the service over the web (broad network access) And there is a risk that other people penetrate the service and get hold of or even change your data.
- Another security example In many organizations people use mandatory, standard desktops, configured with security policies like an automatic screen lock after a time out But: more and more people want to use there own device, because of which organizations decide to use cloud services Using own devices introduces security risks. Show of hands: who has an automatic screen lock on their smartphone or tablet that needs to be unlocked with a pin code? Of course I cannot see you, but based on experience in other audiences I guess that NOT all of you have your device secured with this simple mechanism to prevent others to access your data or the company’s data on your device
- Ja het is anders!
- Ja het is anders!
- Ja het is anders!
- En dat concept – van risico’s naar testmaatregelen – vormt dan ook de basis en de architectuur van Cloutest en het bijbehorende boek.
- En dat concept – van risico’s naar testmaatregelen – vormt dan ook de basis en de architectuur van Cloutest en het bijbehorende boek.
- A well known risk example in the area of legislation deals with this question Where is my data? Before the Cloud this would have been easy to answer: somewhere in house in the data centre. Access to data centres is normally limited to well known maintenance people But what about ‘in the cloud’? Again a request for a virtual show of hands, who of you uses Google Docs? So was a European local government: they started to use it to enable easy sharing of documents to enable government employees to work at home Until someone found out they were seriously violating European rules These rules requires guarantees that all privacy sensitive data stays within the borders of the European Community It was not possible at that time for Google to provide these guarantees, because the data is stored in different data centres in the world part of which were outside the borders of the European Community In the cloud means my data is somewhere, but I need to know: is that OK? What legislation risks apply?
- A well known risk example in the area of legislation deals with this question Where is my data? Before the Cloud this would have been easy to answer: somewhere in house in the data centre. Access to data centres is normally limited to well known maintenance people But what about ‘in the cloud’? Again a request for a virtual show of hands, who of you uses Google Docs? So was a European local government: they started to use it to enable easy sharing of documents to enable government employees to work at home Until someone found out they were seriously violating European rules These rules requires guarantees that all privacy sensitive data stays within the borders of the European Community It was not possible at that time for Google to provide these guarantees, because the data is stored in different data centres in the world part of which were outside the borders of the European Community In the cloud means my data is somewhere, but I need to know: is that OK? What legislation risks apply?
- Ja het is anders!
- Alle overige individuele risico’s – in totaal 143 – worden in Cloutest beschreven op een vergelijkbare manier… That where a number of examples of cloud related risks How to mitigate Cloud specific risks with testing?
- We determined test measures for all 143 identified risks. Certain risks can be mitigated by traditional testing and reviewing Other risks needed additional measures to mitigate for intance auditing, proof of concepts and simluation
- We determined test measures for all 143 identified risks. Certain risks can be mitigated by traditional testing and reviewing Other risks needed additional measures to mitigate for intance auditing, proof of concepts and simluation
- We organized the test measures in the following groups Later on I will give some examples of test measures
- Risico’s en maatregelen. Wat Cloutest UNIEK maakt is de dat er een directe link wordt gelegd tussen INDIVIDUELE risico’s en de bijbehorende INDIVIDUELE maatregelen. --- The next step is to link the risks with test measures.
- The various performance risks link to test measures from four groups Kijk bijvoorbeeld naar performance. Voor de meesten van ons als begrip weinig nieuws. Wat blijkt is dat de performance eisen al nadrukkelijk tijdens de selectie moeten worden meegenomen. Maar ook dat naast de set van maatregelen voor het testen van performance, het testen van beheerbaarheid van belang is: heb ik een representatieve omgeving? En tijdens productie? Meer dan voorheen moet de performance niet alleen gemonitord worden maar het kan zelfs van belang zijn om het regelmatig te testen. De cloud services veranderen continu, veel en vaak en ongecontroleerd en dit kan impact hebben op uw performance. Monitoring is dan niet voldoende maar regressietest op uw performance kan noodzakelijk zijn.
- The number of security risks is overwhelming They link to a large variety of test measures Neem nu bijvoorbeeld Beveiliging. Hoewel daar niet bewust op gestuurd is, blijkt dat het testen van beveiliging in elke testmaatregele terug komt… Van selectiecriteria (welke beveiligingsmaatregelen heeft de leverancier genomen en over welke certificaten beschikt hij), via het testen van autorisatie en autenticatie (terug te vinden in het testen van beveiliging en het testen van functionaliteit), Naar het testen van bv dataschoning: onbedoeld blijven gegevens bestaan die gewist hadden. Tot en met het "live" bewaken want het gebeurt domweg dat er onbedoeld wijzigingen worden doorgevoerd die impact hebben op de beveiligingsmaatregelen.
- For each of the individual 143 risks we identified one or more individual test measures that will help mitigating each risk --- Cloutest Architectuur Risico’s handig gegroepeerd, maatregelen effectief geclusterd INDIVIDUELE risicos gekoppeld aan INDIVIDUELE maatregelen
- Many of the test measures are similar to testing on traditional, I mean non-cloud, software. By tweaking, tuning and combining known test approaches a part of the cloud specific risks can be mitigated. But there are also risks that are not regular in the traditional testing scene that need new test measures. Let’s look through some examples of old and new test measures applied in the cloud context.
- Part of the test measures must be executed during the selection phase, when the cloud service has not been chosen yet Other measures are executed when the service is up and running (in production) But most types of test measures are executed in the implementation phase, when the cloud service is already available but not in production yet. I will give some examples of the numerous test measures that we have described for covering all the cloud related risks we have identified --- De essentie van Cloutest is in feite: Testen begint eerder (tijdens selectie), kijkt breder (anders en met meer diepgang tijdens implementatie), gaat langer door… “Doet de SW het volgens de specs?” verandert in “Gaat dit straks wel werken voor de gebruiker?”. Dus kruip al tijdens de selectie in de huid van de (eind)gebruiker. Voorkom dat je achter de feiten aanloopt en dat je achteraf vaststelt dat je de verkeerde keuze hebt gemaakt. Dus moet je aanwezig zijn bij de selectie want DAAR worden belangrijke beslissingen genomen die grote invloed hebben op de risico’s bij gebruik van de service. Risico’s (de kans dat) veranderen in feiten (issues). Goed en breed testen tijdens implementatie, voor life gang En dan houdt het niet op, want ook nadat de service in productie is genomen blijven er taken voor test. Want als je services gaat afnemen uit de Cloud verdwijnt de vertrouwde rol die beheer tot nu toe heeft gehad.
- Part of the test measures must be executed during the selection phase, when the cloud service has not been chosen yet Other measures are executed when the service is up and running (in production) But most types of test measures are executed in the implementation phase, when the cloud service is already available but not in production yet. I will give some examples of the numerous test measures that we have described for covering all the cloud related risks we have identified --- De essentie van Cloutest is in feite: Testen begint eerder (tijdens selectie), kijkt breder (anders en met meer diepgang tijdens implementatie), gaat langer door… “Doet de SW het volgens de specs?” verandert in “Gaat dit straks wel werken voor de gebruiker?”. Dus kruip al tijdens de selectie in de huid van de (eind)gebruiker. Voorkom dat je achter de feiten aanloopt en dat je achteraf vaststelt dat je de verkeerde keuze hebt gemaakt. Dus moet je aanwezig zijn bij de selectie want DAAR worden belangrijke beslissingen genomen die grote invloed hebben op de risico’s bij gebruik van de service. Risico’s (de kans dat) veranderen in feiten (issues). Goed en breed testen tijdens implementatie, voor life gang En dan houdt het niet op, want ook nadat de service in productie is genomen blijven er taken voor test. Want als je services gaat afnemen uit de Cloud verdwijnt de vertrouwde rol die beheer tot nu toe heeft gehad.
- Starting with testing during selection.
- The earlier risks are mitigated, the better it is Early involvement is key. Nothing new there. One of the first actions that can be taken is making sure that the selection criteria are complete and specific enough. Complete means: including cloud specific risks. For which I provided a number of examples earlier. Specific means: it is clear how to verify that a criterion is met or not Cloud services are not bought but are rented. So an important advantage of cloud services is that one can use it for a limited period of time, for evaluation. The cost is low. This enables a the execution of a proof of concept. That means using the candidate cloud service and try the intended user processes to find out that it can work Or finding out that some blocking issue is found (there is a showstopper for selecting this service). In this way risks are mitigated early, before the final choice for a cloud service is made and the implementation phase is started. Mitigating risks by applying test measures is the typical job of a test manager. It is strongly recommended that someone with the role of test manager participates during the selection of cloud service to help mitigating the risks. ------ Selectie wordt beheerst door de afdeling inkoop en die kijken – naast andere aspecten – met name naar kosten Hebben vaak minder ook voor de aansluiting op het business proces. En vergis u niet: SAP kent ABAP, maar uw service in de cloud kent helemaal niets… Maar wie let op al die cloud risico’s? Het waren er 143, weet u nog wel. En vergeet niet: Na selectie zit je aan de service vast en is een risico geen risico meer maar een feit (issue). Dus moet test moet zich in het selectieproces aan tafel “vechten”! --- Wat speelt er dan allemaal? Zorg dat de selectiecriteria compleet zijn (inclusief de cloudrisico’s) Criteria gericht op de service: dat moet het doen Criteria gericht op de leverancier: ook DAAR zit je straks aan vast Leverancier waaraan je vast zit… door een slimme strategie Gestapelde leveranciers – Hosting! --- Een belangrijk pluspunt is overigens dat de een “life test” van de service kunt doen. Deze zogenaamde Proof of Concept is mogelijk door tijdelijk de service af te nemen.
- The earlier risks are mitigated, the better it is Early involvement is key. Nothing new there. One of the first actions that can be taken is making sure that the selection criteria are complete and specific enough. Complete means: including cloud specific risks. For which I provided a number of examples earlier. Specific means: it is clear how to verify that a criterion is met or not Cloud services are not bought but are rented. So an important advantage of cloud services is that one can use it for a limited period of time, for evaluation. The cost is low. This enables a the execution of a proof of concept. That means using the candidate cloud service and try the intended user processes to find out that it can work Or finding out that some blocking issue is found (there is a showstopper for selecting this service). In this way risks are mitigated early, before the final choice for a cloud service is made and the implementation phase is started. Mitigating risks by applying test measures is the typical job of a test manager. It is strongly recommended that someone with the role of test manager participates during the selection of cloud service to help mitigating the risks. ------ Selectie wordt beheerst door de afdeling inkoop en die kijken – naast andere aspecten – met name naar kosten Hebben vaak minder ook voor de aansluiting op het business proces. En vergis u niet: SAP kent ABAP, maar uw service in de cloud kent helemaal niets… Maar wie let op al die cloud risico’s? Het waren er 143, weet u nog wel. En vergeet niet: Na selectie zit je aan de service vast en is een risico geen risico meer maar een feit (issue). Dus moet test moet zich in het selectieproces aan tafel “vechten”! --- Wat speelt er dan allemaal? Zorg dat de selectiecriteria compleet zijn (inclusief de cloudrisico’s) Criteria gericht op de service: dat moet het doen Criteria gericht op de leverancier: ook DAAR zit je straks aan vast Leverancier waaraan je vast zit… door een slimme strategie Gestapelde leveranciers – Hosting! --- Een belangrijk pluspunt is overigens dat de een “life test” van de service kunt doen. Deze zogenaamde Proof of Concept is mogelijk door tijdelijk de service af te nemen.
- Implementation means: the selection is been done, but the cloud service is not in use yet. It is now time to tests to verify that everything is in order before starting to use the service. Blocking incidents found during the implementation phase need to be solved before the production phase. In an extreme situation a show stopper could cause that selection decision needs to be reconsidered. Oude tekst: Wat is implementatie? Het lijkt op wat we gewend zijn maar in feite is het in productie maar nog niet life… Het inrichten, aansluiten, configureren, in gebruik nemen, etc van de service In de omgeving met bestaande informatiesystemen Met alles erop en eraan En dan met name het testen er van Wordt waargemaakt waar de selectie op was gebaseerd? Proof of the pudding is in the eating Feitelijk constateren IN de productiesituatie Voor performance, beveiliging, functionaliteit, migratie niet heel erg nieuw, maar op diverse onderdelen Bewuster, scherper, completer, preciezer, kritischer, ook minder en anders. Beheerbaarheid, continuiteit waren geen veeleisende onderwerpen voor testen In de Cloud komen die veel nadrukkelijker op de agenda Wet- en regelgeving is helemaal nieuw Kortom: een combinatie van bestaande, aangescherpte en nieuwe aanpakken voor testen
- Implementation means: the selection is been done, but the cloud service is not in use yet. It is now time to tests to verify that everything is in order before starting to use the service. Blocking incidents found during the implementation phase need to be solved before the production phase. In an extreme situation a show stopper could cause that selection decision needs to be reconsidered. Oude tekst: Wat is implementatie? Het lijkt op wat we gewend zijn maar in feite is het in productie maar nog niet life… Het inrichten, aansluiten, configureren, in gebruik nemen, etc van de service In de omgeving met bestaande informatiesystemen Met alles erop en eraan En dan met name het testen er van Wordt waargemaakt waar de selectie op was gebaseerd? Proof of the pudding is in the eating Feitelijk constateren IN de productiesituatie Voor performance, beveiliging, functionaliteit, migratie niet heel erg nieuw, maar op diverse onderdelen Bewuster, scherper, completer, preciezer, kritischer, ook minder en anders. Beheerbaarheid, continuiteit waren geen veeleisende onderwerpen voor testen In de Cloud komen die veel nadrukkelijker op de agenda Wet- en regelgeving is helemaal nieuw Kortom: een combinatie van bestaande, aangescherpte en nieuwe aanpakken voor testen
- Voorbeeld van aangescherpt, verdwenen en nieuw Load testing is more or less a standard these days. Especially when there are many users involved and fast performance is business critical. Such as a web shop. A load test is based on an operational profile: how many people are doing what at which time. This is put in some automated tool which tests the software against the profile. If the test passes: the risk of insufficient performance is covered. What if the software runs in the cloud? The test with the tool can still be executed. The problem now is: the operational profile needs to include also the use of the service by all other customers on the shared cloud resource. In general there is no knowledge about other customers, so what to do? A solution is to execute the load test in real time. That means: test the various loads at the moment they are expected to happen in reality. If there is always a peak load in the morning: execute the test it in the morning assuming that other customers are having their typical every day usage too. This is the closest you can get to reality. There is no guarantee though that the load on the service by other customers stays the same over time, so it necessary to repeat a load test regularly (in production!)
- Voorbeeld van aangescherpt, verdwenen en nieuw
- Traditionally stress testing is following load testing. This is to verify that there sufficient room for growth and to see what happens when the load peaks to extremes? Does the performance degrades or does the system break down completely? Finding stress limits of a cloud service is not an obvious thing to do. Stressing a cloud service for a start cannot be done right away, since it may affect the service to other customers. And the terms and conditions may pose restrictions on going beyond normal usage of a cloud service. So is stress testing not applicable to cloud services? Well yes, in another form: testing elasticity Rapid elasticity is the mechanism that provides the customer of cloud services with the opportunity to accommodate growth. We combine load testing with testing elasticity This can be tested as follows Wat is implementatie? Het lijkt op wat we gewend zijn maar in feite is het in productie maar nog niet life… Het inrichten, aansluiten, configureren, in gebruik nemen, etc van de service In de omgeving met bestaande informatiesystemen Met alles erop en eraan En dan met name het testen er van Wordt waargemaakt waar de selectie op was gebaseerd? Proof of the pudding is in the eating Feitelijk constateren IN de productiesituatie Voor performance, beveiliging, functionaliteit, migratie niet heel erg nieuw, maar op diverse onderdelen Bewuster, scherper, completer, preciezer, kritischer, ook minder en anders. Beheerbaarheid, continuiteit waren geen veeleisende onderwerpen voor testen In de Cloud komen die veel nadrukkelijker op de agenda Wet- en regelgeving is helemaal nieuw Kortom: een combinatie van bestaande, aangescherpte en nieuwe aanpakken voor testen
- Voorbeeld van aangescherpt, verdwenen en nieuw
- Voorbeeld van aangescherpt, verdwenen en nieuw
- Voorbeeld van aangescherpt, verdwenen en nieuw
- Voorbeeld van aangescherpt, verdwenen en nieuw
- Wat is implementatie? Het lijkt op wat we gewend zijn maar in feite is het in productie maar nog niet life… Het inrichten, aansluiten, configureren, in gebruik nemen, etc van de service In de omgeving met bestaande informatiesystemen Met alles erop en eraan En dan met name het testen er van Wordt waargemaakt waar de selectie op was gebaseerd? Proof of the pudding is in the eating Feitelijk constateren IN de productiesituatie Voor performance, beveiliging, functionaliteit, migratie niet heel erg nieuw, maar op diverse onderdelen Bewuster, scherper, completer, preciezer, kritischer, ook minder en anders. Beheerbaarheid, continuiteit waren geen veeleisende onderwerpen voor testen In de Cloud komen die veel nadrukkelijker op de agenda Wet- en regelgeving is helemaal nieuw Kortom: een combinatie van bestaande, aangescherpte en nieuwe aanpakken voor testen
- Testing during implementation is convenient in the sense that the service is not in production yet and no users are bothered by it. But we realized that positive test results from test measures during the selection and implementation phases do not guarantee that the this results stay valid forever Due to all kinds of reasons things continually change like There are all kinds of origins of changes, like The cloud service provider that develops and updates the service Other customers that intensified their use of the service More intensive use by yourself Changes in the user processes Etc.
- Unfortunately It is not very likely that each customer is always aware of or informed about the changes So continuous regression testing is required To detect changes and To check the impact of changes It is best to do this end-to-end to have the best chance to discover all changes and to monitor the full impact on the user processes And it should include functional as well as non-functional tests. ------- Applicaties (Apps) veranderen voortdurend Leveranciers en Services veranderen voortdurend Business wil en kan voortdurend veranderen. Veel meer ongecontroleerde changes vanuit de services… Beheer van de service ligt bij de leverancier dus geen eigen beheerafdeling… Na implementatie kan alles wel goed werken MAAR: er verandert van alles DUS: risico’s komen terug DUS: tester houdt een rol na live-gang --- Zo goed mogelijke schaduw omgeving! Door continue E2E/Ketentest detecteren van veranderingen --- Functioneel; veranderingen in de functionaliteit Performance: ook hier kunnen er veranderingen plaatsvinden Beheerbaarheid: veranderingen in werk instructies
- What did we learn? I showed you that various cloud related risks are introduced when introducing cloud services And we have identified a large variety of test measures to mitigate these risks --- U wordt bij uw manager geroepen en die verteld: “we gaan naar de cloud” en we willen graag dat jij het testen voor je rekening neemt! Allerlei vragen: Welke delen gaan naar de cloud? Wat is de impact? Past mijn huidige testaanpak op deze nieuwe context? Wat moet ik veranderen? Na de eerste opwinding (of teleurstelling), grijp je al snel terug op de vertrouwde principes: laten we eens kijken welke risico’s er nou eigenlijk zijn verbonden aan het gebruiken van cloud services… Misschien verdwijnen bekende risico's wel als sneeuw voor de zon… Maar goed, het kan ook zijn dat nieuwe risico hun plaats innemen en bestaande risico’s op een andere manier alsnog optreden… KLIK Het goede nieuws is dat wij dat ook gedaan hebben… Op basis van onze eigen praktijk en de ervaringen van onze mensen, aan de hand van allerhande informatiebronnen, uiteraard gezond verstand en onze ervaring in testen en IT, hebben wij de meest voorkomende risico's (143) hebben geïnventariseerd.