Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How to Test the
Internet of Everything
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
Slide 1
Agenda
• The Testing Challenge
• The Scope of IoE Testing
• Test Strategy for the IoE
• The Future forTesting and Testers
...
Background
• The IoE will dominate our industry for 10 to 20 years
• Right now, it is very confusing
• Standards emerging,...
Introduction
• IoE will be 'everything on the internet' soon
• A strategy for 'Testing Everything'?
– Dimensions, influenc...
Scope of the Internet of Everything
• The scope of IoE is growing and includes:
– Vast number of static devices, mostly se...
The BiggestThing in 30Years?
• Continuous delivery and DevOps bring factory
automation processes into software
• After Wat...
The Testing Journey
(from the 1980s to the present)
• Screen based applications running on dumb terminals –
1980s
• GUI ap...
If CMOs hold the budget…
• There is a shift in power in the mobile space
• If marketers hold the money for building IoE:
–...
Connecting the Unconnected
• 'Internet.org by Facebook' wants to provide cheaper
access to data by (to perhaps 1/100th of ...
The Scope of IoETesting
Functional testing
• Hardware-Level functionality
– Most of this testing will be performed by manufacturers
• Object and S...
Testing @ Scale
• Scalability
– We'll have to do some large-scale load,
performance and stress testing
• Wearables and emb...
Networks
• Moving networks
– We'll have to test security and resilience of our systems against
alien or rogue networks
• N...
Big Data
• Logistics
– A substantial data storage service will have to be
tested (performance, backup/recovery etc.)
• Ana...
The Scope of IoE Testing
• Not all IoE systems will be huge or complex
• But we face a wider range of concerns than
ever b...
Test Strategy for the IoE
Modelling,Testing and Test Data
• Tools to model the IoE are only just emerging
• We'll need to use Big Data test techniqu...
Test Environments;Testing in the
Field
• Test environments will vary widely
– A test lab for a home could be set up in any...
Tool Support
• Automated support will be essential and
probably dominate testing
• Most components tested through APIs, we...
Test Analytics
• DevOps processes and production analytics
monitor system vital signs like sensors
– These analytics are e...
PerformanceTesting andTest Data
• Local network protocols
– IP based (XMPP, MQTT); few tools, but cloud services emerging
...
The Future for Testing
and Testers
We need to change the way we think
about testing
New Model for Testing
"We explore sources of knowledge to build test models
that inform our testing"
• Forget logistics; f...
New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
Slide 24
Manual v AutomatedTesting?
• Automation will not make testing easy; it will
make testing possible
• More and more, DevOps ...
Shift-Left
• Every company seems to be 'shifting left'
– Developers responsible for testing – no testers
– Testers embedde...
Close
• Testing the IoE is the next level of testing in terms of
complexity and scale
• The tools we need don't yet exist
...
How to Test the
Internet of Everything
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
Slide 28
Upcoming SlideShare
Loading in …5
×

How to Test the Internet of Everything

904 views

Published on

Доклад Пола Геррарда на конференции SQA Days-17,
29-30 мая 2015 г., Минск
www.sqadays.com

Published in: Education
  • How long does it take for VigRX Plus to start working? ■■■ http://t.cn/Ai88iYkP
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

How to Test the Internet of Everything

  1. 1. How to Test the Internet of Everything @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com Slide 1
  2. 2. Agenda • The Testing Challenge • The Scope of IoE Testing • Test Strategy for the IoE • The Future forTesting and Testers • Close Slide 2Intelligent Definition and Assurance
  3. 3. Background • The IoE will dominate our industry for 10 to 20 years • Right now, it is very confusing • Standards emerging, but current applications are bleeding edge, speculative or exploratory • Security and privacy risks dominate today • But functional and integration complexity challenges must also be overcome • There are also broader societal risks • Government, companies and individuals need to pay attention. Slide 3Intelligent Definition and Assurance
  4. 4. Introduction • IoE will be 'everything on the internet' soon • A strategy for 'Testing Everything'? – Dimensions, influences, and opportunities – Can only paint a picture – Suggest what testers need to think about • The challenge we face is quite different from what we have tested before. Slide 4Intelligent Definition and Assurance
  5. 5. Scope of the Internet of Everything • The scope of IoE is growing and includes: – Vast number of static devices, mostly sensors – Mobile devices: cars, buses, trains, planes, ships, satellites... • It will include: – Objects in homes, cities, work, hospitals, social venues – Wearable, human-embedded devices, phones and tablets • IoE integrates with 100s of services, payment and legacy systems • Eventually, IoE will be called the Internet • The test strategy for the IoE is the test strategy for everything. Slide 5Intelligent Definition and Assurance
  6. 6. The BiggestThing in 30Years? • Continuous delivery and DevOps bring factory automation processes into software • After Waterfall and Agile, CD and DevOps become 'the third way' • The IoE will affect everyone on the planet. • From 50-700bn devices over the next 5 to 20 years; no one knows • We are on a journey that will increase the scale of the internet by one hundred times • How will we test it? Slide 6Intelligent Definition and Assurance
  7. 7. The Testing Journey (from the 1980s to the present) • Screen based applications running on dumb terminals – 1980s • GUI applications and Client/Server – 1990s • The emergence of the Internet - 1995 • 'Mobile Revolution' - today • Internet of Everything - emerging – Smart phones, buildings, cars, ships, planes and cities – BigData, pervasive, wearable computing – Massive scale • IoE brings new levels of complexity and scale • Non-functional risks are well-understood – what's new? • Functional testing and simulation at very large scale. Slide 7Intelligent Definition and Assurance
  8. 8. If CMOs hold the budget… • There is a shift in power in the mobile space • If marketers hold the money for building IoE: – IT will have to meet short and long term business goals – Development must deliver, experiment and learn • Market testing and experimentation to drive rapid evolution • DevOps, Continuous Delivery, high levels of automation and simulation • Test analytics blended with production analytics. Slide 8Intelligent Definition and Assurance
  9. 9. Connecting the Unconnected • 'Internet.org by Facebook' wants to provide cheaper access to data by (to perhaps 1/100th of the current price) • Developing economies will probably lead the way in implementing the IoE with mobile internet • But what networking technology/protocols will be used for local connectivity? • "No technology dominates yet.We are waiting for technologies to improve in range and performance.The standard technology for connecting the IoT hasn't been invented yet." • We are still at the start of the IoE journey Slide 9Intelligent Definition and Assurance
  10. 10. The Scope of IoETesting
  11. 11. Functional testing • Hardware-Level functionality – Most of this testing will be performed by manufacturers • Object and Server level functionality – We'll have to test simple web-apps to systems with many complex sub-systems • Integration and Interaction at all levels – Companies, testers will have to use judgement on deciding what to test and how much • Collaboration confusion – Use expectations and user testing of collaborating systems will be difficult. Slide 11Intelligent Definition and Assurance
  12. 12. Testing @ Scale • Scalability – We'll have to do some large-scale load, performance and stress testing • Wearables and embedded – We will need to simulate and test thousands of devices carried by or embedded in people • Everything connected – We'll need to do location-based tests at very large scale. Slide 12Intelligent Definition and Assurance
  13. 13. Networks • Moving networks – We'll have to test security and resilience of our systems against alien or rogue networks • Network security risks at multiple levels – Networks will need to be hardened and tested at all levels, not just 'perimeter' • Device registration, provisioning, failure and security – Power-down, power-up and automated authentication, configuration and registration processes will need to be tested • Mobile objects – Power, interference, network strength, roaming and jamming issues will all need testing. Slide 13Intelligent Definition and Assurance
  14. 14. Big Data • Logistics – A substantial data storage service will have to be tested (performance, backup/recovery etc.) • Analysis and visualisation – These outputs and processes will need to be tested. • Personal and corporate privacy – Testing will need to embrace the ethical use of data and it's protection from misuse. Slide 14Intelligent Definition and Assurance
  15. 15. The Scope of IoE Testing • Not all IoE systems will be huge or complex • But we face a wider range of concerns than ever before: – Most bring a new technology and risk profile – The range of testing issues we need to consider – The scale of the testing required has increased • The approaches we will need to tackle the risks of failure will have to change. Slide 15Intelligent Definition and Assurance
  16. 16. Test Strategy for the IoE
  17. 17. Modelling,Testing and Test Data • Tools to model the IoE are only just emerging • We'll need to use Big Data test techniques • 'Pattern-based test design' – Generate tests from patterns to simulate scenarios – Oracles will derive from conventions or common- sense – Personas as patterns of user or system behaviour – Goal: to feed automated tools large numbers of scenarios • Developers will need to make systems testable through HighVolume AutomatedTesting. Slide 17Intelligent Definition and Assurance
  18. 18. Test Environments;Testing in the Field • Test environments will vary widely – A test lab for a home could be set up in any office – An urban environmental management system will be harder to create • Scale of field testing will vary widely, but will become a common (and possibly mandatory) testing stage • Barcelona has set up an urban-scale testing facility – Barcelona Urban Lab is positioned as a field test environment to prove the viability of innovative systems. • Expect other cities around the world to offer similar services. Slide 18Intelligent Definition and Assurance
  19. 19. Tool Support • Automated support will be essential and probably dominate testing • Most components tested through APIs, web or other services • New protocols emerging; custom drivers may be required • The challenge will be, "how do we design the thousands or millions of tests to feed the tools?" Slide 19Intelligent Definition and Assurance
  20. 20. Test Analytics • DevOps processes and production analytics monitor system vital signs like sensors – These analytics are essentially the same • Production systems are platforms for experimentation – The code that serves marketers can also serve developers and testers • Analytics code is your sensor network • DevOps processes are 'things', too. Slide 20Intelligent Definition and Assurance
  21. 21. PerformanceTesting andTest Data • Local network protocols – IP based (XMPP, MQTT); few tools, but cloud services emerging – Low Power/Lossy Network (ZigBee, 6LowPAN, DASH7/RFID, Bluetooth and NFC); only need simple drivers to 'chirp' • Coherent data may be a problem, e.g. location data – Cars must follow the road network – People can only walk through certain places in a shopping mall – So you will have to capture the GPS data using a tool perhaps • To create unique paths, you could edit together segments of intersecting paths. Slide 21Intelligent Definition and Assurance
  22. 22. The Future for Testing and Testers We need to change the way we think about testing
  23. 23. New Model for Testing "We explore sources of knowledge to build test models that inform our testing" • Forget logistics; focus on 10 thinking skills • Not a process with sequence, inputs/outputs, entry/exit criteria • What is the model useful for? – Separates logistics from the core testing skills – Clarifies two modes: exploration and testing – Supports all testing, regardless of methodology, manual v automated, structured or agile. Intelligent Definition and Assurance Slide 23
  24. 24. New Model Testing 29 page paper: http://dev.sp.qa/download/newModel Slide 24
  25. 25. Manual v AutomatedTesting? • Automation will not make testing easy; it will make testing possible • More and more, DevOps processes are a source of data for analysis just like production • You might not need to be a programmer but you may need to: – Write your own utilities, download, configure and adapt open source tools or create automated tests – Have informed conversations with developers • Should testers learn how to write code?Yes! Slide 25Intelligent Definition and Assurance
  26. 26. Shift-Left • Every company seems to be 'shifting left' – Developers responsible for testing – no testers – Testers embedded in development teams – Testers morphing into business or systems analysts • Testing thinking is being moved left and earlier in the process • The New Model identifies these thinking activities • Testers need to learn how to 'let go' of testing, so view this as an opportunity, rather than a threat. Slide 26Intelligent Definition and Assurance
  27. 27. Close • Testing the IoE is the next level of testing in terms of complexity and scale • The tools we need don't yet exist – Test models, test data generators and automatic oracles – Simulation/HighVolume Test Automation tools • Testing in the lab and experimentation in production will become one discipline – test analytics • Some new test approaches will be required • The New Model is an attempt to trigger new thinking in this area. Slide 27Intelligent Definition and Assurance
  28. 28. How to Test the Internet of Everything @paul_gerrard Paul Gerrard paul@gerrardconsulting.com gerrardconsulting.com Slide 28

×