'Test Automation in the Cloud' with Jonathon Wright

Today’s webinar is presented by Jonathan Wright and he will
discuss Test Automation in the Cloud.
Welcome to the EuroSTAR July
Webinar Series
www.eurostarconferences.com
This webinar is due to start at 2pm. Make sure you stick around at the end for the
Q&A session and continue the conversation with the speaker on Twitter after the
show!
@esconfs
#esconfs

Test Automation in the Cloud
Jonathon Wright, Founder,
Automated Development Services, UK
Jonathon Wright has over 13 years of commercial automation experience with a number of
international organisations. Currently working on providing ‘Test Automation as a Service’ to
a number of global clients. Jonathon also contributed to the recent "Experiences of Test
Automation: Case Studies of Software Test Automation (2012)" and a number of upcoming
books on test automation in the cloud. He is an active blogger on "Test Automation as a
Service" (TaaaS.net) as well as presenting at various international testing conferences
(StarWEST(California)/Fusion(Sydney)/ANZTB(Melbourne)/BCS-SIGIST(London)).
@esconfs
#esconfs

Q&A
If you have a question, the control panel on the right has a Questions
box. All questions will be answered at the end of the presentation.
The GoToWebinar attendee interface is made up of two parts. The
Viewer Window shows the presenter’s screen. The Control Panel is
where attendees can interact and ask questions.
The webinar will be recorded and you will be notified
via email when it is available.
@esconfs
#esconfs

How can I get the slides?
Slides and all materials will be posted on the EuroSTAR website as well as emailed to you.
@esconfs
#esconfs

Join the conversation on Twitter
#esconfs
@esconfs
@Jonathon_Wright
Continue the conversation with the speaker on Twitter after the show!
@esconfs
#esconfs

Jonathon Wright (jlwright@iee.org)
www.linkedin.com/in/automation
EuroSTAR (Tuesday@ 2:00 PM, 23rd July 2013)
Test Automation in the Cloud
@jonathon_wright
#esconfs

@jonathon_wright
#esconfs
The current economic climate is making companies
review their approach to IT even more closely
“We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant
scalability, flexibility, and availability for testing-on-demand. - But how well does this work in practice?”
• Why (Business Benefits & Value)
• What (Cloud Test Automation)
• When (Timeline)
• Agile Portfolio Management (Scope)
• Transition (AMMi)
• Dependencies (Cloud Type)
• Design (First Day Testing)
• Design (Manual’mation)
• Implementation (Agnostic)
• Cloud Platform (IaaS)
• High Volume Automation (HVAT)
• Continuous Build & Delivery
• Actionable Insight (Funnel Virtualisation)
• Dashboards (Real-time)

Background – Test Automation as a Service (TaaaS.net)
@jonathon_wright
#esconfs
“Over the three years we have been developing a azure cloud based solution to provide true
"Test Automaton as a Service" using a hybrid automation approach specifically designed with the Hybrid
Cloud (Private/Public) in mind to utilise the instant scalability, flexibility and availability of the Cloud.”
www.TaaaS.net

@jonathon_wright
#esconfs
“The current implementation is a service-oriented architecture (SOA) based on utilising business process
modelling to create business level keywords written in natural language (domain specific languages)
validated against content sensitive validation to create reusable, jigsaw pieces representing the Solution
Under Test (SUT).”
www.slideshare.net/Jonathon_Wright

platform, technology, client, browser,
version, language, test type agnostic
Executing Cloud Tests
feat. VM Dispenser Technology
Cloud Test Environments
natural language validated against
content sensitive validation
Creating Cloud Tests
service-oriented architecture
Born in the Cloud
@jonathon_wright
#esconfs

Relevant
Clear traceability of the business value of Automation through the visualisation
of the tests via Business Process Modelling (BPMNv2.2 compliant)
Effective
Self-validating test assets achieved using natural language with context sensitive
validation against business and testing rules, workflows and data
Maintainable Self-maintaining test asset loader/scraper
Efficient
Reports on SUT health including ratings such as percentage availability since
build/release, reported errors over time and traffic to error ratio
Manageable
Unified platform which non-domain experts can use a natural language to
represent business processes and user story acceptance criteria
Portable
Technology agonistic - Platform, client, component, browser, version &
language
Test type agnostic – smoke, regression, integration & performance
Reliable
Fault tolerance is built in to report and continue on different levels of fuzzy
matching combined with the non-technology specific test definition language
Diagnosable
Actionable defects provided by environment under test (EUT) live pause-
playback supported by dynamic data adapters (DDA) for accelerated defect
investigation and resolution
Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)
Summary - Test Automation as a Service (TaaaS.net)
@jonathon_wright
#esconfs

RIP Automation Frameworks
(1993-2013)
This years EuroSTAR topic ‘Questioning Testing’
 Q1) Do you really need an office?
(Does your organisation support VPN, video conferencing, online collaboration tools)
 Q2) Do you really need an IT department?
(Does your organisation have access communication tools (e.g. email/IM) business
platforms (e.g. SalesForce) or internal applications (e.g. Citrix)
 Q3) Do you really need an internal Test/QA department?
(Does your organisation need full-time testing specialists. (e.g. security, penetration,
performance, compliance, usability & automation engineers)
 Q4) Do you really need a bespoke Automation Framework?
(What is the core purpose / goals of your organisation? i.e. do they sell food products?
are they a bank? or should they be specialists in creating bespoke test harnesses)
@jonathon_wright
#esconfs
“Here lies beloved architects, engineers and developers who invested a disproportional amount of time and
effort searching for the holy grail of test automation. Careers ended suddenly after a portfolio test manager
asked them to provide the actual business value of what they were doing.”

Q1) How can we move to ‘Test Automation in the Cloud’?
@jonathon_wright
#esconfs
“Existing technologies of virtualisation, business process modelling, cloud based test automation solutions and
rapid and easy internet access allow for the development of approaches that allow companies
to order ‘Testing as a Service’ and pay only for what they use (PAYU).”

Automation Ready
Cloud Power
Test Automation
in the Cloud
Where are we going next?@jonathon_wright
#esconfs

Example Automation Projects Automation Level Investment in Effort Business Benefit
Description AMMi Gen Implementation Planning Execution Accel. ROI
Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112%
Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85%
Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26%
Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200%
Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186%
TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306%
Q2) Are we ready for ‘Test Automation in the Cloud’?
Automation Maturity Model index (AMMi®)
Assessment to provide a current automation maturity level and the target maturity level:
* Source: AMMi®(http://www.AMMi.org.uk) + Test Automation Patterns (http://TestAutomationPatterns.wikispaces.com)
AMMi Phase Target Criteria
Level 0 Accelerating Automated Test Lifecycle Methodology (ATLM)
Level 1 Traditional Framework Driven (Gen 1-5)
Level 2 Managed Processes are planned, performed, measured, and controlled
Level 3 Sustaining
Automation process defined and validated against international
standards (ISO-29119)
Level 4 Quantified Predictability of automation process performance
Level 5 Optimising Automation process variation and statistical predictability
@jonathon_wright
#esconfs
“Sometimes half the battle just starting companies on their automation journey by taking them one step
closer to becoming ready for automation.” AMMi Foundation 2003

Q3) – Which ‘Automation Tool’
@jonathon_wright
#esconfs
“I was recently discussing roles at two of the worlds largest investment banks. Their landscape consisted of over
100+ solutions under test (SUT), representing 500+ applications under test (AUT), across 50+ global locations
with over 400 automation practitioners worldwide. They asked me how to measure the value of automation?”

A1
Login.Process
Request.VM
Access.VM
Manage.VM
Logout.Process
A3
A2
C1
C3
D2
C2D1
C4
E2
D4 E1
E3 E4
D3
Business Process
Scenarios (BPS)
Business Process
Data (BPD)
Business Process
Modelling (BPM)
Business Process
Test (BPT)
Quick Overview - ‘First Day Automation’ Approach
@jonathon_wright
#esconfs www.eurostarconferences.com
Moment in time

Start Cloud Tester
Modelling Tool
(BizAgi or Visio)
Static Source
Dynamic Source
Cloud Source
Direct Source
(xPDL or XAML)
Test Asset Loader
(Test Asset Cube)
Business Process Modelling
BPT Tool
(UFT or Specification)
Business Process Tests
Test Scheduler
(WCF or REST)
TaaaS.net
(Test Designer)
Q4) How do we become ‘First Day Automation’ Ready?
Agile Portfolio Management – BPM Export (Visio)
@jonathon_wright
#esconfs
Agile Portfolio Management- Business Process Modelling
@jonathon_wright
#esconfs
Agile Portfolio Management – Business Process Modelling
@jonathon_wright
#esconfs
BPS Tool
(BizAgi or ALM/QC)
Direct Source
(Workflow or XML)
Business Process Scenarios
Hyper-Test.com
(Test Asset Journals)
www.eurostarconferences.com@jonathon_wright
#esconfs
Solution Lifecycle Management – Business Process Scenarios
HP Quality Centre (10+) / Application Lifecycle Management (11.5)
@jonathon_wright
#esconfs
34 35
36 37
38
39

Feature(s) Work ID
Code
Team Foundation Service
TFS - Gated Check-in
!
CancelOK
Build Server
Backlog
VS2013 x64 Cross Tools Command
Microsoft Windows [Version 7.1.7000]
C:Automation>mstest
/testlist:MasterDriver
/testmetadata:”TestProject.vsmdi”
/testsettings:”MobileDevices.run
settings”
Unit Tests
Feature-Driven Exploratory Test Case(s)
Issue(s) Action(s) Log ScreenCaptureIntelliTrace
Traffic Proxy
Continuous Deployment Test Controller(s) Test Agent(s) Browser(s)
Client(s)Load Agent(s)
Interface(s)
Solution Under Test
Application Under Test
ASP.net | DynoForms (Ajax/Json/Spoc)
Application Server(s)
Database Server(s) Shim/Stub(s)
Shim/Stub(s)
Business Process Scenarios (BPS)Business Process Tests (BPT) Business Process Model (BPM)
Business Process Data (BPD)Business Process Reporting (BPR)
Q5) What is the ‘Manual’mation’ Approach?
@jonathon_wright
#esconfs
Agile Portfolio Management – Agile Feature(s)
@jonathon_wright
#esconfs
Agile Portfolio Management – Product Backlog
Microsoft Project 2013 (Connected to TFS)Team Foundation Service (Product Backlog)
@jonathon_wright
#esconfs
Agile Portfolio Management – Work Items > Tasks
Team Foundation Service TFS Workbench
Digital Analogue
@jonathon_wright
#esconfs
Agile Portfolio Management – Feature-Driven
Agile Portfolio Management – Exploratory Testing
@jonathon_wright
@jonathon_wright
#esconfs
Agile Portfolio Management – Test Case(s)
@jonathon_wright
#esconfs
Application Lifecycle Management – Screen Capture(s)
@jonathon_wright
#esconfs
Application Lifecycle Management – Action(s) Log
Hyper-Test.com
Automation Migration Tool
40
39
41 42 43 44 45
4647

Feature(s) Work ID
Code
!
CancelOK
Build Server
Backlog
C:Automation>mstest
settings”
Unit Tests
Traffic Proxy
Interface(s)
Solution Under Test
Shim/Stub(s)
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Digital Analogue
@jonathon_wright
#esconfs
@jonathon_wright
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Solution Lifecycle Management - Interfaces
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Solution Under Test – Traffic Proxy
@jonathon_wright
#esconfs
Application Lifecycle Management – IntelliTrace
Solution Lifecycle Management – Issue(s)
Test Case Migrator Plus (TFS)
@jonathon_wright
#esconfswww.eurostarconferences.com
Solution Lifecycle Management – Code
@jonathon_wright
#esconfs
Application Lifecycle Management – Unit Test(s)
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Agile Portfolio Management – Business Process Data
ALTOVA XML Spy 2013 (R2 SP1)
Agile Portfolio Management – Reporting - Burndown
TFS Workbench
@jonathon_wright
#esconfs
Monitor velocity & capacity
@jonathon_wright
#esconfs
Agile Portfolio Management – Reporting - Test Execution
Microsoft Test Manager 2012.R3
Q6) How to provide ‘Actionable Insight / Automation Value’
@jonathon_wright
#esconfs
Solution Lifecycle Management – Business Process Tests
“verbs and nouns in the test
specific language are written in
natural language which using
context sensitive validation.”
Login.Process
Action Object Event
Enter Username Text
Enter Password Text
Press Login Button
Domain Specific Language
Actions
Objects
Events
Press | Login | Button
Nouns
Verbs
Test Specific Language (TSL)
“The test editor is designed to be low waste, it will only offer nouns and verbs based on
discovered artefacts not programming objects.”
Agile Portfolio Management – Business Process Scenarios
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
40 41 42 43 44 45
464748495051
5253
5455
565758
38

Dashboards (Real-time) providing ‘Actionable Insight’
Theme: Travel@jonathon_wright
#esconfs

Automation Ready
Cloud Power
Test Automation
in the Cloud
“To the cloud”@jonathon_wright
#esconfs

Q7) Should we move to the cloud?
New Challenges & Opportunities ?
and/or
A match made
in heaven?
@jonathon_wright
#esconfs

New Challenges
already tested?
Community Clouds
@jonathon_wright
#esconfs
“Don’t assume the ecosystems (community or commercial) and
other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)

@jonathon_wright
#esconfs
New Challenges & Opportunities

Consumer Freedom
@jonathon_wright
#esconfs
“Enable sophisticated modelling of the components of cloud deployments,
including servers, storage, databases and data.
For the consumption of cloud resources” - PlanForCloud.com (2012)
CompareTesting.com
Instant scalability
Flexible Test Execution
Flexible Test Assets Environments on-demand
Geo-based execution
Instant Accessibility
99.9% Availability (100% Web)
Instant Portability
Best of Breed (BoB) tools on demand
No upfront investment
Pay As You Use (PAYU)
New Opportunities

@jonathon_wright
#esconfs
Private
Combination
Public
SUT SUTSUT
Community
Clouds
Public
Clouds
Private
Clouds
TaaaS.net TaaaS.net
“... the prediction that 60% of the average enterprise will
have 60% of its applications in the cloud.”
- Karl Flinders, Computer Weekly (2011)
“IBM are working with other cloud suppliers to establish
the standards for cloud service provider compliance.”
- Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011)
Q8) Cloud Type

Automation Ready
Cloud Power
Test Automation
in the Cloud
The Final Frontier@jonathon_wright
#esconfs

Hybrid
Cloud
Private
Cloud
Storage
Networking
Servers
Virtualization
Applications
Infrastructure
(as a Service)
Runtimes
Databases
Continuous Integration,
Build & Deployment
Public
Cloud
Community
Cloud
Q9) How can we move to ‘Test Automation in the Cloud’?
Lab Management
Test Controller(s)
Test Agent(s)
@jonathon_wright
#esconfs
Test Agent(s) – Telerik Art of Testing Framework
VMM LibraryVM Factory
High Volume
Automated Testing (HVAT)
High Volume Mobile
Automated Testing (HVMAT)
High Volume Mobile Automated Testing (HVMAT)
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Solution Lifecycle Management – VMM Library
59
60
61

Feature(s) Work ID
Code
!
CancelOK
Build Server
Backlog
C:Automation>mstest
settings”
Unit Tests
Traffic Proxy
Interface(s)
Solution Under Test
Shim/Stub(s)
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Digital Analogue
@jonathon_wright
#esconfs
@jonathon_wright
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Solution Under Test – Traffic Proxy
@jonathon_wright
#esconfs
@jonathon_wright
#esconfswww.eurostarconferences.com
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
TFS Workbench
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Login.Process
Action Object Event
Enter Username Text
Enter Password Text
Press Login Button
Actions
Objects
Events
Nouns
Verbs
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Q10) How can we execute ‘Test Automation in the Cloud’?
Solution Lifecycle Management – Build Server
@jonathon_wright
#esconfs
Solution Lifecycle Management – Continuous Deployment
@jonathon_wright
#esconfs
Solution Lifecycle Management – Test Controller(s)
@jonathon_wright
Solution Lifecycle Management – Test Agent(s)
@jonathon_wright
#esconfs
Load Agent(s) – Visual Studio 2013 – Load Test Manager
@jonathon_wright
#esconfs
@jonathon_wright
#esconfs
Application Lifecycle Management – Client(s)
Real Mobile Devices (PerfectoMobile) Test Studio for iOS (Telerik)
@jonathon_wright
#esconfs
Cross Browser Testing – Telerik (Art of Testing - Framework)
40 41 42 43 44 45
464748495051
5253
5455
565758
38
626364
65 66 67 68

@jonathon_wright
#esconfs
“The best aspects of these proven test approaches
demonstrate how they have evolved over the past
decade, and this echoes some of the changes towards
more lean and agile business methodologies. They are in
a constant state of evolution – just as the underpinning
technology evolves over time.”
My section in the ‘Experiences in Test Automation’ book
(Dorothy Graham & Mark Fewster) 2012.
Final Thoughts - Test Automation in the Cloud

Contact Details – Jonathon-Wright.com
Testing Blog (www.Right-Testing.com)
Automation Blog (www.TaaaS.net)
@jonathon_wright
#esconfs
LinkedIn + Email (jlwright@iee.org)
www.linkedin.com/in/automation
Testing-Alliance.com

@jonathon_wright
#esconfs
Demo - Test Automation in the Cloud

Agile Portfolio Management – BPM Export (Visio)
@jonathon_wright
#esconfs
34

www.eurostarconferences.com@jonathon_wright
#esconfs
HP Quality Centre (10+) / Application Lifecycle Management (11.5)
35

Solution Lifecycle Management - Business Process Scenarios
@jonathon_wright
#esconfs
36

Solution Lifecycle Management – Business Process Testing
@jonathon_wright
#esconfs
37

@jonathon_wright
#esconfs
38

Solution Lifecycle Management – Hyper-Test.com
@jonathon_wright
#esconfs
39

@jonathon_wright
#esconfs
Estimate Effort?
40

@jonathon_wright
#esconfs
41

@jonathon_wright
#esconfs
Digital Analogue
42

@jonathon_wright
#esconfs
43

@jonathon_wright
#esconfs
44

45
@jonathon_wright
#esconfs

@jonathon_wright
#esconfs
46

@jonathon_wright
#esconfs
47

@jonathon_wright
#esconfs
48

@jonathon_wright
#esconfs
49

@jonathon_wright
#esconfs
50

@jonathon_wright
#esconfs
51

@jonathon_wright
#esconfs
52

@jonathon_wright
#esconfs
Solution Lifecycle Management – Traffic Proxy
53

@jonathon_wright
#esconfs
54

@jonathon_wright
#esconfs
Login.Process
Action Object Event
Enter Username Text
Enter Password Text
Press Login Button
Actions
Objects
Events
Nouns
Verbs
55

@jonathon_wright
#esconfs
56

@jonathon_wright
#esconfs
57

TFS Workbench
@jonathon_wright
#esconfs
58

@jonathon_wright
#esconfs
Solution Lifecycle Management – VMM Library
59

@jonathon_wright
#esconfs
Test Agent(s) – Telerik Art of Testing Framework
60

High Volume Mobile Automated Testing (HVMAT)
@jonathon_wright
#esconfs
61

@jonathon_wright
#esconfs
Application Lifecycle Management – Client(s)
Real Mobile Devices (PerfectoMobile) Test Studio for iOS (Telerik)
62

Load Agent(s) – Visual Studio 2013 – Load Test Manager
@jonathon_wright
#esconfs
63

Solution Lifecycle Management – Build Server
@jonathon_wright
#esconfs
64

Solution Lifecycle Management – Continuous Deployment
@jonathon_wright
#esconfs
65

Solution Lifecycle Management – Test Controller(s)
@jonathon_wright
#esconfs
66

Solution Lifecycle Management – Test Agent(s)
@jonathon_wright
#esconfs
67

@jonathon_wright
#esconfs
Cross Browser Testing – Telerik (Art of Testing - Framework)
68

Performance Testing As A Service – PTaaS.net
@jonathon_wright
#esconfs
69
www.PTaaS.net

Questions?
@esconfs
#esconfs

EuroSTAR Webinar Archive
Access almost 70 on-demand software testing webinars by the world's leading
testing experts. Topics include….
Agile, Automation, Test Management, Process Improvements, Risk-based Testing, Performance Testing, Test
Documentation, Cloud Testing, Session-based Testing, Weekend Testers
Visit the archive!
http://www.eurostarconferences.com/community/member/webinar-archive
@esconfs
#esconfs

The EuroSTAR Community.
Are you a member?
Did you know we have an online resource area with almost …
600 software testing presentations, 70 topical webinars, over 40 videos
and podcasts and 25 expert eBooks
Do you know that membership is completely free and you can access these premium
testing resources anytime?
Join today!
http://www.eurostarconferences.com/community/
member
@esconfs
#esconfs

Are you coming to Gothenburg?
Join us in the Swedish Exhibition &
Congress Centre, Gothenburg,
Sweden from the 04-07 November
for the 21st annual EuroSTAR
Conference.
Visit www.eurostarconferences.com
for more information on what you
can expect to see at Europe’s largest
software testing event!
@esconfs
#esconfs
Book before Sept

Have you seen the latest competition?
You and 3 of your team could be joining us at the 2013 EuroSTAR Conference!
Entering is easy! Just send us a short video (2 minutes approx.) demonstrating how your team works together to meet
goals or solve problems.
Deadline for entries is Wednesday, August 14th!
More information here: http://www.eurostarconferences.com/content/teamstar-2013@esconfs
#esconfs

Don’t miss these great savings, book your ticket today!
Plus Make even bigger savings when you register for the Group discount
where every 5th attendee goes free!
For more information on the Conference Programme and pricing options
go to www.eurostarconferences.com@esconfs
#esconfs

'Test Automation in the Cloud' with Jonathon Wright

Recommended

Recommended

More Related Content

Similar to 'Test Automation in the Cloud' with Jonathon Wright

Similar to 'Test Automation in the Cloud' with Jonathon Wright (20)

More from TEST Huddle

More from TEST Huddle (20)

Recently uploaded

Recently uploaded (20)

'Test Automation in the Cloud' with Jonathon Wright