Beta studio Reimagining Beta Testing -Samir Dash - 07-12-2017

Samir Dash
Principal So,ware Engineer (UXD), RedHat

Originally Presented at
RedHat QE Camp X, Bangalore
7 DEC 2017
Re-imagining UAT & Beta Testing
© 2017, Samir Dash

•  Exploring Beta-Tes?ng space and points to ponder.

•  The Big-Picture of the solu?on.
•  A small step – the demo of the Beta-Studio POC.
•  Q&A
What we will cover!
BetaStudio - Re-imagining UAT & Beta Testing sdash@redhat.com RedHat QE Camp X, Bangalore 7 DEC 2017
© 2017, Samir Dash

“
User
Acceptance
Testing
(UAT)
Beta Test
A test of a product performed by
real users in real environment.
Customer
Acceptance
Testing
(CAT)
Customer
Validation
(CV)
Field
Trials
(Common Term
used in Europe)
Pre-
Release Release
Preview
Images: Crea6ve Commons – No A;ribu6on Required (CC0), Pixabay BetaStudio - Re-imagining UAT & Beta Testing sdash@redhat.com RedHat QE Camp X, Bangalore 7 DEC 2017
© 2017, Samir Dash

Typical Focus Area in Beta-Testing.
Running a beta test doesn’t have to be an
intimidating process.It’s the perfect opportunity to hear from your
target market and plan for the road ahead.
Beta-Tes?ng and/or User Acceptance Tes?ng
is very important phase in SDLC.
© 2017, Samir Dash

What it involves in typical approaches?
Front-End Testing User Testing
•  UI Functionality
•  Script Issues
•  Environment Issues
•  UI Cosmetic – Alignment
•  Formatting
•  Spacing
•  Font
•  Color
•  Grid
•  Design Principle Compliance
•  UI Interaction
•  No interaction Error
•  Appropriateness of the
interaction in specific
environment/device
•  Visual Look n Feel
•  Consistency
•  Visual hierarchy
•  Consisitet Iconography
•  Visual Clutter
•  UX & Usability
•  Herustics alignment
•  Redability
•  Information Hierarchy
•  Appropriateness of Patterns used
•  Consistency in UI Behaviour
•  UI Flow Validation (entry & exist points,
ambiguity etc. )
•  Information Overload
•  User goal alignment
•  Device/Platform Specific HCI
compliance
•  Cognitive Load assesment
•  Accessibility
•  Web Content Accessibility Guideline
•  Mobile Web Accessibility Specs
•  ARIA
•  Device Specific Interaction Specs
•  Section 508
•  A/B Testing or Split Testing
•  User Behaviour tracking & analysis
•  Heat maps, click flows
•  User work flows
•  Segment Preference study
•  Feedback
© 2017, Samir Dash

Why Beta-Tes?ng is
Important?
© 2017, Samir Dash

Popular beliefs on
for Beta Testing Tools
Dimensions
of Goodness
© 2017, Samir Dash

Dimensions
of Goodness
Popular beliefs on
for Beta Testing Tools
That sounds like a professional
‘gyaan’
© 2017, Samir Dash

Let’s view
from 2 widely
different
angles
1.  Left - Right Brain Analogy and the
Intersection of Human &
Technology
2.  ISO/IEC 9126-4 (2001) standard that
defines the difference between
usability and the quality as a
percep6ve thin line.
Here is my take on it …
© 2017, Samir Dash

Bug Reporting
API-UI Functional Bugs
Usability IssuesUI/Front-end Bug
Cosmetic Issues User Testing
Explorative Testing A/B or Split Testing
Accessibility Issues
Customer Validations
Backend Issues
API IssuesBackend Functional Issues
Code Quality Checks
Data Testing
End-user FeedbackCode Optimization Checks
LeX - Right Brain Analogy
LOGICAL EMOTIONAL
© 2017, Samir Dash

When these are of low cri?cality...
why so much
focus on
these?
© 2017, Samir Dash

These are important because
these touch the
core Human
aspects.
Human side of the technology
Here is a Case Study:
h;ps://www.youtube.com/watch?v=-jHDyhJdDb8
h;ps://www.youtube.com/watch?v=27WGQFwrH-w
SOURCE:
© 2017, Samir Dash

How
can you as
a developer
or designer
help
this father & son
?
© 2017, Samir Dash

98%
In 2011, almost 98%* of the
websites failed the W3C’s
accessibility Guideline.
Every 1 in 5 in world population is affected by disability.
Many of whom have difficulty in accessing the web due to
bad design or code.
When Accessibility Failures in Websites is in Focus …
* Internet World Stats 2011
© 2017, Samir Dash

In the ISO/IEC 9126-4 (2001) standard,
the difference between usability and
the quality in use is a ma;er of
context of use.
Usability == Quality
In the ISO/IEC 9126-4 (2001) standard,
the difference between usability and the
quality in use is a ma;er of context of
use.
The ISO
9241-11 (1998) standard iden6fied
efficiency, effec6veness, and
sa6sfac6on as major a;ributes of
usability.
The ISO/IEC 14598-1 (1999) standard
suggested a model for measuring quality in
use from the perspec6ve of internal
sodware quality a;ributes.
2001 1999 1998
© 2017, Samir Dash

So without
Usability / HCI in the
right context the
Quality process is
incomplete.
© 2017, Samir Dash

So without
right context the
Quality process is
incomplete.
real users in
real environment.
© 2017, Samir Dash

So without
right context the
Quality process is
incomplete.
real users in
real environment.
… now we know why BetaTesting is important!
Metrics for external quality can be obtained
only by execu6ng the sodware product in the
system environment for which the product is
intended.
The ISO/IEC 9126-4 (2001) standard also
dis?nguished external versus
internal quality and deﬁned related
metrics. Second, most of the soXware quality models
just described, including the ISO/IEC 9126
standard, are sta?c. That is, none of these
models really describes the rela?on between
phases in the product development cycle and
appropriate usability measures at speciﬁc
project milestones.
© 2017, Samir Dash

Challenges
in Beta-Tes?ng Scenario
© 2017, Samir Dash

Defini6on and percep6ons of
usability as a quality factor
“A set of a;ributes that bear on the effort needed for use and on the
individual assessment
of such use, by a stated or implied set of users” (ISO/IEC 9126, 1991)

“The extent to which a product can be used by specified users to
achieve specified goals
with effec6veness, efficiency and sa6sfac6on in a specified context of
use” (ISO 9241−11,1998)

“The ease with which a user can learn to operate, prepare inputs for,
and interpret outputs
of a system or component” (IEEE Std.610.12-1990)
Usability measurement and metrics: A consolidated model
Ahmed Seffah · Mohammad Donyaee · Rex B. Kline ·Harkirat K. Padda C
Springer Science Business Media, Inc. 2006
…standard also gives
rela6vely few guidelines about
how to interpret scores from
specific usability metrics Usability is
that aspect
of quality
where the
metrics has
to be
interpreted
© 2017, Samir Dash

Top Beta-Tes?ng
tools leave the
interpreta?on to the
customer/end-
user’s discre?on.
real users in
real environment.
© 2017, Samir Dash

Real users in real environment are
unless empowered are hanicapped
to comprehend all the aspects of beta
testing and react.
Also it’s all perspective and all of them
can not be validated with real data
from some bench mark/standards.
As most of the issues are related to user-testing, split-
testing and front-end testing, there is no optimized
single window solution that is smart enough to handle
this in an effective way.
How to ﬁlter-out
pure percep?on
from the actual &
valid issues and
concerns?

© 2017, Samir Dash

h;ps://techbeacon.com/node/623
World Quality Report 2015-16
This indicates that the
customers are looking for
more product insights
through a reliable way to
test quality, functionality
along with the regular
usability and
user-testing in the real
customer facing
environment.
Dil Maange
‘More’
© 2017, Samir Dash

Raising Complexities &
Challenges for Testing is
Increasing.
Mood of
the ‘Public’
© 2017, Samir Dash

It’s about
Automa?on,
Mobility &
Ubiquity
… and of course
Smart-ness!
© 2017, Samir Dash

Let’s Explore
Exis?ng Popular
Solu?ons
© 2017, Samir Dash

Existing Beta Testing Solutions
•  For iOS Apps
•  Provides screen captures as the user is using
and records it
•  User ﬂow and Heat map recording
•  Forming a team to collaborate
•  It is integrated to the app via a few lines of
code.
h;ps://uxcam.com/
-
Commercial
© 2017, Samir Dash

•  A mobile testing platform, providing
companies with detailed video recordings, logs
and crash reports of mobile sessions
•  Provides screen captures as the user is using
and records it
.
•  User ﬂow and Heat map recording
•  Has a time line of user’s action
•  Integrates with Jira , Slack or any popular Bug
Tracking Tool
•  App Distribution - Invite users by email, SMS,
push notifications or share your app with our
automatically generated.
h;ps://tesoairy.com/
-
Commercial
© 2017, Samir Dash

•  Promoted as a crowd-sourced testing platform
•  Exploratory Testing – Based on the
demographics, User type, Device Type and
Location the the right testers can be found.
•  Offers localization testing, exploratory testing,
and security testing.
•  Integrates with many popular bug tracking
tools, keeping track of reported bugs is
managed through a single dashboard
h;ps://99tests.com/
-
Commercial
© 2017, Samir Dash

•  A/B Testing
•  Specify tasks to test
•  Access to videos (and audio) of real-world
users interacting with the app
•  Just uploading the app to the UserTesting
platform -- and then creating the test parameters.
•  No SDK integration needed.
•  Using testers from UserTesting's own pool of
users
•  Exploratory testing – customer is able to
specify the type of user you are targeting - from
region, through to web expertise and other key
demographics.
h;ps://www.usertes6ng.com/
-
Commercial
© 2017, Samir Dash

•  A Mobile testing platform for apps with beta
distribution, crash reporting, user metrics,
feedbacks.
•  User metrics in HockeyApp encompass
everything from number of active users and
engagement, through to information on
devices on which your app crashed in the last
30-days.
•  Via Visual Studio App Center, HockeyApp is
your continuous integration, delivery and
learning solution for Android, iOS, Windows,
and macOS apps.
h;ps://www.usertes6ng.com/
Microsod (aquired in 2014)
Commercial
© 2017, Samir Dash

Existing Test Management Solutions
•  Customer validation through beta testing and
market testing tool.
•  Provides
h;ps://www.centercode.com/
-
Commercial
© 2017, Samir Dash

User
ﬂow
Heat
Map
Cross
Platform/
Cross
Browser
Cosmetic –
Layout /
Grid
Cosmetic
& Other
Visual
A/B
Testing
Usability
Tools
Manual
Usability
Auto
Script
Func.
UI issues
CSS Accessibility
/ Sec 508
Standards
Metrics
Bench
marks
Design
Bench
marks
Auto
generated
Meaningful
Insights
Most Popular Beta-Tes?ng Tools
© 2017, Samir Dash

We lack
Smartness and
proper
Automa?on in
Beta Tes?ng
Solu?ons.
Photo source: ‘The Simpsons’ h;p://pandawhale.com
© 2017, Samir Dash

What’s inside
the bonnet ?
The Product owner at the Beta-phase need the
confidence to sign-off the product. In reality in
most of cases this persona is completely not
aware about the nitty grity of the technology
invoved in the development of the product he
is about to sign-off.
Moreover...

...even when it
comes to
recognize
func?onal issues
in frontend. BetaStudio - Re-imagining UAT & Beta Testing sdash@redhat.com RedHat QE Camp X, Bangalore 7 DEC 2017
© 2017, Samir Dash

So many partial solutions available to deal with specific aspect of
beta testing separately. So many Extensions , indpendednt
applications to handle A/B Testing, scripting, csss analysis , bug
reporting etc.
No Single-window
extension/widgets
based solu?on
available.
Also not all widgets are
connected.
Not all comprehendible by
the customer/end user.
Browser Extension for web
page proﬁling etc. are
speciﬁcally designed for
func6onal tes6ng… they
don’t help solving a
customer’s perception.
© 2017, Samir Dash

Automa?on solu?ons in
tes?ng are engaged and
eﬀec?ve in mostly pre-
beta stages and require
speciﬁc skills to run
them .
With the focus on DevOps, in many cases the CI-CD
solutions are getting developed and integrated with the
new age solutions looking at the rising complexities of
technology stacks, languages , frameworks etc. But most of
them are for the skilled dev or test specialists to use and
execute them.
© 2017, Samir Dash

Automa?on solu?ons in
tes?ng are engaged and
eﬀec?ve in mostly pre-
beta stages and require
speciﬁc skills to run
them .
With the focus on DevOps, in many cases the CI-CD
solutions are getting developed and integrated with the
new age solutions looking at the rising complexities of
technology stacks, languages , frameworks etc. But most of
them are for the skilled dev or test specialists to use and
execute them.
Assuming we can have all
these automa?on features
enabled in BETA it s?ll
points to another
limita?on in the exis?ng
solu?ons.
So much information, no
way to get a consolidated
and meaningful for the
specific product context.
Customer/End-User BetaStudio - Re-imagining UAT & Beta Testing sdash@redhat.com RedHat QE Camp X, Bangalore 7 DEC 2017
© 2017, Samir Dash

We need smart , connected,
single window beta tes?ng
solu?on with automa?on
that can be comprehendible
by the end-users in real
environment without the
help from the geeks.
© 2017, Samir Dash

We need smart , connected,
single window beta tes?ng
solu?on with automa?on
that can be comprehendible
by the end-users in real
environment without the
help from the geeks.
1.  Computer vision to help recognising the patterns implemented.
2.  Analyse the design and export measurable metrics to compare with the benchmarks from standards
3.  Userstand the User’s behaviour from environment and heatmap, pattern bahiviour etc.
4.  Analyze the bunch of data to interpret the the persona and the needs and effectiveness of the product.
© 2017, Samir Dash

•  Beta-Testing that utilizes data from all stages of SDLC and PLM
along with the standards + specs and user testing data to provide
more meaningful insights to the customer.
•  Test real application in real environment by the real users.
•  Customer as well as end-user centric.
•  Test soft aspects of the application -- Usability , Accessibility ,
Cosmetic etc.
•  Smart enough to compare and analyze these soft aspects of the
application against functional testing data.
•  Use machine-learning & cognitive to make more meaningful
recommendation and not just dump info about bugs and
potential issues.
TESTING
DEVELOPMENT
DESIGN
PLANNING
MAINTAINANCE
RELEASE
Product Life-cycle Management
PRODUCT
LIFE-CYCLE
MANAGEMENT
Vision
© 2017, Samir Dash

TESTING
DEVELOPMENT
DESIGN
PLANNING
MAINTAINANCE
RELEASE
CVNLP
DEVELOPER
UX DESIGNER
VISUAL DESIGNER
PRODUCT MANAGER
PROJECT MANAGER
TESTER
END-USER
COGNITIVE
A.I.
DESIGN
BENCHMARKS
END-USER DEVELOPER
CVNLP
CVNLP
CVNLP
CVNLP
NLP
CV
INSIGHTS
&
RECOMMENDATION
CUSTOMER
CUSTOMER
CUSTOMER
CUSTOMER
AUTOMATED
TRACKING
AUTOMATED
TRACKING
PRODUCT
LIFE-CCLE
MANAGEMENT
STANDARDS
&
SPECIFICATIONS
BETA / UAT TESTING
BetaStudio Vision of Beta-Testing
© 2017, Samir Dash

Collect Benchmark
data from design
stage
DESIGN
BENCHMARKS STANDARDS
& SPECIFICATIONS
MODEL
& MTERICSCOMPUTER VISION
NLP
AUTOMATED
TRACKING
Track & Collect data
from product
Collect Standards &
Specifications
Process
Unstructured data
to get inference.
Model the data and
Map against the
Metrics.
A.I.
& COGNITIVE
AI helps finding the
outliers &
generate Insights /
recommendations.
FEEDBACK
& USER TESTING
Data from User
feedback & User
Testing
Web Content Accessibility
Guideline (WCAG)
Section 508
Web Accessibility Initiative Specs
ARIA, Design Principles,
W3C Compliance
JS Standards
CSS Standards & Grids
Code Optimization Metrics
Error codes & Specs
Device Specific Guidelines (e.g.
Apple Human Interface Guideline)
etc.
?
?
?
?
? ?
Auto Generate
Design Specification
from Any Design files
Pattern-AI
In-App Tracking
?
?
?
Journey has just started ...
POCPOC POC
POC
© 2017, Samir Dash

POC
We started with small needs
© 2017, Samir Dash

Developer Developer
Is there an easy way
to find out all
consolidated front
end related issues
and errors ?
© 2017, Samir Dash

Is there a single tool that
helps me finding out the
usability and cosmetic
issues and check if the
design is implemented
correctly?
Designer Designer Images: Crea6ve Commons – No A;ribu6on Required (CC0), Pixabay BetaStudio - Re-imagining UAT & Beta Testing sdash@redhat.com RedHat QE Camp X, Bangalore 7 DEC 2017
© 2017, Samir Dash

Tester Tester
Is there an easy
way to find out all
consolidated front
end related issues
and errors ?
© 2017, Samir Dash

Customer Customer
Can I sign-off with
confidence without
much bothering about
the technology or know
how about the
standards in use?
© 2017, Samir Dash

The Numbers.
Development Design 2
© 2017, Samir Dash

When more than one third
issues are non-functional and
mostly Cosmetic , there is no
reliable solution that helps in
bench marking these kind of
issues against some specific
standards .
On minimum one third of the issues found during the beta /
UAT stages of testing are mostly cosmetic and alignment
issues including category 3 and 4 types.
When Cosmetic Issues is in Focus…
30%

45%
Quick first-aid approach using
scattered UI patterns and
solutionsis popular among
dev-community to compensate
design aspect of the code.
Many don’t know on how to implement the design principles
while coding. Many depend on scattered tools, solutions, UI
patterns to come up with some design first aid for their
code, which in many cases are not holistic.
When Developer is in Focus…
Images: Crea6ve Commons – No A;ribu6on Required (CC0), Pixabay
© 2017, Samir Dash

50%
When Designer is in Focus…
More than half of the designers
are not aware about more than
half of the evolving
technological solutions about
design
Many don’t know about the how to implement CSS grid with
a design grid under different devices with varied screen
resolution, dpi etc.
© 2017, Samir Dash

70%
How to
bench-mark with design
standards and specs?
Detailing out a spec for design comes with a cost and skills.
In more than two-third of the cases of development there is
absence of detailed design with specs. Wherever it exists
many of the designs are not standardised, also most of them
do not have clear and detailed specs. Also being design is
carried out by different tools it is not always easy to have a
centralized place where all the designs info is available for
bench marking.
When Design Specs is in Focus…
© 2017, Samir Dash

Specstra is a Automation POC
that is more like a cloud based
Visual Design Style Guide
Generator from the third party
design source files – this is a case
where the user would like to
continue use his existing Design
tools like Photoshop/Sketch or
Illustrator, PDF etc.

‘Specstra’ — Experimen?ng with Automa?on in Design Industry
h;ps://medium.com/design-ibm/specstra-experimen6ng-with-automa6on-in-design-industry-4641c0b4244d

DEMO :h;ps://youtu.be/H-o3vb9MZtg

Specstra helps in collect benchmark
data from design stage.
BetaStudio - Re-imagining UAT & Beta Testing sdash@redhat.com RedHat QE Camp X, Bangalore 7 DEC 2017 © 2017, Samir Dash

h;ps://
samirshomepage.ﬁles.wordpress.com/
2016/08/sample-generated-style-guide-
output1418308769-en.pdf
SAMPLE GENERATED GUIDE
Specstra helps in collect benchmark
data from design stage.
© 2017, Samir Dash

Demo - SPECSTRA

The goal of is to facilitate
and empower the user/tester of the
target applica6on in carrying out such
tes6ng and converge incident repor?ng ,
issue tracking & management in the
areas of Client-side poten?al issues
ﬁnding, Accessibility & Usability tes?ng
as well as Cosme?c bug repor?ng.
Dashboard
Tracking Workspace
Proof of Concept

Runtime Tracker
•  Tracks real 6me script issues.
•  Detects environmental details and logs
•  Captures events and issues for debugging.
•  Check for inline Style blocks
•  Analysis of CSS files
•  Automa6c report genera6on
CSS Inspector
Screen-Reader Sim
•  Provides screen reader simula6on
•  Supports detec6ng sequence issues
•  Highlights poten6al issue areas.
•  Helps in finding out understanding how the
page is consumed by screen readers.

Accessibility Inspector
•  Automa6c analysis & report genera6on
•  Web Content Accessibility Guidelines 2.0
(WCAG2) supported
•  508 Accessibility Checks
Color-Blind Sim
•  Provides op6ons to check the different
colorblind tests on the page.
•  Supports 6 types of color blind scenarios.
X-Ray Tool
•  Help finding cosme6c issues in UI
•  Find alignment, pixel level details in
alignments.
•  Script syntax checks.
•  Formatng and errors.
•  JS Lint support
JS Inspector
On-Screen App
•  On-screen Ruler
•  On-Screen Guides
•  On-Screen Distance Measure
•  On-screen CSS Grid Checker
•  On-screen Baseline Checker

AUTOMATATED
AUTOMATATED
AUTOMATATED
AUTOMATATED
MANUAL
MANUAL
MANUAL
MANUAL
TRACKING
WORKSPACE
Major Features:
•  Automated tools
generate report on
potential issues that are
categorised under error,
warning etc.
•  Track Accessibility, Style
sheet, Java Script errors,
Run time Client side
issue tracker, Cosmetic
issues finder for
alignment, margin,
padding etc.
•  Attach screenshot with
redlines while reporting
•  Screen-reader simulator
•  Color blindness
simulator etc.

DASHBOARD
Consolifdated
view of issues,
warnings, data
visualization
PROJECTS
All projects can
be managed
separately and
stats of reports,
review cycles
etc. can be
tracked.
REVIEW-CYCLES
Each time user
tests a project, it
is considered a
review-cycle. All
such cycles along
with their reports
can be found
here
REPORTS
Each url or screen
the user tests, a
consolodated
report on all
aspect is tested
against it. Those
can be accessed
with list of issues
in them.
Download PDF/
Excel etc.
BUGS
When user
reviews any
page, the
automated tools
logs potential
issues, along
with it user can
use other tools
to manually log
bugs.
GITHUB
User selected bugs
can be pushed and
managed in
Github.
In future more
tools will be
supported.

Demo

MVP 1
Public Beta
Basic Framework Stable Code Scaled version
JUL-DEC 2017 JAN –JUN 2018 JUL – DEC 2018 JAN – JUN 2019
MVP 0
POC
MVP 2

MVP 3

Cross-plaoorm Client
User Management & Dashboard
Git-Hub Repo + Ac6ve Public IP/ Domain
Thin client/Browser Support
Produc6on Ready
Produc6on Ready
Social Network
Standardized APIs
Mobile Version
In App Feedback
CSS INSPECTOR
XRAY APP
JS INSPECTOR
ACCESSIBILITY INSPECTOR
RUNTIME TRACKER
SCREEN-READER SIMULATOR
COLORBLIND SIMULATOR ISSUE TRACKING IN OSIO
ONSCREEN SIMULATOR
IN-BROWSER TESTING ISSUE TRACKING IN GITHUB
POC ROADMAP
Browser Extension InApp Version/ In Browser
© 2017, Samir Dash

Desktop Client
(OSX, Windows, Linux)
Browser Extension / Plugin
(Mozzila, Chrome, IE)
In-App
(Cross plasorm & cross- devices)
ROADMAP

I am not able to
replicate the issue
on the desktop!
Many front-end & cosmetic issues are
device specific involving triggering points
from screen-densities, resolution and
phyiscal as well environmental interaction
factors.

In-App

In App tries to take BetaStudio into the
Webpage and mobile browsers.

Patern-AI
This experiment is about utilizing
Computer Vision to recognize UI
patterns from screenshots of a
webpage.
Using Open-CV output Haar files, the
initial attempt is to create Haar Cascade
files for web patterns.

•  Integra6on of Specstra (How to set benchmark in cosme?c issue detec?on area?)
•  A/B Tes?ng / Split Tes?ng
•  Screen Annota?ons
•  Member real ?me chat/messaging (Slack / Matermost/ Skype / BlueJeans)
•  Mouse hit map
•  Eye tracking on screen
•  Survey On screen/project speciﬁc speciﬁc
•  Star ra?ng
•  Report bug
•  Timeline history
•  W3C Valida?on
•  Screen reader
•  InApp
Wish-list ...

h;ps://www.youtube.com/watch?v=kItqD5wc4_4

Connect & Contribute....

OpenSource Project @GitHub :
https://github.com/betaStudio-online
Homepage: https://betastudio.online
LINKS
h;ps://www.youtube.com/watch?v=kItqD5wc4_4

THANK YOU.

•  Test Automation and Management.
•  Mostly functional & load testing in nature and
layout etc. is termed as user experience testing.
•  Supports Mobile App Test Automation, Dev-OPS,
Risk based Testing,
•  Execute a batch of automated tests with zero
training and only a few clicks - automation across
browsers & mobile devices.
•  Uses Selenium (browser + driver) to click and
point and interact with windows.
•  Layout measurement uses Selenium Grid.
h;p://www.qsometech.com/
-
Commercial
APPENDIX

•  Test Management for JIRA
•  Easily organize, and reuse complex test cases
for any use case, ensure seamless test execution
on multiple platforms, integrate with leading
automation tools, and gather comprehensive
test coverage and traceability reporting on all
testing activities.
h;p://www.qmetry.com/
-
Commercial
APPENDIX

Existing Open Source Test Automation Solutions
•  Java-based Test automation framework for
Acceptance Testing (AT) and Regression Test
•  Integrates with behavior-driven development
tools such as Cucumber and Jbehave
•  Uses BDD to create test scripts.
•  Write tests that are more ﬂexible and easier to
maintain.
•  Produce illustrated, narrative reports about
tests.
•  Map your automated tests back to
requirements
•  Test Code Coverage
h;p://www.thucydides.info/
Wakaleo Consul6ng
Open Source
APPENDIX

•  Test automation framework for Acceptance
Testing (AT) and Acceptance Test Driven
Development (ATDD)
•  The framework has a rich ecosystem around it
consisting of various generic test libraries and
tools that are developed as separate projects.
•  Developer-centric approach and more focused
on functional testing automation
•  Uses Selenium for Web Application Testing
h;p://robooramework.org/
Nokia , Robot Framework Founda6ons
Open Source
APPENDIX

•  Test automation framework that works with any
tool (Selenium, Appium, Silk, etc.)and works with
anything Java/Groovy, Python and C# to test
things like API SOAP/REST, Database, CLI, etc.
•  Provides Action/Keyword Driven Framework
to run tests.
•  Import existing TestNG/JUnit test cases and
develop new ones.
•  Developer-centric approach and more focused
on functional testing automation
h;p://redwoodhq.com/
RedwoodHQ
Commercial + Open Source
APPENDIX

•  Focuses on automation and testing tool for web
applications, with the facility to record and
playback scripts.
•  Provides Special Syntax to run tests.
•  It easily automates applications built using
ExtJS, ZK, Dojo, YUI or any other framework.
•  Claims to be Testers Centric
•  Good for initial test on relatively simple html, but
need to customize it to make it reliable and
maintainable.
h;p://sahipro.com/
Sahi Pro
Commercial + Open Source
APPENDIX

•  Focuses on Layout Testing - opens a browser,
resizes it to a defined size and then tests the
page according to specifications
•  Provides Special Syntax to run & measure
layouts in different view sizes and captures
screen shots
•  Command line Approach
•  Uses Selenium (browser + driver) to click and
point and interact with windows.
•  Layout measurement uses Selenium Grid.
•  No Intelligence
•  No True UX testing
h;p://galenframework.com/
-
Apache License
APPENDIX

•  Developer-centric approach and more
focused on functional testing automation.
•  Provides Syntax to run & execute tests.
•  supports C#, Java, Ruby, IntelliJ and Visual
Studio out-of-the-box.
•  Uses Selenium Web driver for browsers and
Sahi for web -app
h;ps://getgauge.io/
ThoughtWorks
GNU Public License version 3.0
APPENDIX

Beta studio Reimagining Beta Testing -Samir Dash - 07-12-2017

Recommended

Recommended

More Related Content

Similar to Beta studio Reimagining Beta Testing -Samir Dash - 07-12-2017

Similar to Beta studio Reimagining Beta Testing -Samir Dash - 07-12-2017 (20)

More from Samir Dash

More from Samir Dash (20)

Recently uploaded

Recently uploaded (20)

Beta studio Reimagining Beta Testing -Samir Dash - 07-12-2017