Innovations in Testing - Book

KALILUR RAHMAN
ii
Innovations in Testing
Testing and Quality Engineering Innovations, Disruptive Tools,
Techniques and Processes needed for success in the new digital age
KALILUR RAHMAN

Free to reproduce – with a credit reference to the author under Creative Commons Licenses

DEDICATION
I dedicate this book to my family. My wonderful spouse and the two angelic daughters who made me
a complete man showered with full of blessings and happiness. It’s the cherubic smile of your
children and the giant sequoia like support from your spouse that motivates simpletons like me to
dream and aspire to move mountains!

CONTENTS
PREFACE
1. WHY IS QUALITY IMPORTANT?
2. INTELLIGENT TESTING SKILLS - PRIMARY NEED OF THE HOUR FOR DIGITAL
INNOVATION
3. INTELLIGENT TESTING SKILLS NEEDED FOR THE NEXT GENERATION -
UPSKILL OR RETIRE
4. TOP TIPS - HOW TO ESTABLISH A SUCCESSFUL TCOE / QCOE (TESTING /
QUALITY CENTRE OF EXCELLENCE)
5. ARTIFICIAL INTELLIGENCE (AI) IS THE NEW ELECTRICITY! IS THERE
ANYTHING ARTIFICIAL OR INTELLIGENT ABOUT IT?
6. IMPACTS OF DEVOPS ON TESTING
7. HOW TO RUN EFFICIENT API TESTING FOR IOT, WEB AND MOBILE APP
INTERFACES?
8. "CROWD SOURCED TESTING" – A NEW WAVE IN DIGITAL REVOLUTION - A
POINT-OF-VIEW
WHAT NEXT?9. NOUVEAU SKILL NEEDS FOR TESTING – FOR NEW SOFTWARE
DRIVEN BUSINESSES
10. HOW DIGITAL INNOVATION IMPACTS TESTING AND COMMUNICATIONS
INDUSTRIES? – A POV

i
ACKNOWLEDGMENTS
I acknowledge all my friends, current and ex-colleagues who helped me shape-up my writing skills.
A special thanks to all my social network followers - especially in LinkedIn who motivated me to
become a better writer and as a result – a better person and a human being. Appreciate all the
inspiring authors, leaders and stalwarts who make learning a happy and progressive one on a daily
basis. I wish all the humans who contribute to make a better place, in spite of all the challenges we
face on a day-to-day basis. I would acknowledge my teachers, coaches and mentors who made me
become who I am today.

KALILUR RAHMAN
2
PREFACE
I’ve been posting blogs on various topics for the past few years. I was contemplating collating all my
thoughts into an e-book format and publish for the wider forum. Based on the inputs I received from
my friends and family, I have assimilated all my posts in the public Internet and categorized them
under Testing, Technology and General Management. As a part of this journey, Innovations in
Testing is my first e-book I target to publish via Kindle Direct Publishing. All the proceeds associated
to my account from this book will be donated to charities focusing on educational welfare of
underprivileged children.
I have categorized this book in the following groups.
1. First things first
a. Given there are hundreds of great books on Testing fundamentals and concepts, I’ve
decided to include only key skills and knowledge centric thoughts I’ve posted. These
include – Why Quality is important, What are the next generation skills needed for
testers, How a corporate can embark on a journey of excellence with a Quality
Centers of Excellence
2. Impact of Digital Innovations in Testing
a. Focuses primarily on real impact the digital innovations are making in the industry.
Covers the impact of DevOps, Artificial Intelligence, API evolution, Crowd Sourcing,
Big Data on Testing. It also covers some of the ways to address the challenges and
opportunities, covering briefly on methodology, tools and learnings to do things better
3. What Next?
a. Covers largely on what are the areas of focus one should have to innovate in the new
age testing, what are the skills one needs to acquire, how the testing world is
eventuating in the near term, scope of testing in future
Again, this is a collection of already published blogs posted by me. Most of the illustrations were
hand-crafted and some of the photos or images are available freely in the public internet.

INNOVATIONS IN TESTING
3
1. WHY IS QUALITY IMPORTANT?
"A thing of beauty is a joy forever" quoted John Keats. "Beauty lies in the eyes of the beholder"
touted Plato. Is Beauty and Quality Interlinked? I think so. Towards this, let me start off with some
stories.
Have you ever been to Florence, Italy and saw a brilliant statue (touted to be the best marble statue
ever to have been made in history)? If you have strolled through the streets of the Tuscan capital
that pull in well over 13 million visitors every year – Florence, you will certainly understand why it is
called as a UNESCO world heritage site, or as one of the most beautiful cities in the world. Florence
was at the center of the renaissance movement and it had one brilliant master amongst them.
On the cobblestone streets near Palazzo Vecchio, you would see a giant replica of a monumental
sculpture of David (Original one is in Galleria Dell ‘Academia). This was built by one of the greatest
artisans of all time “Michelangelo di Lodovico Buonarroti Simoni”. His other masterpieces include
Pieta, Sistine Chapel, and St. Peter’s Basilica's popular murals. All of these are great crafts of
pristine quality with the sheer brilliance of genius that is enthralling tourists visiting Italy for six
centuries since they were commissioned. All due to Quality, Perfection, Excellence. I had a chance
to view some of these masterpieces in quality, in person. David is a masterpiece piece of work.
Standing 17 feet (which is nearly 3 times a human) tall, this gigantic statue was commissioned in
1504. This was at built from a marble that was up at about 80 meters from the ground.
The story goes like this. A Young, energetic, inquisitive, talented and creative, 25 year old
Michelangelo was asked by the seniors of the Vestry Board to complete an unfinished project. A
Project that was started in 1464 by Agostino di Duccio to be dropped; and later continued shortly
by Antonio Rossellino in 1475. Both the talented artists backed out from sculpting this huge piece of
marble. The reason cited is that they rejected the work of chiseling the humongous marble stone
was due to the presence of too many “taroli”, or imperfections, (or bugs in testing parlance)
threatening the stability of a huge statue. This block of marble of exceptional dimensions remained
therefore neglected for 25 years, lying within the courtyard of the Opera del Duomo (Vestry Board).

KALILUR RAHMAN
4
As per some quotes, it is said, after his masterpiece work – David was done, Michelangelo was
asked a question. He was asked about his work
Question: “How did you make such a genius piece of art from a lame, huge piece of marble stone”?
Michelangelo’s response (whose famous quotes includes quotes such as “Trifles make perfection,
and perfection is no trifle.”) was a classic one.
Answer: “I saw the angel in the marble and carved until I set him free from imperfections (taroli)”
A lot of stories of similar kind can be said. A tester’s key craftsmanship is to remove
imperfections of a product or a software and make it a high quality one.

5
This is a classic case of having an end-state in mind. The artist had a vision of what he wanted to
enthrall the world with! Quality exemplified.
On the other hand, let me give you an example of where ignorance of Quality results in major
disaster. Two decades ago, Ariane-5 Rocket Launch in 1996 was a disaster. This project was a
decade-long, $7 billion investment (worth more than $10.7B in current currency) that was gone
under 40 seconds. The target was to launch two 3-ton satellites to orbit, following a series of
successful launches earlier. The issue was a software bug caused by a floating point exception
arising due to an incorrect Exception handling of a 64-bit number into a 16-bit handler. This was
never tested or properly modeled before the launch. Think about this – lack of attention to details
causing a big failure. This major accident (or incident for a better analogy) happened due to lack of
attention to details. To quote John Ruskin,
“Quality is never an accident. It is always the result of intelligent effort”.
How many of us do the right thing even when no one is watching? Taking a seminal quote from
Henry Ford
“Quality means doing it right when no one is looking“.
The quality of your work will stay long after you are gone. Quality in simpler terms is Craftsmanship.
To borrow from popular culture, when you take global sobriquets for quality “German
Engineering”, “Swiss Watches” and “Japanese Electronic Goods and Cars” for example, all are
popular world over for excellence in quality. Some of them have come up with innovations to take
their industries to the next level. Barring a few headline-making deviations, all of them bring so many
quality names to mind.
Why would quality matter to people? If you study the resurgence of Apple Inc. Steve Jobs’ 2.0 CEO
tenure, you will understand all of it is attributed to one term. “Quality”. Quality minded producing
high-quality goods. Consumers who love quality buy quality products. It becomes a popular culture

KALILUR RAHMAN
6
as there is a uniform belief that quality of work reflects a person’s character and attribution of one’s
capability with the quality of the products they use, endorse and associate with. While this is a
debatable topic, people nowadays try to autograph themselves with excellence. This comes with
affiliation to quality. Steve Jobs and Apple cashed in on this very fact. They focused on Making
Complex things look a lot simple - Using multiple perspectives and analysis with the customers in
mind. They did it with the right rigor in doing things right - with a focus on the quality of things
unseen. Whilst a thousand music players were already in the marketplace, they made a huge
difference with a brilliant iPod. Similar success was repeated with iPhone.
How did Steve embark on this journey? A couple of his quotes summarizes it – from his biography
with Walter Isaacson and public Internet.
“For you to sleep well at night, the aesthetic, the quality, has to be carried all the way through.”
“I thought my dad’s sense of design was pretty good. He loved doing things right. He even cared
about the look of the parts you couldn’t see.”
His attention to detail, experience with his creative aesthetics during his calligraphy session, quality
focus in picking up the best team and always thirsty attitude for customer-centric innovation were all
having one theme in common – Quality.
Quality is a result of Authenticity, Accuracy, Beauty, Making complex things simpler, Higher Level of
Thinking with the end in mind, learning from the Best, Practicing with different types, Attention to
details and use of science and technology or aesthetics to make it an impeccable
product/service/experience or highest quality. Good companies have thrived because of the quality
of service/product they produce. It could be a restaurant, a sweet shop, a boutique tailor or it could
be a major brand. Quality is their business plan and without it, they will be out of business.
Based on economic viability, corporates or individuals choose product or service that comes with 1)
high quality at a premium or 2) a cheaper one at lesser quality, or 3) a high-value proposition one
that gives good value for money with great quality. It will be difficult to see a market hegemony for
one of the types to rule the market as there are niche areas. Similar things can be said about job
satisfaction and quality of workforce and quality of job satisfaction that get attributed due to so many
factors. The quality of thinking also gets impacted by other people’s opinions and “societal
influence”. The way one think, behave, dress, the drive gets impacted by the quality of influence.
Being unique in terms of value quality or aligned to societal conformance is also a factor of quality.
At the end of the day,
• Quality is a mindset and it is everyone’s responsibility.
• Quality is not the last line of defense before a product or service goes out to the end target. It
is the beginning and the end.
• The quality of One’s work is dependent of Quality of One’s work which gets influenced by
Quality of Training.
• The quality of One’s behavior gets influenced by Quality of one’s surroundings.
• The quality of a student’s result is an outcome of Quality of the learning / studies / tutors and
effort.

7
• The quality of a craftsman’s output is directly dependent on the quality of one’s practice. The
quality of input determines the Quality of Output.
Hence Quality is not a silo but everyone’s responsibility. Some more quotes on Quality highlights the
same
”Quality is a habit and not an act”.
“Be a yardstick of QUALITY. Some people aren't used to an environment where excellence is
expected” - Steve Jobs
“When you're out of quality you're out of business.” - Anonymous
Quality is the best business plan.” - John Lasseter
One might say "Quality of a product or service lies in the eyes of the beholder" but we have
enough examples to repeat over and over again to highlight how quality products or inventions
outlive time immortally. There is enough art, scientific examples out there that surmount as
evidence.
What is your view on Quality?

8
2. INTELLIGENT TESTING SKILLS - PRIMARY NEED OF THE HOUR
FOR DIGITAL INNOVATION
“If you are not learning something new each day, you are not TESTING”
- Gerald Weinberg
What are the intelligent testing skills needed for the hour?

9
1. “What’s in it for me?”
• We are intelligent anyways –
• Why should we really bother?
• What is changing?
• How do we address these questions?
2. “How can we be Relevant Technologically?”
• What are the Skills we need to focus on in a rapidly changing eco-system?
• What are hypes and fads vs real must-have skills?
• Given the rapid advances in technology, what is that we need to do to stay relevant? Is
testing dead by any chance, is another question we can ask ourselves.
3. Finally – What are the “Soft Skills Needed to Succeed”
• What will make us more intelligent – business and customer centrist?
• Why are these the most important skills/traits to master?
• A Tester is only as good as the skills he/she possesses. What are the essential soft skills
needed by the tester?
A perfectionist of sorts, Steve Jobs quoted - “Be a yardstick of quality. Some people aren't used
to an environment where excellence is expected”
Without a focus on quality, simplicity and efficiency, APPLE wouldn’t have become the most
valuable company on earth, a brilliant turnaround from a company that was almost dead before Jobs
2.0 began.

KALILUR RAHMAN
10
Carrying on Quality - Some statistics or a tip of the iceberg
• Over 4.4 billion people got affected by a software fail – which is greater than 50% of Human
Population. It is almost a number similar to the people not having access to a Toilet – But
less than the number of mobile phones in use in the world!
• More than a $Trillion worth of assets affected and a cumulative impact of 315 years.
• A leading airlines lost 3% market cap due to a botched software upgrade infested with bugs
While the defects and bugs are making a dramatic impact, the world is leaping ahead. Business is
expecting agility in business delivery.
Take these for some stats
1. Google – which supposedly has a single code repository, refactors code by upwards of
50% each month. They have ~2 Billion LOC (and counting). Even taking a 75-95% test
coverage taken up by empowered teams (as claimed by some of the engineers in published
artifacts), this is a humongous testing effort. If you have a backlog of code to be verified, it
could be a disaster exceeding the size of a titanic by all means. When the Cyclomatic
complexity of testing is so huge, how can you test the entire code base and application

11
flawlessly? This is a brilliant example of how one can run an efficient test strategy.
2. Take Netflix that currently has over 5 Billion API Calls per day (up from Billion+ a few years
ago). How would you do effective Load, Stress, and Performance Testing and ensure
Availability, Redundancy and Reliability of service is not impacted?
3. A lot of firms are moving towards a bi-modal IT (doing a transformation while running the
legacy apps running) and doing continuous delivery and Testing all the time, leveraging all
the fancy words such as Agile, DevOps DevQAOps etc. etc.
4. Additionally, nearly 41% of Global corporate workload is shifting to cloud, to ease out on
Capital Expense and controlled Operational Expense strategies.
5. By some means, Augmented Reality, Gestural Computing, IoT is expected to take the world
by storm. How are we going to test all these permutations?
Taking the prior discussion forward, if you look at how businesses are getting innovative, it is
largely due to rapidity in delivering code faster, with high velocity, high quality deliveries.
There are two schools of thought – In Systems of Engagement or Systems of Innovation, you
need to rapidly deliver. You will not have time to test all permutations. As Testing is typically a linear
process, with the increasing complexity, the testing process becomes exponential but the capacity is
linear in terms of number of resources and hours you have. You need to do testing in an intelligent
manner.
This is where creativity plays a major role. It's a question of How and Can.
• Can you do more with less?
• Can you do effective automation?
• Can you deliver a release in 11.2 seconds like Amazon does and roll-back in case of issue,
with a production testing?
• Can you do testing for 1 billion lines of code every month?
• At the same time can you take risk of compliance and regulatory norms in systems of
records that have very strict rules to abide by? In these cases where scrutiny is very
critical, you cannot do what an Amazon or a Netflix or an Etsy or a Google does. You need
to do thorough validation. How can you achieve this?
• How can you reduce the cycles from 4 releases a year to 12 releases a year?
• How can you test optimally when the testing scope grows exponentially?
• How can you increase the quality while velocity increases?
If you take the Gartner’s Hype Cycle for Emerging Technologies for 2017 – You see a pattern.

KALILUR RAHMAN
12
Some are in the slope of enlightenment but majority in the curve of inflated expectations and
disillusionment. For the technologies emerging stronger, we need to have some solid test approach
/ strategy to deliver high quality outcomes
• Artificial Intelligence
• Machine Learning / Deep Learning
• Internet of Things (or Everything)
• Drones & Vehicles
• Connected Devices
• AR/VR & Wearables
• Gestural Computing
• Human Augmentation
• Robotics
• Block Chain
• Algorithms
• Smart Assistants
Are we ready for these and do testing rapidly, smartly, and creatively with high quality?
Broadly speaking – New Age Test Innovation focuses on the following needs with Intelligent
Testing
• Rapid High Quality and Innovative Test Delivery
• Test Suite Creation and Optimization (Risk Coverage)
• Useful Automation – Test Smart, Self-Healing, Script less, Purposeful automation using
cognitive and machine learning intelligence
• Predictive and Cognitive Testing – Foresee issues reduce reactive time, resolve rapidly
• Rapid Impact Defect Finding - Intelligent Defect Detection, Pattern Analysis, Predictive Modeling
• Intelligent Environment Provisioning
• Management with Intelligent Metrics and Dashboard
Are we capable of building intelligent automated frameworks and leverage cognitive models to
optimize our test strategy and test suites to do proactive application health analytics via rapid defect
finding and scale up rapidly to do niche and special areas of testing? That remains the key.

13
Coming forward, given testing is an exponential function and not a linear function any more due
to rapid explosion of technology? The key is to have a pragmatic view of testing that is applicable to
the industry, firm or department you are in.
Leading firms have unique test approach for
• SoE (Systems of Engagement)
• SoI (Systems of Innovation)
• SoR (Systems of Records)
There is no One Size Fits all approach for testing and Intelligent Approach needs to be adopted
as per needs
• Currently, majority of the organizations focus majorly on Business testing (a heavy
investment) which is time consuming
• An inverted pyramid of more testing in earlier phases will deliver better value to the
organization
How can a testing organization catch-up to exponential growth by inverting the testing pyramid to
do more testing upfront so that actual test effort is reduced closer to deployment time?
This is exactly what the leading organizations are doing to propel their technological hegemony.
Testers Career path

KALILUR RAHMAN
14
A 5 step ladder to excellence for a tester includes
1. Getting Stronger in fundamentals
2. Building the needed techno functional skills
3. Building an excellent Soft Skills – as Testers will deal with a lot of distinct stakeholders and
need to navigate different layers
4. Specialization in a particular area to build a strong I in a T shaped expertise
5. Have a vision and thought leadership to become one of the best in the industry
Tester's IKIGAI (Reason for Being)
How would you drive towards a life of self-actualization? There is a saying that goes like – “If you
do work that you love doing (or a profession), then you will not work but love doing your work”.
Ikigai is a Japanese Concept meaning “A Reason for being”. A concept that gives various
perspectives on Satisfaction, Excitement, Passion, Delight, Recognition and eventually fulfillment.
What would be Ikigai for a tester? How would you leverage your Passion, Vision, Skills and

15
Expertise to derive a sense of being -? How would you remove the demotivating acts such as feeling
of uselessness, emptiness, uncertainty, lack of wealth while pursuing these higher objectives that
are aided by getting paid for doing what the world needs by leveraging what you are good at and
love doing? It needs a fine balance of things to derive utmost happiness and a sense of
accomplishment.
Top-3 Leadership Skills needed for a Tester
Let’s see a Test Leadership survey outcome by Testing Planet few years back
As per a research done a few years back, the top-3 leadership skills a tester needs is primarily on
the following 3 parameters
1. Communication Skills
2. Strategy Skills
3. Analytical Skills
Top-5 Technology Focus areas for a Tester
From a technology standpoint, Key focus areas from a technology for testers include
1. Testing (and Automation) Skills for AI, Machine Learning, Deep Learning
2. Testing (and Automation) Skills for API and Micro Services Landscape
3. Automation / Testing and Programming Skills to Test in an Agile / DevOps / Rapid Delivery
Projects
4. Industry Specific Skills to Test End-to-End Functional Flows – Rapidly and Accurately
5. Skills to Test Big Data Analytics, Cognitive Computing, Behavior Driven, Test Driven Design
Skills
Do no mistake – Good manual testing is still a niche heavily pursued by many organizations – just
like mainframe skills were a couple of decades back? However, it needs to be augmented for a
future focused need. This is a shift a career manual tester may need to make to stay relevant in the
industry.
Top-5 Soft Skills for a Tester
Coming to Top-5 Soft Skills for an Intelligent Tester – needs from my perspective. I call it - 5 Cs
of Continuous Success relates to 5 most important Soft Skills needed by an Intelligent Tester. They
will be valuable regardless of the time. This is similar to the new Bugatti Chiron – a $2.6 Million car
made by Bugatti. They decided not to have any infotainment system in such an expensive car. They
wanted the car to be timeless car – like their 1957 model. Likewise these soft skills for a testers are
timeless.
1. Communication
2. Critical Thinking
3. Creativity
4. Curiosity
5. Collaboration
Without a doubt they are the most important. Not to say other skills are not important, but these 5
C are Core for a successful Career
1. Communication Skills

KALILUR RAHMAN
16
• Are you able to handle Communication from others point of view in a way they understand
(be it a 5 year old or an 85 year old or a CEO or a technical guru or a functional expert or a
designer or a developer or a peer) – Regardless of How Good, bad or ugly the testing is
going?
• Do you Understanding the business Rules
• Do you realize What is the problem –If there is a Problem
• Are you able to communicate well with the product/project managers, Developers,
Architects, Designers, Business Analysts, CIOs and even CEOs about what you do and
found?
• Do you have skills to communicate to different layers like the intensity and content for
different layer of onions differ?
2. Critical Thinking skills
As a tester are you able to think quality by focusing on how a thing can be broken, what all could
go wrong?

17
With a belief in XYZ we trust but everything else needs to be proven and thinking critically on
what the problem is, how it can be communicated, why is it failing, when and how is it failing and
where? Who could fix it or face an issue? The questions are in numerous. Think critically is
important for everyone but for a tester is absolutely important.
3. Creativity
Do not mistake creative testing to creative accounting that led Enron, Worldcom and Satyam to
trouble. Creativity is definitely important to propel the testing world to keep pace, innovate and
reduce the cycle time. Creativity is useful across the life-cycle and is not confined to one area.
• In Design and Execution
• In Analysis
• Problem Solving
• Tooling and Automation
4. Curiosity
Take the brilliant quote by Walt Disney
We keep moving forward, opening new doors, and doing new things, because
we're curious and curiosity keeps leading us down new paths.

KALILUR RAHMAN
18
If we have curiosity to know the end at the beginning – by understanding the vision, mission and
objectives of what the customers want, what the business wants and try to propel the cause, we can
certainly deliver a better outcome.
5. Collaboration
A typical challenge in a testing team’s life is the challenges and being at cross-fire of multiple
teams. You are always at the front-line, faced with multiple difficult questions.
But like the famous Omega Sector tag-line, you (Testing Team) are the Last Line of Defense at all
times. You need to
• Know when to fight a battle.
• Always have integrity to stand-up and speak with courage
• Always collaborate to achieve the common objectives
• Have a one team mentality with a vision and a purpose

19
What is your view?

20
3. INTELLIGENT TESTING SKILLS NEEDED FOR THE NEXT
GENERATION - UPSKILL OR RETIRE
Introduction
Over the past 25 years, testing has gained prominence as a critical phase that determines the
success of any project or program. Testing as a discipline/science is approximately a US$ 65–75
billion industry with an estimate of 9 to 15 % year-on-year growth (Ovum and other sources).
Given the stature of the testing industry, a crucial ingredient to make testing work is the people
who deliver testing – testers. Like any major discipline, skills are of paramount importance in
successful testing. Testing skills are crucial in aligning the test approach with the requirements of
end users, making the outcome more relevant to the business context. Customer experience and
stakeholder management are vital.
As the velocity of change in technology and the associated delivery methods have increased over
the last 30 years, testing has kept pace. This combination of criticality (to the delivery and the
business outcome) and the pace of change make testing itself the zenith of technology delivery.
As far as technical evolution and need for advanced testing skills needed for AI, Machine
Learning, Big Data, Multi-Channel, Multi-Device Testing and DevOps driven innovation, I shall write
another article. This article focuses more on general aspect of testing skills.
Testing involves deep thinking, hard work, and meticulous planning. Testing can be quicker with
smart work in the form of continuous improvements and clever planning. A tester’s reward is the
identification of genuine, showstopper-critical defects (early in the life cycle) that results in changes
to the software, application, program, project requirements, design, or code. Testing is an
assimilation of features delivered by the requirements, design, and development phases, and
determines the product quality. In the context of quality management, testing is the cornerstone of
quality and provides motivation for quality improvements upstream in the lifecycle. Rephrasing a
JRD Tata quote, “a tester must forever strive for excellence, or even perfection, in any test, however,
small and never be satisfied with the second best result or performance”.
This article touches on the key skills a tester should have to succeed in the era of the next
generation of testing. Some of the key testing skills (this is not a comprehensive or exclusive list)
are:
• Capability areas – skills including testing knowledge, technical knowledge, analytical and
logical reasoning (ability to map Requirements for testing and ability to validate and verify
results), end-to-end business, functional and domain knowledge (this is for
specialists/specialized phases of testing and is not mandatory for all testers).
• Soft skills including clear communication (written and verbal), skills in stakeholder
management, negotiation, diplomacy, team bonding with global multi-site multi-vendor
teams, and empathy with the clients (i.e. both testing and communicating in a manner
sympathetic to the end user requirements and demands). Soft skills are very important for a
tester to be successful.
• Other skills including observation, transparency, focused drive towards success, curiosity,

21
exploration, inquisitiveness, critical analysis, creativity, ability to toggle between purism and
pragmatism, cognitive psychology with an ability to think in other people’s shoes, attention to
detail, knowledge of applied statistics, ability to detect patterns, knowledge gathering skills,
ability to withstand pressure, good memory for reconstruction.
Core Skills Needed by Testers
• First things first – a tester should be familiar with testing concepts. At a bare minimum, an
entry-level tester should have a theoretical understanding of the testing concepts, test
phases, SDLC, requirements, as well as basic industry, functional, and product knowledge.
Like any other mature discipline, testing has solid literature, tools, and standards available. A
tester can learn in an iterative structured manner based on their level of knowledge and skills
by making use of the numerous learning methods and tools available (online, classroom,
simulations) in the market. However, theoretical technical testing alone will not be sufficient,
as the testers need to develop an ability to strike a good balance between perfectionism and
pragmatism that comes with experience.
• A professional tester should have a basic understanding of the following testing terms and
concepts, depending on geography and industry. Some of the nice-to-learn topics are:
• Testing terms and concepts (TMMi, BS 7925-1, NIST 500-234 or similar industry/global
standard)
• Test types, test phases, test prioritization, risk-based testing, regression test selection,
test automation, test traceability to requirements, shift-left testing, model-based
standards, application based standards, application/product-specific functional
standards, stakeholder management, requirements integrity levels, to name but a few
• Test policies (of testing organization, client firm)
• Test strategies (IEEE 1012), approaches and documentation (IEEE 829, ISTQB)
• Software engineering standards (ISO 12207, 15026, 15288, IEEE 1028, 1044, CMMi)

KALILUR RAHMAN
22
• IEEE Computer Society –SWEBOK focusing on software testing knowledge area and
quality management, software (cognitive) ergonomics, and systems engineering
discipline.
In addition to the basic knowledge, professional testers will be required to acquire knowledge in
many areas within a program or project. These include:
• Vision, mission, goals of the program/project
• Customer experience and usability expectations
• Scope of the project/program/test phase
• Program or project test plan/test approaches
• Test phase plan (For system, domain/continuous integration/assembly/link, systems
integration testing, user acceptance testing, operational acceptance testing, usability testing,
etc.)
• Test process (generic or specific to a test phase)
• Defect management ( as per test policy or industry standard)
• Environment, data and support management, and test documentation
• Test tools and the skills to use them
• Test process and tester productivity improvement measures using tools, processes, and
templates
• Lessons learned, dos and don’ts from past experiences in the firm, project, or earlier phases
• Challenge the constraints a team faces
• Navigate and understand the goals and mission of the project/program/test type
• Collaborate meticulously with the various teams/partners/vendors involved
• Manage risks and issues proactively with the project team(s)
• Provide excellent observation, recording, and documentation
• Be a cautious judge with critical thinking and the ability to be pragmatic when needed
• Be flexible enough to adapt to challenging situations without deviating from the original test
strategy
• Handle ambiguity with a continuous learning mentality
• Take accountability for testing the relevant task in hand
• Treat clients with a ‘client first’ mentality
It is important for testers to understand the importance of their work – there are major
examples of testing-related defects causing newspaper headlines, such as an aircraft being
grounded due to battery issues, a financial institution facing IT integration issues, a floating
point bug in a processor, a space shuttle failure, an airport baggage handling system going
berserk, to name but a few.
From these categories, tests need to have three key inputs to succeed:
• Project/program/solution context
• Technical/functional knowledge
• Personal experience and knowledge gathered These three key essentials give the testers a
grip on:
• Stakeholder context
• Project understanding from a technical, functional, and business architecture perspective
• Testing methodology and good practices to follow
• Soft skills needed to succeed
• Pragmatism oriented Shortcuts and quick wins for effective testing
Once mastered, testers graduate to possess the skills (as a testing or subject matter expert) that
allow them to:

23
• Challenge the constraints a team faces
• Navigate and understand the goals and mission of the project/program/test type
• Collaborate meticulously with the various teams/partners/vendors involved
• Manage risks and issues proactively with the project team(s)
• Provide excellent observation, recording, and documentation
• be a cautious judge with critical thinking and the ability to be pragmatic when needed
• be flexible enough to adapt to challenging situations without deviating from the original test
strategy
• Handle ambiguity with a continuous learning mentality
• Take accountability for testing the relevant task in hand
• Treat clients with a ‘client first’ mentality
Requirement and Development Skills Mapped to the Tester’s Skills
Requirements determine the key testing scope of a project/program/ system/solution.
Requirements are an integral pointer to the success or failure of a project or program. A quality
business requirement will result in a high-quality development and a quality product/solution that
meets the end user’s expectations. As per industry studies, a requirement- related bug found in
production could cost between 10 and 100 times more than a defect found earlier in the life-cycle
and as much as 10 times one found during the test phase. Similarly, a design or architectural miss
can cost 25 to 100 times the initial cost4 if found in production, and approximately 10 to 15 times
more if found during the build/test phases. Based on information on the Internet and various studies,
the highest number of delivered defects is requirements-related5 and better value is delivered by
improving the efficiency of defect removal during the requirements phase. This is largely due to a
lack of a disciplined requirements management process. A disciplined, tools-managed process will
help companies and programs prevent budget overruns by reducing the cost of poor quality. The
tester needs to work with the key business analysts (BA), business transformation leads, and
designers who have the business context and knowledge and help shape the final end user
deliverables.
The quality of the requirements determines the quality of the end product and, hence, proper
scrutiny of the requirements using static testing techniques such as reviews, walkthroughs, and
inspection, visual validation (all manual), or using automated tools for requirements analysis for
compliance is a must. Currently, standard tools are available in the market to convert requirements
to test models and create test scenarios and conditions in an automated manner – covering
boundary value conditions and negative scenarios. While it is good for testers to understand the
fundamentals, it is equally important to check the output of the automated tools for consistency and
accuracy. Experienced testers who understand the business function or industry very well can carry
this out.

KALILUR RAHMAN
24
Requirements and Test Scope Management Skills
One of the key skills of the tester is to perform requirement verification checks. Some of the
checks include singularity, ambiguity, generalization, solidity, comprehensiveness, style,
extensibility, maintainability, reliability, integrity, robustness, scalability and testability. Terms and
definitions differ between various companies and automated requirements verification tools. Testers
need to be familiar with the terminologies to perform relevant requirement verification checks during
the static testing phase or for test design inputs.
For test requirements, one of the key Six-Sigma concepts – SMART – can be used for any testing
task or requirement. A SMART requirement is a good requirement. A test designer should have
certain skills to define a test requirement in a SMART manner. Test requirements need to be very
specific about a particular task or an activity, results or output should be Measurable, and Attainable
with Relevancy and a Time-bound response.
Additionally, a good test designer will be familiar with leading test design techniques such as
Equivalence partitioning, Logical Combination, Taguchi Method, Orthogonal Array, State Transition
Model, Karnaugh Maps(K-Map), Risk-Based Testing, Exploratory, Fail-First to convert the
requirements to a solid set of requirements, resulting in high quality test coverage with an optimum
number of test cases. This will result in a high-quality product at optimal cost. Testing stops and it
never finishes. As the number of tests, it is possible to execute is large. Hence, a tester must
understand the client’s accepted level of risk under different client circumstances will drive different
testing behaviors, and this is where having industry experience for similar clients can lead to
behaviors resulting in higher value for the client.
Smarter test scope management can be achieved by using industry- leading standard test
requirements and test management tools. Key requirements can be understood implicitly by testers
by keeping constantly aware of industry trends and having the ability to incorporate new testing
methods (innovative and intelligent testing). Building such a proactive team of testers will help build
a very high performing test team.
Testing Specialization
Testing Generalist
• To become a generalist, it would be good for a communications industry aligned tester to be

25
familiar with eTOM (enhanced telecoms operations map) and key standards for the type of
operator (e.g. 3G/4G for a wireless, PSTN/xDSL for a fixed line, OTT, DOCSIS standard for a
cable MSO, etc.). For Pharma - Focus on GxP Methodologies, How Research &
Development, Manufacturing or Commercial Operations work etc.
Testing Specialist
• To becoming a specialist, it would be good for a tester to focus on a particular type of testing
and have in-depth knowledge of and capability in the key tools needed to perform testing.
For example, to become a good performance tester, it is good for testers to have expertise in
some/ all of the concepts.
• Requirements gathering and analysis of non-functional performance measures
• Performance modelling, and design and architecture
• Workload, performance architecture and capacity models
• Virtualization and extrapolation using performance technologies, such as low-latency tools,
in-memory, solid-state devices, network tuning, data analytics, and caching using big-data
• Performance test execution and statistics capture
• Performance tuning measures, such as code and hardware
• Performance testing using tools and automation
• Skills to use probes/tools to measure and optimize performance
It is common to see deeply skilled specialists in one particular area or tool. Hence specialization
is a good thing to have if a tester chooses the path of the depth of expertise.
Options for Testing Career Path
Test Career Trajectory
• Testing generalist track
 Tester → test module lead (design/execution) → test lead → work stream test
manager → project test manager → program test manager →portfolio test director
→ testing practice lead
• Test design/process specialist track
 Tester → test design/process lead → test architect lead → work stream test manager
→ project test manager → program test manager →portfolio test director → testing
practice lead
• Test data/support generalization track
 Test data team member → test data module lead → test lead → test support
manager → project test support manager → project test manager → program test
manager → portfolio test director → testing practice lead
• Performance test specialization track
 Performance test specialist → performance work modeling lead → performance test
lead → performance test manager → performance test expert → program non-
functional/performance test manager → portfolio test director → testing practice lead
• Sample Functional Specialization: Telecoms billing specialization track
 Telecoms mediation tester → telecoms bill run module lead → billing test module
lead → billing test lead → billing testing work stream lead/ manager → program
billing test manager → billing test portfolio director → testing practice lead
Next Generation Testing Industry – Skills Needed
Taking an example of the telecommunications industry (communications) has grown
phenomenally over the past 20 years. The number of innovations and advancements made by the
communications industry is second to none and is one of the engines for growth in the economy,
the IT industry, and testing in general. With such phenomenal growth, numerous challenges come

KALILUR RAHMAN
26
onboard.
From a testing skills standpoint, the challenges posed by this growth include the need to keep the
various types of testing requirements for different communications industry projects and roll-outs.
The telecommunications industry is vast and there are different functional domains. Testers tend to
be specialists in one or more of these areas, but will seldom master everything because the
technologies, frameworks, and standards are numerous.
Testing Career Progression Graph
Testing as a career has a lot of options from a growth perspective. Some choices for a tester to
progress along the test management path as a generalist areas seen in the figure above.
In order to progress along a testing career track as a specialist, a tester can develop his career
as a specialist in one of the following:
▪ Test data subject matter expert (SME), test architect, test metrics SME, test environment
discovery, test environment delivery, defect management, war-room/problem manager, incident
manager, test automation, test support tools, test requirements, report management, performance
testing, security or penetration testing, Web/ usability testing, compatibility and accessibility testing,
operational acceptance testing (OAT).
A tester can complete training and become a certified specialist in testing from standards
organizations such as BCS, QAI, ISTQB, IIST, ASQ, ISEB, CSTM, CMST, PTCRB, and GCF. There is
specialization certification for tools such as HP Performance Centre, Selenium, Tricentis Tosca,
Worksoft Certify, Panaya, SOAP, SilkTest, Rational Quality Manager – Performance Tester, and
Rational Robot, to name but a few.
A tester should have a probing mindset and should be capable of dis- covering the features of a
product, process, or service and understand what the customers want (from the requirements), how

27
the product should work ( requirements and design), design the tests using various design
techniques, select tests with a fail-first mentality with the aim of identifying defects in order to fix
them, run the tests (using tools or manually), observe the test outcome, and validate the results
against the expected behavior.
Conclusion
Testing as a career has become a mainstay and is the pinnacle of success for large-scale
product development or program delivery. Testing is a key contributor to the development of the IT
industry and the phenomenal growth in technology which has taken place over the past three
decades. Testing as a horizontal (across the SDLC life-cycle) or a vertical (industry-specific) is a
hotbed of innovation, creativity, and productivity improvement in terms of processes (with processes
such as TMMi), products (automation tools, intelligent testing tools, productivity tools), and people
(thought leaders and innovators).
Testers have the opportunity to become generalist test experts with good industry knowledge of
the testing life cycle, or a highly skilled expert or specialist in a particular test type, test phase, test
activity, tool, or industry, or an expert in test delivery and management before taking charge of a test
centre or test practice.
Especially in communications industry related testing:
▪ there will be growth in some markets and maturity in others. Both will drive rapid technical and
business changes, with the business and operating model proving to be very dynamic.
▪ In the next 5 to 10 years, the outlook is for continued change as the market for data further
drives mobile towards being a commodity market and seamless, always on the move with the
Internet of Things becoming a de-facto lifestyle.
▪ In content delivery, the changes in platform/delivery method mean increasingly interactive and
mobile applications will continue to stretch testing methods both for media providers and telecoms
companies.
It will be a nice challenge for the testing experts and testing services firms to keep up with the
phenomenal growth in the industry and the variations and permutations for service options,
commercial models, career growth, innovation, and transformation in various industries. Testing as a
tipping point has been and will be a major contributor to the success of the IT industry. If there is
one area where skills can be an evergreen option with continuous opportunities for learning and
growth, it is testing. For one to be successful in the next generation of testing, it is important to keep
learning and keep innovating. A career in testing comes with infinite options – akin to a famous
quote by James Bach, which states: “Testing is potentially an infinite process”.

28
4. TOP TIPS - HOW TO ESTABLISH A SUCCESSFUL TCOE / QCOE
(TESTING / QUALITY CENTRE OF EXCELLENCE)
"Before you start some work, always ask yourself three questions - WHY am I doing it, WHAT the
results might be and WILL I be successful. Only when you think deeply and find satisfactory answers
to these questions, go ahead." - Chanakya
A brilliant quote by Chanakya (who is definitely a peer equivalent of Lao Tzu or Machiavelli for
political/strategic sagacity relevance, or beyond) from Artha Shastra is relevant and is accurate, for
large-scale transformation programs.
I've had opportunities to work on some complex transformation programs and thought of sharing the
wisdom I gathered via my experiences.
There are plenty of examples of the pros and cons of having a centralized testing unit or a services
unit. For those proposing to start a Quality / Testing Centre of Excellence for your customer needs
to be understood and addressed from your customer's point of you. Drawing from the quote from
Chanakya - You need to understand what your customer is currently doing and have an
understanding of the Business
What is the customer's position, business, finance, competition, roadmap, product and
services vision etc.?
What is the maturity of the delivery framework? This may vary for different customers and this
is very important to be understood before we can recommend. Some of your customers may
have very solid process and methodology in place - could be less agile
• Some customers could have best of both worlds with the limited process with a sense of
urgency
• Some may have a sense of urgency and get things done with a lesser focus on quality with a
fix-it-in-live mentality
• Some may have a tremendous focus on quality but do not have enough resources, know-
how in-house to do it
• Some may have a stable delivery process but are in expansion mode with a focus to
streamline operations with best-in-class service and reduced cost of operations
• Etc.
What is the Roadmap planned?
• What does your customer want to achieve in 3 years, 5 years, and 10 years?

29
• How does the QCoE/TCoE fit into the entire mix?
• What are the Cost, Quality, Maturity, Agility levels for the customers in terms of the
roadmap?
• Challenges your customers want to address?
How and Where to Start?
How do you start well? There are plenty of new age challenges posed by the emergence of
technology and how things are eventuating in digital technology. The need to have a separate
testing unit becomes redundant in some areas. DevOps makes things a bit more iffy from a testing
standpoint. How do we start? Aristotle once famously quoted
"Well Begun is half done".
This profound statement is an eternal truth in pretty much everything we do. Once you understand
what your customer wants to achieve, you are on a good track to recommend how you could help
them. You can perform an assessment - using your tools/processes or generic frameworks such
as TMMi, TMap etc. and understand where they fit and address what you can address.
You also need to understand that having good sales personnel may get you the deal over your
competitors. However, unless you have levers to address an "apples vs. oranges" scenario between
due-diligence and delivery phase to address commercial challenges, it will be a difficult challenge
from day-1, to begin with. Plan with the end in mind. Plan and Prices the Solution to deliver what you
promise to the client to reach a win-win situation for both you and clients.
Some of the typical pitfalls you need to avoid include are

31
Now coming to the TIPS to avoid some of these challenges
Pitfall / Suggestion - 1
• Pitfall: Incorrect estimation to meet customer's budget limits for a contract winning
price
• Impacts: Customer Impact due to poor quality / Customer experience impact / Loss of
Credibility
• Suggestions / Mitigation Plans: Perform a good due-diligence plan / Add levers for scope
creeps / Get buy-in from customers on what is getting priced and make a collaborative
decision on deviation / Stay away from bidding in a cost competition to win the deal at any
cost. This is a recipe for disaster.
• Pitfall: Poor Stakeholder Management
• Impacts: Customer experience impact / Loss of Credibility / Project Delivery Risks /
Impact during transition resulting in Quality Impact
• Suggestions / Mitigation Plans: Identify A-Team for your deal / Map the A-Team to
Customer team / Have a customer experience mapping plan / Have a smooth
stakeholder management plan leveraging power maps and experience maps./ Have regular
connects with stakeholders / Institute a Governance meeting that works for both sides
• Pitfall: Lack of Governance
• Impacts: Project Delivery Risks / Lack of Ownership / Delays due to lack of clarity
• Suggestions / Mitigation Plans: Establish a solid Governance framework at operational,
tactical, strategic and at thought leadership level / Make joint decisions with the customers /
Make the customers a key influencer, decision maker for innovative processes,
improvements in TCoE
• Pitfall: Incorrect mapping of team members - Halo effect resulting in wrong roles for team
members
• Impacts: Project Delivery Risks / Lack of Ownership / Revolving door team
structure with volatile team structure – resulting in poor customer experience
• Suggestions / Mitigation Plans: A Critical factor for the success of a TCoE is the
identification of right resources in right roles with the right attitude. / Having a team of alpha-
performers may not work very well. Neither is a team formed with "Halo effect assumption”. /
You need to evaluate the resources for fitment and ensure right role/right attitude/right
person/right level right mapping is done for the team with good resource capability leveling. /
This is normally ignored by vendors when forming a team with whatever available resources
they have in bench or coming from a Greenfield training program. Unless the intake is of
high quality, you cannot expect solid output as well
• Pitfall: Incorrect and Audacious Assumptions
• Impacts: Customer experience impact / Loss of Credibility / Project Delivery Risks /
Impact during transition resulting in Quality Impact / Ambitious Goals/Promises that can’t be
met or delivered
• Suggestions / Mitigation Plans: This is caused due to Incorrect Solutions / Pricing to win
and cutting the corners / Lack of understanding about the customer landscape / incorrect
understanding of the internal delivery mechanisms
• Pitfall: Fragmented leadership
• Impacts: Teaming /Disconnect and politicking / Customer Visible disconnects within the

KALILUR RAHMAN
32
team / Morale impacts / Possibility of Revolving Door for critical resources / Customer
Experience Impact / Loss of Credibility
• Suggestions / Mitigation Plans: Involve Key (Named) individuals from the solution stage
itself / Conduct teaming exercises from early on / Map the customers based on social
styles/power maps / Tag key leads to Customer leads – Have a solid Customer Experience /
Stakeholder Management Plan / Keep the key leads accountable / Have regular connects till
the QCoE/TCoE becomes a BAU operation
• Pitfall: Silo Delivery Units
• Impacts: Teaming /Disconnect and politicking / Customer Visible disconnect / Possible
“Us vs. Them” issues / Possibility of disconnect for leading practices & common process
tools / Customer Experience Impact / Lack of Consistency
• Suggestions / Mitigation Plans: Have a good and working Matrix Org structure where
possible /Build a cross unit SME group and ensure common methodology / regular connect
sessions for knowledge sharing – QCoE/TCoE/Project Level Connects/Teaming Events
/Group units of commonality under one leadership with Cross-Unit management inputs from
a specialist SME group
• Pitfall: Lack of a Service Management Framework
• Impacts: Loss of Credibility about Professionalism of the team/resources
/ Customer Experience Impact / “Re-inventing” the wheel / Loss of Productivity / Team
Morale Impacts
• Suggestions / Mitigation Plans: Establish a standard Operating Model / Establish a
Standard Service Management Framework / Build Common
Process/Tools/Templates/Methods very early in QCoE Setup / Ensure standardization is
made mandatory / Train the QCoE Team / Run a structured Continuous improvements
Programme
• Pitfall: Urgent Implementation of tools/processes without a clear vision
• Impacts: Business Benefit Realization Impact / Loss of Customer Trust / Cost Impacts
/ Impact to reuse / productivity targets
• Suggestions / Mitigation Plans: Perform Tools Evaluation Process / Involve customers
for a Fair Assessment / Decide on Tools delivering Benefits and continuous value / Roll
out an Implementation Plan / Implement Governance Approach / continuously assess and
make business savvy decisions
• Pitfall: Lack of Clearly defined Roles and Responsibilities (RACI) for the team
• Impacts: Customer frustration due to / Wild-goose chase / Merry-go-round for issues
resolution / Lack of clear ownership
• Suggestions / Mitigation Plans: Define the Roles and Responsibility for the entire team
/ Define Organization Charts, Deliverables Responsibility Matrix, stakeholder Mapping
Matrix etc. in the beginning
• Pitfall: Lack of Team Engagement / Morale due to multiple reasons
• Impacts: Delivery Impact / Customer Experience Impact / Resource
Turnover
• Suggestions / Mitigation Plans: Transition is a challenging process and a solid team
engagement process needs to be put in place even before the start of transition or
enablement of a TCoE

33
• Pitfall: Incorrect Customer Expectation Setting
• Impacts: Customer Experience Impact / Loss of Credibility
• Suggestions / Mitigation Plans: Alignment of Sales / Delivery / Capability / Customer
Engagement teams throughout the lifecycle / Action plan to address disagreements in
expectations between customer and internal teams – Price Points, Transformation,
Resourcing, Capabilities, Metrics and Schedule
• Pitfall: Disconnect with Business Stakeholders
• Impacts: QCoE/TCoE team getting in the line of crossfire between two internal customer
teams / Possibility of one group being unhappy at the expense of another
• Suggestions / Mitigation Plans: Ensure business stakeholders are involved in decision
making, QCoE/TCoE Roadmap, and transformation objectives / Share transformation
roadmap, delivery roadmap and business benefits cases to both IT and Business
stakeholders
Pitfall / Suggestion - 14 & 15
• Pitfall: "US vs. THEM” mentality with 3rd Party teams, Delivery partners and teams
located in multiple geographies & Lack of “One team mentality” to customers
• Impacts: Impact on delivery / visible politicking to customers resulting in perception
about professionalism of teams involved
• Suggestions / Mitigation Plans: Ensure – across vendor alignment meetings are
conducted by customers to ensure all parties are aligned / Rekindle the “One customer”
mentality in all cross-vendor meetings / Form a “Delivery partnership” with the vendors for
the relationship timeline without a conflict of interests
• Pitfall: What is the right dilemma? - "Contract First Vs Customer First" dilemma /
"Employee First Vs Customer First" dilemma / "Contract First Vs Employee First" Dilemma
• Impacts: Customer escalations / Perception about the firm being price centric or being
insensitive to customers / Potential risk of customer taking advantages of “free lunches”
• Suggestions / Mitigation Plans: Establish right behaviors for the team / Enforce
“Customer First” mentality with clear boundaries for deviations in scope, cost, and
contractual deviations / Establish clear responsibility, accountability, permissible levers
across levels to deal the situation
• Pitfall: Changes in customer Organizational Leadership
• Impacts: Lack of direction / Changes in Scope / Contractual impact
• Suggestions / Mitigation Plans: Brainstorm and ensure backup options are planned /
Have a “leadership working style specific” engagement plan / Ensure regularly connects with
the stakeholders
• Pitfall: Changes in customer business priorities
• Impacts: Lack of direction / Changes in Scope / Contractual impact
• Suggestions / Mitigation Plans: Look for creative options to support customer’s business
priorities / Look at ways to help the customer’s delivery/achieve what they want / Renegotiate
and look for incremental business (where possible)!
If we do not focus well on the objectives, a famous situation similar to "Project Management Tire -
Swing Cartoon" may eventuate. A historic perspective of this variant is very much applicable to a
TCoE/QCoE setup as well. http://www.simpleapply.com/blog/how-it-projects-fail/

KALILUR RAHMAN
34
Walt Disney once famously quoted “You can dream, create, design and build the most wonderful
place in the world… but it requires people to make the dream a reality…” -
Building a dream QCoE/TCoE for your customers could be addressed implementing the two brilliant
quotes from Chanakya and Disney. The key to success is ensuring that you have the right people
who have the knowledge, understanding, creativity, and the right tools/attitude/talent to design and
build it for you the way you want it. Nothing more, Nothing Less!
"Learn the Rules like a Pro, so that you can break them like an Artist"– Pablo Picasso
"Creativity is the one of the last remaining legal ways of gaining an unfair advantage over the
competition" – Ed McCabe

35
Impact of Digital Innovations in
Testing

KALILUR RAHMAN
36
5. ARTIFICIAL INTELLIGENCE (AI) IS THE NEW ELECTRICITY! IS
THERE ANYTHING ARTIFICIAL OR INTELLIGENT ABOUT IT?
"AI is the new ELECTRICITY" - Andrew Ng - AI Guru, Founder Coursera, Ex-
Baidu
Ever since the term Artificial Intelligence - AI was coined by John McCarthy in 1956, a lot of
innovations have taken place. With the incremental innovations in the landscape over last 60 years
and explosive growth in the last decade, there is a varied level of understanding of the concept.
However, the understanding is varied across the spectrum. A Tautological simile could be the
concept of "6 Blind-folded men explaining what they feel about an elephant based on what the feel"

37
However, a brilliant prediction was made by one of the greatest Computer Science Genius - Alan
Turing.
Humans have grown leaps and bounds since the initial prediction by Turing. On how AI would
transcend into Singularity, is given in a brilliant intro as a trailer in TRANSCENDENCE
For 130,000 years, our capacity to reason has remained unchanged. The combined intellect of
the neuroscientists, mathematicians, and hackers in this auditorium pales in comparison to the most
basic A.I. Once online, a sentient machine will quickly overcome the limits of biology. And in a short
time, its analytic power will become greater than the collective intelligence of every person born in
the history of the world. So imagine such an entity with a full range of human emotion. Even self-
awareness. Some scientists refer to this as "the Singularity." I call it "Transcendence."
However, NVidia has a brilliant diagram depicting the growth of AI in 3 distinct wrappers - AI,
Machine Learning and Deep Learning and how it has progressed in the past 60 years.

KALILUR RAHMAN
38
AI has already bounced far ahead in terms of innovations and has been beating the best of
humankind in CHESS, Jeopardy! (A Top US Quiz Game) and the most complex of strategy game -
Go! They all failed in various ways to show that they to err on certain basic moves or questions
resulting in a human victory.
It proves the point that AI can certainly leap forward exorbitantly and drive us towards Singularity.
So - How far can AI take us forward? Will it be like the following scene in the movie Matrix?
https://www.youtube.com/watch?v=zatL4uFRpC0
Can we download instructions in real-time and do things like a pro? May be yes (after our lifetime
perhaps...) or May be not! But how the technology has grown, it is very much feasible.
From a testing stand point, How about Fast Testing – Hey – Can I test this brand new “thing” in 2
minutes? Is this humanly possible? Well - if we can download and fly a helicopter in 30 seconds,
why not?

39
We have AI Systems that can write algorithms itself, write programs to reduce development time,
invent a new language for processing (like Facebook BOTs), trainable AI BOTs that could be as
fragile as a human with biases (like the Microsoft Twitter BOT that got shut down post a racial error)
https://www.youtube.com/watch?v=Bsh5rh78y0E
While many of these tools have been in existence at a much lower scale and capacity such as
Integrated Development Environments (IDEs) helping write faster code, CASE Tools, code
generators etc., the increasing intelligence and potential of these tools will take us to the next level.
Recently a YouTube and WhatsApp video of a Japanese Humanoid Robot crossing the street like
a normal human - with typical human emotions and moves created a big stir and went viral. So,
practical use of these technologies are very much real and are in use in a variety of ways!
Other good links I want to share are
• Real time Translator of Things (with a photo of the thing using a camera)
https://oxism.com/thing-translator/
• AI Experiments by Google - https://g.co/aiexperiments
• World’s Largest "Neural Network"-driven, Deep learning site for doodle/drawing collection -
https://quickdraw.withgoogle.com/
• Use of Deep Learning collated via Quick Draw for rapid drawings
https://www.autodraw.com/
• Identification of emotions via text typed - https://deepmoji.mit.edu/#
• Morality/Ethics detection in challenging circumstances - such as Self-driving cars, robots,
drones and other tools aided by technology or AI - http://moralmachine.mit.edu/
Again, with these, we can certainly say that AI is here to stay and it may become the "electricity"
of the century this point on. Obviously, there may be a digital disparity - just like places without
electricity, when it comes to adoption of AI.
AI Startups are all over, all the industries, technologies and what not and is a hot bed for VC
activities in firms delivering AI Driven Bots, Automobile, Computer Vision, Core / Functional AI ,
Commerce, IIoT/IOT, Healthcare, Fintech, Robotics, Analytics, Cyber Security, Sales & Marketing.
“Be a yardstick of QUALITY. Some people aren't used to an
environment where EXCELLENCE is expected” - Steve Jobs
AI Uses in Testing
Coming to Testing, How is AI helping Testing? How can we test better with AI? How can we test
AI systems better?
Take these stats for a moment
 As per information available on the Internet, Google - refactors code by 50% each
month and they have a single repository. Going by a rough estimate of about 2 Billion
Lines of Code, they refactor about 1 Billion LOC. How is it humanly possible to test
the code changes in a month? The sheer volume is definitely a lot more than
combined code developed and maintained by all the biggest firms combined over a
few decades, perhaps. It is also outlined that the teams have permissions to use
anywhere from 79% to 95% test coverage to validate the quality of the code. No one
likes to boil the ocean right? It would become a vaporware if you boil the ocean.
 Netflix has over 5 Billion+ API Calls per Day (and increasing daily). How will you test
an application or a micro-services infrastructure with exponential growth in volume in
a test environment?

KALILUR RAHMAN
40
 About 52% of Fortune 500 companies have disappeared from the list & Average S&P
500 span reduces from 61 Years to 17 Years in 60 years. Soon the average could be
reduced to 4-5 Years. All the top-5 firms in terms of Market Capitalization are
technology firms and all OIL and Retail Behemoths have been outsmarted in the past
10 years.
 As per a Gartner report, in 2020, 100 million consumers will shop via augmented
reality and 30% web browsing will be done without a screen; and that by 2022 - $1
Trillion a year to be saved through IoT
Given that the world is moving away from year/quarter/month long testing process into minutes
and hours based testing mode, a Tricentis shared view explains it brilliantly.
The answer to the questions is outlined in two words - "Intelligent Machines" or "Artificial
Intelligence" - All the top technology firms use these extensively to propel their hegemony to the
next level in the industries they serve in.

41
AI will help in many ways to reduce the pain of huge coverage gap arising due to two key drivers
- Increasing Complexity and Reduction in Time Available to Test and the ever increasing Coverage
Gap.
AI can also help in addressing key questions the testing leaders face. (i.e.) How do I balance the
Time/Cost/Scope/Quality tetrahedron for my project/product/app and Resources/Test coverage/Test
Effort/Costs/#Test Cases? How do I run an optimal test coverage for Minimum Viable Product with
acceptable quality by the stakeholders?
AI has some solutions readily available that can be picked up from the progress done so far. A
Brilliant article from Futurism site catches the essence of some of the artificial intelligence (AI)
algorithms currently helping machines think. (Original Credit: CIO Journal / Narrative Science)

KALILUR RAHMAN
42
If you take it further, following are some of the concepts applied for leveraging all these
techniques such as Reinforcement Learning / Passive Reinforcement /Regression Algorithms /
Linear Regression / Gaussian Processes / Supervised Learning / Neural Networks / Unsupervised
Learning / Independent Component Analysis / Principle Component Analysis / Natural Language
Understanding / Morphological, , Semantic, syntactic , Discourse analysis / Natural Language
Generation / Deep planning / Syntactic generation / Clustering Algorithms / K-Means Clustering /
KPCA – Kernel Analysis / Statistical Algorithms / Support Vector Machines / K-Nearest Neighbor
algorithm / Native Bayes Classifier Method / Maximum Entropy Classifiers / Other Techniques /
Spanning Trees and Graphs / Neural Network – Multi-Level Perceptron’s / Pattern Recognition
Techniques / Statistical , Syntactic approaches / Template Matching / Neural Networks / Labeling /
Hidden Markov Models / Maximum Entropy MM / Conditional Random Fields / Parsing Algorithms.
Out Of these if you just take the following 5 approaches, you can accomplish some of the
advanced testing techniques for productivity. They are
1. Monte Carlo Simulation Tree
2. Support Vector Machines
3. Search Trees / Deep Neural Networks
4. Bayesian Classification and Clustering
5. Probabilistic Models

43
Using these, we can accomplish the following advanced techniques such as Automated Defect
Detection / Automated Exploratory Testing / Test Coverage Heat map / Self-Healing Automation /
Predictive Modeling for Test Design Defects / Self Adjusting Regression / Pattern Recognition for
various Metrics and Trends / Risk & Coverage Optimization / Diagnostic, Prescriptive and Predictive
Analysis / Deep Learning of systems under Test and forecasting / Root-Cause Analysis / Sentiment
Analysis of User behavior for enhancements.
One of the ways we can build an AI model for testing is by feeding inputs for learning and
enhancement of the models, building algorithms for Deep Learning and enhance the engine on an
ongoing basis.
Case Study of AI Usage in Testing
A brilliant case study for AI usage in testing can be seen in the YouTube Video by KING.com
https://www.youtube.com/watch?v=wHlD99vDy0s
The Key take away from the session is the following.
• Deep Artificial Neural Network

KALILUR RAHMAN
44
• Use of Monte Carlo Tree Simulation
• Use of Advanced Automation by BOTS
• Use of BOTS to perform Testing
• Use of AI engine for continuous Feedback Loop
• Constant Machine learning model updates.
• Hybrid Test team (150-200+ Testers) with unique skills covering test automation, data
science programming
• Use of Data Scientists for Domain Knowledge, Fun (using historic info and user behavior,
Game Balancing)
• Regular Crash Testing, Performance Testing, Regression Testing
• Regular Maintenance of AI Bot for Testing
Given the popularity of Candy Crush Saga, how they leveraged AI to do testing efficiently, Code
Refactoring and quality engineering by behemoths such as Google, Amazon, Facebook etc., AI will
enter mainstay of the next wave of digital revolution.
CONCLUSION
Some closing thoughts on what we can focus on would be the following
• Let AI do Exhaustive Testing - Such as non-creative, non-value adding, effort intensive,
repeatable automation and data creation, predictive exploratory testing etc.
• Leave Qualitative Aspects to Human Creativity - as it will be necessary for continuous
innovation by humans.
• Be Creative and Business/Customer Centrist in all aspects you are planning. Instead of
joining the bandwagon with a "me-too" quip, see the relevance, usefulness and long-term
business and customer centrist gains before taking the leap - leveraging the learning by
other implementations
• Automate / BOTomate early for getting returns from early phases itself.
Having said that, there are two sides of the coin. There is still a lot of nay-sayers for AI and given
Testing comes at the end, there could be even more skepticism. However, only way to innovate is to
find ways to automate and propel your capabilities by leveraging the opportunities AI gives us. At
the end of the day, it will be Augmented Intelligence, Adaptive Intelligence, and Automated
Intelligence that will propel human intelligence forward - more than anything else. It will be a great
time ahead. Whether it would be an "Eye (AI) Wash" as skeptics say or an "I wish" from them for
starting late on the journey, only time will tell. It is a matter of when and how long, instead of an If.

45
6. IMPACTS OF DEVOPS ON TESTING
I always come across a standard question from testers in any DevOps related presentation. “Will
DevOps remove the need for Testers?” A simple answer is “No”. DevOps is not going to replace
Testing as a discipline. Firms implementing DevOps have a vision for better product quality
workable by brilliant Testing.
DevOps is an enabler for the following.
• Reduction in IT Budgets
• Efficient end-to-end delivery
• Lean delivery framework
• Working end-to-end automation framework
Successful businesses today are design thinking driven. Businesses have a strong sense of
urgency, aided by innovative software engineering practices. Businesses today need to have a
working continuous delivery practice in place. Continuous delivery also mandates better product
quality and zero touch end-to-end delivery. Automaton is key to success across
DevOps processes. Delivery by co-located, integrated teams using innovative tools
prevents collaboration and communication issues.
Manual errors, Stove-piped communication issues get sorted with a good DevOps Governance
model.
DevOps mandates “Fail Fast, Fail Often” driven by “Test First” concept. This means that testing is
a continuous process and more extensive in nature. Testing allows for cohesiveness between
various teams, automation of everything possible. Testing also allows for continuous improvement in
test process delivered via iterations. Intelligent test automation, improved regression, continuous
test augmentation are some benefits delivered by DevOps.
• Key benefit delivered by DevOps is the cost reduction in end-to-end delivery. Businesses
achieve cost reduction by Automation, manual error reduction and reduction of duplicate
effort.
• Continuous delivery enables streamlined governance and reduces time-to-market.
• With fail-first, fail-fast, fail-often philosophy, shift-left testing is a reality. With early
defect identification, product quality increases. With increased automation,
continuous testing is a feasibility.
• With co-location, productivity of DevOps teams (design, test, operations) improves.
Implementing DevOps could be tricky in organizations having monolithic systems and processes.
Culture change is critical for successful initiation of DevOps and Continuous Delivery.
From a testing standpoint, DevOps addresses the following key testing challenges.
• Test Environment – Virtualization, Automated Deployment, Automated build
and configuration management, data refresh and maintenance becomes simple
• Continuous Integration, Automation and Verification – Test First centric test driven
development, behavior driven development concepts allow test processes to improve in a
DevOps Model
As per a report published in 2002 by NIST – Cost of Poor Quality (in US) was $59.5 Billion per
year. Cost of poor quality is higher in large organizations running legacy infrastructure. By some
industry guesstimates average cost of a production bug is about $7000. Customer experience and
agility for quick delivery takes a hit if the organizations aren’t quicker. DevOps becomes a positive
catalyst here.

KALILUR RAHMAN
46
Some skeptics may say “DevOps looks easy on paper but difficult to put in place”. Any innovative
change is difficult to implement at first. It takes discipline and drive to run the operations watertight
to achieve the goals.
Some key factors to implement for Successful DevOps are
• Cohesive, One Team mentality between teams
• Implementation and Use of Tools that take away manual effort
• Continuous Delivery Phases
• Superior, Automated Configuration Management Approach
• Continuous Improvements across phases
• Continuous Automation
• Continuous Integration
• Continuous Testing
• Continuous Monitoring
Use of tools is key to success of a DevOps program. Plethora of tools are available to support
automation. Tools are available for Environment Build, System Configuration, Application Build,
Database Setup, Artifact/Document Generation, Testing, Requirement Management, Collaboration,
Production Monitoring, and Reporting across DevOps activities.
Like Agile Delivery Programs DevOps needs a strong Team Cohesion to be successful. A
DevOps team need to have a Product Life-cycle Management mindset as well. This also means that
team needs to unlearn standard SDLC processes. This means, DevOps team should have a clear
vision. Vision to Ideate, Implement, and improvise and retire the product. An integrated DevOps
team with clear vision will make DevOps projects successful.
Conclusion:
DevOps testing needs to address the need of Business and Technology needs. Automation plays
a major role to achieve this aim. DevOps in principle has a goal for better product quality albeit the
need for faster delivery. Terms such as “Shift-Left”, “Fail-First, Fail-Fast, Fail-Often” will be
regular vocabulary in DevOps. DevOps Test team comprises of Cross Functional Specialists and
Testing Purists. A Performing Testing Practice for DevOps will take some time. Testing as a practice
has matured over the past two decades and is well equipped to support DevOps. Use of automation
and productivity enhancement tools helps testing move faster.
Successful DevOps implementation in large firms with legacy processes and technology is a
challenge. A big culture change is the only way to achieve success in such firms. It will be a hybrid
DevOps Model as purist implementation will be a challenge.

49
7. HOW TO RUN EFFICIENT API TESTING FOR IOT, WEB AND
MOBILE APP INTERFACES?
Technology landscape is growing exponentially in the Business-to-Consumer Segment and in
Machine-to-Machine and Internet of Things. While IOT has some roads to cross in terms of
Machines-to-Consumer (M2C) market, it is definitely going to get a lot of mileage in Machine-to-
Enterprise or IIOT (Industrial Internet of Things) segment. One of the key enabler for this expansive
growth is the advances made in Application Integration and underlying API growth (Application
Programming Interface). As per a Gartner study done, Forecast for API testing growth is nearly 3
folds the growth of automated testing, for the next 5 years.
If you take a look at Google Trends for IOT, REST and SOAP searches from 2004, you will see that
REST and IOT are getting a lot of prominence. SOAP is still a choice of enterprise application
integration (probably for not long) due to ease of product integration albeit the complexity for tightly
controlled one-to-one interfaces. Whereas, REST is the choice for new age architects, designers and
developers for the simplicity it brings to the table. A report published by InfoQ has validated this fact
in 2011 itself. The stats and trends are giving a lead to REST as the thing to be till it will be replaced
by its successor champion.
The FUTURE always comes too FAST and in the wrong ORDER – Alvin Toffler
Forecast for API testing growth is nearly 3 folds the growth of automated testing, for the next 5
years
One nice Infographic to look at for some of the API design and trends is published by Layer7, a
subsidiary of CA who also own market leading tool for service virtualization (CA LISA).
One of the key aspects of a successful API testing is to ensure that the response times of the API for
various aspects of integrations (such as synchronous or asynchronous, real-time request-response,
CRUD (Create, Read, Update, Delete) style Operations, Bulk Transactions). As per a study done by
Nielson Norman Group (http://www.nngroup.com/articles/website-response-times/ ), the key aspect
of a successful customer experience is the speed with which responses are received in a website or
an API, the usability of an application from a varying Gaze Plot, different expectations to be
supported for various users.
Taking the website tautology to a mobile application or to a “Thing/Device” in IOT Parlance,
response times do matter. If we are talking about trillion devices communicating over APIs, what will
happen if there is an API traffic jam? Wouldn’t that be like Internet going offline for days together?
Wouldn’t the businesses, governments come to a stand-still? While on this subject, yes, of course,
security is of prima-facie importance in API development and testing, especially at the edge devices
connected and accessible by users. Security and Usability are the primary concerns of most API
developers as per few researches carried out by some leaders in this area.
“It’s not the BIG that eats SMALL, It’s the FAST that eats SLOW”
There are plenty of good tools available in the market that helps users test APIs effectively – be it
Open Source or Commercial. Some of the good ones to try are SMARTBEAR SOAPUI + SWAGGER
+ ALERTSITE, RUNSCOPE, Rest Assured, PostMan, JMeter.
Let’s come to the point about how can we run a successful test approach for IOT or a Mobile
Application with Billions of Transaction per second or a website handling millions of transactions per
second. How will you ensure that these work well together?

KALILUR RAHMAN
50
Obviously, you need to go for an approach that may be radical or unique to your organizational
architecture/infrastructure and how you’ve designed your APIs, There are standards built that
recommends suitable architectural principles to be followed such as MQTT for IOT (http://mqtt.org)
supported by OASIS framework. Similarly, global standards will emerge with certain native
implementations of the framework for efficiency, accuracy and usability. API testing is much more
than error code checking and data rendering in UIs.
A nice article covered in API-Hierarchy of Needs covers the key aspects one needs to test -
Functionality, Reliability, Proficiency, Usability and Creativity.
There are plenty of good tools available in the market that helps users test APIs effectively – be it
Open Source or Commercial. Some of the good ones to try are SMARTBEAR SOAPUI + SWAGGER
+ ALERTSITE, RUNSCOPE, RestAssured, PostMan, JMeter. There are other good tools available as
well, I’ve shared the list I’ve known and/or used by my team.
Key to a successful API delivery is good API Architecture.
• APIs need to be designed with simplicity, usability and test-ability in mind. This can be done
by Test Driven Development (TDD) or Behavior Driven Development (BDD) and building
the features progressively.
o Testers can play a key role in validating the APIs right from design and development.
o Use of automation and tools is recommended for API testing. Testing complexity
increases with progressive feature additions to the APIs.
• APIs need to be secure – as highlighted by the developers themselves and needs to be
tested extensively.
• APIs need to be responsive – especially when you are using responsive User Interfaces (UI)
with a requirement for faster load and response times.
• APIs need to be reliable with Integrity – This is required for heavy usage APIs (private or
public) that are likely to be accessed extensively. The responses of the APIs need to be
predictable and reliable. Data and Response integrity needs to be impeccable. This means
that there shouldn’t be any race conditions or invalid responses coming back.
• APIs need to be simple – They need to be simple to develop, test and document. Public
APIs need to have very good documentation for developers to integrate the APIs with their
applications, devices etc. The APIs need to be well document and easy to programme. This
is also relevant when you need to maintain the APIs post the launch.
How would you implement a good API test strategy?
Some ideas for building a good API strategy would be
• Build an excellent Testing team with domain/functional expertise, have solid knowledge of
testing and ability to automate with a good programming skills.
o If having testers with all the skills are not possible, go for a match that allows you to
get the best of all worlds
• Have a solid API Governance Framework – akin to an architecture/data/SOA governance you
may have in your organizations
o Ensure that API management principles are followed
o Ensure everyone in the team (and partners) understand the concepts
o Have a long term strategy for API Maintenance -> Real challenges begin post the
successful release of the initial versions
• Ensure you have a test governance framework aligned to the API governance covering all
the test phases (a set applicable not comprehensive though)
o Functional Regression
o Integration Testing for new features
o Usability and User Interface
o Security

51
o Compliance and Compatibility
o Load, Performance and Stress
o Device and Platform Compatibility
o Batch Testing (where required)
o Data Integrity and Reliability
o User Acceptance Testing
o Operational Readiness
Future will be a myriad of public, private APIs integrations, connected devices communicating real-
time in disparate manner API testing will be a very big factor in next few years when devices move
from desktop/laptop to hand-held devices. Users may not have an appetite for 10 second responses
but are likely to switch if the responses are not received within a second or so. How do we handle
this?
The challenge is complex, intriguing and a welcome one to address. The fastest to respond will
become the winner.

52
8. "CROWD SOURCED TESTING" – A NEW WAVE IN DIGITAL
REVOLUTION - A POINT-OF-VIEW
Introduction
Recently, Crowdsourcing is becoming a big item to monitor and an area of interest for me.
Crowdsourcing and innovations in Software Engineering as a practice is not new. Ever since the
advent of GNU Project and BSD Unix innovations led by the Open Source Movement spearheaded
by Richard Stallman, Lot of innovations are happening in the Open Innovation, Co-Creation areas.
Oxford English Dictionary can be considered as one of the earliest beneficiary of crowdsourcing.
Commercial success of LAMP (Linux, Apache, MySQL, and Python) is an example and open source
movement has led droves of creative and intelligent developers to support its cause. A lot of firms
are encouraging their employees to support open source, product innovation with the advent of 20%
project time, objective to commit/build an Open Source product of their choice etc. Google,
Thoughtworks are a couple of examples in this area.
Similarly, industry bodies are driving a collaborative crowdsourcing, co-creation and innovation. An
example is TeleManagement Forum’s Catalyst forum for Communications Service Providers. I’ve
had an opportunity to be a part of this industry, systems integrator and product vendor collaboration
to solve an industry problem with a mix of standard, systems integration approaches and
demonstrate to wider fora of industry, academia, vendors and suppliers. Another example is industry
bodies such as CMMi, TMMi, TISPAN, ETSI etc. where standards are built with collaborative,
collective intelligence.
Crowdsourcing – “Crowd Sourced Product Innovation” – Leverage Best of Breed “Driven by
Intelligent Insourced Leadership”
In today’s world, talent is spread across the globe with best talents having their own diverse needs.
It is difficult to get best talent collocated in a single location to drive-up innovative product
development – for companies with lesser financial means. This opens up door for crowdsourcing the
innovation. Crowdsourcing could address some of the challenging areas in multiple spheres of IT
Industry such as
• Product Innovation
• Process Innovation
• Complex Business System Transformation
• Virtual Global Consulting
By leveraging the best-in-breed, best-in-class, best-in-industry products, services and leaders to
provide a winning service to clients.
Given the fact that services firms help clients reduce the effort to deliver through the use of field-
tested tools that automate repetitive, time-consuming tasks—reducing implementation time during
development by 10-35%, and improving execution consistency and quality, Innovative firms need to
continue to make the machines/computers work harder for us, by cutting out the repetition, the
manual effort, the repetition the repetition. This means we shall have less time writing code, testing
applications, finding and raising defects and it would mean the focus will be on delivering cutting-

53
edge technology and services. This cutting edge cannot be industrialized overnight. This can be
driven by a process centric, best-in-breed, world-class specialists spread across the globe.
Crowd sourced Testing
From a testing Point of view, out of interest I wanted to answer the following question with some
finding.
• Where does "CROWD SOURCED TESTING" stand in grander scheme of Testing in general?
• What is the reception for Crowd sourced testing?
• How the future looking like is for Crowd sourced testing?
While doing some analysis, I came across pretty good information. Let’s see what they are
1. Where does CROWDSOURCED TESTING stand in grander scheme of Testing in general?
While doing some analysis, I came across some solid pieces of information. Let’s see what they are.
• Consider a situation where a product launch needs to happen in a very short amount of time,
and needs to be validated in multiple platforms, multiple locations, multiple devices, different
permutations, different usability parameters for a diverse set of end-users with a need to use,
different types of security and compliance validations. How would you test such a big ticket
launch, if you are pressed for time? Of course, you have the product that is near final (in
terms of hardware and software) and you have innumerous combinations to validate, rules to
abide by. Ten years back, people would have thought about this being a 10 year project of
sorts. Not any longer. Every 3-4 months things change in digital world.
o One of the solution to look at is CROWDSOURCING as an option, if testing such a
big testing engagement is not feasible to complete in a short window.
o Cost, Geography, Configurations (Device, Platform, Languages, and Localizations
etc.), User Diversity restrictions can be a hindrance if an in-house or a vendor based
options are considered/looked at.
 Key ingredient is efficient crowd programme management skills
o With a crowdsourcing option (Managed in-house or proven Vendors/alliance partners
with Crowdsourcing options), Platform coverage, User demography and Global
coverages can be addressed in an easy manner.
o We can have a faster, round-the-clock testing service. Fix delivery could be a
challenge to handle – as testing will be very fast.
o We can have a set of highly qualified testers available to test , after a rigorous vetting,
scrutiny by Crowdsourcing selection process (either in-house or with a partner)
o It could be cheaper – As there will be competition for the work/availability and
therefore, the cost of getting the work done will be a lot cheaper than an employee or
contractor base.
o It removes resource dependency (in-house, contractors or vendor) and gives access
wider base
o Flexibility of resourcing is a big plus (ramp-up and ramp-down demands quickly)
o With an output/deliverable based approach, we can improve quality. By incentivizing
the crowd sourced experts with a pay-per-bug, it instills a healthy gamification and
improving quality of the output. If implemented in a solid manner – this will be
removing wastage and increase quality rapidly.

KALILUR RAHMAN
54
There are good number of players providing Crowd sourced Testing as a service. Some of the
notable ones are “Applause (uTest earlier), PassBrains, and Crowd sourced Testing, 99 Tests,
BugFinders, Pay4Bugs, Mob4Hire, Test Birds, Test Bats”. Some of the firms have marquee
clients (household names to speak) consuming their Crowdsourcing services.
Given the challenges forced by these niche vendors, big players need to move to a model that
allows them to maintain or increase the large chunk of testing pie that can be opened up by existing
clients.
There are big outsourcing services firms who are offering Testing as a service in a crowd-sourced
model. As some examples, with the opening of WATSON platform by IBM, Amazon Mechanical Turk
by Amazon, this is crowdsourcing platform is really picking up momentum. Crowd sourced testing is
not behind.
2. What is the reception for Crowd sourced testing?
Well, the reception is pretty good and decent, I would say. Given the success APPLAUSE (earlier
uTEST) has enjoyed, “boutique, niche and specialist” crowdsourcing test players can become big,
charge a premium and could eventuate to solid players.
A second eco-system emerging is the software programmes that help Crowdsourcing delivery. This
involve Crowd sourced project management, Vendor/Contractor/SME management tools, online
verification/validation and payment mechanisms etc. There are players who support these services.
Some of the platforms such as CrowdFlower, Elance-ODesk, IBM Watson – Elance, Amazon
Mechanical Turk etc.
With the popularity for DevOps increasing, Crowd Sourcing could become a good choice to utilize
globally diverse talent pool (internal and external) and leverage bench/free-time efficiently. Some
firms are using Crowdsourcing (aided by gamification) for testing internal applications effectively.
There are some drawbacks in using crowdsourcing for testing.
3. How the future looking like is for Crowd sourced testing?
For many years, specialized contracted labor was a sought after service. For short term specialty
services, contracted labor was the way to go and it was a costly (at times) services in cases of niche
and/or mundane tasks. When enterprises evolved, part of the contracting became a corporate led
services offering in the form of a services industry.
Hence, current fad of Crowdsourcing is nothing new but a reminder of an old adage “Old wine in a
new bottle”. It is a packaged delivery of a tried and tested model with the help of advanced tools and
technologies to manage the crowd efficiently.
Drawing from success of Open Source programmes that relied on crowdsourcing implicitly (albeit
with a vetting process), When it comes to crowd sourced testing, it will work well as talent and skills

55
will play a key role. If performance or output based fees/payment are tied, people will get paid for
what they produce instead of the volume or hours.
Crowdsourcing will work very well in case of output based test deliverable services such as test
case design, test execution (results), automation , bug reports and more importantly identification of
critical bugs. There could be a tier based payment systems depending on the “niche” of the test
activity.
Based on the interest shown in Applause by successful venture capitalists, one can be certain that
Crowd sourced testing services is a viable, successful model to go after, provided a solid business
framework is setup.
One key aspect to look at will be the benefits for crowd sourced talent pool. Success of
Crowdsourcing depends on the success of a company in engaging the talent pool. A fair and just
rewards and recognition platform. Smooth and transparent gamification platform. A well designed
online crowdsourcing management platform for the crowds and clients. One more key aspect will be
a solid intermediary, payment and intellectual property and legal aspects of delivery.
Crowd sourced delivery mechanism, tools and technologies are evolving on their own right and it
will eventuate to a solid ecosystem on its own.
Conclusion
Coming to Crowd sourced testing – it will be a solid ground for growth and it will be contributing to
and be an integral part of the Digital revolution. Soon, a lot of global firms may start to offer these
services as a consolidated pay-per-use or a volume/commodity model based on test output.

Innovations in Testing - Book

Recommended

Recommended

More Related Content

Similar to Innovations in Testing - Book

Similar to Innovations in Testing - Book (20)

More from Kalilur Rahman

More from Kalilur Rahman (11)

Recently uploaded

Recently uploaded (20)

Innovations in Testing - Book