Test Articles Compilation from 2009-
Oct to 2013-Aug by Johan Hoberg
This document is a compilation of test articles ( or articles relevant to testing) I read and sent out as a
news mail during 2009-2013.
The idea of this document is to use it when you are searching for information, find the relevant
articles and then go and read those articles from the original source.
When I started in 2009 I did not have this in mind, and did not copy parts of the text from the original
source of the article, which makes those articles more difficult to search unless the topic is in the
title/link text, since those first article links are mostly the actual link and maybe a title, while later
articles will have a small part of the article copied with the title and the link.
The first articles are at the end of the document, and the latest articles are in the beginning, so the
most relevant articles are quick to find.
Test Articles 2009-2013
Fake your way to better tests
”After many years of blogging, you decide to try out your blog platform's API. You start playing
around with it, but then you realize: how can you tell that your code works without having your test
talk to a remote blog server?
Fakes to the rescue! A fake is a lightweight implementation of an API that behaves like the real
implementation, but isn't suitable for production.”
Know your test doubles
”A test double is an object that can stand in for a real object in a test,
The most common types of test doubles are stubs, mocks, and fakes.
A fake doesn’t use a mocking framework: it’s a lightweight implementation of an API that behaves
like the real implementation, but isn't suitable for production (e.g. an in-memory database).”
Thoughts about automation from Microsoft
“Automation” is an overloaded and abused word that often implies that test code only exists
to automate human tasks – that’s stupid
If you are a tester who codes, you write code. Period. Some of your code may automate tasks.
Most will not (or shouldn’t)
“The point is, if your testing strategy is entirely to automate a suite of user tasks, you are doing way
more to keep “automators” busy than you are actually providing valuable testing.”
How long does it take to find the root cause of a failure in your system? Five minutes? Five days? If
you answered close to five minutes, it’s very likely that your production system and tests have great
logging. All too often, seemingly unessential features like logging, exception handling, and (dare I say
it) testing are an implementation afterthought.
Dumb Monkey testing is when the automation randomly sends keystrokes or commands to the
System Under Test (SUT).
Smart Monkey testing is, well, smarter. Instead of doing random things, the automation system
knows something about your program:
Where it is
What it can do there
Where it can go and how to get there
Where it’s been
If what it’s seeing is correct
Top Secrets from Top Testers
As part of a new series that we’re calling The Top Secrets of Top Testers, we’ll be asking a panel of
testing experts, uTest community members and internal team members a new question every month.
As you might have guessed, this month’s question is:
“If you could offer a new tester one piece of advice, what would it be?”
This mindset, although common, kills me:
Testers who code write automation
Automation is primarily made up of a suite of user tasks
Automation is primarily done through the GUI
All of these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test
teams in the wrong place and is, frankly, insulting to testers who know coding.
If you’re a tester who writes code (or a coder who cares about testing), try this:
Write tools that help you find important bugs quickly
Write a few high level tests that ensure user actions “work”, but leave most of it for
exploratory testing, user testing (assuming you have great monitoring tools), or both.
If you must write GUI tests for the above, write fewer of them (that way you’ll spend less
time investigating their flaky behavior).
If you’re a tester who writes code, you’re not a test automator, you’re not a test automation
engineer, and you don’t write test automation or test code.
You write code. And you test.
Modeling Programmer Productivity
"There is a perception in some tech circles that older programmers aren’t able to keep pace with
rapidly changing technology, and that they are discriminated against in the software field. But a new
study from North Carolina State University indicates that the knowledge and skills of programmers
actually improve over time—and that older programmers know as much (or more) than their
younger peers when it comes to recent software platforms."
”This study didn't surprise me. Nor did it surprise many of my programmer friends. Why, then, do so
many managers still believe that older programmers are less skilled than their younger co-workers?”
State Transition Testing
State Transition Diagrams, or STDs as they’re affectionately called, are effective for documenting
functionality and designing test cases. They should be in every testers bag of tricks, along with
Decision Tables, Pair-Wise analysis, and acting annoyed at work to appear busy.
One benefit of State Transition Diagrams is that they describe the behavior of the system in a
complete, yet easy-to-read and compact, way.
Often, the best strategy is to create tests that cover all transitions (the arrows) at least once. This
guarantees you will test every state, event, action, and transition. It gives you good coverage in a
reasonable amount of tests.
Metric Driven Development
So what is MDD? I would define MDD as a practice where metrics are used to drive the entire
application development. In a company which uses MDD, everything from performance and usage
patterns to revenue is measured. Moreover, every single decision taken by developers, operations or
even business people is based on metrics. Metrics are used to monitor team performance, solve
performance bottlenecks, estimate hardware needs or to accomplish other purposes at any stage of
Asshole Driven Development
The software industry might be the world’s greatest breeding ground for new systems of
management. From Agile, to Extreme Programming , to Test Driven Development (TDD), the
acronyms and frameworks keep piling up. Why?
Some say it’s immaturity: that software is still a young industry and all the change is the path to some
true fundamentals. Others say it’s because software people like making things up and can’t help
themselves. Well I say this: if we’re going to have dozens of models we may as well have some that
are honest, however cynical, to what’s really going on much of the time.
Tear down the Wall
There’s a wall between testers and programmers. Programmers “build”, testers “break” – but
passing features back and forth between one group and another is wasteful. It takes time and extra
communication to traverse the wall.
What software teams need in the future is team members who can perform the activities of
programming, testing, and analysis – together.
I saw a slide deck recently that stated, “Agile Testers frequently can’t keep up with Programmers”.
Good software development can’t happen when you serialize tasks – the team needs to work on this
stuff – together.
I’ll always be deeply involved in the activity of software testing, but I don’t know if the role or title
exists in my future. After nearly 20 years of being a tester (and likely several more with that title), I’m
admittedly going out on a bit of a limb with that statement. Despite my title, I want the walls to go
GTAC (Google Test Automation Conference) 2013 Slides and Presentations
Topics such as
Android UI Automation
How Facebook tests Facebook on Android
How do you Test a Mobile OS?
Finding Culprits automatically in failing builds, i.e. Who broke the Build?
Evolution from Quality Assurance to Test Engineering
Escape from Brainstorm Island
Innovation is not about specific innovation events or workshops – it is a continuous work which
everyone needs to be involved in on a daily basis.
Emphasize Good Practice
However, I noticed a strange thing when I urged people to stop screwing up. I found this didn’t
motivate them at all! I realized getting better isn’t just about reducing what goes wrong (making
mistakes). It’s also about increasing what is right (using good practices). And every now and then
people need a reminder that they’re doing just fine.
Test Management is Wrong (Don’t forget to read the comments, which are equally or more valuable
than the article)
Basically, don’t get so bogged down in test management (the process) that you overlook actual
quality. Checking boxes on your testing to-do list isn’t the same thing as working to make sure
software is actually of high quality – bug free (at least mostly), efficient, effective and achieving its
Testing the Limits with Fiona Charles
How to rescue a failing test project:
Interview a range of people to establish the facts and perceptions.
Set a clear direction and make sure everyone is working productively on tasks that are
aligned with it.
Dig for and lay out the facts about the system quality or project process issues, and report
the findings in a blame-free way.
Make sure there’s a test strategy oriented to the stakeholders’ values and the business risks
and a viable plan to execute on it.
Fill any skills gaps on the team.
Help build communication bridges and coach the test team to develop working relationships
across the project.
Make sure every tester has good reason to feel empowered and valued for his/her
Kanban and Scrum – Making the most of both
Book about differences and similarities between Kanban and Scrum and how they complement and
conflict with each other. Published 2010.
Testing State vs. Testing Interaction
There are typically two ways a unit test can verify that the code under test is working properly: by
testing state or by testing interactions. What’s the difference between these?
That Damn Gorilla
Last week I happened to log on to twitter just as some test folks were marveling over the the massive
parallels between the video with the gorilla and the basketball players and software testing. I made
some cracks about the absurdity of the hype, and more than a few testers freaked out.
Software Testing is like …
When you work in technology, you will inevitably have to explain your job to non-technical people.
This isn’t always easy (and it’s certainly not much fun). Worse yet, most the time we don’t give
ourselves enough credit when explaining our profession to friends and relatives; we sell ourselves
Purpose over targets
That’s the difference between a purpose and a target. And that’s why I don’t like how many apply
the SMART model for goals, because the SMART model moves people’s attention away from a
purpose to a target. Your targets are useless if you can’t identify your purpose.
Samsung trying out 3 CEOs
The announcement last week that the Samsung Electronics is elevating two executives, Boo-Keun
Yoon and J. K. Shin, to the CEO role was met with interest in leadership circles. We have seen this
kind of co-CEO arrangement grow over the years. All the more interesting: the South Korean giant's
current CEO and vice chairman Oh-Hyun Kwon isn't even vacating the seat. All three men will now
share the role.
Testing vs. Checking Redefined – by James Bach
Testing is the process of evaluating a product by learning about it through experimentation,
which includes to some degree: questioning, study, modeling, observation and inference.
Human checking is an attempted checking process wherein humans collect the observations
and apply the rules without the mediation of tools.
Machine checking is a checking process wherein tools collect the observations and apply the
rules without the mediation of humans.
Human/machine checking is an attempted checking process wherein both humans and tools interact
to collect the observations and apply the rules.
Automated vs. Exploratory – we got it wrong!
”There is a misconception floating around that Agile Testing is mostly about automated checking. On
the other way the whole testing community fights since decades about the value of test automation
vs. the value of exploratory testing. I find these two different extreme position unhelpful to do any
constructive testing. Most recently I ended up with a model that explains the underlying dynamics
quite well, to testers, to programmers, and to their managers. Here’s the model.”
Why organizations are so afraid to simplify
”It's one of the great contradictions of organizational life: People are great at starting new things —
projects, meetings, initiatives, task forces — but have a much harder time stopping the ones that
The Perimeter Test
“I want to include a test to click around the button to check if the focus of the button is restricted to
the button area or extends outside the button area. According to me, the name for the test idea:
"The Perimeter Test".”
Good Requirements Deliver a high ROI
”A while back, Ellen Gottesdiener from EBG Consultants assembled a list of citations about ROI and
requirements titled Do You Know Your ROI on Your Requirements Work. Ellen did an excellent job
researching this topic and presents a comprehensive list of citations in her article.”
Management Myth 15: People need to work overtime
”When managers ask people to work overtime, they are not thinking about what a technical day is
like. Especially if you work on an agile team, they are not thinking of what an agile day is like. A
manager’s day is fraught with interruption and context switching every ten minutes. Many managers
feel as if they need to wait until 5 p.m. to get anything done.
But if you are a technical person who has arranged your day so you have a good two-hour chunk in
the morning, and a couple of more two-hour chunks in the afternoon, you have accomplished your
work for the day. Your brain is tired and you need to leave. If you have paired or swarmed or worked
with a cross-functional team, you are tired. You need to leave, to refresh your brain for the next day.”
Stop Blaming the System
”I see it again and again. The suggestion of systems thinkers and Agile writers to stop blaming people
and instead try blaming the system. Seriously, what is the difference between blaming other people
versus blaming the system? It seems to me it’s all the same, as long as we’re not blaming ourselves!”
Why programming classes need to cover software testing too
”Software testing has been on a path toward convergence with programming for some time,
especially when it comes to test automation. At the same time, many vendors have tried to create
tools that allow non-programmers to exercise code in ways they would not know how to do by
”At the same time, software testers provide much-needed input and perspective from exploratory
testing and user acceptance testing. In my opinion, nothing can replace the “human perspective” as
part of the quality assurance cycle.”
The Meaning of Models
”There were several other documents with boxes and arrows, definitions and categories, quadrants
and matrices, circles and pyramids, and they didn’t mean much to me.
I’m a bit tired of such models, I noticed.
I’ve probably seen too many.”
High Level Overview of Software Testing
”Thanks for your request of a high level summary of software testing. You would get different
answers from each tester, and here’s what I think you should know.”
The 60% Rule
”I’ve heard some coaches suggest a 60% rule: everyone should spend 60% of their time on the same
team, the same department, or the same organization. The rest of their time they can spend working
An Overview of High Volume Automated Testing
”This overview describes the general concept of high volume automated testing (HiVAT) and twelve
examples of HiVAT techniques. The common thread of these techniques is automated generation,
execution and evaluation of arbitrarily many tests. The individual tests are often weak, but taken
together, they can expose problems that individually-crafted tests will miss. The simplest techniques
offer high coverage: they can reach a program’s weakness in handling specific special cases. Other
techniques take the program through long sequences of tests and can expose programs that build
gradually (e.g. memory leaks, stack corruption or memory corruption) or that involve unexpected
timing of events. In my experience, these sequences have exposed serious problems in embedded
software, especially in systems that involved interaction among a few processors. As we embed more
software into more consumer systems, including systems that pose life-critical risks, I believe these
tests will become increasingly important.”
Checklist for Software Testing Outsourcing
”When lawyers negotiate a contract, they often find it handy to work from a checklist of
key issues. Fewer things slip through the cracks, and the negotiator doesn’t have to
rethink the same issues in every contract. Further, the checklist grows and cumulates
Being an Effecivte Spec Reviewer (Microsoft)
”The first time testers can impact a feature is often during functional and design reviews. This is also
when we make our first impression on our co-workers. If you want to make a great initial impact on
both your product and peers, you have to be an effective reviewer”
Stakeholder Matrix for Stakeholder Analysis
A stakeholder matrix, like the one in the above diagram, is a great tool to identify the stakeholders
and also classify them appropriately. The classification based on influence of and impact of the
change on stakeholder, helps to reduce the number of stakeholders to a manageable size.
6 Testing Mistakes to Avoid
Mistake #1: Testing too late
Mistake #2: Testing with amateurs
Mistake #3: Testing without a scope
Mistake #4: Testing “one and done”
Mistake #5: Testing in a controlled environment
Mistake #6: Testing too fast/slow
Black Box vs. White Box
”I have heard and seen the difference between Black Box Software Testing and White Box Software
Testing being described as “whether or not you have knowledge about the code”; and that Gray Box
Software Testing is a mix of the two.
But is it really about how much of the code you see?”
Value Creation over Value Extraction
”But new value can only be created when that which is already valuable remains intact. When you
delight customers while screwing suppliers, you’re not creating value, you’re just moving it from one
stakeholder to the other. When you increase productivity while cutting corners in quality, you’re also
not creating value, you’re just stealing it from the future.”
The Insapience of Anti-Automationism
”The distinction between manual and automated is a false dichotomy. We are talking about a matter
of degree (how much automation, how much manual), not a distinction of principle.”
Don’t let Scrum make you Fragile
”A complex system benefits from not following the same practices over and over again. By adding a
little bit of stress, and continuous variability in the environment, the system learns to become fit and
Firmware vs. Software
”I have nothing against firmware, or firmware engineers (I’ve been known to write some firmware
myself). But what we really need is less firmware and more software. Actually, I am disappointed that
firmware engineers write so much firmware!”
Should business people care about clean code?
”Do business people care about clean code? They behave like they don’t. Well, they do and I can
prove it using proof by contradiction. Suppose that a business person does not care about clean code.
The business person wants features and they want them fast. So they pressure engineers to just get
the code working. This causes the code to deteriorate. The business person wants more features, but
the code is a mess so features take longer to add and things that work start to break. If the business
person wants features fast, they need clean code and if they need clean code they must care about it
too, which contradicts the statement that business people don’t care about clean code.”
Exploratory Testing = Fun Productivity
”Exploratory testing can be valuable in specific situations and reveal certain categories of defects
more readily than other approaches. The overall effectiveness of behavioral testing approaches is
heavily influenced by the tester’s in-depth system and domain knowledge and experience. Of course,
the effectiveness of any test method eventually plateaus or becomes less valuable and testers must
employ different approaches to further investigate and evaluate the software under test”
Please reproduce this issue
“… how many times have you seen a test error, and immediately re-run the test to see if you could
reproduce the error yourself?
Is it a big number? If it is, you’re not going to like what I have to say. If you need to reproduce all of
your bugs to figure out what’s going on, you screwed up. Your logging is bad. Your diagnostics don’t
exist. You wrote crummy code. You’re wasting time!”
”I think the idea of double testing is about what models we use in testing. More precisely our mental
models on our test approach/perspective on testing, system boundaries, system parts, levels of
testing, terminology in testing, ideas of test coverage, usage of test techniques, test idea sources,
test planning techniques and so on.”
The Merit System
“It is clear that the annual bonus system doesn't work. A tremendous amount of research says that it
demotivates people, destroys collaboration, and causes dysfunctional behavior among managers and
What also doesn't work is the flat system, where everyone simply gets the same compensation or
bonus. This also demotivates most people. And it makes organizations inflexible in times when agility
What we need is a merits system...”
Root Cause Analysis Template
“When something gets released to Production which adversely affects a customer, we sometimes
write up a formal Root Cause Analysis to send to the customer. It basically explains what happened,
and how we intend to prevent it from happening again.”
Future of Testing
”Regardless of how you feel about the health of software testing (which depends largely on your
ability to interpret a metaphor), for me, it’s getting easier and easier to see that testing is changing.
Granted, it’s changing along with the software under test, so if you’re testing the same sort of
desktop software products you have been for years, using development practices even older, the
good news is that you’re safe – your testing world probably isn’t going to change either.”
Test Engineers @ Google
”In order to get a clearer picture of what Test Engineers are responsible for, I chatted with three of
my colleagues. We were able to identify the underlying Test Engineers’ similarities, while highlighting
How to recognize a problem
“So: sometimes as we’re testing, we recognize a problem by a direct feeling, or through a principle
that we’re aware of. Sometimes our recognition of a problem comes from a document or tool;
sometimes from another person. That may entail embodied mental processes, rational principles, or
mechanisms, but all that’s kind a mouthful. So lately, James and I have started to say that an oracle is
simply “a way to recognize a problem”.”
Debugging for Testers
“I came across a few comments and statements recently that set off a slightly red (orange?) flag for
me. The gist was that debugging was for coders, and that testers “just found the issues”. I get the
context (or at least I think I get it) – I agree that all testers don’t have to be coders, and that all
testers certainly don’t have to be debugging masters, but I
don’t think that any testers should run away from a debugger in fear either.”
Guide to Technical Testing
Presentation about what a technical tester is.
Managing your requirements
”The world of requirements management has developed significantly in the last decade or so and has
increasingly become one of the corner stones of successful software and systems engineering
projects. We have been discussing various aspects of the domain from a best practices perspective
and how tools can help managing your requirements efficiently and effectively.
Starting today we will discuss various aspects of the requirements management discipline at a bird’s
eye view level. These are meant to be introductory in nature and also intend to serve as refreshers
for those who are already in the field.”
[Swedish Article] Tala är Silver, Skriva är Guld
”När det agila manifestet ses som universallösning på systemutvecklingens utmaningar har vi en
tendens att slänga ut barnet med badvattnet och övertolka principer som ”fungerande programvara
framför omfattande dokumentation”.
Omfattande dokumentation vantolkas inte sällan som ”så gott som ingen dokumentation – i alla fall
inte mer än vad jag behöver nu” och plötsligt har vi dragit undan mattan för möjligheten att
kommunicera bakomliggande behov och krav i skriftlig form över lång tid.”
Scaled Agile Framework
This is a model for implementing Agile on an enterprise level. This is not new, but was mentioned in
the article below. I thought it was a nice overview.
Agile at VersionOne
Continuous integration is an important part of the process. They use Jenkins to manage the builds
for each team, tracking both checkin and nightly test runs which comprise both unit, and acceptance
tests. They are currently iterating through the use of cucumber to create scenario tests. Steve
reports that they haven’t had much success at using the tests as “living documentation” but they
have seen value in the collaboration it fosters in helping refine stories.
I asked about UI automation, and they do have some, but mostly used for high level scenario and
process flow checks. After a recent STPCon, the lone dedicated tester of the group discovered the
glories of Session-Based Testing. This method of managing Exploratory Testing is quickly being
adopted by the teams.
Rules I’ve Learned and Live by to Motivate People and Organizations
It is your career – Lead it! (Microsoft)
No review system is perfect and it turns out the Peter Principle (“people get promoted to the point of
incompetency”) is true. It was proven in 2010 by the Nobel Peace prize winners of Management
Science (here). Microsoft follows a general “Up or Out” strategy, which is considered a typical
“defense” to Peter Principle happening in your org.
These same guys proved that promoting the most competent people from the lower levels up is not
as successful of a strategy as alternating between the Best and Worst when promoting.
Organisational Culture Defined (Old, but a good overview of organisational culture)
I hear people referring to the culture of a place as being “good” or “bad”, managers and consultants
speak of changing culture, and employees speak of being part of culture. What is this culture they all
refer to? My studies introduced me to Edgar Schein’s answer, and confirm that the task of getting
culture “right” is a tough target to hit.
Risk Based Testing – Two parts
”Now, when we test a product with a limited number of testers in a small amount of time, we have
to curtail the amount of testing that can be performed. This is where the concept of Risk Based
Testing comes in. Let us understand the concept in detail.”
The RBT process can be described by these major steps:
Step1 – Define all requirements in terms of Risk associated to them
Step2 – Based on risk assessment, prioritize the requirements
Step3 – Plan and define tests according to requirement prioritization
Step4 – Execute test according to prioritization and acceptance criteria.
The Most important advantages of this approach are as follows:
Linking the product risks to the requirements identifies gaps.
With limited time, money and qualified resources, testing concentrates on the most
important matters first, thus delivering the most optimal test.
The focus lies on risks to the business instead of just the functionality of the information
During testing, communication (Test Reporting) takes place in a language (risks) that all
stakeholders understand, hence involvement of testing is greater in comparison to mere list
of issues communicated.
It offers a negotiating instrument to client and test manager alike when available means are
Experts say it is time to give Java the boot
”Java, the programming language designed to make the web fun and interactive, has become one of
the weakest links in a PC’s and Mac’s defenses against external threats. Consider the most recent
Java vulnerability, a weakness currently being exploited by malware distributors: When Oracle, Java’s
maker, released an emergency update to fix the software, security analysts reported that even the
hot-off-the-presses code contains additional vulnerabilities.”
Decision Table Testing
“Decision tables are an effective tool for describing complex functional requirements and designing
test cases. Yet, although they’re far from a new concept, I’ve only seen a handful of functional specs
and test plans that included one.”
What’s Comparable – Part 1
“People interpret requirements and specifications in different ways, based on their models, and their
past experiences, and their current context. When they hear or read something, many people tend to
choose an interpretation that is familiar to them, which may close off their thinking about other
possible interpretations. That’s not a big problem in simple, stable systems. It’s a bigger problem in
software development. The problems we’re trying to solve are neither simple nor stable, and the
same is true with the software that we’re developing.”
The Kudo Box
”There are many wrong ways to reward employees. A simple but effective approach is to install a
“kudo box”, which enables people to give each other a small reward.”
Tell a Story at the Daily Standup
”We should tell three stories during test reporting:
a story about the product
a story about testing
a story about the process”
Software testing training and gaming
Discussion about using games to teach about software testing
Censoring people for disagreeing with Context driven testing
Interesting drama in the test world
TDD ignores the design – No it doesn’t
”TDD ignores design; that is a frequently stated misconception. Many people get this idea from the
code focus of TDD. TDD does not call for the creation of any non-executable design documentation.
So the questioning developer gets the idea that there is no design. But I say, “yes there is”.”
“But make no mistake about it, it you can’t get your code into a unit test, it’s not well designed.”
TDD guides you to improved design
”My premise from these first two articles is that TDD is radar for approaching code rot, as well as a
guiding beacon to modular design.”
”When your code depends on some third party code, or code from some other subsystem or group
that causes testing inconvenience, you have a choice to make. Should I create a test double for the
problem dependency, or write an abstraction layer above the problem dependency. Given new code,
the abstraction layer, or adapter, is usually a good choice.”
Sikuli – Some first impressions
”Sikuli started as a project at MIT. It's now an open source tool that can be used with a number of
different applications. It can be used with the web, it can be used with Flash apps, it can be used with
compiled applications on a number of different platforms. Yeah, OK, that's all fine and good, but
aren't there plenty of tools out there that already fit that space? Why use another one?”
How to manage conflict in virtual teams
“Employee conflicts can be poisonous. We have all experienced the damage to productivity, crushed
creativity, and squashed morale. As Kevin M. Campbell, Accenture's Group Chief Executive,
Technology, notes, "All too often, I've seen that personal conflicts derail costly projects and
important initiatives." Unresolved employee conflicts are bad enough in a traditional, physical
workplace. They are all the more dangerous in a virtual environment, where people don't have the
luxury of proximity to work their differences out face-to-face.”
The Future of Software Security Testing
“In recent years it has been repeatedly proven that vehicle computer systems and wireless medical
devices are susceptible to hacking. Though a malicious attack has never been reported, several
research groups and security organizations have proven that security vulnerabilities exist that allow
people to hack insulin pumps, pacemakers, implantable cardiac defibrillators (ICD) and wireless
vehicle communication systems.”
Another certification, another scam?
This is a continuous discussion in the testing world. Good to keep up-to-date about it if you are
Two ways to deal with Bias
First part of the article is about creating a mind map of different areas of a
function/application/feature and using that as a start for exploratory testing. Could be an interesting
Where does all the time go
“So since you’re a tester, and since the door on the test lab says ‘Test Lab’, and your business card
says ‘Testing’, they figure that’s all you do. The illusion is what Jerry Weinberg calls the Lumping
Problem. All of those different activities—administrative compliance, setup, bug investigation and
reporting, and test design and execution—are lumped into a single idea for them.”
Why are there so many C++ testing frameworks
”So, I think a major reason that we have many C++ testing frameworks is that C++ is different in
different environments, making it hard to write portable C++ code.
Another reason is that some limitations of C++ make it impossible to implement certain features
really well, and different people chose different ways to workaround the limitations.
Finally, many C++ testing framework authors neglected extensibility and were happy just providing
The Less is Best Approach to Innovation
”In our world of excess everything, savvy innovators realize that less is actually best. They know that
delivering a memorable and meaningful experience hinges on user engagement, which is best
achieved through a subtractive approach. Anything excessive, confusing or wasteful is intelligently
and cleverly removed, or never added in the first place.”
The End of a Bug’s Life (Microsoft)
”Some testers work on teams that reward employees based on the number of bugs they find. What
effect does this have on testers? It encourages the finding of bugs, not the Closing of them. If
anything, testers should be rewarded based on the number of bugs fixed and closed, which would at
least encourage the resolution of quality bugs.”
Test Managers and why you should not spend time trying to avoid blame
”This *negative blame+ cycle flourishes only in organizations that perpetuate a culture of fear and
blame. Fix the culture and you’ll fix the cycle.”
The Problem of Organizational Change - People
”Although myriad factors are cited, the inability to engage people is the factor noted longest and
most often. As organizational behavioral experts Kenneth Thompson and Fred Luthans noted almost
20 years ago, a person's reaction to organizational change "can be so excessive and immediate, that
some researchers have suggested it may be easier to start a completely new organization than to try
to change an existing one."”
100% Automation Pass Rate
“Automated testing is now more pervasive throughout the industry as compared to the past.
Automated tests running against daily builds are an essential element in an Agile lifecycle.
Continuous integration (CI) requires continuous feedback from automated tests. So, let’s dispel some
myths and explain why 100% pass rates are not only a good thing, but are and important goal to
Why people don’t instantly buy into Agile
“When "selling" their methods, Agile evangelists often stress the strength of Agile methods at
removing, and even preventing, errors. I used to do this myself, but I always wondered how people
could resist this sales pitch. I would plead, "Don't you want quality?" And, of course, they always said,
"Yes, we want quality," but they didn't buy what I was selling. Eventually, I learned the reason, or at
least one of the reasons. In today's blog, I want to help today's evangelists (coaches, team leaders,
managers, or whomever) by sharing what I've learned about why Agile methods can be so difficult to
The Rise of the Customer
”I spoke recently on the topic of “Customer Focused Test Design” (synopsis: customers don’t really
care how much functional testing testers do, or how many bugs they find. A test approach that favors
scenarios, -ilities, and customer feedback is better; and a bunch of examples for emphasis/proof).”
”If you want to be a successful software company, you have to care about customers. In addition to
keeping them happy once they have used your software or service, you need to respond to their
needs and give them what they need”
Making meetings productive
“First, get the right attendees and be aggressive about it.
Second, make coming and going kosher. Halfway through a meeting and you realize you can't
contribute or don't need to be there? Leave.
Third, vote with your feet by walking away from the more useless meetings. Just don't go to those
meetings where the organizer holds court.
Fourth, heed the warning signs. The first such sign is the meeting agenda. Big agendas mean lots of
time switching topics.
Fifth, follow up a scheduled meeting with scheduled work time.
Finally, all this points to building an anti meeting culture within your organization”
The Definition of Quality
”In short, quality does not exist in a non-human vacuum, but every statement about quality is a
statement about some person(s).
That statement may be explicit or implicit. Most often, the "who" is implicit, and statements about
quality sound like something Moses brought down from Mount Sinai on a stone tablet. That's why so
many discussions of software quality are unproductive: It's my stone tablet versus your Golden Calf.
When we encompass the relativity of quality, we have a tool to make those discussions more fruitful.
Each time somebody asserts a definition of software quality, we simply ask,
"Who is the person behind that statement about quality."”
Premises of Rapid Software Testing
”These are the premises of the Rapid Software Testing methodology. Everything in the
methodology derives in some way from this foundation. These premises derive from our experience,
study, and discussions over a period of decades. They have been shaped by the influence of two
thinkers above all: Cem Kaner and Jerry Weinberg, both of whom have worked as programmers,
managers, social scientists, authors, teachers, and of course, testers.”
Conversation with a Test Engineer at Google
”Alan Faulkner is a Google Test Engineer working on DoubleClick Bid Manager, which enables
advertising agencies and advertisers to bid on multiple ad exchanges. Bid Manager is the next
generation of the Invite Media product, acquired by Google in 2010. Alan Faulkner has been focused
on the migration component of Bid Manager, which transitions advertiser information from Invite
Media to Bid Manager. He joined Google in August 2011, and works in the Kirkland, WA office.”
Bugs spread disease
”There’s a lesson here about the real cost of tolerating bugs, of supporting practices that involve
triaging defects. The cost of carrying a bug is far far greater than most people realize. That trivial little
“cosmetic” issue will cost a little time to fix. If you have to argue about whether or not to fix it, then
track it, then revisit it, then prioritize it, then argue again about whether to fix it or defer it yet again,
you will spend hours upon hours on it.
This is why I have no patience for the “bugs are inevitable; you can’t fix them all” attitude. Bugs kill.
They do it slowly, painfully, but relentlessly.”
I am an exploratory tester
”I am curious about how the system works
I look at details and the whole, and at many places
I use many sources to learn what is important
I am creative and see many testing possibilities
I test in many different ways, and adjust to the situation
I am good at finding important problems fast
I am aware of models enabling and limiting my testing
I can evaluate a system from many dimensions
I can communicate my thinking and findings
I am an exploratory tester”
Our obsession with scale is failing us
In the industrial era and even the early web era, size was central to scale.
The new reality is this: The ability to scale is no longer a direct function of size.
Traditional Test Engineering …
Turning Quality on its Head
”In summary, the closer testing is related to real use the better equipped we will be to find the types
of issues that are by definition the bugs that will most annoy users. Early cycle testing should thin the
bug herd and eliminate the simple bugs and coding mistakes so that late cycle testing can focus on
recall-class issues. Early cycle testing is like crowd control; no matter how diligent we are early in the
cycle, later cycles of manual, dogfood, and beta testing are not skippable phases. Like it or not, we
will benefit from putting more time and innovative thinking into late cycle testing.”
Here is the original youtube video
How to get your bugs fixed (Microsoft)
”To release good quality software, then, you not only have to be proficient at finding bugs, but you
also have to be proficient at getting them fixed. Unfortunately some testers are good at the former,
but not the latter.”
Big Data’s Management Revolution
”The evidence is clear: Data-driven decisions tend to be better decisions. In sector after sector,
companies that embrace this fact will pull away from their rivals. We can't say that all the winners
will be harnessing big data to transform decision making. But the data tell us that's the surest bet.”
Recording My Testing Sessions
”Can you replicate the bug again for me?
How many times have you answered the above question? It might be asked by testers, programmers,
managers and anyone interested in the bug. Sometimes, even you will like to replicate the bug.”
Ten Reasons to Fix Software Bugs Right Away
1. Unfixed bugs camouflage other bugs
2. Unfixed bugs suggest quality isn’t important
3. Discussing unfixed bugs is a waste of time
4. Unfixed bugs lead to duplicate effort
5. Unfixed bugs lead to unreliable metrics
6. Unfixed bugs distract the entire team
7. Unfixed bugs hinder short-notice releases
8. Unfixed bugs lead to inaccurate estimates
9. Fixing familiar code is easier than unfamiliar code
10. Fixing a bug today costs less than tomorrow
Getting 360 Reviews Right
”But there is one thing we've personally seen that profoundly and consistently changes lives —
what's generally referred to as the 360-degree feedback process. In the course of completing tens of
thousands of these reviews as part of our strength-based leadership programs, we had an up-close
view of many people who were teetering on the edge of job termination, and have seen them
blossom into extremely valuable contributors. It's been one of the really gratifying parts of our work.”
”It’s amazing what has happened in the field of test in the last 20 years... a lot of “art” has turned
into “science”. Computer scientists, engineers, and many other disciplines have worked on provable
systems and calculus, pioneered model based testing, invented security fuzz testing, and even settled
on a common pattern for unit tests called xunit.”
Three Tips for Leaders about to Miss their Forecasts
”It's a scary moment when a company realizes that the goals it has set for a quarter or a year are
simply not attainable. And yet in today's continually choppy environment, leaders are encountering
that moment with increased frequency. And it is at this moment that leaders truly earn their pay,
because many short-term actions can have devastating long-term consequences, particularly when it
comes to innovation.”
Interview Questions to ask your Future Employer
”As interviewing is a two-way street, it’s always a good idea to ask your potential employer a few
basic questions before accepting the gig. After all, you need to make sure that the company is as
much a fit for you, as you are for them.”
The Trojan Form of Change
”I believe real change in traditional management will not come from people telling managers how
green the grass is on the other side of the fence. It will come from people who can inject an idea
virus in a traditional organization, which is not destroyed by the current mental model, and which
then transforms it from the inside, using the current mental model against itself."
Looking at two different models and seeing how they intertwine.
Musings on Test Design
” The point is, and this is worth repeating so often, is that thinking of test automation and “human”
testing as separate activities is one of the biggest sins I see in software testing today. It’s not only
ineffective, but I think this sort of thinking is a detriment to the advancement of software testing.
In my world, there are no such things as automated testing, exploratory testing, manual testing, etc.
There is only testing. ”
Testers caught sleeping on the job
”At Microsoft, we submit every code change to a peer-review before it’s checked in. I’ve performed
hundreds (although it feels like millions) of code reviews in the six years I’ve been in Test. One of my
biggest red flags during a code review is the Sleep statement.”
A tool for mapping your goals
”With multiple options on the table, it's hard to decide what's best for you and your future. But by
assessing your career objectives and defining which are most important, you can reach a career
decision with confidence. An easy but structured exercise like the career matrix can give you the
answers you need all in one place.”
Introducing the Software Testing Ice Cream Cone
Are you sure you are not a bad boss?
“When we analyzed the behavior of 30,000 managers, as seen through the eyes of some 300,000 of
their peers, direct reports, and bosses on 360-degree evaluations, we found that the sins of the bad
boss are far more often those of omission, not commission. That is, bad bosses are defined not so
much by any appalling things they do as by certain critical things they don't do.”
Orchestrating Test Automation
”Designing good tests is one of the hardest tasks in software development. That’s worth saying again.
Designing good tests is one of the hardest tasks in software development.”
”I fear that testers are worrying too much about the effort and skills required to automate tests –
and not enough about what it takes to design reliable, portable, accurate, and trustworthy tests that
Yet another Software Testing Pyramid
We Have Met the Enemy and He Is PowerPoint
“It’s dangerous because it can create the illusion of understanding and the illusion of control,”
General McMaster said in a telephone interview afterward. “Some problems in the world are not
Quality is a Four Letter Word
”In my experience, metrics of this sort serve to do one thing and one thing only. They seek to modify
behavior of the test team. This is due to the Hawthorne Effect. Peoples’ behavior change in
accordance to how they are being measured. As an experienced manager, I know the ones I chose
above, if used “right”, will make sure more and more tests are being produced, but they tell me
nothing of the quality of the code or product.”
Initial Thoughts on Group Testing
”I want to open for a discussion on pair testing or at least widen the concept. By saying pair, we say
two people, but why should we limit us to that? Depending what your objective is a different number
might be more applicable?”
A List of Distorted Thinking
“Being human means that if you have a bad lunch or some cloud system goes down which keeps you
from your testing, the physical reaction of frustration leaves you very open to distorted thinking. In
the case of testing, some of this distorted thinking seems useful for uncovering test ideas, and yet we
also need to recognize when it’s time to stop catastrophizing and time to start collaborating with our
Actually there are best practices
Discussion about what Best Practice means.
The Testing Quadrants – We got it wrong
”With the new labels the quadrants are a bit different than the ones that Brian Marick originally
wrote down. On the left-hand side the confirmatory tests that are technology-facing are of course
the unit tests and those class-level integration tests that developers write. These help drive the
development of the project forward.
In the second quadrant now are the expectations of the business. Usually I try to express most of
them in the form of executable specifications, that I can automate. And I can also imagine other
business-facing expectations that I cannot automate that probably fall in here, like reliability of the
In the third quadrants on the top right are tests that help to investigate risks concerning the external
quality of the product. For me Exploratory Tests fall in this category, but also usability concerns, and
The fourth quadrant then probably consists of internal qualities for the code like changeability (static
code analysis comes to mind), maintainability (how high is your CRAP metric these days?), and
design flaws like code smells.”
Creating a Test Lab for Mobile Apps Testing
”If we have a testing burden when we develop mobile apps, what sort of considerations should we
take into account when we are setting up test labs, when using real devices? Here are some pointers
from my own experience.”
Three Keys to Mobile Application Design
“If you are new to designing mobile applications, that often means you will look at web apps and PC
apps first, because that is what you are most familiar with. Then you pull out your smartphone, and
look at the apps you like and use the most there. That’s exactly what I did when I started out, and
while I felt like I had adjusted to a new paradigm (smaller screens, less powerful devices, different
network technology, etc.) I felt like something was missing. I couldn’t quite put my finger on what
was different, but the apps I use on my mobile devices, and the apps I helped create demonstrated it
to me. We are going through a major shift in technology, and the software we use is moving toward a
combination of mobility, social connections, and gaming and entertainment.”
FEW HICCUPPS Heuristics
Several years ago, I wrote an article for Better Software Magazine called Testing Without a Map. The
article was about identifying and applying oracles, and it listed several dimensions of consistency by
which we might find or describe problems in the product.
However, it’s been a long time since the article was published, and since then, James Bach and I have
observed testers using other oracle principles, both to anticipate problems and to describe the
problems they’ve found.
Keep experts on tap, not on top
Among the many qualities that distinguish successful leaders from millions of less-successful
executives in the world is an awareness of the limits of their knowledge. They know what they know,
they appreciate what they don't know, and they have a healthy respect for what they don't know
they don't know. In short, they have great meta-knowledge.
In my experience, experts are among the least successful predictors in times of massive uncertainty.
Valve Handbook for New Employees
Tracking Testing on the Scrum Taskboard
”How do you treat bugs on the taskboard that are found during testing? Create a new test for each
bug, and put the test task back in ToDo? Or create a bug, and a bug-follow-up testing task? ”
Two problems with Context Driven Testing
The most interesting thing about this articles is actually the comments by Cem Kaner:
“As I see it, my responsibility (as a tester) is to serve the project and its stakeholders, in ways that
advance their goals for the project, and if I can’t focus on that and do that well, I shouldn’t be on that
Test Estimation Tips
”Estimating the time it takes to test a feature is an important, yet often overlooked, skill. If your
estimate is too low, it can affect your product’s quality and release date. If your estimate is too high,
it can lead to an inefficient use of resources.
The topic of project estimation is complex enough topic to fill entire books. But here are a few simple
tips that can quickly improve your estimation skills, not to mention your work/life balance and
The Growing Queue at the IT Morgue
”For one, hardware and user-experience companies like Apple, who make their money selling
products and apps, are more valuable than web companies like Google who make their money
advertising web content. This value gap is only going to get wider. As the web recedes into the
background, the opportunity shifts from making money on ads to providing actual value to users
monetized through direct sales and subscriptions. Devices and data centers are the future and it is a
very compelling future.”
Estimates are often helpful, Estimates are often not
”Estimating helps when the process of estimating builds shared understanding among the people
who want the work done and the people doing the work..
Collaborative estimating gives the best results. Diverse experience yields a broader range of
perspectives and questions. Questions and perspectives build understanding of the what, why, and
who related to the request. That’s helpful.
Group estimating reveals differences in knowledge and understanding. Finding those gaps early is
Group estimating surfaces assumptions. When we are aware of our assumptions, we can verify–or
The Hallmark of a Great Test Team
“One hallmark of a great test team is how they behave in the final days before a release. I’m not
referring to working late to automate the last few tests, validate bug fixes, and run a final test pass.
These long hours are an expected part of the job, and when you think about it, these tasks are really
just bookkeeping. I’m talking about something that actually affects the product: does your test team
still try to find bugs just before a release?”
The elements of Improvement (Three pre-requisites for improvement)
Information. People need information about the context and how their work fits into the big
picture. They need information from the work so they can self-correct. Without this
information, systematic improvement is impossible.
A desire to improve. Most people want to do their best and learn to do better–until that
impulse is squashed. One-sided evaluations, organizational hurdles, relentless pressure
strangle the desire to improve.
Time to reflect and learn. People need time to design and implement new processes,
and practice new skills. Relentless deadline pressure makes this impossible. People under
pressure are less likely to try something new, or think clearly about anything.
What do middle managers do in Agile?
”Organizations moving to agile still need management, and often still need people in management
roles, especially in large complex organizations. In traditional hierarchies, middle managers look up
the hierarchy for direction, and focus down the hierarchy to accomplish cascading goals. When
teams pull work from queues and self-organize to meet goals, the real opportunity for middle
managers is to look across the organization to improve the system and develop people and teams.”
Test Plans & Test Cases in Agile/Scrum
”First, yes, the “where’s the testing in Agile?” question is a good one. Many of the people involved in
defining Agile methodologies seemed very focused on automated unit testing, and perhaps missed
the fact that unit testing by itself is not enough. According to Capers Jones’ studies, even the best
unit testing removes only 50% of defects, so testing needs to be added to the methodology.”
Personal Kanban Tool (Free for personal use or for small teams, costs money for more users)
Can TDD Help Reduce Integration Time?
”It has been known for decades that the sooner you find a problem the easier it is to fix. TDD takes
that idea and applies it to the minute to minute activity of programming. Not that TDD is only about
defect prevention, but each defect prevented can potentially save considerable time in the future,
including time spent in integration.”
Manual Test is Unsustainable
”Creating automated tests can be very difficult, especially when the code has gotten long in the tooth
and was not created with automated tests to begin with. Many product development teams don’t
invest in automated tests. They think they cannot afford them. They think their product is different
and can’t be manually tested. This thinking is flawed.”
Management 3.0 Is a Way of Work
”My core belief is that organizations are complex adaptive systems, and that managers must care for
the system, instead of bossing people around.”
Ethics of Testing (James Bach)
”I have an ethical code. Actually, several! The Association for Software Testing adopted the ACM
Code of Ethics, verbatim. It’s okay. I would have preferred the simpler IEEE Code, personally. I also
almost completely follow Jerry Weinberg’s code. I take comfort from all of them, plus I have my own.”
Microsoft vs. Google Part 1: Innovation
Their grassroots efforts at innovation clearly show they believe (correctly) that creativity isn't
governed by a person’s title or the org in which they sit. Creative people can literally be anyone and
are often those without advanced degrees (Gates, Jobs, Zuckerberg …) and without an
entrepreneurial track record (many top innovators have been one hit wonders). However, they take
a very different approach to encouraging the masses to innovate. The bottom line: with 20% time
Google lowered the bar for innovators and thereby watered down the value of the results; with its
Garage, Microsoft raised the bar and created a more sustainable model.
Sleepy Automated Tests
“One of the most common problems with automated tests is synchronizing the automated test and
the application or service under test. Automated tests sometimes race through their instruction set
faster than the application or service can respond leading to false positive (test reports a failure but
there is no bug). Often times testers use a Band-Aid approach to solve the problem by sprinkling
Sleep() methods throughout their test code in an attempt to synchronize the test code and the
application or service under test. Unfortunately, these Sleep() methods may negatively impact the
performance of an automated test and artificially increase the time of the automated test pass.”
The Goal and the Path
”I’ve seen many people with “visions” fail to succeed because they spent their time looking for
shortcuts through the system rather than understanding that navigating the system is the key to
achieving a vision (conversely, I’ve seen many others fail because they focused on navigating the
system rather than their initial goals).”
Management 2.0 is the right thing done wrong
”There’s also nothing wrong with the idea behind balanced scorecards. The problem with
measurements is that one metric will lead to sub-optimization, and therefore you need multiple
metrics to have a holistic view of the organization. It is useful to have multiple perspectives on
Unfortunately… managers see the organization as a collection of parts. They try to optimize
departments. But in a complex system the performance is in the relationships between the parts.
And thus you need metrics that cover perspectives of multiple stakeholders, or “measure from-the-
Proving the Embedded Software Test Team
”I am afraid that I do not have a lot of quantitative data, as most software engineering metrics lack
construct validity. (Or, in different words, "All bugs are not created equal.")
Three arguments I have for embedded teams:
1) Track the amount of time testers spent blocked on different clients over a period of years, I see a
correlation between tester-is-expensive-paperweight waiting for a build and external test teams. Of
course, there are too many independent variables to track here; the improved teams didn't just
embed the testers, they also generally moved toward one-piece flow, pair programming, etc.
2) Track the number of handoffs that occur, from dev-to-test-to-dev. When people sit next to each
other, it is easier to show the dev your test methods, so the programm can 'poke test' the app before
handing it back - not just fix the first blocking bug you found as a tester. Also, allowing mentions-in-
passing and desk pairing increses the chance the bug will get fixed-fixed on the first go round.
3) Track the costs of hand-offs. If testers have to file a bug report that needs to get triaged before it
gets fixed, then costs increase over a conversation and a fix. The most project management layers on
top of this, the more the fixes cost.”
Using Project Artefacts to their full potential (The same thoughts are applicable to many test
artefacts as well)
“Problem: People in projects are very seldom using the basic artefacts to it's full potential.
Which artefacts? Time plans. Resource committments. Budget plans. Processes and delivery plans.”
Visualizing Stakeholder relationships
”Just a thought about stakeholder maps. If you're into 6 Sigma, you know what I mean. Quite
possibly you know about them anyway. Four quadrants, Interest or Support on the x-axis, power or
influence on the y-axis. Map people according their interest in the project and their power in the
Anyway, when I did my 6 Sigma Green Belt training, we were advised not to show the stakeholder
map to the people actually listed there.
What if we actually did the opposite and took the stakeholder map to all concerned parties and
showed it to them and asked them to comment it?”
A sense of ownership
”A friend of mine recently told me that his off-shore team was producing terrible results, and slowly
too. When he traveled there to discuss the quality and productivity problems, he found out that the
same people had their own little startup on the side, working on it in their spare time. And the
product they were making as a startup was amazing. While the product they were making as off-
shore employees was crap. I think these people had an ownership problem.”
We Cannot Choose Between Management And Leadership
“I subscribe to a number of services that look for pithy quotes from Big Names, authors, and other
people who are looking for publicity. I saw one about moving from manager to leader.
Ok, so these are writers or reporters, and they may not know. Choosing to be a manager without
being a leader is like choosing to drive across the country without a map.”
Metrics & Ethics
They (management) understood that the metrics were imperfect. But they felt that they needed
ways to summarize what the organization knew about projects. They felt they needed ways to
compare progress, costs, priorities, and risks. They felt they needed ways to organize the information
so that they could compare several projects or groups at the same time. And they felt they needed to
compare what was happening now to what had happened in the past. Hung then made three points:
1. These are perfectly legitimate management goals.
2. Quantification (metrics) is probably necessary to achieve these goals.
3. The fact that there is no collection of metrics that will do this perfectly (or even terribly well)
doesn’t eliminate the need. Without a better alternative, managers will do the best they can
with what they’ve got.
It is important (I think very important) to pay attention to the validity of our metrics. It is important
to improve them, to find ways to mitigate the risks of using them, and to advise our clients about the
characteristics and risks of the data/statistics we supply to them.
More adventures in magic automation numbers
‘When will our *functional+ automation remove the need for our extensive manual regression
This is of course a trap.
There is no way to determine this number with any degree of certainty.
But what if you really needed to provide an argument around the inability to predict this?
Wherever You Go: Testing Mobile Applications
Joey McAllister: What are some particular concerns you’ve encountered with popular mobile
products like games, social applications, and mapping software?
Jonathan Kohl: A couple of the big things that are difficult are security and privacy. The devices
themselves aren’t that secure with their network connections, so it’s easy for people to snoop in,
crack your accounts, and find out personal information. But, privacy is a huge issue. Some of the big
social applications get in trouble in Canada because they violate our privacy laws and they have to
clean things up.
Two of the most dangerous terms in Agile; Done and Whole Team
I have come to realize in the last two weeks that there are two words in the Agile lexicon that really
bug me. Not because they are bad per se, but that they are so commonly misused and abused. Not
to mention being insanely context sensitive.
The Secrets of Faking a Test Project
This presentation is satirical – we challenge people to think about how they would approach a
project where the goal is to release bad software, but you make it look as if you really tried to test it
well. It didn’t take much effort on our part, we just looked to typical, all too common practices that
are often the staple of how people approach testing projects, and presented them from a different
Why 100% Test Pass Rates are bad
When you change your mindset and write tests to prove the product doesn’t work, rather than to
prove that it does work, you’ll be surprised how many more bugs you expose. This simple change will
make you a better tester, and your products will be better for it.
Testing the Limits with Zynga’s Galina Kramer (Localization Testing)
uTest: Aside from being challenging, localization is arguably one of the most misunderstood testing
types. We don’t expect you to disclose any specific bugs, but what are some of the types of issues
that are uncovered during the course of a localization test? Misspellings? Poor translations? Others?
And while we’re at it, how do you define localization testing?
GK: Localization testing covers all of the above and more. We definitely look for misspellings, poor
translations, truncation issues… We need to make our games equally enjoyable in every language.
This means keeping in mind culture differences, religious beliefs, language intricacies – all of it.
Testing with surrogate code points
Surrogate pairs are often problematic in string parsing algorithms. Unlike “typical” 16-bit Unicode
characters in the base multilingual plane (BMP or Plane 0) surrogate pairs are composed of 2 16-bit
Unicode code points that are mapped to represent a single character (glyph). (See definition D75
Section 3.8, Surrogates in The Unicode Standard Version 6.1) Surrogate code points typically cause
problems because many string parsing algorithms assume 1 character/glyph is 1 code point value
which can lead to character (data) corruption or string buffer miscounts (which can sometime lead to
buffer overflow errors).
As an example of string buffer miscounting let’s take a look at Twitter.
Seven Ways to Find Software Defects Before They Hit Production
This article provides a quick list of test ideas in seven techniques, along with hints of where to go for
1. Quick Attacks
2. Equivalence and Boundary Conditions
3. Common Failure Modes
4. State-Transition Diagrams
5. Use Cases and Soap Opera Tests
6. Code-Based Coverage Models
7. Regression and High-Volume Test Techniques
Quality Assurance is a Process, not a Department
“Of course every great development company will have a final step in the process called Quality
Control or Quality Assurance, but it is my sense that the QA formal group is there to be the standard-
bearer for Quality and rally the company around it, putting a final go or no-go procedure in place
before the world gets its hands on a product, but not accepting proxy status for an otherwise poor
To ignore or be ignorant of a showstopper in one’s own product is a reflection of a process that
needs to be re-engineered. When you’re working with world-class engineers, it is much easier and far
more fruitful to make sure the process is engineered correctly before the products go through it. The
material cost of discovering a bug early in development is a tiny fraction of what it can cost you in the
hands of the public. Give your engineers a voice and they will save you every time.”
The Embedded Tester
”I hear it a lot at conferences, work meetings, interviews, discussions with managers, etc., the idea of
“embedded testers.” The utterances are along the lines of, “Our testers are embedded on
development teams!” or “How do we embed our testers with our developers?” It unfailingly puts me
in the mind of a reporter from say, CNN wearing an ill-fitting helmet and flak jacket*, standing there
as well-drilled troops go about their business.
In Agile, testers aren’t “embedded” on teams any more than programmers are, or analysts, or any
other role that is needed on the team. To say that they are suggests that this is an option, or a
particular strategy you might employ to help with Agile development. It’s not! It’s an essential part of
Eliminate boring testing: Automating visual comparison
”It was a pretty straightforward process to set up an automated script that would import a campaign
and show the preview screen. Then I just needed a way to capture the previews in each environment
as a screenshot for easy comparison. Visually comparing two screenshots was still pretty annoying
and time-consuming. So I added a step to automatically compare the two images pixel by pixel, and
tell me if they’re identical. If they’re not identical, then I want a new screenshot showing me how
The Secrets of why Testers are valued and proud at Microsoft
“The short answer is, hire great people, give them challenging work, and give them a vision for
Testing Dojo Example
”The dojo started with the introduction of the rules and our mission statement. The attendees were
going to deal with touring heuristics FCC CUTS VIDS and pass as many of the tours as possible.”
Why internet search as we know it today will disappear
”The problem with Internet search is that being stupid about it is profitable. The more ugly blue links
you serve up, the more time users have to click on ads. Serve up bad results and the user must
search again and this doubles the number of sponsored links you get paid for. Why be part of the
solution when being part of the problem pays so damn well?”
Misusing Software Process Metrics as People Metrics
“I wish I could say that I never saw organizations make this kind of mistake with process metrics (i.e.,
mistaking a software process metric for a people metric), but unfortunately it is all too common.”
“MacCready took a different approach. He knew that no matter how good his designs, the likely
result would be failure, just like all the others before him. As a result, he expected failure and built
his prototypes with that in mind. He use common and cheap materials that were easy to work with
and easy to repair and replace. As a result he could test new designs in a matter of weeks rather than
Indicators, Testing & Wine Tasting
”Essentially, it was about how to judge feedback from a continuous integration system next to the
other information that the testers in the team were producing. The reason I'd started asking
questions was that with the myriad of information available there was a risk (as I saw it) to ignore
some information - or cherry-pick the information to use...”
Everything You Wanted to Know About SCRUM, But Were Afraid to Ask
Want to learn about SCRUM, but don’t have 10 minutes of spare time? Then checkout this 9-minute
video on the basics of this popular methodology. A nice introduction to a very important subject.
Test doesn’t understand the customer
”On the first day of our mentorship, I asked his opinion around something I had casually observed at
Microsoft. What I did not expect was how fundamentally his answer would shake my world.
Me: “…In addition, given your experience, there’s another pattern I’ve observed that I’d love your
opinion on. I’ve noticed that around 15 out of 20 folks in your role (Product Business Manager) got
there via Program Management, about 4 via development, and around 1 out of 20 via test.”
Mr. R: “I don’t know if that’s totally accurate, but seems about right.”
Me: “So why do you think test so rarely gets the role?”
Mr. R: “Oh, That’s easy! It’s because Test doesn’t understand the customer.” ”
What is “leadership” anyway?
“It's easy to see the absence of leadership - but how do you define it's presence?”
“We've replaced leadership with management, and we suffer from it. This replacement pushes us, to
a great extent, to many of the maladies you've seen on this blog: The desire to institutionalize
process, to have easy answers, to manage to a template or a spreadsheet or process instead of
getting out there, understanding the problem, and, you know, fixing it!”
Why I joined Microsoft by James Whittaker
“I suppose that pointing out that the Google to Microsoft transition is not particularly rare won't
suffice as an explanation so what follows is a more long winded account. For those uninterested in
the finer points, here's the short version: I think much of what is happening in the mobile and web
space is broken and is trending toward becoming more broken. Users have their privacy under
assault and are losing control of their identity and personal data. Independent developers are facing
walled gardens around the data and services necessary to move the web forward. Solving this
problem is going to require a diverse set of intellectual property, technical and information assets
and an inclusive attitude toward software developers. In my opinion, Microsoft is among the best
positioned companies to lead such an effort.”
We're Not So Different, You And I
”No matter where I go, people have the same problems with management & leadership, the same
problems with self-organizing teams, and the same problems with organizations & departments.”
“In my experience, cultural differences between countries pale in comparison to the cultural
differences between industries.”
”Our test cases shouldn't be written to prove there are bugs in our code. They should be written to
prove there are no bugs in our code. So having bugs associated with 97% of our test cases in one of
our projects means there are way too many bugs in our code and we need to push quality upstream. ”
Look to the Data
“I was recently asked to investigate my team’s automated daily test cases. It was taking more than 24
hours to execute our “daily” test run. My job was to find why the tests were taking so long to
complete, and speed them up when possible.”
Exploring Test Roles
”Roles that testers play on teams vary. They vary a lot. You can’t compare them. That’s ok, and (IMO)
part of the growth of the role. I find it disturbing and detrimental when testers not only assume that
their version of testing is “the way”, or that some roles (or people in those roles) do not qualify as
Every Test Failure Means Something
”The point of this story is simple. Every test failure means something. The failure may mean a
product failure. It may mean you have flaky tests. When you start to assume flaky tests or
environments, you’re heading into the land of broken windows and product failures you could have
found earlier (actually, you probably did – you just ignored them).”
“Relying on test automation for any part of your testing is pointless if you don’t care about the
results and look at failed tests every time they fail.”
The Robots are Taking Over
“Normally, I don’t blog about news stories, but on the way to work this morning I heard an
interesting discussion on a local talk radio station. It turns out that there are people up in arms over
the purchase because using robots in the warehouse puts people out of jobs.
I laughed out loud in my car as I realized that these people calling into the radio station were having
the same discussion many people have about test automation.”
Testing the Limits with Gerald Weinberg
Bug Title Crash Course
* As short as possible, but no shorter than that.
* Start the sentence with the most important, to capture the reader’s interest.
* Don’t overdo “externalization” to capture interest, rather describe the dry facts, and let the readers
* If it’s difficult, try many times, or write the title after everything else is done (you might find the
right words on the way.)
* Include a brief description of what happens.
* Include where the problem happens.
* Describe observations rather than (presumed) facts.
* Use a fair description, don’t exaggerate or understate the consequence of the problem
The Linear Filter Trap
”People like to solve problems that they can see - which means we have a tendency to "fix" the
problem we see in front of us. This occurs more often if the problem appears to have a
"straightforward" fix -> I think of this as a form of cognitive ease in action. Digging for root causes is a
challenging activity, and we sometimes want to believe that the cause we identify is good enough to
Delegation does not mean Negation
”With delegation of work you do not delegate your accountability.
Instead, you share it.”
Context Driven Testing
Sleeping with the enemy - Gojko Adzic describes why independent testing should be a thing of the
Traditional software delivery models are based on a lack of trust. Because the business doesn't trust
developers, testers are asked to provide independent validation. Because developers don't trust
testers, everyone wastes a lot of time arguing about whether a problem is in the code or in the tests.
And testers are taught not to trust anyone! All of this distrust even though we share the same end-
goal—delivering a product that satisfies our customers. Gojko Adzic describes why independent
testing should be a thing of the past.
Focusing Testing on Business Needs
Testers often forget that they are service providers whose role is to provide critical information to
the project’s stakeholders. Testing must focus on business needs to add the most value and gain
In this presentation I will investigate what happens when we infuse testing with aesthetics. Can the
fine arts in any way support or complement our testing efforts?
How I wish users knew how I help them through context driven testing
This talk will be about the journey of being a context driven tester and the opposition I faced and
how I paddled through it to win in most of my assignments.
Context-driven Testing is not a religion
Over the past few years, several people have gained visibility in the testing community who express
ideas and values that sound context-driven to me. Some call themselves context-driven, some don’t.
My impression is that some are being told they are not welcome. Others are uncomfortable with a
perceived orthodoxy. They like the approach but not the school. They like the ideas, but not the
We don’t manage the project managers. We don’t decide what information they have to give to the
people they report to.
Context-driven testing at a crossroads (James Bach and Cem Kaner part ways)
”Cem Kaner, who controls www.context-driven-testing.com, has announced an interesting change in
his view of the Context-Driven School. He says he prefers to think of it in terms of the Context-Driven
approach, not a school of thought. This is a significant change from his original view, which was that
CDT is a different paradigm.”
Lessons Learned from Context-driven testing (Follow up to the above article – with comments from
Cem Kaner which are interesting)
There has been some fluff and rumor around context-driven testing yesterday. Some folks even
talked about the death of context-driven testing. Most of it was issued by the about page from Cem
Kaner. If you haven’t read it yet, go ahead, read it now, I will wait here.
The Hindsight Trap
Root cause analysis (RCA) activities are an important part of many retrospectives - whether as part
of a team activity, analysis of test reports, analysis of fault (bug) reports or as standalone activities to
investigate organizational and process problems.
The most common form of this trap in testing circles (where documentation is involved) usually crops
up as a question, "why didn't you/someone spot this problem earlier (as it's specified in document
X)", or statement, "this requirement should been tested because it's specified in document X".
60% of Users Expect Mobile sites to load in 3 Seconds or Less (Importance of Load testing for mobile
That’s right, if your mobile site doesn’t load within THREE seconds 60% of visitors will abandon it.
And, seeing how we’re not a very patient society anymore, 57% of mobile users will only give your
site an extra two seconds (bring the grand total up to five) to load before aborting the mission.
Test Reporting by Michael Bolton
Calculating a ratio of passing tests to failing tests is a relatively easy task. If it is used as a means of
estimating the state of a development project, though, the ratio is invalid, irrelevant, and misleading.
At best, if everyone ignores it entirely, it’s simply playing with numbers. Otherwise, producing a
pass/fail ratio is irresponsible, unethical, and unprofessional.
Testing is like investigative journalism, researching and delivering stories to people. The newspaper
business knows how to direct attention efficiently to the stories in which we’re interested, such that
we get the level of detail that we seek.
As testers (or, heaven help us, quality assurance) people, we tend to be chartered to look for
problems, and to investigate them in ways that are most helpful to programmers, managers, and our
other clients. A failing test on its own tells us little, and a failing check even less; a failing test is only
an allegation of a problem. Investigation and study of a failing test is likely to inform us of something
more useful: whether someone will perceive a problem that threatens the value of the product.
What Metrics do you Use in Agile?
This is a great question. Too many organizations transitioning to Agile don’t ask it. They continue
using the same metrics they were using for traditional projects even though the process is
The problem is that traditional measures don’t work in Agile. Sometimes they’re actively harmful.
First, I only use metrics to get a 50,000 foot view on what’s going on.
Second, I do not use metrics to compare teams or individuals. Ever.
Third, I am much more focused on qualitative than quantitative measures.
Where do bad managers come from?
"Real" executives take the long view. They are the kind of people who, at age 60, can be found
planting trees. IT managers who think that fast moving technology requires short-term quick ﬁxes are
stuck in a middle management mentality. They will never become real executives.
How to Measure Productivity
That’s why it’s also a bad idea to rely on one metric. Use a combination of tools to focus on areas of
concern and improve them. If velocity is going up, but cycle time is staying the same, there may be
some problems with story estimation – maybe one voice has started to dominate the process. If
output is up, but velocity is down, there may be a problem with prioritization (we’re only working on
Take the Crappy Path