Test Articles Compilation from 2009-
Oct to 2013-Aug by Johan Hoberg
This document is a compilation of test articles ( or articles relevant to testing) I read and sent out as a
news mail during 2009-2013.
The idea of this document is to use it when you are searching for information, find the relevant
articles and then go and read those articles from the original source.
When I started in 2009 I did not have this in mind, and did not copy parts of the text from the original
source of the article, which makes those articles more difficult to search unless the topic is in the
title/link text, since those first article links are mostly the actual link and maybe a title, while later
articles will have a small part of the article copied with the title and the link.
The first articles are at the end of the document, and the latest articles are in the beginning, so the
most relevant articles are quick to find.
Test Articles 2009-2013
Fake your way to better tests
”After many years of blogging, you decide to try out your blog platform's API. You start playing
around with it, but then you realize: how can you tell that your code works without having your test
talk to a remote blog server?
Fakes to the rescue! A fake is a lightweight implementation of an API that behaves like the real
implementation, but isn't suitable for production.”
Know your test doubles
”A test double is an object that can stand in for a real object in a test,
The most common types of test doubles are stubs, mocks, and fakes.
A fake doesn’t use a mocking framework: it’s a lightweight implementation of an API that behaves
like the real implementation, but isn't suitable for production (e.g. an in-memory database).”
Thoughts about automation from Microsoft
“Automation” is an overloaded and abused word that often implies that test code only exists
to automate human tasks – that’s stupid
If you are a tester who codes, you write code. Period. Some of your code may automate tasks.
Most will not (or shouldn’t)
“The point is, if your testing strategy is entirely to automate a suite of user tasks, you are doing way
more to keep “automators” busy than you are actually providing valuable testing.”
How long does it take to find the root cause of a failure in your system? Five minutes? Five days? If
you answered close to five minutes, it’s very likely that your production system and tests have great
logging. All too often, seemingly unessential features like logging, exception handling, and (dare I say
it) testing are an implementation afterthought.
Dumb Monkey testing is when the automation randomly sends keystrokes or commands to the
System Under Test (SUT).
Smart Monkey testing is, well, smarter. Instead of doing random things, the automation system
knows something about your program:
Where it is
What it can do there
Where it can go and how to get there
Where it’s been
If what it’s seeing is correct
Top Secrets from Top Testers
As part of a new series that we’re calling The Top Secrets of Top Testers, we’ll be asking a panel of
testing experts, uTest community members and internal team members a new question every month.
As you might have guessed, this month’s question is:
“If you could offer a new tester one piece of advice, what would it be?”
This mindset, although common, kills me:
Testers who code write automation
Automation is primarily made up of a suite of user tasks
Automation is primarily done through the GUI
All of these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test
teams in the wrong place and is, frankly, insulting to testers who know coding.
If you’re a tester who writes code (or a coder who cares about testing), try this:
Write tools that help you find important bugs quickly
Write a few high level tests that ensure user actions “work”, but leave most of it for
exploratory testing, user testing (assuming you have great monitoring tools), or both.
If you must write GUI tests for the above, write fewer of them (that way you’ll spend less
time investigating their flaky behavior).
If you’re a tester who writes code, you’re not a test automator, you’re not a test automation
engineer, and you don’t write test automation or test code.
You write code. And you test.
Modeling Programmer Productivity
"There is a perception in some tech circles that older programmers aren’t able to keep pace with
rapidly changing technology, and that they are discriminated against in the software field. But a new
study from North Carolina State University indicates that the knowledge and skills of programmers
actually improve over time—and that older programmers know as much (or more) than their
younger peers when it comes to recent software platforms."
”This study didn't surprise me. Nor did it surprise many of my programmer friends. Why, then, do so
many managers still believe that older programmers are less skilled than their younger co-workers?”
State Transition Testing
State Transition Diagrams, or STDs as they’re affectionately called, are effective for documenting
functionality and designing test cases. They should be in every testers bag of tricks, along with
Decision Tables, Pair-Wise analysis, and acting annoyed at work to appear busy.
One benefit of State Transition Diagrams is that they describe the behavior of the system in a
complete, yet easy-to-read and compact, way.
Often, the best strategy is to create tests that cover all transitions (the arrows) at least once. This
guarantees you will test every state, event, action, and transition. It gives you good coverage in a
reasonable amount of tests.
Metric Driven Development
So what is MDD? I would define MDD as a practice where metrics are used to drive the entire
application development. In a company which uses MDD, everything from performance and usage
patterns to revenue is measured. Moreover, every single decision taken by developers, operations or
even business people is based on metrics. Metrics are used to monitor team performance, solve
performance bottlenecks, estimate hardware needs or to accomplish other purposes at any stage of
Asshole Driven Development
The software industry might be the world’s greatest breeding ground for new systems of
management. From Agile, to Extreme Programming , to Test Driven Development (TDD), the
acronyms and frameworks keep piling up. Why?
Some say it’s immaturity: that software is still a young industry and all the change is the path to some
true fundamentals. Others say it’s because software people like making things up and can’t help
themselves. Well I say this: if we’re going to have dozens of models we may as well have some that
are honest, however cynical, to what’s really going on much of the time.
Tear down the Wall
There’s a wall between testers and programmers. Programmers “build”, testers “break” – but
passing features back and forth between one group and another is wasteful. It takes time and extra
communication to traverse the wall.
What software teams need in the future is team members who can perform the activities of
programming, testing, and analysis – together.
I saw a slide deck recently that stated, “Agile Testers frequently can’t keep up with Programmers”.
Good software development can’t happen when you serialize tasks – the team needs to work on this
stuff – together.
I’ll always be deeply involved in the activity of software testing, but I don’t know if the role or title
exists in my future. After nearly 20 years of being a tester (and likely several more with that title), I’m
admittedly going out on a bit of a limb with that statement. Despite my title, I want the walls to go
GTAC (Google Test Automation Conference) 2013 Slides and Presentations
Topics such as
Android UI Automation
How Facebook tests Facebook on Android
How do you Test a Mobile OS?
Finding Culprits automatically in failing builds, i.e. Who broke the Build?
Evolution from Quality Assurance to Test Engineering
Escape from Brainstorm Island
Innovation is not about specific innovation events or workshops – it is a continuous work which
everyone needs to be involved in on a daily basis.
Emphasize Good Practice
However, I noticed a strange thing when I urged people to stop screwing up. I found this didn’t
motivate them at all! I realized getting better isn’t just about reducing what goes wrong (making
mistakes). It’s also about increasing what is right (using good practices). And every now and then
people need a reminder that they’re doing just fine.
Test Management is Wrong (Don’t forget to read the comments, which are equally or more valuable
than the article)
Basically, don’t get so bogged down in test management (the process) that you overlook actual
quality. Checking boxes on your testing to-do list isn’t the same thing as working to make sure
software is actually of high quality – bug free (at least mostly), efficient, effective and achieving its
Testing the Limits with Fiona Charles
How to rescue a failing test project:
Interview a range of people to establish the facts and perceptions.
Set a clear direction and make sure everyone is working productively on tasks that are
aligned with it.
Dig for and lay out the facts about the system quality or project process issues, and report
the findings in a blame-free way.
Make sure there’s a test strategy oriented to the stakeholders’ values and the business risks
and a viable plan to execute on it.
Fill any skills gaps on the team.
Help build communication bridges and coach the test team to develop working relationships
across the project.
Make sure every tester has good reason to feel empowered and valued for his/her
Kanban and Scrum – Making the most of both
Book about differences and similarities between Kanban and Scrum and how they complement and
conflict with each other. Published 2010.
Testing State vs. Testing Interaction
There are typically two ways a unit test can verify that the code under test is working properly: by
testing state or by testing interactions. What’s the difference between these?
That Damn Gorilla
Last week I happened to log on to twitter just as some test folks were marveling over the the massive
parallels between the video with the gorilla and the basketball players and software testing. I made
some cracks about the absurdity of the hype, and more than a few testers freaked out.
Software Testing is like …
When you work in technology, you will inevitably have to explain your job to non-technical people.
This isn’t always easy (and it’s certainly not much fun). Worse yet, most the time we don’t give
ourselves enough credit when explaining our profession to friends and relatives; we sell ourselves
Purpose over targets
That’s the difference between a purpose and a target. And that’s why I don’t like how many apply
the SMART model for goals, because the SMART model moves people’s attention away from a
purpose to a target. Your targets are useless if you can’t identify your purpose.
Samsung trying out 3 CEOs
The announcement last week that the Samsung Electronics is elevating two executives, Boo-Keun
Yoon and J. K. Shin, to the CEO role was met with interest in leadership circles. We have seen this
kind of co-CEO arrangement grow over the years. All the more interesting: the South Korean giant's
current CEO and vice chairman Oh-Hyun Kwon isn't even vacating the seat. All three men will now
share the role.
Testing vs. Checking Redefined – by James Bach
Testing is the process of evaluating a product by learning about it through experimentation,
which includes to some degree: questioning, study, modeling, observation and inference.
Human checking is an attempted checking process wherein humans collect the observations
and apply the rules without the mediation of tools.
Machine checking is a checking process wherein tools collect the observations and apply the
rules without the mediation of humans.
Human/machine checking is an attempted checking process wherein both humans and tools interact
to collect the observations and apply the rules.
Automated vs. Exploratory – we got it wrong!
”There is a misconception floating around that Agile Testing is mostly about automated checking. On
the other way the whole testing community fights since decades about the value of test automation
vs. the value of exploratory testing. I find these two different extreme position unhelpful to do any
constructive testing. Most recently I ended up with a model that explains the underlying dynamics
quite well, to testers, to programmers, and to their managers. Here’s the model.”
Why organizations are so afraid to simplify
”It's one of the great contradictions of organizational life: People are great at starting new things —
projects, meetings, initiatives, task forces — but have a much harder time stopping the ones that
The Perimeter Test
“I want to include a test to click around the button to check if the focus of the button is restricted to
the button area or extends outside the button area. According to me, the name for the test idea:
"The Perimeter Test".”
Good Requirements Deliver a high ROI
”A while back, Ellen Gottesdiener from EBG Consultants assembled a list of citations about ROI and
requirements titled Do You Know Your ROI on Your Requirements Work. Ellen did an excellent job
researching this topic and presents a comprehensive list of citations in her article.”
Management Myth 15: People need to work overtime
”When managers ask people to work overtime, they are not thinking about what a technical day is
like. Especially if you work on an agile team, they are not thinking of what an agile day is like. A
manager’s day is fraught with interruption and context switching every ten minutes. Many managers
feel as if they need to wait until 5 p.m. to get anything done.
But if you are a technical person who has arranged your day so you have a good two-hour chunk in
the morning, and a couple of more two-hour chunks in the afternoon, you have accomplished your
work for the day. Your brain is tired and you need to leave. If you have paired or swarmed or worked
with a cross-functional team, you are tired. You need to leave, to refresh your brain for the next day.”
Stop Blaming the System
”I see it again and again. The suggestion of systems thinkers and Agile writers to stop blaming people
and instead try blaming the system. Seriously, what is the difference between blaming other people
versus blaming the system? It seems to me it’s all the same, as long as we’re not blaming ourselves!”
Why programming classes need to cover software testing too
”Software testing has been on a path toward convergence with programming for some time,
especially when it comes to test automation. At the same time, many vendors have tried to create
tools that allow non-programmers to exercise code in ways they would not know how to do by
”At the same time, software testers provide much-needed input and perspective from exploratory
testing and user acceptance testing. In my opinion, nothing can replace the “human perspective” as
part of the quality assurance cycle.”
The Meaning of Models
”There were several other documents with boxes and arrows, definitions and categories, quadrants
and matrices, circles and pyramids, and they didn’t mean much to me.
I’m a bit tired of such models, I noticed.
I’ve probably seen too many.”
High Level Overview of Software Testing
”Thanks for your request of a high level summary of software testing. You would get different
answers from each tester, and here’s what I think you should know.”
The 60% Rule
”I’ve heard some coaches suggest a 60% rule: everyone should spend 60% of their time on the same
team, the same department, or the same organization. The rest of their time they can spend working
An Overview of High Volume Automated Testing
”This overview describes the general concept of high volume automated testing (HiVAT) and twelve
examples of HiVAT techniques. The common thread of these techniques is automated generation,
execution and evaluation of arbitrarily many tests. The individual tests are often weak, but taken
together, they can expose problems that individually-crafted tests will miss. The simplest techniques
offer high coverage: they can reach a program’s weakness in handling specific special cases. Other
techniques take the program through long sequences of tests and can expose programs that build
gradually (e.g. memory leaks, stack corruption or memory corruption) or that involve unexpected
timing of events. In my experience, these sequences have exposed serious problems in embedded
software, especially in systems that involved interaction among a few processors. As we embed more
software into more consumer systems, including systems that pose life-critical risks, I believe these
tests will become increasingly important.”
Checklist for Software Testing Outsourcing
”When lawyers negotiate a contract, they often find it handy to work from a checklist of
key issues. Fewer things slip through the cracks, and the negotiator doesn’t have to
rethink the same issues in every contract. Further, the checklist grows and cumulates
Being an Effecivte Spec Reviewer (Microsoft)
”The first time testers can impact a feature is often during functional and design reviews. This is also
when we make our first impression on our co-workers. If you want to make a great initial impact on
both your product and peers, you have to be an effective reviewer”
Stakeholder Matrix for Stakeholder Analysis
A stakeholder matrix, like the one in the above diagram, is a great tool to identify the stakeholders
and also classify them appropriately. The classification based on influence of and impact of the
change on stakeholder, helps to reduce the number of stakeholders to a manageable size.
6 Testing Mistakes to Avoid
Mistake #1: Testing too late
Mistake #2: Testing with amateurs
Mistake #3: Testing without a scope
Mistake #4: Testing “one and done”
Mistake #5: Testing in a controlled environment
Mistake #6: Testing too fast/slow
Black Box vs. White Box
”I have heard and seen the difference between Black Box Software Testing and White Box Software
Testing being described as “whether or not you have knowledge about the code”; and that Gray Box
Software Testing is a mix of the two.
But is it really about how much of the code you see?”
Value Creation over Value Extraction
”But new value can only be created when that which is already valuable remains intact. When you
delight customers while screwing suppliers, you’re not creating value, you’re just moving it from one
stakeholder to the other. When you increase productivity while cutting corners in quality, you’re also
not creating value, you’re just stealing it from the future.”
The Insapience of Anti-Automationism
”The distinction between manual and automated is a false dichotomy. We are talking about a matter
of degree (how much automation, how much manual), not a distinction of principle.”
Don’t let Scrum make you Fragile
”A complex system benefits from not following the same practices over and over again. By adding a
little bit of stress, and continuous variability in the environment, the system learns to become fit and
Firmware vs. Software
”I have nothing against firmware, or firmware engineers (I’ve been known to write some firmware
myself). But what we really need is less firmware and more software. Actually, I am disappointed that
firmware engineers write so much firmware!”
Should business people care about clean code?
”Do business people care about clean code? They behave like they don’t. Well, they do and I can
prove it using proof by contradiction. Suppose that a business person does not care about clean code.
The business person wants features and they want them fast. So they pressure engineers to just get
the code working. This causes the code to deteriorate. The business person wants more features, but
the code is a mess so features take longer to add and things that work start to break. If the business
person wants features fast, they need clean code and if they need clean code they must care about it
too, which contradicts the statement that business people don’t care about clean code.”
Exploratory Testing = Fun Productivity
”Exploratory testing can be valuable in specific situations and reveal certain categories of defects
more readily than other approaches. The overall effectiveness of behavioral testing approaches is
heavily influenced by the tester’s in-depth system and domain knowledge and experience. Of course,
the effectiveness of any test method eventually plateaus or becomes less valuable and testers must
employ different approaches to further investigate and evaluate the software under test”
Please reproduce this issue
“… how many times have you seen a test error, and immediately re-run the test to see if you could
reproduce the error yourself?
Is it a big number? If it is, you’re not going to like what I have to say. If you need to reproduce all of
your bugs to figure out what’s going on, you screwed up. Your logging is bad. Your diagnostics don’t
exist. You wrote crummy code. You’re wasting time!”
”I think the idea of double testing is about what models we use in testing. More precisely our mental
models on our test approach/perspective on testing, system boundaries, system parts, levels of
testing, terminology in testing, ideas of test coverage, usage of test techniques, test idea sources,
test planning techniques and so on.”
The Merit System
“It is clear that the annual bonus system doesn't work. A tremendous amount of research says that it
demotivates people, destroys collaboration, and causes dysfunctional behavior among managers and
What also doesn't work is the flat system, where everyone simply gets the same compensation or
bonus. This also demotivates most people. And it makes organizations inflexible in times when agility
What we need is a merits system...”
Root Cause Analysis Template
“When something gets released to Production which adversely affects a customer, we sometimes
write up a formal Root Cause Analysis to send to the customer. It basically explains what happened,
and how we intend to prevent it from happening again.”
Future of Testing
”Regardless of how you feel about the health of software testing (which depends largely on your
ability to interpret a metaphor), for me, it’s getting easier and easier to see that testing is changing.
Granted, it’s changing along with the software under test, so if you’re testing the same sort of
desktop software products you have been for years, using development practices even older, the
good news is that you’re safe – your testing world probably isn’t going to change either.”
Test Engineers @ Google
”In order to get a clearer picture of what Test Engineers are responsible for, I chatted with three of
my colleagues. We were able to identify the underlying Test Engineers’ similarities, while highlighting
How to recognize a problem
“So: sometimes as we’re testing, we recognize a problem by a direct feeling, or through a principle
that we’re aware of. Sometimes our recognition of a problem comes from a document or tool;
sometimes from another person. That may entail embodied mental processes, rational principles, or
mechanisms, but all that’s kind a mouthful. So lately, James and I have started to say that an oracle is
simply “a way to recognize a problem”.”
Debugging for Testers
“I came across a few comments and statements recently that set off a slightly red (orange?) flag for
me. The gist was that debugging was for coders, and that testers “just found the issues”. I get the
context (or at least I think I get it) – I agree that all testers don’t have to be coders, and that all
testers certainly don’t have to be debugging masters, but I
don’t think that any testers should run away from a debugger in fear either.”
Guide to Technical Testing
Presentation about what a technical tester is.
Managing your requirements
”The world of requirements management has developed significantly in the last decade or so and has
increasingly become one of the corner stones of successful software and systems engineering
projects. We have been discussing various aspects of the domain from a best practices perspective
and how tools can help managing your requirements efficiently and effectively.
Starting today we will discuss various aspects of the requirements management discipline at a bird’s
eye view level. These are meant to be introductory in nature and also intend to serve as refreshers
for those who are already in the field.”
[Swedish Article] Tala är Silver, Skriva är Guld
”När det agila manifestet ses som universallösning på systemutvecklingens utmaningar har vi en
tendens att slänga ut barnet med badvattnet och övertolka principer som ”fungerande programvara
framför omfattande dokumentation”.
Omfattande dokumentation vantolkas inte sällan som ”så gott som ingen dokumentation – i alla fall
inte mer än vad jag behöver nu” och plötsligt har vi dragit undan mattan för möjligheten att
kommunicera bakomliggande behov och krav i skriftlig form över lång tid.”
Scaled Agile Framework
This is a model for implementing Agile on an enterprise level. This is not new, but was mentioned in
the article below. I thought it was a nice overview.
Agile at VersionOne
Continuous integration is an important part of the process. They use Jenkins to manage the builds
for each team, tracking both checkin and nightly test runs which comprise both unit, and acceptance
tests. They are currently iterating through the use of cucumber to create scenario tests. Steve
reports that they haven’t had much success at using the tests as “living documentation” but they
have seen value in the collaboration it fosters in helping refine stories.
I asked about UI automation, and they do have some, but mostly used for high level scenario and
process flow checks. After a recent STPCon, the lone dedicated tester of the group discovered the
glories of Session-Based Testing. This method of managing Exploratory Testing is quickly being
adopted by the teams.
Rules I’ve Learned and Live by to Motivate People and Organizations
It is your career – Lead it! (Microsoft)
No review system is perfect and it turns out the Peter Principle (“people get promoted to the point of
incompetency”) is true. It was proven in 2010 by the Nobel Peace prize winners of Management
Science (here). Microsoft follows a general “Up or Out” strategy, which is considered a typical
“defense” to Peter Principle happening in your org.
These same guys proved that promoting the most competent people from the lower levels up is not
as successful of a strategy as alternating between the Best and Worst when promoting.
Organisational Culture Defined (Old, but a good overview of organisational culture)
I hear people referring to the culture of a place as being “good” or “bad”, managers and consultants
speak of changing culture, and employees speak of being part of culture. What is this culture they all
refer to? My studies introduced me to Edgar Schein’s answer, and confirm that the task of getting
culture “right” is a tough target to hit.
Risk Based Testing – Two parts
”Now, when we test a product with a limited number of testers in a small amount of time, we have
to curtail the amount of testing that can be performed. This is where the concept of Risk Based
Testing comes in. Let us understand the concept in detail.”
The RBT process can be described by these major steps:
Step1 – Define all requirements in terms of Risk associated to them
Step2 – Based on risk assessment, prioritize the requirements
Step3 – Plan and define tests according to requirement prioritization
Step4 – Execute test according to prioritization and acceptance criteria.
The Most important advantages of this approach are as follows:
Linking the product risks to the requirements identifies gaps.
With limited time, money and qualified resources, testing concentrates on the most
important matters first, thus delivering the most optimal test.
The focus lies on risks to the business instead of just the functionality of the information
During testing, communication (Test Reporting) takes place in a language (risks) that all
stakeholders understand, hence involvement of testing is greater in comparison to mere list
of issues communicated.
It offers a negotiating instrument to client and test manager alike when available means are
Experts say it is time to give Java the boot
”Java, the programming language designed to make the web fun and interactive, has become one of
the weakest links in a PC’s and Mac’s defenses against external threats. Consider the most recent
Java vulnerability, a weakness currently being exploited by malware distributors: When Oracle, Java’s
maker, released an emergency update to fix the software, security analysts reported that even the
hot-off-the-presses code contains additional vulnerabilities.”
Decision Table Testing
“Decision tables are an effective tool for describing complex functional requirements and designing
test cases. Yet, although they’re far from a new concept, I’ve only seen a handful of functional specs
and test plans that included one.”
What’s Comparable – Part 1
“People interpret requirements and specifications in different ways, based on their models, and their
past experiences, and their current context. When they hear or read something, many people tend to
choose an interpretation that is familiar to them, which may close off their thinking about other
possible interpretations. That’s not a big problem in simple, stable systems. It’s a bigger problem in
software development. The problems we’re trying to solve are neither simple nor stable, and the
same is true with the software that we’re developing.”
The Kudo Box
”There are many wrong ways to reward employees. A simple but effective approach is to install a
“kudo box”, which enables people to give each other a small reward.”
Tell a Story at the Daily Standup
”We should tell three stories during test reporting:
a story about the product
a story about testing
a story about the process”
Software testing training and gaming
Discussion about using games to teach about software testing
Censoring people for disagreeing with Context driven testing
Interesting drama in the test world
TDD ignores the design – No it doesn’t
”TDD ignores design; that is a frequently stated misconception. Many people get this idea from the
code focus of TDD. TDD does not call for the creation of any non-executable design documentation.
So the questioning developer gets the idea that there is no design. But I say, “yes there is”.”
“But make no mistake about it, it you can’t get your code into a unit test, it’s not well designed.”
TDD guides you to improved design
”My premise from these first two articles is that TDD is radar for approaching code rot, as well as a
guiding beacon to modular design.”
”When your code depends on some third party code, or code from some other subsystem or group
that causes testing inconvenience, you have a choice to make. Should I create a test double for the
problem dependency, or write an abstraction layer above the problem dependency. Given new code,
the abstraction layer, or adapter, is usually a good choice.”
Sikuli – Some first impressions
”Sikuli started as a project at MIT. It's now an open source tool that can be used with a number of
different applications. It can be used with the web, it can be used with Flash apps, it can be used with
compiled applications on a number of different platforms. Yeah, OK, that's all fine and good, but
aren't there plenty of tools out there that already fit that space? Why use another one?”
How to manage conflict in virtual teams
“Employee conflicts can be poisonous. We have all experienced the damage to productivity, crushed
creativity, and squashed morale. As Kevin M. Campbell, Accenture's Group Chief Executive,
Technology, notes, "All too often, I've seen that personal conflicts derail costly projects and
important initiatives." Unresolved employee conflicts are bad enough in a traditional, physical
workplace. They are all the more dangerous in a virtual environment, where people don't have the
luxury of proximity to work their differences out face-to-face.”
The Future of Software Security Testing
“In recent years it has been repeatedly proven that vehicle computer systems and wireless medical
devices are susceptible to hacking. Though a malicious attack has never been reported, several
research groups and security organizations have proven that security vulnerabilities exist that allow
people to hack insulin pumps, pacemakers, implantable cardiac defibrillators (ICD) and wireless
vehicle communication systems.”
Another certification, another scam?
This is a continuous discussion in the testing world. Good to keep up-to-date about it if you are
Two ways to deal with Bias
First part of the article is about creating a mind map of different areas of a
function/application/feature and using that as a start for exploratory testing. Could be an interesting
Where does all the time go
“So since you’re a tester, and since the door on the test lab says ‘Test Lab’, and your business card
says ‘Testing’, they figure that’s all you do. The illusion is what Jerry Weinberg calls the Lumping
Problem. All of those different activities—administrative compliance, setup, bug investigation and
reporting, and test design and execution—are lumped into a single idea for them.”
Why are there so many C++ testing frameworks
”So, I think a major reason that we have many C++ testing frameworks is that C++ is different in
different environments, making it hard to write portable C++ code.
Another reason is that some limitations of C++ make it impossible to implement certain features
really well, and different people chose different ways to workaround the limitations.
Finally, many C++ testing framework authors neglected extensibility and were happy just providing
The Less is Best Approach to Innovation
”In our world of excess everything, savvy innovators realize that less is actually best. They know that
delivering a memorable and meaningful experience hinges on user engagement, which is best
achieved through a subtractive approach. Anything excessive, confusing or wasteful is intelligently
and cleverly removed, or never added in the first place.”
The End of a Bug’s Life (Microsoft)
”Some testers work on teams that reward employees based on the number of bugs they find. What
effect does this have on testers? It encourages the finding of bugs, not the Closing of them. If
anything, testers should be rewarded based on the number of bugs fixed and closed, which would at
least encourage the resolution of quality bugs.”
Test Managers and why you should not spend time trying to avoid blame
”This *negative blame+ cycle flourishes only in organizations that perpetuate a culture of fear and
blame. Fix the culture and you’ll fix the cycle.”
The Problem of Organizational Change - People
”Although myriad factors are cited, the inability to engage people is the factor noted longest and
most often. As organizational behavioral experts Kenneth Thompson and Fred Luthans noted almost
20 years ago, a person's reaction to organizational change "can be so excessive and immediate, that
some researchers have suggested it may be easier to start a completely new organization than to try
to change an existing one."”
100% Automation Pass Rate
“Automated testing is now more pervasive throughout the industry as compared to the past.
Automated tests running against daily builds are an essential element in an Agile lifecycle.
Continuous integration (CI) requires continuous feedback from automated tests. So, let’s dispel some
myths and explain why 100% pass rates are not only a good thing, but are and important goal to
Why people don’t instantly buy into Agile
“When "selling" their methods, Agile evangelists often stress the strength of Agile methods at
removing, and even preventing, errors. I used to do this myself, but I always wondered how people
could resist this sales pitch. I would plead, "Don't you want quality?" And, of course, they always said,
"Yes, we want quality," but they didn't buy what I was selling. Eventually, I learned the reason, or at
least one of the reasons. In today's blog, I want to help today's evangelists (coaches, team leaders,
managers, or whomever) by sharing what I've learned about why Agile methods can be so difficult to
The Rise of the Customer
”I spoke recently on the topic of “Customer Focused Test Design” (synopsis: customers don’t really
care how much functional testing testers do, or how many bugs they find. A test approach that favors
scenarios, -ilities, and customer feedback is better; and a bunch of examples for emphasis/proof).”
”If you want to be a successful software company, you have to care about customers. In addition to
keeping them happy once they have used your software or service, you need to respond to their
needs and give them what they need”
Making meetings productive
“First, get the right attendees and be aggressive about it.
Second, make coming and going kosher. Halfway through a meeting and you realize you can't
contribute or don't need to be there? Leave.
Third, vote with your feet by walking away from the more useless meetings. Just don't go to those
meetings where the organizer holds court.
Fourth, heed the warning signs. The first such sign is the meeting agenda. Big agendas mean lots of
time switching topics.
Fifth, follow up a scheduled meeting with scheduled work time.
Finally, all this points to building an anti meeting culture within your organization”
The Definition of Quality
”In short, quality does not exist in a non-human vacuum, but every statement about quality is a
statement about some person(s).
That statement may be explicit or implicit. Most often, the "who" is implicit, and statements about
quality sound like something Moses brought down from Mount Sinai on a stone tablet. That's why so
many discussions of software quality are unproductive: It's my stone tablet versus your Golden Calf.
When we encompass the relativity of quality, we have a tool to make those discussions more fruitful.
Each time somebody asserts a definition of software quality, we simply ask,
"Who is the person behind that statement about quality."”
Premises of Rapid Software Testing
”These are the premises of the Rapid Software Testing methodology. Everything in the
methodology derives in some way from this foundation. These premises derive from our experience,
study, and discussions over a period of decades. They have been shaped by the influence of two
thinkers above all: Cem Kaner and Jerry Weinberg, both of whom have worked as programmers,
managers, social scientists, authors, teachers, and of course, testers.”
Conversation with a Test Engineer at Google
”Alan Faulkner is a Google Test Engineer working on DoubleClick Bid Manager, which enables
advertising agencies and advertisers to bid on multiple ad exchanges. Bid Manager is the next
generation of the Invite Media product, acquired by Google in 2010. Alan Faulkner has been focused
on the migration component of Bid Manager, which transitions advertiser information from Invite
Media to Bid Manager. He joined Google in August 2011, and works in the Kirkland, WA office.”
Bugs spread disease
”There’s a lesson here about the real cost of tolerating bugs, of supporting practices that involve
triaging defects. The cost of carrying a bug is far far greater than most people realize. That trivial little
“cosmetic” issue will cost a little time to fix. If you have to argue about whether or not to fix it, then
track it, then revisit it, then prioritize it, then argue again about whether to fix it or defer it yet again,
you will spend hours upon hours on it.
This is why I have no patience for the “bugs are inevitable; you can’t fix them all” attitude. Bugs kill.
They do it slowly, painfully, but relentlessly.”
I am an exploratory tester
”I am curious about how the system works
I look at details and the whole, and at many places
I use many sources to learn what is important
I am creative and see many testing possibilities
I test in many different ways, and adjust to the situation
I am good at finding important problems fast
I am aware of models enabling and limiting my testing
I can evaluate a system from many dimensions
I can communicate my thinking and findings
I am an exploratory tester”
Our obsession with scale is failing us
In the industrial era and even the early web era, size was central to scale.
The new reality is this: The ability to scale is no longer a direct function of size.
Traditional Test Engineering …
Turning Quality on its Head
”In summary, the closer testing is related to real use the better equipped we will be to find the types
of issues that are by definition the bugs that will most annoy users. Early cycle testing should thin the
bug herd and eliminate the simple bugs and coding mistakes so that late cycle testing can focus on
recall-class issues. Early cycle testing is like crowd control; no matter how diligent we are early in the
cycle, later cycles of manual, dogfood, and beta testing are not skippable phases. Like it or not, we
will benefit from putting more time and innovative thinking into late cycle testing.”
Here is the original youtube video
How to get your bugs fixed (Microsoft)
”To release good quality software, then, you not only have to be proficient at finding bugs, but you
also have to be proficient at getting them fixed. Unfortunately some testers are good at the former,
but not the latter.”
Big Data’s Management Revolution
”The evidence is clear: Data-driven decisions tend to be better decisions. In sector after sector,
companies that embrace this fact will pull away from their rivals. We can't say that all the winners
will be harnessing big data to transform decision making. But the data tell us that's the surest bet.”
Recording My Testing Sessions
”Can you replicate the bug again for me?
How many times have you answered the above question? It might be asked by testers, programmers,
managers and anyone interested in the bug. Sometimes, even you will like to replicate the bug.”
Ten Reasons to Fix Software Bugs Right Away
1. Unfixed bugs camouflage other bugs
2. Unfixed bugs suggest quality isn’t important
3. Discussing unfixed bugs is a waste of time
4. Unfixed bugs lead to duplicate effort
5. Unfixed bugs lead to unreliable metrics
6. Unfixed bugs distract the entire team
7. Unfixed bugs hinder short-notice releases
8. Unfixed bugs lead to inaccurate estimates
9. Fixing familiar code is easier than unfamiliar code
10. Fixing a bug today costs less than tomorrow
Getting 360 Reviews Right
”But there is one thing we've personally seen that profoundly and consistently changes lives —
what's generally referred to as the 360-degree feedback process. In the course of completing tens of
thousands of these reviews as part of our strength-based leadership programs, we had an up-close
view of many people who were teetering on the edge of job termination, and have seen them
blossom into extremely valuable contributors. It's been one of the really gratifying parts of our work.”
”It’s amazing what has happened in the field of test in the last 20 years... a lot of “art” has turned
into “science”. Computer scientists, engineers, and many other disciplines have worked on provable
systems and calculus, pioneered model based testing, invented security fuzz testing, and even settled
on a common pattern for unit tests called xunit.”
Three Tips for Leaders about to Miss their Forecasts
”It's a scary moment when a company realizes that the goals it has set for a quarter or a year are
simply not attainable. And yet in today's continually choppy environment, leaders are encountering
that moment with increased frequency. And it is at this moment that leaders truly earn their pay,
because many short-term actions can have devastating long-term consequences, particularly when it
comes to innovation.”
Interview Questions to ask your Future Employer
”As interviewing is a two-way street, it’s always a good idea to ask your potential employer a few
basic questions before accepting the gig. After all, you need to make sure that the company is as
much a fit for you, as you are for them.”
The Trojan Form of Change
”I believe real change in traditional management will not come from people telling managers how
green the grass is on the other side of the fence. It will come from people who can inject an idea
virus in a traditional organization, which is not destroyed by the current mental model, and which
then transforms it from the inside, using the current mental model against itself."
Looking at two different models and seeing how they intertwine.
Musings on Test Design
” The point is, and this is worth repeating so often, is that thinking of test automation and “human”
testing as separate activities is one of the biggest sins I see in software testing today. It’s not only
ineffective, but I think this sort of thinking is a detriment to the advancement of software testing.
In my world, there are no such things as automated testing, exploratory testing, manual testing, etc.
There is only testing. ”
Testers caught sleeping on the job
”At Microsoft, we submit every code change to a peer-review before it’s checked in. I’ve performed
hundreds (although it feels like millions) of code reviews in the six years I’ve been in Test. One of my
biggest red flags during a code review is the Sleep statement.”
A tool for mapping your goals
”With multiple options on the table, it's hard to decide what's best for you and your future. But by
assessing your career objectives and defining which are most important, you can reach a career
decision with confidence. An easy but structured exercise like the career matrix can give you the
answers you need all in one place.”
Introducing the Software Testing Ice Cream Cone
Are you sure you are not a bad boss?
“When we analyzed the behavior of 30,000 managers, as seen through the eyes of some 300,000 of
their peers, direct reports, and bosses on 360-degree evaluations, we found that the sins of the bad
boss are far more often those of omission, not commission. That is, bad bosses are defined not so
much by any appalling things they do as by certain critical things they don't do.”
Orchestrating Test Automation
”Designing good tests is one of the hardest tasks in software development. That’s worth saying again.
Designing good tests is one of the hardest tasks in software development.”
”I fear that testers are worrying too much about the effort and skills required to automate tests –
and not enough about what it takes to design reliable, portable, accurate, and trustworthy tests that
Yet another Software Testing Pyramid
We Have Met the Enemy and He Is PowerPoint
“It’s dangerous because it can create the illusion of understanding and the illusion of control,”
General McMaster said in a telephone interview afterward. “Some problems in the world are not
Quality is a Four Letter Word
”In my experience, metrics of this sort serve to do one thing and one thing only. They seek to modify
behavior of the test team. This is due to the Hawthorne Effect. Peoples’ behavior change in
accordance to how they are being measured. As an experienced manager, I know the ones I chose
above, if used “right”, will make sure more and more tests are being produced, but they tell me
nothing of the quality of the code or product.”
Initial Thoughts on Group Testing
”I want to open for a discussion on pair testing or at least widen the concept. By saying pair, we say
two people, but why should we limit us to that? Depending what your objective is a different number
might be more applicable?”
A List of Distorted Thinking
“Being human means that if you have a bad lunch or some cloud system goes down which keeps you
from your testing, the physical reaction of frustration leaves you very open to distorted thinking. In
the case of testing, some of this distorted thinking seems useful for uncovering test ideas, and yet we
also need to recognize when it’s time to stop catastrophizing and time to start collaborating with our
Actually there are best practices
Discussion about what Best Practice means.
The Testing Quadrants – We got it wrong
”With the new labels the quadrants are a bit different than the ones that Brian Marick originally
wrote down. On the left-hand side the confirmatory tests that are technology-facing are of course
the unit tests and those class-level integration tests that developers write. These help drive the
development of the project forward.
In the second quadrant now are the expectations of the business. Usually I try to express most of
them in the form of executable specifications, that I can automate. And I can also imagine other
business-facing expectations that I cannot automate that probably fall in here, like reliability of the
In the third quadrants on the top right are tests that help to investigate risks concerning the external
quality of the product. For me Exploratory Tests fall in this category, but also usability concerns, and
The fourth quadrant then probably consists of internal qualities for the code like changeability (static
code analysis comes to mind), maintainability (how high is your CRAP metric these days?), and
design flaws like code smells.”
Creating a Test Lab for Mobile Apps Testing
”If we have a testing burden when we develop mobile apps, what sort of considerations should we
take into account when we are setting up test labs, when using real devices? Here are some pointers
from my own experience.”
Three Keys to Mobile Application Design
“If you are new to designing mobile applications, that often means you will look at web apps and PC
apps first, because that is what you are most familiar with. Then you pull out your smartphone, and
look at the apps you like and use the most there. That’s exactly what I did when I started out, and
while I felt like I had adjusted to a new paradigm (smaller screens, less powerful devices, different
network technology, etc.) I felt like something was missing. I couldn’t quite put my finger on what
was different, but the apps I use on my mobile devices, and the apps I helped create demonstrated it
to me. We are going through a major shift in technology, and the software we use is moving toward a
combination of mobility, social connections, and gaming and entertainment.”
FEW HICCUPPS Heuristics
Several years ago, I wrote an article for Better Software Magazine called Testing Without a Map. The
article was about identifying and applying oracles, and it listed several dimensions of consistency by
which we might find or describe problems in the product.
However, it’s been a long time since the article was published, and since then, James Bach and I have
observed testers using other oracle principles, both to anticipate problems and to describe the
problems they’ve found.
Keep experts on tap, not on top
Among the many qualities that distinguish successful leaders from millions of less-successful
executives in the world is an awareness of the limits of their knowledge. They know what they know,
they appreciate what they don't know, and they have a healthy respect for what they don't know
they don't know. In short, they have great meta-knowledge.
In my experience, experts are among the least successful predictors in times of massive uncertainty.
Valve Handbook for New Employees
Tracking Testing on the Scrum Taskboard
”How do you treat bugs on the taskboard that are found during testing? Create a new test for each
bug, and put the test task back in ToDo? Or create a bug, and a bug-follow-up testing task? ”
Two problems with Context Driven Testing
The most interesting thing about this articles is actually the comments by Cem Kaner:
“As I see it, my responsibility (as a tester) is to serve the project and its stakeholders, in ways that
advance their goals for the project, and if I can’t focus on that and do that well, I shouldn’t be on that
Test Estimation Tips
”Estimating the time it takes to test a feature is an important, yet often overlooked, skill. If your
estimate is too low, it can affect your product’s quality and release date. If your estimate is too high,
it can lead to an inefficient use of resources.
The topic of project estimation is complex enough topic to fill entire books. But here are a few simple
tips that can quickly improve your estimation skills, not to mention your work/life balance and
The Growing Queue at the IT Morgue
”For one, hardware and user-experience companies like Apple, who make their money selling
products and apps, are more valuable than web companies like Google who make their money
advertising web content. This value gap is only going to get wider. As the web recedes into the
background, the opportunity shifts from making money on ads to providing actual value to users
monetized through direct sales and subscriptions. Devices and data centers are the future and it is a
very compelling future.”
Estimates are often helpful, Estimates are often not
”Estimating helps when the process of estimating builds shared understanding among the people
who want the work done and the people doing the work..
Collaborative estimating gives the best results. Diverse experience yields a broader range of
perspectives and questions. Questions and perspectives build understanding of the what, why, and
who related to the request. That’s helpful.
Group estimating reveals differences in knowledge and understanding. Finding those gaps early is
Group estimating surfaces assumptions. When we are aware of our assumptions, we can verify–or
The Hallmark of a Great Test Team
“One hallmark of a great test team is how they behave in the final days before a release. I’m not
referring to working late to automate the last few tests, validate bug fixes, and run a final test pass.
These long hours are an expected part of the job, and when you think about it, these tasks are really
just bookkeeping. I’m talking about something that actually affects the product: does your test team
still try to find bugs just before a release?”
The elements of Improvement (Three pre-requisites for improvement)
Information. People need information about the context and how their work fits into the big
picture. They need information from the work so they can self-correct. Without this
information, systematic improvement is impossible.
A desire to improve. Most people want to do their best and learn to do better–until that
impulse is squashed. One-sided evaluations, organizational hurdles, relentless pressure
strangle the desire to improve.
Time to reflect and learn. People need time to design and implement new processes,
and practice new skills. Relentless deadline pressure makes this impossible. People under
pressure are less likely to try something new, or think clearly about anything.
What do middle managers do in Agile?
”Organizations moving to agile still need management, and often still need people in management
roles, especially in large complex organizations. In traditional hierarchies, middle managers look up
the hierarchy for direction, and focus down the hierarchy to accomplish cascading goals. When
teams pull work from queues and self-organize to meet goals, the real opportunity for middle
managers is to look across the organization to improve the system and develop people and teams.”
Test Plans & Test Cases in Agile/Scrum
”First, yes, the “where’s the testing in Agile?” question is a good one. Many of the people involved in
defining Agile methodologies seemed very focused on automated unit testing, and perhaps missed
the fact that unit testing by itself is not enough. According to Capers Jones’ studies, even the best
unit testing removes only 50% of defects, so testing needs to be added to the methodology.”
Personal Kanban Tool (Free for personal use or for small teams, costs money for more users)
Can TDD Help Reduce Integration Time?
”It has been known for decades that the sooner you find a problem the easier it is to fix. TDD takes
that idea and applies it to the minute to minute activity of programming. Not that TDD is only about
defect prevention, but each defect prevented can potentially save considerable time in the future,
including time spent in integration.”
Manual Test is Unsustainable
”Creating automated tests can be very difficult, especially when the code has gotten long in the tooth
and was not created with automated tests to begin with. Many product development teams don’t
invest in automated tests. They think they cannot afford them. They think their product is different
and can’t be manually tested. This thinking is flawed.”
Management 3.0 Is a Way of Work
”My core belief is that organizations are complex adaptive systems, and that managers must care for
the system, instead of bossing people around.”
Ethics of Testing (James Bach)
”I have an ethical code. Actually, several! The Association for Software Testing adopted the ACM
Code of Ethics, verbatim. It’s okay. I would have preferred the simpler IEEE Code, personally. I also
almost completely follow Jerry Weinberg’s code. I take comfort from all of them, plus I have my own.”
Microsoft vs. Google Part 1: Innovation
Their grassroots efforts at innovation clearly show they believe (correctly) that creativity isn't
governed by a person’s title or the org in which they sit. Creative people can literally be anyone and
are often those without advanced degrees (Gates, Jobs, Zuckerberg …) and without an
entrepreneurial track record (many top innovators have been one hit wonders). However, they take
a very different approach to encouraging the masses to innovate. The bottom line: with 20% time
Google lowered the bar for innovators and thereby watered down the value of the results; with its
Garage, Microsoft raised the bar and created a more sustainable model.
Sleepy Automated Tests
“One of the most common problems with automated tests is synchronizing the automated test and
the application or service under test. Automated tests sometimes race through their instruction set
faster than the application or service can respond leading to false positive (test reports a failure but
there is no bug). Often times testers use a Band-Aid approach to solve the problem by sprinkling
Sleep() methods throughout their test code in an attempt to synchronize the test code and the
application or service under test. Unfortunately, these Sleep() methods may negatively impact the
performance of an automated test and artificially increase the time of the automated test pass.”
The Goal and the Path
”I’ve seen many people with “visions” fail to succeed because they spent their time looking for
shortcuts through the system rather than understanding that navigating the system is the key to
achieving a vision (conversely, I’ve seen many others fail because they focused on navigating the
system rather than their initial goals).”
Management 2.0 is the right thing done wrong
”There’s also nothing wrong with the idea behind balanced scorecards. The problem with
measurements is that one metric will lead to sub-optimization, and therefore you need multiple
metrics to have a holistic view of the organization. It is useful to have multiple perspectives on
Unfortunately… managers see the organization as a collection of parts. They try to optimize
departments. But in a complex system the performance is in the relationships between the parts.
And thus you need metrics that cover perspectives of multiple stakeholders, or “measure from-the-
Proving the Embedded Software Test Team
”I am afraid that I do not have a lot of quantitative data, as most software engineering metrics lack
construct validity. (Or, in different words, "All bugs are not created equal.")
Three arguments I have for embedded teams:
1) Track the amount of time testers spent blocked on different clients over a period of years, I see a
correlation between tester-is-expensive-paperweight waiting for a build and external test teams. Of
course, there are too many independent variables to track here; the improved teams didn't just
embed the testers, they also generally moved toward one-piece flow, pair programming, etc.
2) Track the number of handoffs that occur, from dev-to-test-to-dev. When people sit next to each
other, it is easier to show the dev your test methods, so the programm can 'poke test' the app before
handing it back - not just fix the first blocking bug you found as a tester. Also, allowing mentions-in-
passing and desk pairing increses the chance the bug will get fixed-fixed on the first go round.
3) Track the costs of hand-offs. If testers have to file a bug report that needs to get triaged before it
gets fixed, then costs increase over a conversation and a fix. The most project management layers on
top of this, the more the fixes cost.”
Using Project Artefacts to their full potential (The same thoughts are applicable to many test
artefacts as well)
“Problem: People in projects are very seldom using the basic artefacts to it's full potential.
Which artefacts? Time plans. Resource committments. Budget plans. Processes and delivery plans.”
Visualizing Stakeholder relationships
”Just a thought about stakeholder maps. If you're into 6 Sigma, you know what I mean. Quite
possibly you know about them anyway. Four quadrants, Interest or Support on the x-axis, power or
influence on the y-axis. Map people according their interest in the project and their power in the
Anyway, when I did my 6 Sigma Green Belt training, we were advised not to show the stakeholder
map to the people actually listed there.
What if we actually did the opposite and took the stakeholder map to all concerned parties and
showed it to them and asked them to comment it?”
A sense of ownership
”A friend of mine recently told me that his off-shore team was producing terrible results, and slowly
too. When he traveled there to discuss the quality and productivity problems, he found out that the
same people had their own little startup on the side, working on it in their spare time. And the
product they were making as a startup was amazing. While the product they were making as off-
shore employees was crap. I think these people had an ownership problem.”
We Cannot Choose Between Management And Leadership
“I subscribe to a number of services that look for pithy quotes from Big Names, authors, and other
people who are looking for publicity. I saw one about moving from manager to leader.
Ok, so these are writers or reporters, and they may not know. Choosing to be a manager without
being a leader is like choosing to drive across the country without a map.”
Metrics & Ethics
They (management) understood that the metrics were imperfect. But they felt that they needed
ways to summarize what the organization knew about projects. They felt they needed ways to
compare progress, costs, priorities, and risks. They felt they needed ways to organize the information
so that they could compare several projects or groups at the same time. And they felt they needed to
compare what was happening now to what had happened in the past. Hung then made three points:
1. These are perfectly legitimate management goals.
2. Quantification (metrics) is probably necessary to achieve these goals.
3. The fact that there is no collection of metrics that will do this perfectly (or even terribly well)
doesn’t eliminate the need. Without a better alternative, managers will do the best they can
with what they’ve got.
It is important (I think very important) to pay attention to the validity of our metrics. It is important
to improve them, to find ways to mitigate the risks of using them, and to advise our clients about the
characteristics and risks of the data/statistics we supply to them.
More adventures in magic automation numbers
‘When will our *functional+ automation remove the need for our extensive manual regression
This is of course a trap.
There is no way to determine this number with any degree of certainty.
But what if you really needed to provide an argument around the inability to predict this?
Wherever You Go: Testing Mobile Applications
Joey McAllister: What are some particular concerns you’ve encountered with popular mobile
products like games, social applications, and mapping software?
Jonathan Kohl: A couple of the big things that are difficult are security and privacy. The devices
themselves aren’t that secure with their network connections, so it’s easy for people to snoop in,
crack your accounts, and find out personal information. But, privacy is a huge issue. Some of the big
social applications get in trouble in Canada because they violate our privacy laws and they have to
clean things up.
Two of the most dangerous terms in Agile; Done and Whole Team
I have come to realize in the last two weeks that there are two words in the Agile lexicon that really
bug me. Not because they are bad per se, but that they are so commonly misused and abused. Not
to mention being insanely context sensitive.
The Secrets of Faking a Test Project
This presentation is satirical – we challenge people to think about how they would approach a
project where the goal is to release bad software, but you make it look as if you really tried to test it
well. It didn’t take much effort on our part, we just looked to typical, all too common practices that
are often the staple of how people approach testing projects, and presented them from a different
Why 100% Test Pass Rates are bad
When you change your mindset and write tests to prove the product doesn’t work, rather than to
prove that it does work, you’ll be surprised how many more bugs you expose. This simple change will
make you a better tester, and your products will be better for it.
Testing the Limits with Zynga’s Galina Kramer (Localization Testing)
uTest: Aside from being challenging, localization is arguably one of the most misunderstood testing
types. We don’t expect you to disclose any specific bugs, but what are some of the types of issues
that are uncovered during the course of a localization test? Misspellings? Poor translations? Others?
And while we’re at it, how do you define localization testing?
GK: Localization testing covers all of the above and more. We definitely look for misspellings, poor
translations, truncation issues… We need to make our games equally enjoyable in every language.
This means keeping in mind culture differences, religious beliefs, language intricacies – all of it.
Testing with surrogate code points
Surrogate pairs are often problematic in string parsing algorithms. Unlike “typical” 16-bit Unicode
characters in the base multilingual plane (BMP or Plane 0) surrogate pairs are composed of 2 16-bit
Unicode code points that are mapped to represent a single character (glyph). (See definition D75
Section 3.8, Surrogates in The Unicode Standard Version 6.1) Surrogate code points typically cause
problems because many string parsing algorithms assume 1 character/glyph is 1 code point value
which can lead to character (data) corruption or string buffer miscounts (which can sometime lead to
buffer overflow errors).
As an example of string buffer miscounting let’s take a look at Twitter.
Seven Ways to Find Software Defects Before They Hit Production
This article provides a quick list of test ideas in seven techniques, along with hints of where to go for
1. Quick Attacks
2. Equivalence and Boundary Conditions
3. Common Failure Modes
4. State-Transition Diagrams
5. Use Cases and Soap Opera Tests
6. Code-Based Coverage Models
7. Regression and High-Volume Test Techniques
Quality Assurance is a Process, not a Department
“Of course every great development company will have a final step in the process called Quality
Control or Quality Assurance, but it is my sense that the QA formal group is there to be the standard-
bearer for Quality and rally the company around it, putting a final go or no-go procedure in place
before the world gets its hands on a product, but not accepting proxy status for an otherwise poor
To ignore or be ignorant of a showstopper in one’s own product is a reflection of a process that
needs to be re-engineered. When you’re working with world-class engineers, it is much easier and far
more fruitful to make sure the process is engineered correctly before the products go through it. The
material cost of discovering a bug early in development is a tiny fraction of what it can cost you in the
hands of the public. Give your engineers a voice and they will save you every time.”
The Embedded Tester
”I hear it a lot at conferences, work meetings, interviews, discussions with managers, etc., the idea of
“embedded testers.” The utterances are along the lines of, “Our testers are embedded on
development teams!” or “How do we embed our testers with our developers?” It unfailingly puts me
in the mind of a reporter from say, CNN wearing an ill-fitting helmet and flak jacket*, standing there
as well-drilled troops go about their business.
In Agile, testers aren’t “embedded” on teams any more than programmers are, or analysts, or any
other role that is needed on the team. To say that they are suggests that this is an option, or a
particular strategy you might employ to help with Agile development. It’s not! It’s an essential part of
Eliminate boring testing: Automating visual comparison
”It was a pretty straightforward process to set up an automated script that would import a campaign
and show the preview screen. Then I just needed a way to capture the previews in each environment
as a screenshot for easy comparison. Visually comparing two screenshots was still pretty annoying
and time-consuming. So I added a step to automatically compare the two images pixel by pixel, and
tell me if they’re identical. If they’re not identical, then I want a new screenshot showing me how
The Secrets of why Testers are valued and proud at Microsoft
“The short answer is, hire great people, give them challenging work, and give them a vision for
Testing Dojo Example
”The dojo started with the introduction of the rules and our mission statement. The attendees were
going to deal with touring heuristics FCC CUTS VIDS and pass as many of the tours as possible.”
Why internet search as we know it today will disappear
”The problem with Internet search is that being stupid about it is profitable. The more ugly blue links
you serve up, the more time users have to click on ads. Serve up bad results and the user must
search again and this doubles the number of sponsored links you get paid for. Why be part of the
solution when being part of the problem pays so damn well?”
Misusing Software Process Metrics as People Metrics
“I wish I could say that I never saw organizations make this kind of mistake with process metrics (i.e.,
mistaking a software process metric for a people metric), but unfortunately it is all too common.”
“MacCready took a different approach. He knew that no matter how good his designs, the likely
result would be failure, just like all the others before him. As a result, he expected failure and built
his prototypes with that in mind. He use common and cheap materials that were easy to work with
and easy to repair and replace. As a result he could test new designs in a matter of weeks rather than
Indicators, Testing & Wine Tasting
”Essentially, it was about how to judge feedback from a continuous integration system next to the
other information that the testers in the team were producing. The reason I'd started asking
questions was that with the myriad of information available there was a risk (as I saw it) to ignore
some information - or cherry-pick the information to use...”
Everything You Wanted to Know About SCRUM, But Were Afraid to Ask
Want to learn about SCRUM, but don’t have 10 minutes of spare time? Then checkout this 9-minute
video on the basics of this popular methodology. A nice introduction to a very important subject.
Test doesn’t understand the customer
”On the first day of our mentorship, I asked his opinion around something I had casually observed at
Microsoft. What I did not expect was how fundamentally his answer would shake my world.
Me: “…In addition, given your experience, there’s another pattern I’ve observed that I’d love your
opinion on. I’ve noticed that around 15 out of 20 folks in your role (Product Business Manager) got
there via Program Management, about 4 via development, and around 1 out of 20 via test.”
Mr. R: “I don’t know if that’s totally accurate, but seems about right.”
Me: “So why do you think test so rarely gets the role?”
Mr. R: “Oh, That’s easy! It’s because Test doesn’t understand the customer.” ”
What is “leadership” anyway?
“It's easy to see the absence of leadership - but how do you define it's presence?”
“We've replaced leadership with management, and we suffer from it. This replacement pushes us, to
a great extent, to many of the maladies you've seen on this blog: The desire to institutionalize
process, to have easy answers, to manage to a template or a spreadsheet or process instead of
getting out there, understanding the problem, and, you know, fixing it!”
Why I joined Microsoft by James Whittaker
“I suppose that pointing out that the Google to Microsoft transition is not particularly rare won't
suffice as an explanation so what follows is a more long winded account. For those uninterested in
the finer points, here's the short version: I think much of what is happening in the mobile and web
space is broken and is trending toward becoming more broken. Users have their privacy under
assault and are losing control of their identity and personal data. Independent developers are facing
walled gardens around the data and services necessary to move the web forward. Solving this
problem is going to require a diverse set of intellectual property, technical and information assets
and an inclusive attitude toward software developers. In my opinion, Microsoft is among the best
positioned companies to lead such an effort.”
We're Not So Different, You And I
”No matter where I go, people have the same problems with management & leadership, the same
problems with self-organizing teams, and the same problems with organizations & departments.”
“In my experience, cultural differences between countries pale in comparison to the cultural
differences between industries.”
”Our test cases shouldn't be written to prove there are bugs in our code. They should be written to
prove there are no bugs in our code. So having bugs associated with 97% of our test cases in one of
our projects means there are way too many bugs in our code and we need to push quality upstream. ”
Look to the Data
“I was recently asked to investigate my team’s automated daily test cases. It was taking more than 24
hours to execute our “daily” test run. My job was to find why the tests were taking so long to
complete, and speed them up when possible.”
Exploring Test Roles
”Roles that testers play on teams vary. They vary a lot. You can’t compare them. That’s ok, and (IMO)
part of the growth of the role. I find it disturbing and detrimental when testers not only assume that
their version of testing is “the way”, or that some roles (or people in those roles) do not qualify as
Every Test Failure Means Something
”The point of this story is simple. Every test failure means something. The failure may mean a
product failure. It may mean you have flaky tests. When you start to assume flaky tests or
environments, you’re heading into the land of broken windows and product failures you could have
found earlier (actually, you probably did – you just ignored them).”
“Relying on test automation for any part of your testing is pointless if you don’t care about the
results and look at failed tests every time they fail.”
The Robots are Taking Over
“Normally, I don’t blog about news stories, but on the way to work this morning I heard an
interesting discussion on a local talk radio station. It turns out that there are people up in arms over
the purchase because using robots in the warehouse puts people out of jobs.
I laughed out loud in my car as I realized that these people calling into the radio station were having
the same discussion many people have about test automation.”
Testing the Limits with Gerald Weinberg
Bug Title Crash Course
* As short as possible, but no shorter than that.
* Start the sentence with the most important, to capture the reader’s interest.
* Don’t overdo “externalization” to capture interest, rather describe the dry facts, and let the readers
* If it’s difficult, try many times, or write the title after everything else is done (you might find the
right words on the way.)
* Include a brief description of what happens.
* Include where the problem happens.
* Describe observations rather than (presumed) facts.
* Use a fair description, don’t exaggerate or understate the consequence of the problem
The Linear Filter Trap
”People like to solve problems that they can see - which means we have a tendency to "fix" the
problem we see in front of us. This occurs more often if the problem appears to have a
"straightforward" fix -> I think of this as a form of cognitive ease in action. Digging for root causes is a
challenging activity, and we sometimes want to believe that the cause we identify is good enough to
Delegation does not mean Negation
”With delegation of work you do not delegate your accountability.
Instead, you share it.”
Context Driven Testing
Sleeping with the enemy - Gojko Adzic describes why independent testing should be a thing of the
Traditional software delivery models are based on a lack of trust. Because the business doesn't trust
developers, testers are asked to provide independent validation. Because developers don't trust
testers, everyone wastes a lot of time arguing about whether a problem is in the code or in the tests.
And testers are taught not to trust anyone! All of this distrust even though we share the same end-
goal—delivering a product that satisfies our customers. Gojko Adzic describes why independent
testing should be a thing of the past.
Focusing Testing on Business Needs
Testers often forget that they are service providers whose role is to provide critical information to
the project’s stakeholders. Testing must focus on business needs to add the most value and gain
In this presentation I will investigate what happens when we infuse testing with aesthetics. Can the
fine arts in any way support or complement our testing efforts?
How I wish users knew how I help them through context driven testing
This talk will be about the journey of being a context driven tester and the opposition I faced and
how I paddled through it to win in most of my assignments.
Context-driven Testing is not a religion
Over the past few years, several people have gained visibility in the testing community who express
ideas and values that sound context-driven to me. Some call themselves context-driven, some don’t.
My impression is that some are being told they are not welcome. Others are uncomfortable with a
perceived orthodoxy. They like the approach but not the school. They like the ideas, but not the
We don’t manage the project managers. We don’t decide what information they have to give to the
people they report to.
Context-driven testing at a crossroads (James Bach and Cem Kaner part ways)
”Cem Kaner, who controls www.context-driven-testing.com, has announced an interesting change in
his view of the Context-Driven School. He says he prefers to think of it in terms of the Context-Driven
approach, not a school of thought. This is a significant change from his original view, which was that
CDT is a different paradigm.”
Lessons Learned from Context-driven testing (Follow up to the above article – with comments from
Cem Kaner which are interesting)
There has been some fluff and rumor around context-driven testing yesterday. Some folks even
talked about the death of context-driven testing. Most of it was issued by the about page from Cem
Kaner. If you haven’t read it yet, go ahead, read it now, I will wait here.
The Hindsight Trap
Root cause analysis (RCA) activities are an important part of many retrospectives - whether as part
of a team activity, analysis of test reports, analysis of fault (bug) reports or as standalone activities to
investigate organizational and process problems.
The most common form of this trap in testing circles (where documentation is involved) usually crops
up as a question, "why didn't you/someone spot this problem earlier (as it's specified in document
X)", or statement, "this requirement should been tested because it's specified in document X".
60% of Users Expect Mobile sites to load in 3 Seconds or Less (Importance of Load testing for mobile
That’s right, if your mobile site doesn’t load within THREE seconds 60% of visitors will abandon it.
And, seeing how we’re not a very patient society anymore, 57% of mobile users will only give your
site an extra two seconds (bring the grand total up to five) to load before aborting the mission.
Test Reporting by Michael Bolton
Calculating a ratio of passing tests to failing tests is a relatively easy task. If it is used as a means of
estimating the state of a development project, though, the ratio is invalid, irrelevant, and misleading.
At best, if everyone ignores it entirely, it’s simply playing with numbers. Otherwise, producing a
pass/fail ratio is irresponsible, unethical, and unprofessional.
Testing is like investigative journalism, researching and delivering stories to people. The newspaper
business knows how to direct attention efficiently to the stories in which we’re interested, such that
we get the level of detail that we seek.
As testers (or, heaven help us, quality assurance) people, we tend to be chartered to look for
problems, and to investigate them in ways that are most helpful to programmers, managers, and our
other clients. A failing test on its own tells us little, and a failing check even less; a failing test is only
an allegation of a problem. Investigation and study of a failing test is likely to inform us of something
more useful: whether someone will perceive a problem that threatens the value of the product.
What Metrics do you Use in Agile?
This is a great question. Too many organizations transitioning to Agile don’t ask it. They continue
using the same metrics they were using for traditional projects even though the process is
The problem is that traditional measures don’t work in Agile. Sometimes they’re actively harmful.
First, I only use metrics to get a 50,000 foot view on what’s going on.
Second, I do not use metrics to compare teams or individuals. Ever.
Third, I am much more focused on qualitative than quantitative measures.
Where do bad managers come from?
"Real" executives take the long view. They are the kind of people who, at age 60, can be found
planting trees. IT managers who think that fast moving technology requires short-term quick ﬁxes are
stuck in a middle management mentality. They will never become real executives.
How to Measure Productivity
That’s why it’s also a bad idea to rely on one metric. Use a combination of tools to focus on areas of
concern and improve them. If velocity is going up, but cycle time is staying the same, there may be
some problems with story estimation – maybe one voice has started to dominate the process. If
output is up, but velocity is down, there may be a problem with prioritization (we’re only working on
Take the Crappy Path
Developers take the Happy Path. Their Unit Tests often consist of using their code in the obvious way in
order to produce the expected results. They instinctively want to see their new creations succeed so they
inevitably focus on success. See how nicely behaved my code is! Happy, happy.
Product Managers take the Happy Path. Put nice values in, take a nice marketing-oriented screenshot of
the results. Show it to the Sales Team. Happiness all around.
End Users typically take the Happy Path during User Acceptance Testing. A little bit of typical clean input
in, typical expected output out. Now let me stop wasting time with this new system and get back to my
real work, so I can leave on time and be Happy.
Cost-Benefit Analysis of Exploratory Testing in Comparison with Scripted Testing
7.2 Scenarios in Which SBTM Can Be More Cost-Beneficial
Most of the survey respondents and interviewees were of opinion that SBTM is adaptable in any
situation of the project. And it could always be cost-beneficial if it is employed in a thoughtful way
that is complementary to the testing activities of the development team. But SBTM could be more
cost-beneficial when compared with TCBT in the following scenarios.
Requirements are vague or insufficient, and not testable
They do not have every bit of information about the system
Cost constrained projects, for example, when testing is time critical
Rapid feedback required to the developers
There is rapid and constantly change (in requirements, design, etc.)
Manual regression testing
7.3 Scenarios in Which TCBT Can Be More Cost-Beneficial
Interviewees stated that, in certain scenarios, TCBT is required.
When test execution is too expensive or too difficult and what we have is a lot of cheap time to think.
If we are working with a test platform that is quite expensive or is available less frequently, it is more
cost-effective and makes sense to think about the tests in advance, because what we have is a lot of
cheap time to think and expensive time to execute. For example, when we have to use very complex
SQL queries to test reports, it is better to prepare it before execution.
When there are regulatory requirements
Large project or long term project
Large project always involves complex settings where the test data needs to be prepared in detail.
When they do not have confidence that testers can do good testing without detailed test cases.
For example, new testers
In traditional mode, there can be a requirement for repeatable test execution
When verification of the requirements is required
When test cases for automated regression testing are required
Management 3.0 in China
In Europe I have asked many people for their favorite management practices. But I never expected to
find the most inspiring example of Management 3.0 practices in China.
37 Sources for Test Ideas
It is often stated, with right, that you should use many, different information sources in order to
come up with good test ideas.
Learning is Dead
Perhaps I’m just being an old fart, but the serendipity of these three events launched a thought that
scared the crap out of me. In these days of search engines and Wikipedia, I wonder if anyone knows
how to think anymore. Knowledge is much more than learning or regurgitating facts.
Many Models – Better Test Ideas
”Henrik Emilsson has convinced me that skilled software testing is based on invisible mental models
that help us see what can be tested.
If we can make these visible, we can sharpen our skills, and also teach testing more effectively.”
User Interface Test Automation and its Challenges in an Industrial Scenario
“The growing demand for UI test automation has triggered the development of many tools.
Researchers and developers have been continuously working to further improvise the existing
approaches. If we look at GUI test evolution we can observe a clear progress from manual testing
towards complete automation.”
utest Success Story (Interesting story about a stay-at-home dad who makes money as a freelance
tester through utest)
”I am a Gold rated uTester who, after 10 years as a stay-at-home dad, has gone from no income to
earning legitimate monthly pay and tremendous professional satisfaction in 3 short months. I felt my
story was one worth sharing and hope it both inspires and encourages those in the uTest Community,
those considering joining and those involved in creating it.”
Word on The Street: 75% of Apps Not Tested
“There’s an obvious reason why that app you just downloaded doesn’t work: it wasn’t tested! In fact,
according to a recent article on The Street, that’s the case for 75% of all mobile applications.”
Exploring Test Automation
”I’m all for saving time and money, but I have concerns with an automation approach based entirely
(or largely) on automating a bunch of manual tests. Good test design considers manual and computer
assisted testing as two different attributes – not sequential tasks.”
The Secrets of Leadership
”When people get to “Senior” roles at Microsoft, one of the things we expect is an ability to influence
without authority, work through others, and basically make those around you better. For our
introverted tech geeks, this is often a big wall to climb. Inevitably, one of these Senior non—manager
folks tracks me down and asks, “How can I get better at influence?”, or “How can I get others to
accept my ideas?”.”
Something I did not expect
”Over the past few years, I have been an ardent, arguably minority voice stating my skepticism about
the value of GUI-driving test automation, developed by someone other than the developers, after
the code is "complete."
At the same time, I've admitted that the reality of an increasingly technological society will mean that,
while we may not all need to be SDETs, we'll need to be more technical simply to participate.
Then something happened that threw me for a loop.”
The Skeptic's Dilemma
”But it’s easy to go overboard with skepticism. Time and time again, I hear good testers apply their
skepticism broadly and liberally. Some (paraphrased) quotes I’ve heard recently include:
“Model-based testing is interesting, but it doesn’t work in a lot of places”
“I’m skeptical of static analysis tools – sometimes they have false positives”
“Metrics are evil, because someone may use them to measure people”
I agree whole heartedly with each of these quotes. However, I worry that folks are throwing the baby
out with the bathwater.”
"In-The-Wild Testing" (ITWT)
"In-The-Wild Testing" (ITWT) is an effort to educate tech leaders about how to help QA teams and
organizations launch higher quality software, quicker, faster, and cheaper. The idea of in-the-wild
testing is about providing organizations with the real-world testing data necessary to make informed
decisions about releasing products to market.
What percentage of testing should be done in a lab, and how much of our testing should be pushed
into the wild, to catch issues before the users do?”
8 Things you ought to know if you do not know anything about hiring a software tester
”I believe there are some unique skills I would look for in a software tester that I would not
necessarily look for in a programmer. So, here it is.”
Maybe it is like building a house?
”A few years ago, I overheard part of a conversation between an academic department on how to
teach project management at the graduate level. One professor thought that project management
for IT projects is "just like building houses" -- you break down the work into very small chunks, add
the chunks up, make a plan, and manage to conformance to plan. The second thought it was more ...
nuanced than that. The argument split the department so much that the department chair ended up
appointing a third person to teach the slot - one of the few people in the department with a couple
of decades of relevant experience.
Which brings us to today's question: Is Project Management for software the same as it is for
building a house?”
That’s a nice theory – How to get people to accept a new approach such as agile
” In many organizations agile suggestions fly in the face of their accepted “best practices.” Such ideas
also tread on political toes. So one response I hear a lot is: “That’s a nice theory, but it won’t work
The simple reframe from “Why not?” to “What would have to change?” opens up possibilities. What
could have become an argument becomes instead a brainstorming session. The result is a chain of
steps we can take to go from where we are now to where we want to be.”
You are not a complexity thinker when …
“You’re not a complexity thinker when… you complain your model is misunderstood or
misrepresented by many people because the point of your model should be to enable sense-making.”
In Search of Code Quality (there is also a link to a fun comic in here
”Quality and care are two different ways of looking at the same thing. Quality comes from care, so
perhaps what I want to measure is the care put into coding. The fact that a developer wanted to
compile at a higher warning level or turn on more static analysis checks, or that they cared enough to
get the testers input on testability, or a peer’s advice on maintainability may be the best ways to
measure code quality of all.”
The Value of Testing
”I received the following message from Helen Huang:
”I am tester .I has work this job of 5 years on China. Now I meet some problem in my working.
1. In china , I find many company always talk the tester value ,what is the tester value ? . eg: Find the
number of bug Or find a bug problem ?”
This is a common problem.”
Testing the Limits with Anne Marie Charrett – Part 1
”uTest: In terms of money, time and effort, what is the biggest waste of resources you see in testing
today? Certifications? Test reports? Automated tools? You tell us.
AMC: Behind this waste lies the concept that there is a simple solution to a complex problem. The
above solutions are wasteful when they aim to easily resolve a complex problem. If we could stop
looking for the silver bullet, then perhaps these wasteful solutions wouldn’t look so tempting.”
But let’s deal first with the two nihilistic categories. I don’t believe that visualising quality is
impossible – it is just difficult. If it were easy we’d all be doing it already, but it is hardly in the same
category as flying through space faster than the speed of light. As for the other category, with
comments like “it’s between the user and the system, we need to put something in front of the user”
etc I disagree. Quality isn’t only between a user and a system, it is between those two and also the
people who are charged with delivering software.
The Brutal Difficulty of Transformation
”2012 has not gotten off to a great start for Eastman Kodak. Three of the company's directors quit
near the end of last year, and word recently emerged that the company was on the brink of filing for
Chapter 11 bankruptcy protection.
The easy narrative is that Kodak is a classic case of a company blind to the disruptive changes in its
marketplace. Like many easy narratives, this one is not quite right.”
On Test Design
“Testing is the dominating method for quality assurance of industrial software. Despite its
importance and the vast amount of resources invested, there are surprisingly limited efforts spent on
testing research, and the few industrially applicable results that emerge are rarely adopted by
industry. At the same time, the software industry is in dire need of better support for testing its
software within the limited time available. Our aim is to provide a better understanding of how test
cases are created and applied, and what factors really impact the quality of the actual test.”
Software Testing in Agile Development : Technological and Organisational Challenges
”The contemporary industrial trend towards agile software development brings
forth new concerns, challenges as well as opportunities. One of the main issues
concerns the quality of the final product, for which testing is the well-known
assurance mechanism. However, how to perform testing using existing expertise
in an agile environment presents a challenging issue for the software
“This result was interpreted as “Respondents would like to use TDD
to a significantly higher extent than they actually do presently”.”
”… identified and
listed seven potential limiting factors of industrial adoption of TDD: increased
development time, insufficient TDD experience/knowledge, lack of upfront design,
domain and tool specific issues, lack of developer skill in writing test
cases, insufficient adherence to TDD protocol, and legacy code.”
Scripted vs. Exploratory Testing
”I have been using the Scripted – Exploratory Testing Continuum in classes to explain how scripted
testing and exploratory testing intertwines; and to explain that most of our testing is somewhere in
the middle of the continuum.”
Model-based Testing – Some Assumptions and Traps
”In December I had a very interesting discussion - about model based testing - I was there as a
'customer' and model based testing was being 'sold'. I made my skepticism known before the
meeting, by my assertion that all testing is based on models - and that some people are better at
observing, constructing, expressing and examining them than others. I also stated I'd seen the power
of modeling techniques that allowed testing earlier but that they are far from the complete picture.”
Looking for the Great Thought about Management
”This Friday and Saturday some 20+ management thinkers and practitioners will gather in Stoos
(Switzerland) and talk about management. In particular, we will discuss why management isn’t
changing fast enough.”
Seven ways to make testing irrelevant on your team
”Wouldn't it be great if everyone on your team loved testing and working with testers? When
developers and testers have a respectful relationship, it's easier for developers to see the value and
take an interest in testing. Instead, many testers pick up a knack of alienating themselves—and
testing in general—from their team. The following is a list of ways in which testers may make
themselves and testing irrelevant to others on their team, along with tips for avoiding getting stuck in
Top Ten Reasons Why Large Companies Fail To Keep Their Best Talent
“Whether it’s a high-profile tech company like Yahoo!, or a more established conglomerate like GE or
Home Depot, large companies have a hard time keeping their best and brightest in house.”
A Christmas Gift From a Tester
”A while back I received a gift from Matt Heusser, my early mentor in the Miagi-Do school. It asked
me to test it. So, I decided to make a series of blog entries over the holidays out of it.”
Testing the Limits with Richard Stiennon (Security expert)
uTest: You’ve said before that mobile will not require its own anti-virus systems. That said, it seems
that mobile threats are multiplying by the hour. In your view, what’s the biggest security challenge in
terms of mobile?
RS: Apps, apps, apps. VPNs, firewalls, and carrier filtering are going to impede network based attacks.
Containing and vetting applications is the biggest security challenge for the platform vendors.
By 2015 the overall threatspace will be ten times worse than it is today. Think about that. There will
be TEN breaches as critical as the RSA attack. There will be TEN Google Aurora’s. There will be TEN
Stuxnets. There will be 300 thousand new malware variants a day.
Management Models, Values and Principles
For example, Peter F. Drucker said there are five tasks for managers:
1. Set objectives
3. Motivate and communicate
5. Develop people
Wilful Ignorance on Parade
What a senior tester does at Microsoft
Decision Making – senior testers make confident decisions, and do not rely on consensus for
making (most) decisions.
Influence – Dwight Eisenhower said, “Leadership is the art of getting someone else to do something
you want done because he wants to do it.” Senior testers make their team better through influence
rather than mandates – they are the point guard of the test team.
Network – senior testers use their network to find answers (or knowledge), and connect their
teammates with their network (i.e. tester matchmaking)
Credibility – senior testers don’t proclaim they’re a leader, and no title demands respect. Leaders
know that respect as a leader comes through establishing credibility with their team and peers. For
me, the moment you demand respect as a leader is the moment you lose my respect as a leader.
Test Levels in Agile Testing
Test levels and Test quadrants. How to classify different types of testing.
Would you hire a completely incompetent tester who needed to be told absolutely everything, in
“We wouldn’t hire anyone like that.”
“Fair enough, and I’d hope not. So, why do you insist that people write instructions for them that
What Exploratory Testing is Not
Michael Bolton tries to define exploratory testing further/more practically by stating what is not
”Status reporting of testing activities is extremely project-dependent. The needs of when and how
and what will differ every time. Maybe that’s why there’s so little good writing about status
communication; you have to make it up every time.”
Why scripted testing is not for novices
”Scripted test execution is quite a bit more difficult than exploratory testing, unless you are assuming
that the tester following the script has exactly the same knowledge and skill as the test designer
(even then it is a qualitatively different sort of cognitive process than designing). An exploratory
tester is following (indeed forming as he goes) his own intentions and ideas. But, a scripted tester, to
do well, must apprehend the intent of the one who wrote the script. Moreover, the scripted tester
must go beyond the stated intent and honor the tacit intent, as well– otherwise it’s just shallow, bad
Ten Must-know tips for Performance Testing
”More than other automation, bad performance test automation leads to:
Undetectably incorrect results
Good release decisions, based on bad data
Surprising, catastrophic failures in production
Incorrect hardware purchases
Significant media coverage and brand erosion
More than other automation, performance test automation demands:
Clear objectives (not pass/fail requirements)
Valid application usage models
Detailed knowledge of the system and the business
External test monitoring
Unfortunately, bad performance test automation is:
Very easy to create,
Difficult to detect, and
More difficult to correct.”
How do I deal with my crappy organization?
”It is one of the questions I get most often… How do I deal with my crappy organization? I like my
work but I don’t like what management is doing. How do I deal with it?
Well, that’s easy…”
”I got to visit another company’s location this week – Allure Global. They make dynamic signage for
movie theaters – concessions and ticketing, etc. They are an Extreme Programming shop, which
means they use the XP practices like TDD, collective code ownership, and others. I took a couple of
The Death of Quality
”“The truth is, bad software makes more money” – Goranka Bjedov, STANZ 2011.
If quality is dead, what are we doing here?
If we are adding value, what is it and how do we measure it?”
Selecting Automation Tools
If you want a tool to do functional test automation (as opposed to unit testing), you will probably
need both a framework and a driver.
The framework is responsible for defining the format of the tests, making the connection between
the tests and test automation code, executing the tests, and reporting results.
The driver is responsible for manipulating the interface.
10 Things about testing that should die
”Face reality testers, neither the product nor the business revolve around you. Think about it. No
business, no product, no developers => no need for testers. In fact, if developers wrote perfect code,
there'd be no need for you. You are a service provider and your primary clients are the managers,
developers, and/or executives. Your secondary clients are product users and investors.”
Don’t Forsake Testing in the Rush to Market
The question if it is better to get an application out early but bugged, and then fix it later, or making
good software that is late to the market.
Feature Phones still dominate World Market
With all the talk of iPhone vs. Android these days, it’s easy to forget how the majority of the world’s
mobile users still make calls and access data: via feature phones. A recently released
report (download) from mobile strategy firm VisionMobile takes a look at today’s mobile
marketplace finding that, despite the sharp rise in smartphone shipments over 2010 and 2011, global
smartphone penetration (by OS) is at just 27%
API Testing: How can it help
Key benefits of API testing include:
Reduced testing costs
Higher functional (business logic) quality
Just Fix It!
Chris McMahon’s latest post (Just Fix It) proposes that as far as bug tracking goes, the best course of
action is to skip the “tracking” part of the workflow and “Just Fix It.” I’m a huge fan of this approach
and think that for the most part, tracking a large number of bugs in a big fat bug system (and often
overemphasizing the church of the bug curve) pretty much encourage a test-last / test-quality-in
The Future is Mobile Technology
”Technology is moving toward a combination of mobility, social connections, and gaming and
That means we need to be aware of combined technologies, combined software and tools, and
combined activities. This requires that we solve new problems with new technology, or different
combinations of technology we are familiar with.”
”In many IT installations today, the number one problem is program maintenance.
Although the total problem is far from simple, there are a number of relatively simple ideas that can
be applied immediately to furnish "prompt relief." One such idea is the disposable program.”
Shape of Actions
”In The Shape of Actions, Collins and Kusch describe key differences between two kinds of intentional
human actions that they call mimeomorphic and polimorphic. In both words, “morph” refers to
shape, or form. “Mimeo-” refers to copying.
That is: software development is a polimorphic activity, and if that’s true, testing needs to respond
Test is Dead
(Test is Dead was the topic of the Keynote of the Google Test Automation Conference this year:
On the alleged death of testing
The problem here, at least in my opinion, is simple.
Testers have become so focused on legitimizing their discipline and their role that they've forgotten
that their role and their discipline exists to serve the companies that employ them.
Conversely, the managers and executives of companies that employ testers are regularly directing
testers to do these things they don't really understand rumored to be "best practices"-- and of
course, then blame the testers when those "best practices" don't provide sufficient value.
There is no "basic training" for testers, and there is no "manager basic training" that covers testing
and test management as it relates to value to the business -- and even if you believe the training
exists, you can't deny that:
a) many folks wouldn't agree that training is of value
b) even more folks who "should" have received that training haven't even heard of it
New Ways of Thinking in Software Testing (Read the comment from Alan Page at the bottom of this
To put it a different way: If your company makes it's money off of giving away free software and
selling advertisements, then, perhaps, if your customers are forgiving, and if you can respond to
defects quickly, you can be successful by hiring automated-test-writers (selenium, etc) and
crowdsourcing your testing with a product like uTest.
The idea here is to have humans find bugs fast - and cheap - perhaps over the weekend - while the
automated tools prevent major regressions.
And hey, that might just work for google, or for Yahoo, or Mozilla.
Would you want your bank, your credit card company, your automobile transmission, or the medical
device company that created your pacemaker to use such a strategy?
Alan Pages comment:
I know I don't always play nice with others, but I sort of feel that the test-o-sphere is overreacting to
the test is dead story. *If* you're shipping a web service, *and* the web service is mostly non-critical
to day-to-day mission critical application, would you pay a tester to test it? Likely, you'd want deep
functional testing and some perf load testing, and who knows what else, but it's certainly ok for
developers to write this stuff.
But I don't think we should take the message so literally. IMO, many testers have been doing
developers work (unit & functional) testing for years - and of course it makes sense for developers to
do that work. If you have the right set of diagnostic and monitoring tools in place, by all means, get
rid of testers.
Of course, as you say, if you're testing a medical device, or desktop software, you probably have to
modify your story a bit - but to me, I still think there are aspects of the story (e.g. have developers do
more testing; or collect customer usage patterns) that can apply to any software project.
The reactions (this, and others) are interesting to watch - it's almost as if testers feel that they have
to defend their job - but I see the topic as one we can all take something away from. Hell, I've seen
some pretty insane ideas come from testers, and I've found some good in nearly all of them.
Looking forward for when we can get to Test is Dead (is dead)?
Testing the Limits with Noah Sussman
uTest: What types of traits/qualities/skills does Etsy look for when hiring testers?
NS: On my team there are two roles for which I hire. These aren’t formal roles and we switch them
I hire software engineers who value quality and who are interested in how complex systems fail.
These are the people who customize our mocks and fixtures, manage our coding standards, build
static analysis tools, develop Jenkins plugins, design the CI system in general other things of that
Here I look for talented QA analysts who are longtime Etsy users and deeply involved in the Etsy
community. These are the people who develop functional testing tools, manage Selenium integration
with CI and work with others in the organization to formulate test plans and resolve bugs. They also
design and improve the process by which we triage bugs found in production.
Complexity vs Lean vs Agile vs Me
System thinking in a bar
If we want to decide whether or not something must be treated as a system, we first need to
consider what part of the world we want to look at, and what questions we want to answer.
API Testing – Functional Testing Below the UI
”Some people assume that API testing is a ‘white-box’ testing activity in which the tester has access
to the product source code. But in reality, API testing is truly black-box testing in the truest sense of
the testing approach. API testers make no assumptions about how the functionality is implemented,
and are not limited by constraints or distracting behaviors of a graphical user interface. ”
A tester’s commitments
”This is the latest version of the commitments I make when I work with a programmer.
1. I provide a service. You are an important client of that service. I am not satisfied unless you
2. I am not the gatekeeper of quality. I don’t “own” quality. Shipping a good product is a goal
shared by all of us.”
RPF Google’s Record Playback Framework
”At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test
Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here
is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as
generating ones, it works most of the time but its not perfect.”
Activities and Roles
”I’ve been thinking about testing activities and tester roles lately, or more lately, since it seems to be
a recurring topic on this blog. I have to admit that I remain a bit surprised how often testers argue
about what is and what isn’t testing.”
Iterative Development: Some History
”We were doing incremental development as early as 1957, in Los Angeles, under the direction of
Bernie Dimsdale [at IBM's Service Bureau Corporation]. He was a colleague of John von Neumann, so
perhaps he learned it there, or assumed it as totally natural. I do remember Herb Jacobs (primarily,
though we all participated) developing a large simulation for Motorola, where the technique used
was, as far as I can tell, indistinguishable from XP.”
”Many years ago, test systems at MS reached a level of maturity and scale that enabled teams to
easily run millions of tests. These systems were the culmination of several innovations, but
eventually led to other problems. The prime example of a problem created by the test system
innovation is the challenge of what to do with all of those failed tests. Think for a moment. If you run
a million tests and have a 99% pass rate (I know, but this is just an example), you have ten-thousand
failures to investigate. Automatic failure analysis is an innovation that greatly reduces the time
testers need to investigate automation failures (my stance is that automation is worthless unless
every aspect of automation beyond test authoring is not also automated). We’ve also made great
strides in test selection (let’s say you only had time to run five thousand of those million tests –
which would you run?).”
Smoke Test vs. Sanity Check
”The distinction between the smoke and sanity testing is not generally important. In fact, it’s one of
the most trivial aspects of testing that I can think of, offhand. Yet it does point to something that is
Build in the Cloud: Distributing Build Outputs (Part 4)
”In the previous post, we saw how distributing build load across many machines and reusing the
results of previous build actions can produce extremely fast build times. This distribution and reuse
itself exposed other performance bottlenecks. Specifically, a full build of a large project may produce
several gigabytes of output files, all of which need to be shipped from the cloud back to the
developer’s local machine. This taxes both our networks and the developer’s local disk, the limited
latencies and bandwidth of which slow down the build.”
Build in the Cloud: Distributing Build Steps (Part 3)
“This post builds on these to describe a system for efficiently distributing build steps across a large
number of machines. As you will see, details of the source file system and the build system are
important to how we achieve fast and efficient distributed builds.”
Build in the Cloud: How the build system works (Part 2)
“In this article we will go into a little more detail in how we tell the build system about the source
code and its dependencies. In a later article we will explain how we take advantage of having this
exact knowledge to distribute the builds at Google across a large cluster of machines and share build
results between developers.”
Build in the Cloud: Accessing Source Code (Part 1)
”Much of our day-to-day activities as software engineers involves source code. When we join a
project one of the first things we do is look at the source. We want to build it, run it, experiment with
changes, test it, and challenge our assumptions about how it works. For most of us this means we
start by “checking out” the source from version control. For small to moderately sized projects
almost any reasonable version control system is adequate. But as the number of engineers increases
and the code base grows, this can put a strain on the version control system and decrease engineer
Tools for Continuous Integration at Google Scale (The youtube video that was the foundation for the
Five orders of Ignorance and Software Testing
”0OI – lack of ignorance. It’s when you know something. I know, for example, that the first track on
the Rolling Stone’s Sticky Fingers album is Brown Sugar.
1OI – lack of knowledge. I don’t know (or remember) what the track is following Brown Sugar, but I
could find the answer quickly.
2OI – lack of awareness. You have 2OI when you don’t know what you don’t know. I know that there
are Stones tunes that I’ve never heard before, but it would be impossible for me to make a list of
Stones songs I’ve never heard before.
3OI – lack of process. You have 3OI when you don’t have a suitable method for discovering 2OI (for
discovering what you don’t know you don’t know).
4OI – meta-ignorance. You have 4OI when you don’t know about the five levels of ignorance.”
”When we test to ensure requirements were implemented, or the stories function as expected, we
are testing for 0OI.”
”Testers also perform a lot of 2OI testing. We explore or run dynamic tests on a system to help
discover what we don’t know we don’t know.”
“How do we know that our Intentions matched the Actual Need, that the Implementation matched
our Intentions, and ultimately that the Implementation matched the Actual Need?”
”Finally, if we want to be sure our Implementation matches the Actual Need we have to watch
customer behavior carefully. That means monitoring usage statistics …”
Testing the Limits with Michael Bolton (Interview)
”uTest: You recently wrote a blog saying that testers should learn to program. Conversely, should
programmers learn to test? Why or why not?
MB: Programmers should learn to test. They do learn to test. Most programmers attempt to run the
code they’ve written and look at it critically to some degree. Great programmers do that testing in a
really diversified and yet focused way. What I’d like to get rid of is the idea “I program, I test, and
therefore I know all I need to know about testing. I’m good.” To test really well requires study of
many ways that things can go wrong, and many ways in which we can be fooled, and many
approaches for defending ourselves against them. The best programmers I’ve ever worked with
were themselves excellent testers, but they were also aware of their own limitations. They were
really respectful of good testers, and, of course, good testers were respectful of them. It was a
Should testers play planning poker?
“Having observed Planning Poker in action, I’m conflicted. Estimating anything is always a bit of a
dodgy business, even at the best of times. That’s especially true for investigation and in particular for
The 3 Criteria for Agile Leadership Practices
”On Twitter I have claimed that a leadership practice is Agile when:
1. It supports people and their interactions;
2. It helps to deliver value to stakeholders;
3. And it does this by improving the system. ”
Google Test Analytics - Now in Open Source (James Whittakers ideas of replacing the test plan)
“A group of us at Google set about creating a methodology that can replace a test plan -- it needed to
be comprehensive, quick, actionable, and have sustained value to a project.”
Behavior-Driven Development vs. Testing (The comments are more interesting than the article)
”This is that BDD scenario turned into testing:
+Scenario 1: Account is in credit+
Given the account is in credit
And the card is valid
And the dispenser contains cash
When the customer requests cash
Then check that the account is debited
And check that cash is dispensed
And check that the card is returned
And check that nothing happens that shouldn’t happen and everything else happens that should
happen for all variations of this scenario and all possible states of the ATM and all possible states of
the customer’s account and all possible states of the rest of the database and all possible states of
the system as a whole, and anything happening in the cloud that should not matter but might
Book Review: Perfect Software and Other Illusions About Testing
“Some common questions from the preface of the book:
Why do we have to bother testing when it just seems to slow us down?
Why can't people just build software right, so it doesn't need testing?
Do we have to test everything?
Why not just test everything?
What is it that makes testing so hard?
Why does testing take so long?
Is perfect software even possible?
Why can't we just accept a few bugs?”
The Liberation of Testing
”In the keynote, Julian talked about how Dr. Barry Boehm has moved on from saying that bugs are
found more cheaply earlier in the software process. Instead, Boehm has been writing that it is now
just as cheap to fix bugs in production. This was the perfect setup for Keith Stobie’s presentation,
“Testing Services in Production”. Keith drove home the point that he is currently much more
comfortable testing in production than testing in any type of “staging” environment.
Aside from moving testing into a production environment, there is some testing that Keith doesn’t
even call testing anymore. He calls it “monitoring.” This is where the wall between testing and
production really crumbles away. He talked about moving asserts from tests into the production
code. I’m sure that this is nothing especially new. What’s new is that now there are people relying on
this code as an indication of system health.
The focus is undoubtedly shifting away from looking for bugs and presuming to know what
customer’s want to making sure real data is available about system health and that we know how to
“Here's my latest presentation (that I'm still working on). It is called Complexity Thinking. I did it for
the first time this week at the Lean Kanban Central Europe conference, and I will repeat it (with an
improved version) at LESS2011.”
Take a BITE out of Bugs and Redundant Labor
This is one way to empower the manual tester when it comes to browser testing of Crome. Tool-
supported manual testing. This was one of the tools James Whittaker talked about when we said that
we cannot rely on quantity when it comes to manual testing, we need expert manual testers – and
they need support from tools to be effective.
Standardized Risks and Charters for Exploratory Testing
This presentation is about standardizing how we work with exploratory testing to make it easier for
junior testers to work with it, and to make the results more easily accessible to management and
Adaptive Software Testing
This presentation is about how to use automated tests together with scripted tests and exploratory
testing in one methodology. Using automated tests to explore the risk space at low cost, and then
use this information as input to your manual and exploratory testing.
Change the environment, not the people
”People's behaviors depend on the environment. If you manipulate the environment, you manipulate
Work the network
”Behaviors spread through a social network like viruses. You must work the network to change a
Fixing bugs should be the highest priority
”I wanted to share a short but interesting video essay from Scrum co-founder Jeff McKenna on the
subject of software bugs – mainly, how we think of them and how we act on them. Here a few items I
found particularly intriguing, see if you agree:
There’s virtually no difference between high or low priority bugs
Fixing bugs is most often based on business factors (i.e. a small bug impacting a big customer)
Bug triage is waste of time in terms of learning how to write less bugs
Fixing bugs should be the highest priority
Unit tests help, but they are not a cure-all
Better to find bugs ASAP, to improve the learning”
Testing as a whole team activity
”Testing is an activity. Testers happen to be really good at it. We need testers on Agile teams. But if
we want real agility, we need to see that completing testing as part of the sprint is the responsibility
of the whole team, not just the testers.”
And some interesting exercises that could be valuable to try out for any team
“I invented a new exercise which allows people to reflect on their motivation, and how this is being
affected by organizational change.”
“The exercise allows players to visualize and discuss organizational structures.”
“The exercise allows players to discuss how to delegate decisions to a team.”
The Dangers of Delegating Discovery – Very interesting article on the dangers of delegating where
there is opportunity for unexpected innovation
”Delegation is a necessary survival skill for senior executives. But when executives delegate their
discovery-related innovation tasks, the odds of them finding the surprising insights that often spur
transformative-growth businesses decrease dramatically. ”
You Can't Analyze Your Way to Growth – Interesting article on that change often cannot be
predicted by using historic data
”But in the majority of businesses, if the available data are crunched, it shows a slowly growing
industry — one growing with GDP or population. That generally convinces the company in question
that there aren't really opportunities for top-line growth, and that in turn becomes a self-fulfilling
Four Ways to Lead a Successful Transformation
1. Make the transformation meaningful
2. Be the change you want to see the mindsets and behavior you want to see
3. Build a strong and committed top team
4. Relentlessly pursue impact
Managing an organizational transformation, executives tell us, is like trying to change the wheels on
a bike while you're riding it. You have to take your organization apart and put it back together in a
new way, but you have to keep the business running at the same time. It's a lot to ask, and the senior
leader bears much of the burden. ”
At least three reasons for testers to learn to program
”Yet there are at least three very good reasons why it might be a good idea for a tester to learn to
Testing Speeds Development
“But the reason I write this post is something Joakim Holm said at his talk, that should be told more
if you do a lot of testing, development goes faster”
The Little Black Book on Test Design
“It contains collections of knowledge, and generalizations of my ten years of testing the same
product suite. I think it can be useful for ambitious testers that want to find any problems that might
Open Lecture by James Bach on Software Testing
Google: The 10 Minute Test Plan
”Every time I look at any of the dozens of test plans my teams have written, I see dead test plans.
Plans written, reviewed, referred to a few times and then cast aside as the project moves in
directions not documented in the plan. This begs the question: if a plan isn’t worth bothering to
update, is it worth creating in the first place? To combat this, I came up with a simple task for my
teams: write a test plan in 10 minutes. The idea is simple, if test plans have any value at all then let’s
get to that value as quickly as possible.”
Microsoft: Design for GUI automation
”First off, let me state my main points for disliking GUI automation:
It’s (typically) fragile – tests tend to break / stop working / work unsuccessfully often
It rarely lasts through multiple versions of a project (another aspect of fragility)
It’s freakin’ hard to automate UI (and keep track of state, verify, etc.)
Available tools are weak to moderate (this is arguable, depending on what you want to do with the
Test Design for Automation
”A common variation of this is the automation of manual tests. It pains me to hear that some testers
design scripted test cases, and then automate those actions. This tells me two things: the manual
test cases suck, and the automation (probably) sucks. A good human brain-engaged test case never
makes a good automated test case, and automating a scripted scenario is rarely a good automated
test (although occasionally, it’s a start). Some teams even separate the test “writers” from the
“automators” – which, to me, is a perfect recipe for crap automation.”
More Thoughts on Leadership
”Leadership is much more than management. A successful leader manages projects by articulating a
clear vision, guiding people towards achieving goals, and motivates people by helping them grow.
When folks ask me what my first priority is as a Test Lead I say it is the people on my team. But, what
does that mean?”
Titles for Testers
”I’ve had a few conversations on twitter over the past few weeks about tester titles. My conclusion
is that test titles are meaningless, but let’s look at why I think that, and why it doesn’t matter.”
Changing People’s Barriers
”The crucial parts of a social system are the people in it. And since all people are different, there’s no
one-approach-fits-all solution for change. If you need an organization to improve, you have to work
with people’s individual needs, and the various barriers people put up in their minds…”
Tomorrow through the Past
”This month, on the ASQ blog Paul Borawski notes that perhaps thirty percent, or less, of the
attendees at a typical American Society for Quality (ASQ) have heard of W. Edwards Deming, the
champion of quality. He asks, in essence "Are we forgetting our history?"
I see similar issues in my own tiny sub-field of software testing, mostly in the area of test automation.
I have seen organizations implement test automation, expecting test cost to go down ... only to see it
Dealing with locale/language specific static test data
”Many regular readers know that I am a strong proponent of pseudo-random test data generation in
conjunction with automated testing to increase the variability of test data used in each test iteration
and generally improve test coverage. But I also understand the value of static test data in providing a
solid baseline, and in some cases enabling access to specific test data in different locales or
In the Middle
”Should you automate everything, or nothing? Should you test everything, or nothing? How about
leadership – should you dictate every detail of what your team should do, or give them no guidance
at all. The answer for all of these questions – as you’d expect, is “somewhere in the middle”.”
Managing Multiple Bosses
”In the movie Office Space — a comedy about work life in a typical 1990s software company — the
protagonist, Peter Gibbons, has eight different bosses. All of them, seemingly unaware of each other,
pass by his desk and tell him what to do. While the film is most certainly a satire, for some, it is not
far from the truth. More and more people report to more than one boss and learning to handle
multiple managers is an essential skill in today's complex organizations. ”
Why You Should Automate Parts of Your Job to Save It
”What is the most important thing you do on your job? What portion of that could be turned into an
app that anyone in your organization could effectively use? What portion of that could be automated
and fed directly into the larger system with only minimal review by you? What's the least valuable
but essential part of your job? Why aren't you figuring out ways to automate it on your iPad or
Exploratory Testing in an Agile Context
Actually a very good presentation on exploratory testing, and especially chapter 2 about writing
charters is very interesting. Recommended read.
Technique: Paired Exploratory Survey
”Paired exploratory survey is a process whereby two testers confront one product at the same time
for the purpose of learning a product, preparing for formal testing, and/or characterizing its quality
as rapidly as possible, whereby one tester (the “driver”) is responsible for open-ended play and all
direct interaction with the product while the other tester (the “navigator” or “leader”) acts as
documentarian, mission-minder, and co-test-designer.”
Re-thinking thinking and planning
”In the tutorial, Hooray We’re Agile Testers! What’s Next?, Janet Gregory apologized a couple of
times for saying upfront thinking or planning. I know Janet wanted to let the audience know that she
isn’t a fan of massive test plans or documents written way ahead…But her remarks got me
wondering. Why in the agile community is it a taboo to recommend or admit to doing any upfront
thinking or planning?”
Who says exploratory testing is good for medical devices? The FDA!
”This is what I’ve been arguing for a couple of years, now. If you want to test a medical device very
well, then you have to test it in an exploratory way. This prepares the way for what the FDA here
calls the “pivotal study”, which in software terms is basically a scripted demonstration of the
Pretotyping: A Different Type of Testing
”Testing typically revolves around making sure that we have built something right. Testing activities
can be roughly described as “verifying that something works as intended, or as specified.” This is
critical. However, before we take steps and invest time and effort to make sure that something built
right, we should make sure that the thing we are testing, whether its a new feature or a whole new
product, is the right thing to build in the first place.”
Criticism against Test Tool Vendors
”Test tool vendors that bug me:
Any vendor who designs tools by guessing what will impress top managers in large companies who
know nothing about testing. In other words: tools to support ceremonial software testing.”
Effective, Efficient, and Elegant Software Testing
”In chapter 8 *of the Advanced syllabus], I was reading about Standards and Test process
improvements. There you talk about Efficiency and Effectiveness. I have one definition about these
terms, but what do I have in mind about these terms in order to be prepared for the advanced exam?”
The test process and important testing techniques
”As software development gets more complex and high paced, the pressure on testing
becomes greater to find business critical defects in the software as fast as possible.
But testing that is only introduced in the last stages of development will most likely
not give any valuable information. The fact is that testing needs to be an active part of
development even at the early stages. Requirements need to be written with the
input from the testers and testers need informed of what requirements matter, in
order to start writing test cases early on. By making the process more effective
testers have the chance to increase the quality of both the testing and software.”
”A Testing Dojo is a meeting where testers come together to work on a testing challenge. The testing
challenge can consist of testing a product, or generate test ideas for a particular software, or even
exercise bug reporting. Mainly the testing challenges will use Exploratory Testing.”
”Testing provides information, so you should find out what is important to different people, and give
them what they need, not necessarily what they want.
Metrics will hide importance, and also skew the development efforts.
Try to take them out, with surgical precision.
Then you are left with one final catch:
Qualitative information is hard to aggregate, trust is needed.”
How Pradeep teaches software testing
How we tested Google instant pages
”How did this testing get organized? As with many things in testing at Google, it came down to
people chatting and realizing their work can be helpful for other engineers. This was bottom up, not
top down.” // “Anytime test automation scales, is repeatable, quantified, and developers can
validate the results without us is a good thing!”
Interview with Michael Cooper of T-Mobile
”uTest: What’s the best way for them to overcome the unique challenges of mobile app testing?
MC: I have found that mobile app testing requires a more technical type of testing, and automation is
key. We are also looking into crowdsource testing to augment our current testing handset
application testing efforts.
uTest: What are the biggest pitfalls they should try to avoid?
MC: Don’t forget about performance and security testing when working with a partner, or a vendor
of mobile testing.
uTest: You’ve managed some pretty large testing teams throughout your career. What advice do you
have for QA managers whose testing teams are growing beyond 3-5 person teams?
MC: I recommend standardizing your testing deliverables and keeping track of metrics so you can
manage by facts. Be independent – by that, I mean do everything in your power to not have to
report to development. Speak your mind…without whining and complaining. Learn from your
mistakes! If you or your team make a mistake or miss a defect, take responsibility and do whatever it
takes to ensure that it doesn’t happen again. And, finally, treat your team and your peers with the
utmost respect and dignity.”
Software Design Engineer in Test vs. Software Test Engineer (Microsoft)
”Testers today face many challenges, and hiring great testers (regardless of the job title) is about
finding people who not only have a passion and drive to help improve our customer’s experience and
satisfaction, but can also solve tough technical challenges to advance the craft and help improve the
Software G Forces: The Effects of Acceleration (Suggested to me as very interesting by Magnus
Ernstsson, former DA at PLD SW)
”As deployment cycles shrink, what constitutes effective software engineering changes radically.
Practices that bring improvement to a quarterly release cycle can be fatal with an hourly release
cycle. This talk outlines the changes required of software engineering and organization at different
cycle times: quarterly, monthly, weekly, daily, and hourly.”
Designing Mobile Apps Using Old School Tools for New School Technology
”If we can figure out what we are building and why early on, and support programming by exploring
the design quickly, that can save a lot of painful re-work later on. Some old school tools in my
Business Analyst design toolbox have turned out to be surprisingly appropriate for mobile application
design. I’ll focus on two of them: context-free questions, and state transition networks.”
Seven Guidelines For Designing High-Performance Mobile User Experiences
”We’ve gone over seven guidelines that address performance relative to different aspects of mobile
app design. Design choices affect performance, and so performance should be considered a key
factor in the design process. Unfortunately, it tends to be looked at too late in the process, which
ends up impairing the user experience significantly.”
”Tours help the tester gain insight into the multidimensional nature of this complex, infinite space.
They help us imagine, envision, and as we gain experience on the tour, prioritize the different
sampling strategies we could use when we do more thorough, more intense testing after finishing
the tours. So which tour is best? The ones that give the testers more insight and that achieve a
greater stretch of the testers’ imagination. For this, some tours will work better for me, others for
Testing the Limits with Jim Sivak (Senior QA Manager at McAfee)
”uTest: In your opinion, is there a law of diminishing returns when it comes to software testing? In
other words, after a certain amount of testing, does it really make a difference?
JS: Ah, probably the most asked question—are you done testing? There are diminishing returns
although my definition might be a bit different than some others. I stop testing when the
development team will no longer fix problems because the business has decided that the release
must happen. In other words, if bugs that are found are just all deferred, then it is time to stop.
uTest: You’ve hired your fair share of software testers over the last 30+ years. What sorts of skills
and attributes do you look for when filling out your roster?
JS: The best people I hire are the ones who have three key attributes—they love to learn, they
constantly look to do things better (not satisfied with the status quo) and they love puzzles. To be a
great QA engineer, you have to continuously learn to make yourself and the team better. The best
teams continually try new approaches and processes. And testing is solving puzzles—how do you
make sure it works, how do you determine what can break and when it does, doing initial looks at
what actually happened. I feel that the other skills—programming, product expertise, etc. can be
learned on the job.”
The Purpose of a Business is NOT customer value
”The aim proposed here for any organization is for everybody to gain - stockholders, employees,
suppliers, customers, community, the environment - over the long term.
W. Edwards Deming – New Economics”
Dance with the System
”PDCA, which stands for Plan, Do, Check, Act, is an iterative four-step management process. Any
change management initiative consists of a sequence of interventions in a system (or its
environment), where you check the effects the interventions have on the system. When your actions
work as intended, you move on to the next step. When they don’t work, you try some other ones. It
is a never-ending cycle of setting goals (Plan), implementing ideas (Do), measuring feedback (Check)
and analyzing results (Act).”
Do software testers need a college education
Best practice for Mobile Application Development
”In order to develop feature-rich mobile applications, there are some key development points which
need to be followed upon at the very early development stage. These points will surely enable an IT
outsourcing software development to achieve an optimal consumer experience as well as
Failing Fast, 20% Time and Project Mobility
”Failing Fast. Nothing destroys morale more than a death march. Projects going nowhere should do
so with the utmost haste. The ability of a company to implode pet projects quickly correlates directly
to a great place to work. Engineers working on these project gain not only valuable engineering
experience, they experience first-hand the company's perception of what is important (and, in the
case of their project, what is not important). It's a built-in lesson on company priorities and it ensures
good engineers don't get monopolized by purposeless projects. You gotta like a company willing to
experiment. You have to love a company willing to laugh at itself when the experiments don't pan
20% Time. Any company worth working for has any number of projects that are worth working on.
It's frustrating for many super-sharding engineers to see cool work going on down the hall or in the
next building and not being part of it. A day job that takes all day is tiresome. Enter 20% time, a
concept meant to send a strong message to all engineers: you always have a spare day. Use it wisely.
Project Mobility. Staying fresh by changing projects is part of mobility. Continuous cycling of fresh
ideas from new project members to existing projects is another part. The downside here is obviously
projects with a steep learning curve but I scoff in the general direction of this idea. Whose fault is it
when a wicked smart engineer can't learn the system fast enough to be useful in some (even a small)
context? Only the weakest organization with the poorest documentation can use that excuse. The
only good reason for keeping people on a project is because they have no desire to leave.”
Levels of Testing
”When I teach fundamental testing concepts I often use the ‘levels of testing’ as a model to help
students conceptualize different types of testing activities and potentially different stages of testing
processes during the software development lifecycle (SDLC).”
Automatic Measurement of Source Code Complexity
”The aim of this master thesis is to explore the area of software metrics and to identify
software metrics related to the code complexity. In this thesis, thorough study is made to
determine whether or not the automatic measurement of source code complexity is
Change Management 3.0
”No matter how big the CEO’s desk is, an organization is not a hierarchy. It is a social network.
Sometimes you would like to change something in that social network. Sometimes you want people
to be nicer, more mindful, or better disciplined. Sometime you just want them to fill out a form, or
give you some money. When you want to change behaviors in something as complex as a social
system, you have to understand the four aspects of change management.”
Fully Automatic Software Testing possible?
”One of the fads that seems to be reappearing is the idea of automating away humans from the
software development work at all. This fad came up with the rise of UML, and the most recent fad
appears to be model-based testing. One of the interesting things I noticed is the ignorance of other
past movements. It seems that universities keep on bringing up new talents from time to time that
claim to save the world because universities favor a particular competition-based learning model in
which everyone wants to be the next hero.”
Beyond Agile Programming
Testing at the Speed and Scale of Google
”Continuous integration systems play a crucial role in keeping software working while it is being
developed. The basic steps most continuous integration systems follow are:
1. Get the latest copy of the code.
2. Run all tests.
3. Report results.
4. Repeat 1-3.
This works great while the codebase is small, code flux is reasonable and tests are fast. As a codebase
grows over time, the effectiveness of such a system decreases. As more code is added, each clean
run takes much longer and more changes gets crammed into a single run. If something breaks,
finding and backing out the bad change is a tedious and error prone task for development teams.”
Basics Revisited: Test Strategy
”I see many test strategy documents in my work with clients. Almost without exception, they are
big—fifty or more pages—and stuffed with soporific boilerplate copied from one project to the next.
Defect management process, test suspension and resumption criteria, entry and exit criteria,
meaningless generic risks ... yawn ... again. One frequently used model even parrots textbook test
definitions, from unit to systems integration to load.”
How do you decide what to test?
”The “Test This” game pops up frequently in testing circles. Sometimes it pops up in the “How would
you test a …” variety – e.g. How would you test a stapler. There are better flavors of the Test This
game. Sometimes, testers play the Test This game with real software. This form has huge potential,
but never really delivers, as the software under test is typically a bug-infested piece of crap that
anyone with a pulse could find bugs. Bugs are part of my problem with this form of Test This – the
goal of this game is to find bugs – not to test or learn testing. That’s an important distinction for me –
the goal of testing isn’t to find bugs. Finding bugs is merely a side-effect of the testing process. I fear
that these exercises encourage testers to dive into bug finding before thinking about how to test. To
be clear, I’m not talking about big-upfront-test-design – but a holistic approach to testing.”
Triple Loop Learning
”The first thing you should do is to gather requirements. What is it that users want your product to
be capable of? Do they want a long battery life? A high quality GUI? Automatic updates? A smooth
grip? A sturdy case in the color pink? You go out of your way to get a decent list of features to
increase your understanding of what is necessary to build the product. It also reduces the risk of
welding the foobar to the plingle when users prefer to attach it to a vringle. (I just made that up.)”
Tester Respect at Microsoft
”Here’s a tip – you don’t "get" respect, you earn it. In my experience, testers "earn" respect by being
credible, showing good results, and making good decisions. You don’t "get" respect from your "level",
or by demanding it. Seriously, stop whining and figure out how to earn respect.”
Testing the Limits with James Whittaker
”uTest: The world of testing moves incredibly fast at times. That said, is it possible to “future-proof”
the theories and methods you write about in your books? Talk about some of the challenges of trying
to stay current on the latest testing trends.
James: That’s easy. You stay current by inventing the latest trends not following them. Invention is
less about being able to read the future and more about a willingness to experiment and be wrong.
You should see our drawing room floor at Google, it’s a graveyard of failed experiments. We
purposefully try to do things differently to make sure we’re not missing some great new idea or
technique. We’ll try almost anything once and we push hard to fail fast. Staying open minded is the
key. If you are willing to experiment and expect everything you try to fail, when something actually
works, well, you know its something special.”
The Illusion of Control
”Your eyes capture only a tiny percentage of what happens around you. Your memories are largely just
a figment of your imagination. And your “free will” is little more than an elaborate illusion created by
your brain. (The Grand Delusion - NewScientist, 14 May 2011)”
The Life of a Test Engineer at Google
”Obviously, the work of a TE varies greatly depending on the project. Some TE’s spend much of their
time programming, much like an SET, but with more of a focus on end-to-end user scenarios. Other
TE's take existing code and designs determine failure modes and look for errors that will cause those
failures. In such a role a TE might modify code but not create it from scratch. TE's must be more
systematic and thorough in their test planning and completeness with a focus on the actual usage
and system experience. TE's excel at dealing with ambiguity in requirements and at reasoning and
communicating about fuzzy problems.”
Test Automation – Beyond rudimentary script-lets
“In the crud example we basically entered a static “hard-coded” string into a textbox and used a
simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the
same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition
functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string
gives great confidence that that string works, but not other strings.”
”Framing is the network of logical statements and connectives that link the three story elements and
their details. To frame a test means to create an unbroken line of logic from the mission—the
information objective within some context—through the product story and our activities, down to
the observed outcome of the test and our interpretation of it.”
Binary Disease (lots of comments on this article – discussions regarding pass/fail and how to present
”We make software for computers, and use computers for planning, execution and reporting.
Our theories reflect this much more, way much more than the fact that each piece of software is
unique, made for humans, by humans.
Take a look at these examples:
* Pass/Fail hysteria; ranging from the need of expected results, to the necessity of oracles.
* Coverage obsession; percentage-wise covered/not-covered reports without elaborations on what is
* Metrics tumor; quantitative numbers in a world of qualitative feelings.
* Sick test design techniques, all made to fit computers; algorithms and classifications disregarding
what is important, common, risky, error-prone.”
Software Requirements Quality
”I’m running an ISTQB Advanced Test Analyst course this week in DC. This course includes a hands-
on set of exercises based on a realistic requirements specification.”
How Google Tests Software – Part Six
”SETs are Software Engineers in Test. They are software engineers who happen to write testing
functionality. First and foremost, SETs are developers and the role is touted as a 100% coding role in
our recruiting literature and internal job promotion ladders. When SET candidates are interviewed,
the “coding bar” is nearly identical to the SWE role with more emphasis that SETs know how to test
the code they create. In other words, both SWEs and SETs answer coding questions. SETs are
expected to nail a set of testing questions as well.”
How Google Tests Software – Q&A
”SETs and SWEs are on the same pay scale and virtually the same job ladder. Both roles are
essentially 100% coding roles with the former writing test code and the latter doing feature
development. From a coding perspective the skill set is a dead match. From a testing perspective we
expect a lot more from SETs. But the overlap on coding makes SETs a great fit for SWE positions and
Automation isn’t bad – Bad automation is bad
”The first problem in almost any discussion about automation is the lack of context. Automated tests
are different and when done well can be effective at helping us identify some types of functional
issues. An automated unit test suite can be a powerful tool for developers during refactoring. Of
course some bugs escape unit testing because unit tests are not intended to be comprehensive. I
have long been a proponent of functional testing at the API layer and this is where my team plays
today. I see first hand the value of our automated tests in the early identification of potentially
critical functional issues during our continuous integration process. While this level of testing is
effective in helping prevent build breaks, it is not comprehensive and does not test the behavior of
the application through the user interface. And then, comes GUI automation.”
More on the Coding Tester
”It’s important to note that the role of a coder-tester is NOT to automate everything (everyone
knows that you should automate 100% of the tests that should be automated). Writing automation is
one task of a coding tester, but certainly not the primary focus (yes, I know that in some companies,
automation “experts” only write automated tests. In my opinion, these people are not really testers,
so they don’t count).”
To automate or not to automate
Good testers test first – or at the very least they think of tests first. I think great testers (or at least
the testers I consider great) first think about how they’re going to approach a testing problem, then
figure out what’s suitable for automation, and what’s not suitable. I say that like it’s easy – it’s not.
My stock automation phrase is "You should automate 100% of the tests that should be automated".
nearly every tester knows that, but finding the automation line can be tricky business.”
Management – The following 30 days
”By day I find my time split between people management (team meetings, 1:1’s, etc.) and project
management (triage meetings, status meetings, external partner sync-ups, etc.) At night I read
through the API specs and brush up on my C++.”
Working on Lync
”The "how" part of our work is (to me) much more interesting (and a big reason why I love working
on this team). We strive to have a culture high in trust. We have few (none that I can think of, but I’m
sure I’m missing something) top-down mandates. We believe that the people closest to the work are
in the best place to make decisions on what the work is that they need to do. We obviously give
more guidance to employees new to the team and managers are involved in coaching, but for the
most part, we trust the people on the team to do what they think is best. For example, we have no
requirements on test case counts, bug find rates, code coverage rates, or anything else like that. If
someone decides that working on a side project for a few days will help them get their job done
better, they don’t need to clear it with anyone – they just do it. Failing is ok (and expected – if
you’re not failing, you’re not trying hard enough). ”
Ensuring software quality (the first link is a review of the second link)
Management – the first 30 days (Microsoft)
”It has been more than 8 years since I directly managed a team. In that time some ‘operational’
things have changed, but most importantly I have changed. Personally, I think my responsibilities as a
lead include helping the people on my team grow their career, providing a vision and guidance
towards clear goals, protecting the team from political fallout and pressure, managing the features(s)
I am responsible for, preventing project and personal ‘fires,’ and also actively participating in the
testing effort. There are skills, experiences, and values that I have learned in the past that will help
me in my transition, there are some adaptations to make, and there is a whole lot to learn.”
”Jurgen continued to develop his management model, which consist of six parts – and that formed
most of the remainder of the course:
Improve everything ”
Structure in explortatory testing
“When someone says “Exploratory testing is unstructured,” I immediately hear, “I have a very limited
view of what ‘structure’ means.””
It’s a software crisis!
“At least that’s what they were saying in the late 1960s. By chance, I stumbled across a brief
Wikipedia entry today on that precise term and suddenly realized that very little has changed in 50
Software testing in the development lifecycle (V- W- model)
”I think an important thing to remember about these lifecycle models–whether V-model, W-model,
Agile model, iterative model, etc.–is encapsulated in a witticism attributed to W.E. Deming: “All
models are wrong; some are useful.””
Questions to ask during debriefs
”What questions do you ask during an Exploratory Testing session debriefing?”
People decide, models inform
”Reducing variety is often not the right approach when trying to handle the complexity of the world.”
Summary of Swedish Exploratory Testing Workshop 2
Does pair programming obviate the need for code review?
”If you're working together with another person to write software, surely you don't need someone
else to look over the code. That's part of the process. Or... is it?”
Three years of Scrum
”More than an article about Agile transition, this is about actually doing Scrum, with a focus on the
daily standup meeting and how it changed over the years.”
Software Engineer in Test career path at Google
”SETs are developers who write test code and automation as their primary task. They are in every
sense of the word a developer. When we interview SETs, SWEs are on the interview loop and SWE
questions are asked. They are not all of the interview, but they are part of it.”
My job as a tester at Microsoft
“I have somewhat of a non-traditional test role at Microsoft. I’m a tester (that’s what my business
card says), but I don’t own any specific components or features. I test, but mostly where I want to
discover something for myself, experiment with an idea, or when I’m coaching others. The bulk of my
time, in fact, is spent coaching and mentoring other testers on the team, as well as working with the
test managers on the team and our test director to make sure we’re doing the "right thing" with both
strategic and tactical decisions and goals.”
It takes complexity to handle complexity
”In other words, in order to survive a system must have an internal model that reflects the variety it
encounters in the world outside.”
Questioning test cases: Part 2
”I often hear this kind of justification for test cases: “We need test cases so that testers can learn
quickly how to test the product.” To me, that’s like saying that we need driving cases so that people
can learn quickly how to drive in a new city. In a new city, a map or a general set of ideas might be
helpful. Driving cases won’t, because to drive through a new city well, you have to know how to drive.
In the same way, test cases are unlikely to be anything more than negligbly helpful. In order to learn
quickly how to test a new program well, you have to know how to test.”
Questioning Test Cases: Part 1
More of what testers find: Part 2
Dissecting the V-model
What is test control
Testing the Untestable
“Robert Scoble recently interviewed Trey Ratcliff (one of my favorite photographers) about his new
photo editing app for the iPad. Trey remarked that because there is no camera on the iPad 1, they
had to “blindly” add the feature for taking pictures using the built-in camera – that is, without testing.
That’s because when they wrote the feature, the iPad 2 (which includes a camera) didn’t yet exist.
Kind of scary, but understandable. No amount of testing in the world would validate that feature
until the iPad 2 went on sale.”
A follow up by Michael Bolton on James Bach’s article on what testers find
”How do we know that something threatens the value of the product? The fact is, we don’t know for
sure. Quality is value to some person, and different people will have different perceptions of
value. Since we don’t own the product, the project, or the business, we can’t make absolute
declarations of whether something is a bug or whether it’s worth fixing. The programmers, the
managers, and the project owner will make those determinations, and often they’re running in
different directions. Some will see a problem as a bug; some won’t. Some won’t even see a problem.
It seems like the only certain thing here is uncertainty. So what can we testers do?”
It was a big deal – an article about unintended consequences
”Look, I've been at companies before with a "save the world" mission, I understand the pace of
development, and I know how hard it can be to take a few minutes to look around, think, and say
something like "Wait ... this feature has some unintended consequences."”
Why information is important (Microsoft)
”In the absence of information, people will make stuff up.”
”I was thinking about some of the thoughts and rumors about testing at Microsoft – particularly
some of the musings about why MS only hires testers with coding knowledge for full time positions.
There are many reasons for this move, and most of them are good reasons – but the role shift hasn’t
been without its problems.”
What testers find
”While testing at eBay, recently, it occurred to me that we need a deeper account of what testers
find. It’s not just bugs. Here’s my experimental list:
Testers find bugs.
Testers also find risks.
Testers find issues.
Testers find testability problems.
Testers find artifacts,
Testers find curios.”
Testing the Limits with Jon Bach
”uTest: Part of your new role at eBay will be to hire and recruit a top-flight team of testers (in
addition to the ones already there). What sort of traits/skills/attributes will you be looking for in
JB: The ability to come up with ideas – either old or new – and execute them in a way that helps us
improve notions of Search. For years, I used the triangle program in test auditions. Now I use
something more simple. I draw a long horizontal rectangle on the whiteboard with a little “Submit”
button below that. I say “this is a text input field for Search, just like the one you see on the eBay site.
Help me create a test plan for it.” I’m hoping that instead of an interview, it comes across more like
an invitation to a real collaboration.”
Bugs that automated tests aren’t good at finding
”Sure, some automated tests can help find bugs, but more importantly it is one approach I might use
defect prevention (esp. computational logic problems),
earlier identification of key integration issues,
potential degradation of critical areas (battery, performance, memory),
efficient execution of redundant ‘checks’ (if necessary or for confidence)
more effective/precise ‘oracles’ as compared to humans
cost reduction in long term sustained engineering”
If you think the primary goal of an automated testing effort is to find lots of bugs, or the same types
of bugs as a person then you probably don’t know very much about test automation.”
Advanced software testing: Bug isolation
”Reader Gianni Pucciani has a good question about a question in the Advanced Software Testing:
Volume 2 book (regarding bug isolation)”
How Google Tests Software – Part 5
”Instead of distinguishing between code, integration and system testing, Google uses the language of
small, medium and large tests emphasizing scope over form. Small tests cover small amounts of code
and so on. Each of the three engineering roles may execute any of these types of tests and they may
be performed as automated or manual tests.”
Testing with Code
”However – if your team only writes regression-ish automation – or if your view of the tester-
developer is someone who only writes automated regression tests, you are completely missing the
point of coding skills for testers. I know (and fear) that this is the case for way-too-many people in
the profession today, and I find it a bit disheartening.”
Innovation at Google
”Patrick Copeland presents the first three principles of the eXtreme innovation approach based on
the Pretotyping Manifesto: Innovators Beat Ideas, Pretotypes Beat Productypes, and Data Beats
Calculating Defect Detection Effectiveness
”You are a test manager in charge of system testing on a project to update a cruise-control module
for a new model of a car. The goal of the cruise-control software update is to make the car more fuel
efficient. Assume that management has granted you the time, people, and resources required for
your test effort, based on your estimate. Which of the following is an example of a motivational
technique for testers that will work properly and is based on the concept of adequate rewards as
discussed in the Advanced syllabus?
A. Bonuses for the test team based on improving fuel efficiency by 20% or more
B. Bonuses for the test team based on detecting 90% of defects prior to release
C. Bonuses for individual testers based on finding the largest number of defects
D. Criticism of individual testers at team meetings when someone makes a mistake”
Are testers made or born?
”So, is having a team of really good testers and turning them loose without any guidance a good
strategy for testing? No.”
Technical Debt and Accidental Complexity
”Introducing technical debt increases accidental complexity, and as a side-effect invalidates previous
estimates and increases the likelihood that future estimates will differ in size drastically.”
Does Quality come from Testing?
”Do QA professionals truly assure the quality of a product, or do they assist in delivering high quality
Simulating your way out of regression testing
”Julian spoke about alternative testing – or better put alternatives to testing – arguing that time to
market is a key competitive advantage for Internet businesses and that exhaustive testing damages
that. He cited examples of Facebook, Flickr and Google as companies that do not perform exhaustive
regression testing before releases but deal with problems effectively when they appear.”
”I came up with the idea for Delegation Poker when I tried to create a game that would teach people
the following ideas (learning objectives):
Delegation is not a binary thing. There are plenty of “shades of grey” between being a dictator and
being an anarchist.
Delegation is a step-by-step process. You hand over accountability to other people, in a controlled
and gradual way.
Delegation is context-dependent. You want to delegate as much as possible; but if you go too far,
chaos might unfold.”
Triangle Test Revisited
Implementing Lean and Agile software development in Industry
You need testers because …
“… testers have a different mindset as compared to developers, and we have different perspectives.
Some testers may write, or partner with developers for component level or API tests, but most
testers generally focus on integration and system levels of testing. Using various approaches in our
craft we often expose functionality issues as well as behavioral or usability issues that might
adversely your customers.”
Beyond regression tests
“In my opinion, an automation strategy that only performs regression testing is short-sighted and
“Regression tests, for all the value they bring, tend to train the system to pass.”
Battling the Bias
”There seems to be a particular word that crops up again and again in blogs, papers and
presentations about software testing and beyond:
How Google Tests Software Part 4
”One of the key ways Google achieves good results with fewer testers than many companies is that
we rarely attempt to ship a large set of features at once. In fact, the exact opposite is often the goal:
build the core of a product and release it the moment it is useful to as large a crowd as feasible, then
get their feedback and iterate.”
Tips on Android Usability Testing
“… it is important for app developers and testers to focus on usability from the very beginning. An
app that is is easy-to-use and intuitive, and similar to industry-accepted interfaces will tend to do
well. Usability for mobile apps is not the same for desktop apps: For mobile apps especially the input
method, screen size and connection speed differs.”
Articles about ISTQB test certificates
Certification is evil?
We don’t need no education – follow up on the above articles about certification
The Optimal Fallacy
“There exists a Lean principle called “Optimize the Whole”, which is too often interpreted as an
instruction to only optimize “the whole of the system” and to refrain from optimizing subsystems in
Test Automation: Checking for Bit-ness
“So, rather than having separate automated tests for 32 and 64 bit operating systems a better test
design approach might be to simply detect the bit-ness and auto-magically set variables or redirect
the control flow path through your test for the appropriate operating system environment.”
Goals, Values, and stuff that makes me mad
”The problem many testers face is that they’re too busy with their "day job" to look up and think
about the future. This is as true for looking at the big picture of testing as it is for thinking about their
career. I call this "The Tester Treadmill". What I do often in my mentoring sessions is try to help
people look up and see a little ways into their future.”
Selecting Software Testing Tools
”As software engineers, we want to build useful things, and tools can make us more effective and
efficient in doing so. Before we start to use a tool, we should understand the business objectives the
tool will promote. Understanding the business case will allow us to properly select a tool. With the
tool selected we can then go through one or more pilot projects with the tool, followed by a wider
deployment of the tool. As we deploy—and after we deploy—we should plan to measure the return
on investment, based on the business case. By following this simple process, you can not only
achieve success with tools—you can prove it, using solid ROI numbers.”
Problems and Future solutions – Keynote from an Indian test conference
Acceptance Test Driven Development
Acceptance Test Driven Development brings teams together
Testing the Limits with Elizabeth Hendrickson
State Transition Testing: Thinking in models
”Of course, some people found 1 or 2 issues using an exploratory approach without first creating a
state diagram. But, after viewing the state diagram those folks were also able to realize paths they
had not considered or traversed and were able to find more issues. The single biggest epiphany that
most people have after participating in this exercise is how much more they understand the feature
they are modeling compared to what they thought they knew.”
The Dual Nature of Context Driven Testing (read the comments for criticism of the article and James’
“Five schools of testing
Context Driven: Because testing (and any engineering activity) is a solution to a very difficult problem,
it must be tailored to the context of the project, and therefore testing is a human activity that
requires a great deal of skill to do well. That’s why we must study it seriously. We must practice our
craft. Context-driven testers strive to become the Jedi knights of testing.
Agile (the only other school that names itself): sees testing as an adjunct to programming, preferably
automated. They accept that exploratory testing is a good thing, but don't feel that there are any
special testing skills that deserve or require development.
Quality Control: sees testing as an admission of process failure. Someone in the Quality Control
school of testing keeps reminding everyone that bugs must be prevented, rather than found. They
are testers but don't want to be testers.
Analytical (mainly academics): see testing as a fascinating mathematical problem that they are
content to solve in well-bounded research exercises, and then hope to scale up.
Factory (the biggest school): sees testing as a matter of systematically manufacturing testing artifacts.
They focus on documentation, metrics, and detailed instructions intended to control tester behavior.
They love automation. T-Map and TPI are great examples of the factory school in action.+”
Risk-Based Testing: How fine grained should we be
”… the question of granularity of the risk analysis is also important. The granularity must be fine-
grained enough to allow unambiguous assignment of likelihood and impact scores. However, if you
get too fine-grained then the effort goes up to an unacceptable level. A proper balance must be
4 Ways to Make the Ship No-ship decision
”In the last decade, I’ve seen a few ways to make the call. I present a four ways I’ve seen for “when
to ship” decisions, along with some of the tradeoffs involved with each technique. (And, unlike that
textbook, I do it without the big words or fancy symbolic notation, thank you very much.)”
Productivity and Projects
”We have a lot to draw on for planning and getting productive, but the concept of staying productive
is often lost on us. How a team can sustain value creation over the long haul can be fascinating, and
we overlook it at our peril.”
This Code if CRAP
” As the CRAP acronym suggests, there are several possible patterns that make a piece of code
CRAPpy, but we had to start somewhere. Here is the first version of the (in)famous formula to help
detect CRAPpy Java methods. Let’s call it CRAP1, to make clear that this covers just one of the many
interesting anti-patterns and that there are more to come.
CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)
Where CRAP1(m) is the CRAP1 score for a method m, comp(m) is the cyclomatic complexity of m,
and cov(m) is the basis path code coverage from automated tests for m.”
The One That Got Away
“Many testers have been involved in post-ship decisions about bugs that "got away"–bugs that
escaped testing and found their way into customer's hands. Often, these post-mortem discussions
end up with finger pointing and threats, but with the right focus, these discussions are a wonderful
opportunity for learning and growth.”
How to test with little to no requirements
”To wrap things up, software testers need to be agile enough to test under both “planned” and “free”
testing environments. And under relatively free environments, testers must draw on creativity, past
experience, common sense, and generally accepted practices from similar applications.” – If only our
complex reality was this easy …
App malware on the rise
”If installed, the Trojan gathers the IMEI and IMSI numbers of compromised devices, uploading this
information to a remote server, before generating counterfeit queries against particular search
results. The malware specifically generated fraudulent clicks on the Baidu ad network, according to
anti-virus firm AVG, which reckons the Trojan is the work of a group also producing malware
targeting Symbian smartphone.”
W. Edwards Deming
“It is not necessary to change. Survival is not mandatory.”
“Quality is everyone’s responsibility.”
"The most important figures that one needs for management are unknown or unknowable, but
successful management must nevertheless take account of them."
"There is no substitute for knowledge."
“Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new
levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the
causes of low quality and low productivity belong to the system and thus lie beyond the power of the
W. Edwards Deming
The Wisdom of W. Edwards Deming
And there is also a short youtube documentary about him
The Five Deadly Diseases
Deming’s 14 Points of Management
How Google Tests Software
“The SWE or Software Engineer is the traditional developer role. SWEs write functional
code that ships to users.”
“The SET or Software Engineer in Test is also a developer role except their focus is on
“The TE or Test Engineer is the exact reverse of the SET. It is a a role that puts testing first
and development second.”
“… stop treating development and test as separate disciplines. Testing and development go
hand in hand. Code a little and test what you built. Then code some more and test some
more. Better yet, plan the tests while you code or even before. Test isn’t a separate practice,
it’s part and parcel of the development process itself. Quality is not equal to test; it is
achieved by putting development and testing into a blender and mixing them until one is
indistinguishable from the other.”
How we test at Microsoft – The most complete book review of all time
Agile Testing Best Practices
Android surpasses Symbian and Blackberry
Judgement in testing – Part II
Why do some testers find the critical problems
Judgement in Testing
Mobile Challenges for Project Management, Part 1
Mobile Challenges for Project Management, Part 2
How Google Tests Software
More on Test Design
Globlization Testing by BJ Rollison
A blog post from Michael Bolton discussing that it is a testers job to not only report bugs, but also
provide information regarding risk
Fuzzing Through the Side Door by Jonthan Kohl
Authority & Delegation
Don’t start a project with scarcity
Career Paths for Testers Using Personas
When a problem isn’t really fixed
Agile Management - Complexity Thinking
New Year’s Resolutions by James Whittaker (Google)
Globalization Testing by BJ Rollison (Microsoft)
Testing Clichés: Testing needs a coverage model (Interesting to think about how his criticism against
test coverage models maps against the test coverage model we use)
Effectiveness of Exploratory Testing
“Using exploratory testing approach testers found far more defects than test case based testing. Although, there
was no statistical significance difference between the two approaches for false defects.”
“The results drawn from experimental and survey data shows that exploratory testing proved effective and
efficient in finding more critical bugs in limited time.”
Investigating Exploratory Testing in Industrial Practice
“One of the biggest misconceptions of exploratory testing from the customer’s perspective is that it does not
provide complete coverage of the system. According to the testers in the study it is possible to completely cover
the system testing however due to little or no reliance on structured documentation and simultaneous design and
exertion of test cases it is easy to forget the test case or commit mistakes.”
“According to the customer perspective the biggest benefit of ET is the ability of providing focused testing. ET
facilitates the customers to perform selective but intensive testing according to the project requirements.”
Google Test Automation Conference 2010: Videos of all talks
Strategic Problem Solving with the 5 Whys (Toyota root cause analysis)
Testing Challenges & Testing Dojos
Manage yourself with measures
”How We Test Software at Microsoft” Review
More bang for your testing buck
Test Design Technique Name Competition
How to deal with unknown unknowns – Black Swans
No Silos – Combining roles (read the comments as well)
Basic International Sufficiency Testing in Action
The Agile Acid Test
Book Review: Exploring Requirements
Testing the Limits with Matt Evans
The biggest difference between a web product and mobile device is the amount of testing and
certification that must precede the launch of a mobile product. A smartphone such as the Pre
is an incredibly complex and highly integrated piece of technology–much more so than a
typical web application. First off, a smartphone contains a fully-functional OS, usually based
on some variant of Linux running on very constrained hardware. It must perform all of its
concurrent services utilizing limited memory and limited CPU horsepower. The smartphone
must also respond correctly to the multitude of many current events, from those generated
from the environment–like switching from wifi to a WAN internet connection–to handling
data input from the user, as well as handling events from the onboard applications.
Launching a mobile product requires exhaustive certification of individual hardware
components such as the CPU, modems, codecs, and displays. Even then, the finished product
is really launched by the carrier and must go through their exhaustive certification tests as
well. Testing an onboard mobile application is also a much harder testing task. There are so
many conditions and constraints that are involved in testing a mobile application. A typical
mobile application is nearly functionally equivalent to any counterpart desktop client-side
web application. Take, for example, a mobile email application. It must behave and interact
with the server-side application in nearly the same way as a desktop web client. The
established protocols were designed for a stable communications environment, but this is just
not the case in a mobile environment. The internet connection may be lost and reconnected
very rapidly. The connection may even be lost for long periods of time. The application may,
at any moment, be swapped out of memory. The system may be shut down abruptly. Lots of
system conditions happen in a smartphone that would rarely or never happen in the context of
a desktop web client application. However, a mobile application must perform its main
functional operations of retrieving and sending messages flawlessly with no loss of data and
full operational integrity. Testing mobile applications under these environmental scenarios is
a huge challenge. In short, testing a web application is no easy task, but mobile
applications and products represent a much tougher and larger testing challenge.
Classification of different tests (integration tests, unit tests, E2E tests, etc) at Google
Exploratory Test Management
Generating random dates
(Un)common testing insights – A list that according to the author presents “truths” that he thinks are
Agile transition and employee retention
Story of a doomed software business
Why is software quality so unmeasured (read the comments as well)
Coding Dojo – Simple instructions on how a Coding Dojo works
Testing Dojo – Applying the coding dojo technique on verification
Combinatorial testing with BJ Rollison
Leadership by Allan Page from Microsoft
The Drake Equaition of Software Testing
Synthesizing Test Ideas
Testing the limits with Rex Black
The Future of Test Management
How outsourcing affects testing
Return of Investment on Software Testing
Testers: Is it time to reinvent the wheel? – So why are testers expendable? Because we’re not
perceived as driving value to the company. And that’s the problem. Next question: How do we go
about fixing this?
Peering into the White Box: A testers approach to Code Reviews
Testers doing Code Review
Areas of Improvement in SW Testing
Manage People, Not Roles
Engineering Management (Old article from 2009 from a manager at Facebook Engineering)
Hiring Technical People: How long does it take you to hire a candidate
Combinatorial testing: Testing highly probable combinations
So, up to now we have discussed:
A key to understanding how to use combinatorial or pairwise testing depends largely on our
ability to effectively model the input parameters that affect a common output condition
We can modify the model to deal with mutually exclusive input value combinations using a
simple built in syntax for conditional or invariant constraints
We can increase the likelihood of testing different values using random data generation
We can increase the test coverage of different combinations by using PICT to randomize the
output of the baseline set of tests
We can increase the potential to use certain values with greater frequency by weighting
We can increase the thoroughness of test coverage between specific input combinations with
We can force the tool to include specific input value combinations in a complete n-way output
of other combinations using a seed file that specifies those combinations
Discussions around the content of a test proposal
Software Quality Characteristics 1.0
Turning the Tide of Bad Testing
Combinatorial testing: Invalid Combinations & Output Condition
Forward thinking in Software testing – Read the comments as well, as there is some discussion
around the post
Combinatorial Testing: Selecting the right values – Part 2
Who owns quality?
How to win a utest bug battle
Combinatorial Testing: Selecting the right values – Part 1
Complexity versus Lean – An interesting 90 page presentation to look through
A Mnemonic for Mobile App Testing – Article by Jonathan Kohl
Putting testing to the test – Using crowsourcing for application testing
Tester Personas at Microsoft
Quality in the Test Automation Review Process and Design Review Template
Test Automation Code Review Guidelines
Rapid Test Preparation
Testing the Limits with Jeff Papows
Estimations Part 3 from Michael Bolton
James Bach’s experience report from SWET 1
Interesting experience report from an test automation conference
Thoughts on Test Strategy
Software Test Estimations VII
Two new articles by Michael Bolton on project estimation
A tool for managing agile projects – interesting to look through how it works
Quality Assurance vs. Test
Old Test Plan Template
1 million $ worth of bugs
Comparing the various crowdsourced testing companies
About Software Engineering – A very interesting lecture regarding software engineering
Test Framing – Michael Bolton and James Bach introduce a new concept
Exploratory Testing Best Practices
Perception of Testing – Future of testing
Trust and Testing
Managers new to agile may not know what to do
Why exploratory? Isn’t it all just testing?
Can exploratory testing be automated
Testing the limits with James Bach – Part 2
Encouraging programmers to be testers
Exploratory Testing and Review
Test Estimations – part six
Pradeep blogs about what aspiring testers should think about in India
Automated Test Case Design – First step
Competence needed for testing – Part 4
What is a Test Strategy?
Testing the limits with James Bach – Part 1
Exploratory Testing – The Learning Part
Managing focus when doing exploratory testing – An article summarizing a workshop, with links to
several presentations regarding exploratory testing – some quite interesting and practical
Agile Backlash – Elizabeth discusses criticism against agile
Definition of Done
The Complete List of Testing Inspiration
On test estimations – 2 more articles on test estimation
The challenge with developers writing automated tests: QA’s perspective
http://www.softwaretestingclub.com/forum/topics/how-do-you-make-sure-that-the - Forum discussion
regarding the above article
Automation problems – Discussions around that automation often costs more than you think
On test estimations
The Role of Leadership in Software Development by Mary Poppendieck
Automating Screen Captures
A picture is not worth a thousand words – This article is mentioned in the above article. Talking about
how a picture can not replace a good bug report.
Putting the pieces together – An interesting discussion regarding testing complex systems
Ingredients for testing – Part Three
The Motive for Metaphor – A discussion regarding the value of using metaphors such as black box
and white box testing
50 Ways to ... Improve Test Automation
Will we survive the future of software
Are we innovative enough when it comes to software testing? Are we improving how we work as
software becomes more complex and harder to test? Are we discussing the same problems today
that we were discussing 5 years ago?
Introducing Thread-based Test Management – New test management ideas by James Bach
How can I tell users what testers did?
Statistician or Journalist – Blog post by Michael Bolton as answer to the above post
Status of software testing professionals
Code Coverage: Unreachable Code and Hard to reach code
The code coverage number is not really useful information to anyone. It is the analysis of the code
coverage results that can help us decide whether we need to design additional tests, identify areas of
the code that can’t be executed without even more expensive testing such as fault injection and/or code
mutation, or refactor the code to improve testability (which often increases the code coverage
Ingredient List for testing – Part two
Testing the limits with Ben Simo
Why the metric?
All testing is confirmatory
http://www.developsense.com/blog/2010/08/all-testing-is-not-confirmatory/ - Reply to the above post
by Michael Bolton
Code Coverage – Finding missing tests
Vilken typ av kompetens en testare bör ha – del 1
Why testers need interpersonal skills
Relationships matter – Tester & Programmer
Exploratory Testing at Microsoft
Code coverage with manual testing
How do I create value with my testing?
The Testing Planet Magazine
No test is also a test
Careers in test (Microsoft)
Code Coverage – Did we run the right tests? (BJ Rollison)
An approach to code coverage (Microsoft)
How to handle regression testing (Bolton)
Exploring regression testing and product line testing
Search-based approaches to software fault prediction and software testing
Structural System-level Testing of Embedded real time systems
Test Driven Code Review
Kaner on testing
Model-based testing for non functional requirements
QA activities in Agile
Let’s change the tune – Agile terminology
Let’s change the title too – Uppföljning på ovanstående artikel av Michael Bolton
Redefining Done – Definition of Done i Agile
What should done mean
How short can your iterations be
Testing lessons from a glass factory
Model-based testing: An evaluation
Investigating Exploratory Testing in Industrial Practice
Testing the limits with Cem Kaner
Why test automation costs too much
Random thoughts on record-and-playback
An analysis of session-based test management
Behind the scenes of the test group
Quality is value to some person at some time
There, but for the grace of testing, go I
How many errors are left to find?
Testign stories from developing MacPaint
To Cert of not to Cert – That is the question
In-the-lab-testing vs In-the-wild-testing
An analysis of session-based test management
Testing the limits: James Whittaker
Agile is not a process; Agile is a mindset
Is unit testing automated?
Challanging each other helps the craft
Transition to Agile Testing
Some test teams may be stumped on how to transition to agile. If you're in such a team, you probably have
manual tests for regression either because you never have had the time to automate them or because you are
testing from the GUI and it doesn't make sense to automate them. You probably have great exploratory testers
who can find problems inside complex applications, yet they tend not to automate their testing and need a final
product before they start testing. You know how to plan the testing for a release, but now everything has to be
done inside a two-, three-, or four-week iteration. How do you make it work? How do you keep up with
Women in Agile
The List is not enough
UI automation beneath the presentation layer
Experience report: Testing vs. Checking
Are testers going away?
Agile: An experience report
We can’t find all the important bugs?
Introduction to Test Driven Design
Test Driven Development
It is better with no model than one model
Heuristics & Leadership
Shipping software is a team effort
A new brand of snake oil for software testing – Cem Kaner
The truth about testing – Jon Bach
Stuart Reid’s Bizarre Plea by James Bach
Handling inexperienced testers
What is a good test case
Testing the limits with Lanette Creamer
Improve Your Testing and Your Testers with Paired Testing (Test)
Automatic Automation (Test)
Measure progress in Scrum by acceptance tests instead of completed tasks (Scrum/Agile)
Without support you will fall (Ledarskap)
The nonsense of leadership (Ledarskap)
Reconnaissance – A testing story (Test)
Shio or No Ship
Testers: Get out of the Quality Assurance Business av Michael Bolton
How Introverts and Extroverts Perceive Each Other
Testing the limits with Scott Barber
uTest: Hypothetical: All testers are barred from practicing their craft for an entire year. What does
the software industry – nay, the modern world – look like without their services? Please be as
dramatic and hyperbolic as possible.
SB: Sadly, I think most of the world wouldn’t even notice. I think for the period of a year, non-testers
would pick up enough of the slack to keep things moving, and enough individuals would perform enough
“heroics” to keep the software world running. Sure, more bugs would make it into production, more
software would ship late, and there would be more evening news reports of software failures, but I’m not
sure the general public would even notice.
In fact, if software producers were smart, they’d pass some of the savings they were realizing by not having
to pay testers on to the general public, making them even less likely to mind a few extra bugs.
But eventually, some kind of tipping point would be reached and the public would demand less buggy
software. That is when the disaster would actually happen. By the time companies re-engaged with testers,
re-integrated them into their new (clearly, more streamlined, process), started getting results, realizing just
how much re-work needed to be done and how long it would take to get software back up to the “old”
standard, the public would be so furious that the “old” standard would no longer suffice. Of course,
achieving the new, higher, quality standard would take more time, and make the software cost more, and
thus make the public demand yet *more* for their money.
Google's Innovation Factory (and how testing adapts)
Bättre testdesign ger lägre kostnader
Expect the unexpected
Software Testers: A costly necessity
Exploratory Testing - Is this as good as it gets?
BEhavioral Regression Testing
Five top causes of nasty embedded software bugs.
Complex != Better
Working on my first playbook
Bug Reporting Lessons From Toyota: Are Your Brakes Show Stoppers?
Journey of a passionate tester
Demystifying Exploratory Testing
How to reduce the cost of testing
When Extreme Programming (XP) started in 1999, the rhetoric was that XP teams did not need “QA People”
because we were going to automate all the tests. Today, even the founders of XP (Kent Beck, Ward
Cunningham, Ron Jeffries) would admit there can be a role for a tester in an XP shop. Now we’re hearing
the rhetoric again. How are we supposed to respond?
Interview with Jon Bach
JIRA Defect Tracking System
http://www.satisfice.com/blog/archives/420 : Toyota
http://www.satisfice.com/blog/archives/418 : Indian Testers
Joel on Software goes Offline
The Customer is always right
Do I really need to automate this test?
Lessons Learned from Toyota
Interview with James Bach
Interview with Michael Bolton
Interview with James Whittaker
Interview with Patrick Copeland
Scary stories and GUI automation
Lines of Code and Tests in Balance
The Role of the Test Manager in an Agile Organization
Burning Issues of the Day av Michael Bolton
An Overview of “A Visual Approach to Risk-Based Integration Testing av Neil Pandit
Logging: Exploratory Tester’s Friend
Exploratory Testing is accountable
Code Coverage – More than just a number
”there is no correlation between code coverage and quality, and code coverage measures don’t tell us
“how well” the code was tested”
How to create test cases from bugs
Critical Mass of Software
Limitations of TDD
My experiences in testing with an exploratory mindset and methodology
Be an explorer
Structures of Exploratory Testing
Best Bug or Bugs
Moving to an Agile Testing Environment: What Went Right, What Went Wrong
”Why is testing taking so long”
Agile, Testing, and
Quality: Looking Back,
Moving Forward av Elizabeth Hendriksson
Test Management in an Agile organization av Elizabeth Hendriksson
”How to get started with TDD”
James Whittaker on the Future of Testing
http://www.satisfice.com/blog/archives/370 som länkar till http://www.satisfice.com/blog/wp-
Exploratory Testing Dispute