Your SlideShare is downloading. ×
Test articles 2009 to 2013
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Test articles 2009 to 2013


Published on

Compilation of test articles I have read and sent out as a news mail between 2009-2013

Compilation of test articles I have read and sent out as a news mail between 2009-2013

Published in: Technology

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Test Articles Compilation from 2009- Oct to 2013-Aug by Johan Hoberg Introduction This document is a compilation of test articles ( or articles relevant to testing) I read and sent out as a news mail during 2009-2013. The idea of this document is to use it when you are searching for information, find the relevant articles and then go and read those articles from the original source. When I started in 2009 I did not have this in mind, and did not copy parts of the text from the original source of the article, which makes those articles more difficult to search unless the topic is in the title/link text, since those first article links are mostly the actual link and maybe a title, while later articles will have a small part of the article copied with the title and the link. The first articles are at the end of the document, and the latest articles are in the beginning, so the most relevant articles are quick to find. Test Articles 2009-2013 Fake your way to better tests ”After many years of blogging, you decide to try out your blog platform's API. You start playing around with it, but then you realize: how can you tell that your code works without having your test talk to a remote blog server? Fakes to the rescue! A fake is a lightweight implementation of an API that behaves like the real implementation, but isn't suitable for production.” Know your test doubles ”A test double is an object that can stand in for a real object in a test, The most common types of test doubles are stubs, mocks, and fakes. A fake doesn’t use a mocking framework: it’s a lightweight implementation of an API that behaves like the real implementation, but isn't suitable for production (e.g. an in-memory database).” Thoughts about automation from Microsoft
  • 2. “Automation” is an overloaded and abused word that often implies that test code only exists to automate human tasks – that’s stupid If you are a tester who codes, you write code. Period. Some of your code may automate tasks. Most will not (or shouldn’t) “The point is, if your testing strategy is entirely to automate a suite of user tasks, you are doing way more to keep “automators” busy than you are actually providing valuable testing.” Optimal Logging How long does it take to find the root cause of a failure in your system? Five minutes? Five days? If you answered close to five minutes, it’s very likely that your production system and tests have great logging. All too often, seemingly unessential features like logging, exception handling, and (dare I say it) testing are an implementation afterthought. Smart Monkey Dumb Monkey testing is when the automation randomly sends keystrokes or commands to the System Under Test (SUT). Smart Monkey testing is, well, smarter. Instead of doing random things, the automation system knows something about your program: Where it is What it can do there Where it can go and how to get there Where it’s been If what it’s seeing is correct Top Secrets from Top Testers As part of a new series that we’re calling The Top Secrets of Top Testers, we’ll be asking a panel of testing experts, uTest community members and internal team members a new question every month. As you might have guessed, this month’s question is: “If you could offer a new tester one piece of advice, what would it be?” Coding Mindset This mindset, although common, kills me: Testers who code write automation Automation is primarily made up of a suite of user tasks Automation is primarily done through the GUI All of these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test teams in the wrong place and is, frankly, insulting to testers who know coding. If you’re a tester who writes code (or a coder who cares about testing), try this: Write tools that help you find important bugs quickly
  • 3. Write a few high level tests that ensure user actions “work”, but leave most of it for exploratory testing, user testing (assuming you have great monitoring tools), or both. If you must write GUI tests for the above, write fewer of them (that way you’ll spend less time investigating their flaky behavior). If you’re a tester who writes code, you’re not a test automator, you’re not a test automation engineer, and you don’t write test automation or test code. You write code. And you test. Modeling Programmer Productivity "There is a perception in some tech circles that older programmers aren’t able to keep pace with rapidly changing technology, and that they are discriminated against in the software field. But a new study from North Carolina State University indicates that the knowledge and skills of programmers actually improve over time—and that older programmers know as much (or more) than their younger peers when it comes to recent software platforms." ”This study didn't surprise me. Nor did it surprise many of my programmer friends. Why, then, do so many managers still believe that older programmers are less skilled than their younger co-workers?” State Transition Testing State Transition Diagrams, or STDs as they’re affectionately called, are effective for documenting functionality and designing test cases. They should be in every testers bag of tricks, along with Decision Tables, Pair-Wise analysis, and acting annoyed at work to appear busy. One benefit of State Transition Diagrams is that they describe the behavior of the system in a complete, yet easy-to-read and compact, way. Often, the best strategy is to create tests that cover all transitions (the arrows) at least once. This guarantees you will test every state, event, action, and transition. It gives you good coverage in a reasonable amount of tests. Metric Driven Development So what is MDD? I would define MDD as a practice where metrics are used to drive the entire application development. In a company which uses MDD, everything from performance and usage patterns to revenue is measured. Moreover, every single decision taken by developers, operations or even business people is based on metrics. Metrics are used to monitor team performance, solve performance bottlenecks, estimate hardware needs or to accomplish other purposes at any stage of development life-cycle. Asshole Driven Development The software industry might be the world’s greatest breeding ground for new systems of management. From Agile, to Extreme Programming , to Test Driven Development (TDD), the acronyms and frameworks keep piling up. Why? Some say it’s immaturity: that software is still a young industry and all the change is the path to some true fundamentals. Others say it’s because software people like making things up and can’t help
  • 4. themselves. Well I say this: if we’re going to have dozens of models we may as well have some that are honest, however cynical, to what’s really going on much of the time. Tear down the Wall There’s a wall between testers and programmers. Programmers “build”, testers “break” – but passing features back and forth between one group and another is wasteful. It takes time and extra communication to traverse the wall. What software teams need in the future is team members who can perform the activities of programming, testing, and analysis – together. I saw a slide deck recently that stated, “Agile Testers frequently can’t keep up with Programmers”. Good software development can’t happen when you serialize tasks – the team needs to work on this stuff – together. I’ll always be deeply involved in the activity of software testing, but I don’t know if the role or title exists in my future. After nearly 20 years of being a tester (and likely several more with that title), I’m admittedly going out on a bit of a limb with that statement. Despite my title, I want the walls to go away. GTAC (Google Test Automation Conference) 2013 Slides and Presentations Topics such as Android UI Automation How Facebook tests Facebook on Android How do you Test a Mobile OS? Finding Culprits automatically in failing builds, i.e. Who broke the Build? Evolution from Quality Assurance to Test Engineering Escape from Brainstorm Island Innovation is not about specific innovation events or workshops – it is a continuous work which everyone needs to be involved in on a daily basis. Emphasize Good Practice However, I noticed a strange thing when I urged people to stop screwing up. I found this didn’t motivate them at all! I realized getting better isn’t just about reducing what goes wrong (making mistakes). It’s also about increasing what is right (using good practices). And every now and then people need a reminder that they’re doing just fine. Test Management is Wrong (Don’t forget to read the comments, which are equally or more valuable than the article) Basically, don’t get so bogged down in test management (the process) that you overlook actual quality. Checking boxes on your testing to-do list isn’t the same thing as working to make sure software is actually of high quality – bug free (at least mostly), efficient, effective and achieving its goal. Testing the Limits with Fiona Charles
  • 5. How to rescue a failing test project: Interview a range of people to establish the facts and perceptions. Set a clear direction and make sure everyone is working productively on tasks that are aligned with it. Dig for and lay out the facts about the system quality or project process issues, and report the findings in a blame-free way. Make sure there’s a test strategy oriented to the stakeholders’ values and the business risks and a viable plan to execute on it. Fill any skills gaps on the team. Help build communication bridges and coach the test team to develop working relationships across the project. Make sure every tester has good reason to feel empowered and valued for his/her contribution. Kanban and Scrum – Making the most of both NAL.pdf Book about differences and similarities between Kanban and Scrum and how they complement and conflict with each other. Published 2010. Testing State vs. Testing Interaction There are typically two ways a unit test can verify that the code under test is working properly: by testing state or by testing interactions. What’s the difference between these? That Damn Gorilla Last week I happened to log on to twitter just as some test folks were marveling over the the massive parallels between the video with the gorilla and the basketball players and software testing. I made some cracks about the absurdity of the hype, and more than a few testers freaked out. Software Testing is like … When you work in technology, you will inevitably have to explain your job to non-technical people. This isn’t always easy (and it’s certainly not much fun). Worse yet, most the time we don’t give ourselves enough credit when explaining our profession to friends and relatives; we sell ourselves short. Purpose over targets That’s the difference between a purpose and a target. And that’s why I don’t like how many apply the SMART model for goals, because the SMART model moves people’s attention away from a purpose to a target. Your targets are useless if you can’t identify your purpose. Samsung trying out 3 CEOs The announcement last week that the Samsung Electronics is elevating two executives, Boo-Keun Yoon and J. K. Shin, to the CEO role was met with interest in leadership circles. We have seen this kind of co-CEO arrangement grow over the years. All the more interesting: the South Korean giant's
  • 6. current CEO and vice chairman Oh-Hyun Kwon isn't even vacating the seat. All three men will now share the role. Testing vs. Checking Redefined – by James Bach Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference. Human checking is an attempted checking process wherein humans collect the observations and apply the rules without the mediation of tools. Machine checking is a checking process wherein tools collect the observations and apply the rules without the mediation of humans. Human/machine checking is an attempted checking process wherein both humans and tools interact to collect the observations and apply the rules. Automated vs. Exploratory – we got it wrong! ”There is a misconception floating around that Agile Testing is mostly about automated checking. On the other way the whole testing community fights since decades about the value of test automation vs. the value of exploratory testing. I find these two different extreme position unhelpful to do any constructive testing. Most recently I ended up with a model that explains the underlying dynamics quite well, to testers, to programmers, and to their managers. Here’s the model.” Why organizations are so afraid to simplify ”It's one of the great contradictions of organizational life: People are great at starting new things — projects, meetings, initiatives, task forces — but have a much harder time stopping the ones that already exist.” The Perimeter Test “I want to include a test to click around the button to check if the focus of the button is restricted to the button area or extends outside the button area. According to me, the name for the test idea: "The Perimeter Test".” Good Requirements Deliver a high ROI Deliver-a-High-ROI ”A while back, Ellen Gottesdiener from EBG Consultants assembled a list of citations about ROI and requirements titled Do You Know Your ROI on Your Requirements Work. Ellen did an excellent job researching this topic and presents a comprehensive list of citations in her article.” Management Myth 15: People need to work overtime h=DYN&tt=siteemail&iDyn=2
  • 7. ”When managers ask people to work overtime, they are not thinking about what a technical day is like. Especially if you work on an agile team, they are not thinking of what an agile day is like. A manager’s day is fraught with interruption and context switching every ten minutes. Many managers feel as if they need to wait until 5 p.m. to get anything done. But if you are a technical person who has arranged your day so you have a good two-hour chunk in the morning, and a couple of more two-hour chunks in the afternoon, you have accomplished your work for the day. Your brain is tired and you need to leave. If you have paired or swarmed or worked with a cross-functional team, you are tired. You need to leave, to refresh your brain for the next day.” Stop Blaming the System ”I see it again and again. The suggestion of systems thinkers and Agile writers to stop blaming people and instead try blaming the system. Seriously, what is the difference between blaming other people versus blaming the system? It seems to me it’s all the same, as long as we’re not blaming ourselves!” Why programming classes need to cover software testing too ”Software testing has been on a path toward convergence with programming for some time, especially when it comes to test automation. At the same time, many vendors have tried to create tools that allow non-programmers to exercise code in ways they would not know how to do by themselves.” ”At the same time, software testers provide much-needed input and perspective from exploratory testing and user acceptance testing. In my opinion, nothing can replace the “human perspective” as part of the quality assurance cycle.” The Meaning of Models ”There were several other documents with boxes and arrows, definitions and categories, quadrants and matrices, circles and pyramids, and they didn’t mean much to me. I’m a bit tired of such models, I noticed. I’ve probably seen too many.” High Level Overview of Software Testing ”Thanks for your request of a high level summary of software testing. You would get different answers from each tester, and here’s what I think you should know.” The 60% Rule ”I’ve heard some coaches suggest a 60% rule: everyone should spend 60% of their time on the same team, the same department, or the same organization. The rest of their time they can spend working for others.” An Overview of High Volume Automated Testing ”This overview describes the general concept of high volume automated testing (HiVAT) and twelve examples of HiVAT techniques. The common thread of these techniques is automated generation, execution and evaluation of arbitrarily many tests. The individual tests are often weak, but taken together, they can expose problems that individually-crafted tests will miss. The simplest techniques offer high coverage: they can reach a program’s weakness in handling specific special cases. Other
  • 8. techniques take the program through long sequences of tests and can expose programs that build gradually (e.g. memory leaks, stack corruption or memory corruption) or that involve unexpected timing of events. In my experience, these sequences have exposed serious problems in embedded software, especially in systems that involved interaction among a few processors. As we embed more software into more consumer systems, including systems that pose life-critical risks, I believe these tests will become increasingly important.” Checklist for Software Testing Outsourcing ”When lawyers negotiate a contract, they often find it handy to work from a checklist of key issues. Fewer things slip through the cracks, and the negotiator doesn’t have to rethink the same issues in every contract. Further, the checklist grows and cumulates experience.” Being an Effecivte Spec Reviewer (Microsoft) ”The first time testers can impact a feature is often during functional and design reviews. This is also when we make our first impression on our co-workers. If you want to make a great initial impact on both your product and peers, you have to be an effective reviewer” Stakeholder Matrix for Stakeholder Analysis A stakeholder matrix, like the one in the above diagram, is a great tool to identify the stakeholders and also classify them appropriately. The classification based on influence of and impact of the change on stakeholder, helps to reduce the number of stakeholders to a manageable size. 6 Testing Mistakes to Avoid Mistake #1: Testing too late Mistake #2: Testing with amateurs Mistake #3: Testing without a scope Mistake #4: Testing “one and done” Mistake #5: Testing in a controlled environment Mistake #6: Testing too fast/slow Black Box vs. White Box ”I have heard and seen the difference between Black Box Software Testing and White Box Software Testing being described as “whether or not you have knowledge about the code”; and that Gray Box Software Testing is a mix of the two. But is it really about how much of the code you see?” Value Creation over Value Extraction ”But new value can only be created when that which is already valuable remains intact. When you delight customers while screwing suppliers, you’re not creating value, you’re just moving it from one stakeholder to the other. When you increase productivity while cutting corners in quality, you’re also not creating value, you’re just stealing it from the future.” The Insapience of Anti-Automationism
  • 9. ”The distinction between manual and automated is a false dichotomy. We are talking about a matter of degree (how much automation, how much manual), not a distinction of principle.” Don’t let Scrum make you Fragile ”A complex system benefits from not following the same practices over and over again. By adding a little bit of stress, and continuous variability in the environment, the system learns to become fit and healthy.” Firmware vs. Software ”I have nothing against firmware, or firmware engineers (I’ve been known to write some firmware myself). But what we really need is less firmware and more software. Actually, I am disappointed that firmware engineers write so much firmware!” Should business people care about clean code? ”Do business people care about clean code? They behave like they don’t. Well, they do and I can prove it using proof by contradiction. Suppose that a business person does not care about clean code. The business person wants features and they want them fast. So they pressure engineers to just get the code working. This causes the code to deteriorate. The business person wants more features, but the code is a mess so features take longer to add and things that work start to break. If the business person wants features fast, they need clean code and if they need clean code they must care about it too, which contradicts the statement that business people don’t care about clean code.” Exploratory Testing = Fun Productivity ”Exploratory testing can be valuable in specific situations and reveal certain categories of defects more readily than other approaches. The overall effectiveness of behavioral testing approaches is heavily influenced by the tester’s in-depth system and domain knowledge and experience. Of course, the effectiveness of any test method eventually plateaus or becomes less valuable and testers must employ different approaches to further investigate and evaluate the software under test” Please reproduce this issue “… how many times have you seen a test error, and immediately re-run the test to see if you could reproduce the error yourself? Is it a big number? If it is, you’re not going to like what I have to say. If you need to reproduce all of your bugs to figure out what’s going on, you screwed up. Your logging is bad. Your diagnostics don’t exist. You wrote crummy code. You’re wasting time!” Double Testing ”I think the idea of double testing is about what models we use in testing. More precisely our mental models on our test approach/perspective on testing, system boundaries, system parts, levels of testing, terminology in testing, ideas of test coverage, usage of test techniques, test idea sources, test planning techniques and so on.” The Merit System
  • 10. “It is clear that the annual bonus system doesn't work. A tremendous amount of research says that it demotivates people, destroys collaboration, and causes dysfunctional behavior among managers and employees. What also doesn't work is the flat system, where everyone simply gets the same compensation or bonus. This also demotivates most people. And it makes organizations inflexible in times when agility is needed. What we need is a merits system...” Root Cause Analysis Template “When something gets released to Production which adversely affects a customer, we sometimes write up a formal Root Cause Analysis to send to the customer. It basically explains what happened, and how we intend to prevent it from happening again.” Future of Testing ”Regardless of how you feel about the health of software testing (which depends largely on your ability to interpret a metaphor), for me, it’s getting easier and easier to see that testing is changing. Granted, it’s changing along with the software under test, so if you’re testing the same sort of desktop software products you have been for years, using development practices even older, the good news is that you’re safe – your testing world probably isn’t going to change either.” Test Engineers @ Google!/2013/01/test-engineers-google.html ”In order to get a clearer picture of what Test Engineers are responsible for, I chatted with three of my colleagues. We were able to identify the underlying Test Engineers’ similarities, while highlighting the differences.” How to recognize a problem “So: sometimes as we’re testing, we recognize a problem by a direct feeling, or through a principle that we’re aware of. Sometimes our recognition of a problem comes from a document or tool; sometimes from another person. That may entail embodied mental processes, rational principles, or mechanisms, but all that’s kind a mouthful. So lately, James and I have started to say that an oracle is simply “a way to recognize a problem”.” Debugging for Testers “I came across a few comments and statements recently that set off a slightly red (orange?) flag for me. The gist was that debugging was for coders, and that testers “just found the issues”. I get the context (or at least I think I get it) – I agree that all testers don’t have to be coders, and that all testers certainly don’t have to be debugging masters, but I don’t think that any testers should run away from a debugger in fear either.” Guide to Technical Testing Presentation about what a technical tester is. Managing your requirements requirements-management?lang=en
  • 11. -requirements-writing?lang=en ”The world of requirements management has developed significantly in the last decade or so and has increasingly become one of the corner stones of successful software and systems engineering projects. We have been discussing various aspects of the domain from a best practices perspective and how tools can help managing your requirements efficiently and effectively. Starting today we will discuss various aspects of the requirements management discipline at a bird’s eye view level. These are meant to be introductory in nature and also intend to serve as refreshers for those who are already in the field.” [Swedish Article] Tala är Silver, Skriva är Guld ”När det agila manifestet ses som universallösning på systemutvecklingens utmaningar har vi en tendens att slänga ut barnet med badvattnet och övertolka principer som ”fungerande programvara framför omfattande dokumentation”. Omfattande dokumentation vantolkas inte sällan som ”så gott som ingen dokumentation – i alla fall inte mer än vad jag behöver nu” och plötsligt har vi dragit undan mattan för möjligheten att kommunicera bakomliggande behov och krav i skriftlig form över lång tid.” Scaled Agile Framework This is a model for implementing Agile on an enterprise level. This is not new, but was mentioned in the article below. I thought it was a nice overview. Agile at VersionOne Continuous integration is an important part of the process. They use Jenkins to manage the builds for each team, tracking both checkin and nightly test runs which comprise both unit, and acceptance tests. They are currently iterating through the use of cucumber to create scenario tests. Steve reports that they haven’t had much success at using the tests as “living documentation” but they have seen value in the collaboration it fosters in helping refine stories. I asked about UI automation, and they do have some, but mostly used for high level scenario and process flow checks. After a recent STPCon, the lone dedicated tester of the group discovered the glories of Session-Based Testing. This method of managing Exploratory Testing is quickly being adopted by the teams. Rules I’ve Learned and Live by to Motivate People and Organizations by-to-motivate-people-and-organizations people-and-organizations and-organizations through-the-people-they-manage It is your career – Lead it! (Microsoft) No review system is perfect and it turns out the Peter Principle (“people get promoted to the point of incompetency”) is true. It was proven in 2010 by the Nobel Peace prize winners of Management
  • 12. Science (here). Microsoft follows a general “Up or Out” strategy, which is considered a typical “defense” to Peter Principle happening in your org. These same guys proved that promoting the most competent people from the lower levels up is not as successful of a strategy as alternating between the Best and Worst when promoting. Organisational Culture Defined (Old, but a good overview of organisational culture) schein/ I hear people referring to the culture of a place as being “good” or “bad”, managers and consultants speak of changing culture, and employees speak of being part of culture. What is this culture they all refer to? My studies introduced me to Edgar Schein’s answer, and confirm that the task of getting culture “right” is a tough target to hit. Risk Based Testing – Two parts ”Now, when we test a product with a limited number of testers in a small amount of time, we have to curtail the amount of testing that can be performed. This is where the concept of Risk Based Testing comes in. Let us understand the concept in detail.” The RBT process can be described by these major steps: Step1 – Define all requirements in terms of Risk associated to them Step2 – Based on risk assessment, prioritize the requirements Step3 – Plan and define tests according to requirement prioritization Step4 – Execute test according to prioritization and acceptance criteria. The Most important advantages of this approach are as follows: Linking the product risks to the requirements identifies gaps. With limited time, money and qualified resources, testing concentrates on the most important matters first, thus delivering the most optimal test. The focus lies on risks to the business instead of just the functionality of the information system. During testing, communication (Test Reporting) takes place in a language (risks) that all stakeholders understand, hence involvement of testing is greater in comparison to mere list of issues communicated. It offers a negotiating instrument to client and test manager alike when available means are limited. Experts say it is time to give Java the boot ”Java, the programming language designed to make the web fun and interactive, has become one of the weakest links in a PC’s and Mac’s defenses against external threats. Consider the most recent Java vulnerability, a weakness currently being exploited by malware distributors: When Oracle, Java’s maker, released an emergency update to fix the software, security analysts reported that even the hot-off-the-presses code contains additional vulnerabilities.” Decision Table Testing
  • 13. “Decision tables are an effective tool for describing complex functional requirements and designing test cases. Yet, although they’re far from a new concept, I’ve only seen a handful of functional specs and test plans that included one.” What’s Comparable – Part 1 “People interpret requirements and specifications in different ways, based on their models, and their past experiences, and their current context. When they hear or read something, many people tend to choose an interpretation that is familiar to them, which may close off their thinking about other possible interpretations. That’s not a big problem in simple, stable systems. It’s a bigger problem in software development. The problems we’re trying to solve are neither simple nor stable, and the same is true with the software that we’re developing.” The Kudo Box ”There are many wrong ways to reward employees. A simple but effective approach is to install a “kudo box”, which enables people to give each other a small reward.” Tell a Story at the Daily Standup ”We should tell three stories during test reporting: a story about the product a story about testing a story about the process” Software testing training and gaming Discussion about using games to teach about software testing Censoring people for disagreeing with Context driven testing Interesting drama in the test world TDD ignores the design – No it doesn’t ”TDD ignores design; that is a frequently stated misconception. Many people get this idea from the code focus of TDD. TDD does not call for the creation of any non-executable design documentation. So the questioning developer gets the idea that there is no design. But I say, “yes there is”.” “But make no mistake about it, it you can’t get your code into a unit test, it’s not well designed.” TDD guides you to improved design ”My premise from these first two articles is that TDD is radar for approaching code rot, as well as a guiding beacon to modular design.” ”When your code depends on some third party code, or code from some other subsystem or group that causes testing inconvenience, you have a choice to make. Should I create a test double for the problem dependency, or write an abstraction layer above the problem dependency. Given new code, the abstraction layer, or adapter, is usually a good choice.” Sikuli – Some first impressions
  • 14. ”Sikuli started as a project at MIT. It's now an open source tool that can be used with a number of different applications. It can be used with the web, it can be used with Flash apps, it can be used with compiled applications on a number of different platforms. Yeah, OK, that's all fine and good, but aren't there plenty of tools out there that already fit that space? Why use another one?” How to manage conflict in virtual teams “Employee conflicts can be poisonous. We have all experienced the damage to productivity, crushed creativity, and squashed morale. As Kevin M. Campbell, Accenture's Group Chief Executive, Technology, notes, "All too often, I've seen that personal conflicts derail costly projects and important initiatives." Unresolved employee conflicts are bad enough in a traditional, physical workplace. They are all the more dangerous in a virtual environment, where people don't have the luxury of proximity to work their differences out face-to-face.” The Future of Software Security Testing “In recent years it has been repeatedly proven that vehicle computer systems and wireless medical devices are susceptible to hacking. Though a malicious attack has never been reported, several research groups and security organizations have proven that security vulnerabilities exist that allow people to hack insulin pumps, pacemakers, implantable cardiac defibrillators (ICD) and wireless vehicle communication systems.” Another certification, another scam? This is a continuous discussion in the testing world. Good to keep up-to-date about it if you are interested. Two ways to deal with Bias First part of the article is about creating a mind map of different areas of a function/application/feature and using that as a start for exploratory testing. Could be an interesting read. Where does all the time go “So since you’re a tester, and since the door on the test lab says ‘Test Lab’, and your business card says ‘Testing’, they figure that’s all you do. The illusion is what Jerry Weinberg calls the Lumping Problem. All of those different activities—administrative compliance, setup, bug investigation and reporting, and test design and execution—are lumped into a single idea for them.” Why are there so many C++ testing frameworks!/2012/10/why-are-there-so-many-c-testing.html ”So, I think a major reason that we have many C++ testing frameworks is that C++ is different in different environments, making it hard to write portable C++ code. Another reason is that some limitations of C++ make it impossible to implement certain features really well, and different people chose different ways to workaround the limitations. Finally, many C++ testing framework authors neglected extensibility and were happy just providing canned solutions,”
  • 15. The Less is Best Approach to Innovation ”In our world of excess everything, savvy innovators realize that less is actually best. They know that delivering a memorable and meaningful experience hinges on user engagement, which is best achieved through a subtractive approach. Anything excessive, confusing or wasteful is intelligently and cleverly removed, or never added in the first place.” The End of a Bug’s Life (Microsoft) ”Some testers work on teams that reward employees based on the number of bugs they find. What effect does this have on testers? It encourages the finding of bugs, not the Closing of them. If anything, testers should be rewarded based on the number of bugs fixed and closed, which would at least encourage the resolution of quality bugs.” Test Managers and why you should not spend time trying to avoid blame ”This *negative blame+ cycle flourishes only in organizations that perpetuate a culture of fear and blame. Fix the culture and you’ll fix the cycle.” The Problem of Organizational Change - People ”Although myriad factors are cited, the inability to engage people is the factor noted longest and most often. As organizational behavioral experts Kenneth Thompson and Fred Luthans noted almost 20 years ago, a person's reaction to organizational change "can be so excessive and immediate, that some researchers have suggested it may be easier to start a completely new organization than to try to change an existing one."” 100% Automation Pass Rate “Automated testing is now more pervasive throughout the industry as compared to the past. Automated tests running against daily builds are an essential element in an Agile lifecycle. Continuous integration (CI) requires continuous feedback from automated tests. So, let’s dispel some myths and explain why 100% pass rates are not only a good thing, but are and important goal to strive towards.” Why people don’t instantly buy into Agile “When "selling" their methods, Agile evangelists often stress the strength of Agile methods at removing, and even preventing, errors. I used to do this myself, but I always wondered how people could resist this sales pitch. I would plead, "Don't you want quality?" And, of course, they always said, "Yes, we want quality," but they didn't buy what I was selling. Eventually, I learned the reason, or at least one of the reasons. In today's blog, I want to help today's evangelists (coaches, team leaders, managers, or whomever) by sharing what I've learned about why Agile methods can be so difficult to sell.” The Rise of the Customer
  • 16. ”I spoke recently on the topic of “Customer Focused Test Design” (synopsis: customers don’t really care how much functional testing testers do, or how many bugs they find. A test approach that favors scenarios, -ilities, and customer feedback is better; and a bunch of examples for emphasis/proof).” ”If you want to be a successful software company, you have to care about customers. In addition to keeping them happy once they have used your software or service, you need to respond to their needs and give them what they need” Making meetings productive “First, get the right attendees and be aggressive about it. Second, make coming and going kosher. Halfway through a meeting and you realize you can't contribute or don't need to be there? Leave. Third, vote with your feet by walking away from the more useless meetings. Just don't go to those meetings where the organizer holds court. Fourth, heed the warning signs. The first such sign is the meeting agenda. Big agendas mean lots of time switching topics. Fifth, follow up a scheduled meeting with scheduled work time. Finally, all this points to building an anti meeting culture within your organization” The Definition of Quality ”In short, quality does not exist in a non-human vacuum, but every statement about quality is a statement about some person(s). That statement may be explicit or implicit. Most often, the "who" is implicit, and statements about quality sound like something Moses brought down from Mount Sinai on a stone tablet. That's why so many discussions of software quality are unproductive: It's my stone tablet versus your Golden Calf. When we encompass the relativity of quality, we have a tool to make those discussions more fruitful. Each time somebody asserts a definition of software quality, we simply ask, "Who is the person behind that statement about quality."” Premises of Rapid Software Testing ”These are the premises of the Rapid Software Testing methodology. Everything in the methodology derives in some way from this foundation. These premises derive from our experience, study, and discussions over a period of decades. They have been shaped by the influence of two thinkers above all: Cem Kaner and Jerry Weinberg, both of whom have worked as programmers, managers, social scientists, authors, teachers, and of course, testers.” Conversation with a Test Engineer at Google!/2012/09/conversation-with-test-engineer.html ”Alan Faulkner is a Google Test Engineer working on DoubleClick Bid Manager, which enables advertising agencies and advertisers to bid on multiple ad exchanges. Bid Manager is the next generation of the Invite Media product, acquired by Google in 2010. Alan Faulkner has been focused on the migration component of Bid Manager, which transitions advertiser information from Invite Media to Bid Manager. He joined Google in August 2011, and works in the Kirkland, WA office.” Bugs spread disease ”There’s a lesson here about the real cost of tolerating bugs, of supporting practices that involve triaging defects. The cost of carrying a bug is far far greater than most people realize. That trivial little
  • 17. “cosmetic” issue will cost a little time to fix. If you have to argue about whether or not to fix it, then track it, then revisit it, then prioritize it, then argue again about whether to fix it or defer it yet again, you will spend hours upon hours on it. This is why I have no patience for the “bugs are inevitable; you can’t fix them all” attitude. Bugs kill. They do it slowly, painfully, but relentlessly.” I am an exploratory tester ”I am curious about how the system works I look at details and the whole, and at many places I use many sources to learn what is important I am creative and see many testing possibilities I test in many different ways, and adjust to the situation I am good at finding important problems fast I am aware of models enabling and limiting my testing I can evaluate a system from many dimensions I can communicate my thinking and findings I am an exploratory tester” Our obsession with scale is failing us In the industrial era and even the early web era, size was central to scale. The new reality is this: The ability to scale is no longer a direct function of size. Traditional Test Engineering … Turning Quality on its Head ”In summary, the closer testing is related to real use the better equipped we will be to find the types of issues that are by definition the bugs that will most annoy users. Early cycle testing should thin the bug herd and eliminate the simple bugs and coding mistakes so that late cycle testing can focus on recall-class issues. Early cycle testing is like crowd control; no matter how diligent we are early in the cycle, later cycles of manual, dogfood, and beta testing are not skippable phases. Like it or not, we will benefit from putting more time and innovative thinking into late cycle testing.” Here is the original youtube video How to get your bugs fixed (Microsoft) ”To release good quality software, then, you not only have to be proficient at finding bugs, but you also have to be proficient at getting them fixed. Unfortunately some testers are good at the former, but not the latter.”
  • 18. Big Data’s Management Revolution ”The evidence is clear: Data-driven decisions tend to be better decisions. In sector after sector, companies that embrace this fact will pull away from their rivals. We can't say that all the winners will be harnessing big data to transform decision making. But the data tell us that's the surest bet.” Recording My Testing Sessions ”Can you replicate the bug again for me? How many times have you answered the above question? It might be asked by testers, programmers, managers and anyone interested in the bug. Sometimes, even you will like to replicate the bug.” Ten Reasons to Fix Software Bugs Right Away 1. Unfixed bugs camouflage other bugs 2. Unfixed bugs suggest quality isn’t important 3. Discussing unfixed bugs is a waste of time 4. Unfixed bugs lead to duplicate effort 5. Unfixed bugs lead to unreliable metrics 6. Unfixed bugs distract the entire team 7. Unfixed bugs hinder short-notice releases 8. Unfixed bugs lead to inaccurate estimates 9. Fixing familiar code is easier than unfamiliar code 10. Fixing a bug today costs less than tomorrow Getting 360 Reviews Right ”But there is one thing we've personally seen that profoundly and consistently changes lives — what's generally referred to as the 360-degree feedback process. In the course of completing tens of thousands of these reviews as part of our strength-based leadership programs, we had an up-close view of many people who were teetering on the edge of job termination, and have seen them blossom into extremely valuable contributors. It's been one of the really gratifying parts of our work.” Testing 2.0!/2012/08/testing-20.html ”It’s amazing what has happened in the field of test in the last 20 years... a lot of “art” has turned into “science”. Computer scientists, engineers, and many other disciplines have worked on provable systems and calculus, pioneered model based testing, invented security fuzz testing, and even settled on a common pattern for unit tests called xunit.” Three Tips for Leaders about to Miss their Forecasts ”It's a scary moment when a company realizes that the goals it has set for a quarter or a year are simply not attainable. And yet in today's continually choppy environment, leaders are encountering that moment with increased frequency. And it is at this moment that leaders truly earn their pay, because many short-term actions can have devastating long-term consequences, particularly when it comes to innovation.”
  • 19. Interview Questions to ask your Future Employer ”As interviewing is a two-way street, it’s always a good idea to ask your potential employer a few basic questions before accepting the gig. After all, you need to make sure that the company is as much a fit for you, as you are for them.” The Trojan Form of Change ”I believe real change in traditional management will not come from people telling managers how green the grass is on the other side of the fence. It will come from people who can inject an idea virus in a traditional organization, which is not destroyed by the current mental model, and which then transforms it from the inside, using the current mental model against itself." Intertwined Heuristics Looking at two different models and seeing how they intertwine. Musings on Test Design ” The point is, and this is worth repeating so often, is that thinking of test automation and “human” testing as separate activities is one of the biggest sins I see in software testing today. It’s not only ineffective, but I think this sort of thinking is a detriment to the advancement of software testing. In my world, there are no such things as automated testing, exploratory testing, manual testing, etc. There is only testing. ” Testers caught sleeping on the job ”At Microsoft, we submit every code change to a peer-review before it’s checked in. I’ve performed hundreds (although it feels like millions) of code reviews in the six years I’ve been in Test. One of my biggest red flags during a code review is the Sleep statement.” A tool for mapping your goals ”With multiple options on the table, it's hard to decide what's best for you and your future. But by assessing your career objectives and defining which are most important, you can reach a career decision with confidence. An easy but structured exercise like the career matrix can give you the answers you need all in one place.” Introducing the Software Testing Ice Cream Cone
  • 20. Are you sure you are not a bad boss? “When we analyzed the behavior of 30,000 managers, as seen through the eyes of some 300,000 of their peers, direct reports, and bosses on 360-degree evaluations, we found that the sins of the bad boss are far more often those of omission, not commission. That is, bad bosses are defined not so much by any appalling things they do as by certain critical things they don't do.” Orchestrating Test Automation ”Designing good tests is one of the hardest tasks in software development. That’s worth saying again. Designing good tests is one of the hardest tasks in software development.” ”I fear that testers are worrying too much about the effort and skills required to automate tests – and not enough about what it takes to design reliable, portable, accurate, and trustworthy tests that actually matter.” Yet another Software Testing Pyramid We Have Met the Enemy and He Is PowerPoint “It’s dangerous because it can create the illusion of understanding and the illusion of control,” General McMaster said in a telephone interview afterward. “Some problems in the world are not bullet-izable.”
  • 21. Quality is a Four Letter Word ”In my experience, metrics of this sort serve to do one thing and one thing only. They seek to modify behavior of the test team. This is due to the Hawthorne Effect. Peoples’ behavior change in accordance to how they are being measured. As an experienced manager, I know the ones I chose above, if used “right”, will make sure more and more tests are being produced, but they tell me nothing of the quality of the code or product.” Initial Thoughts on Group Testing ”I want to open for a discussion on pair testing or at least widen the concept. By saying pair, we say two people, but why should we limit us to that? Depending what your objective is a different number might be more applicable?” A List of Distorted Thinking “Being human means that if you have a bad lunch or some cloud system goes down which keeps you from your testing, the physical reaction of frustration leaves you very open to distorted thinking. In the case of testing, some of this distorted thinking seems useful for uncovering test ideas, and yet we also need to recognize when it’s time to stop catastrophizing and time to start collaborating with our teammates.“ Actually there are best practices Discussion about what Best Practice means. The Testing Quadrants – We got it wrong ”With the new labels the quadrants are a bit different than the ones that Brian Marick originally wrote down. On the left-hand side the confirmatory tests that are technology-facing are of course the unit tests and those class-level integration tests that developers write. These help drive the development of the project forward. In the second quadrant now are the expectations of the business. Usually I try to express most of them in the form of executable specifications, that I can automate. And I can also imagine other business-facing expectations that I cannot automate that probably fall in here, like reliability of the software product. In the third quadrants on the top right are tests that help to investigate risks concerning the external quality of the product. For me Exploratory Tests fall in this category, but also usability concerns, and end-to-end functionality. The fourth quadrant then probably consists of internal qualities for the code like changeability (static code analysis comes to mind), maintainability (how high is your CRAP metric these days?), and design flaws like code smells.” Creating a Test Lab for Mobile Apps Testing
  • 22. ”If we have a testing burden when we develop mobile apps, what sort of considerations should we take into account when we are setting up test labs, when using real devices? Here are some pointers from my own experience.” Three Keys to Mobile Application Design “If you are new to designing mobile applications, that often means you will look at web apps and PC apps first, because that is what you are most familiar with. Then you pull out your smartphone, and look at the apps you like and use the most there. That’s exactly what I did when I started out, and while I felt like I had adjusted to a new paradigm (smaller screens, less powerful devices, different network technology, etc.) I felt like something was missing. I couldn’t quite put my finger on what was different, but the apps I use on my mobile devices, and the apps I helped create demonstrated it to me. We are going through a major shift in technology, and the software we use is moving toward a combination of mobility, social connections, and gaming and entertainment.” FEW HICCUPPS Heuristics Several years ago, I wrote an article for Better Software Magazine called Testing Without a Map. The article was about identifying and applying oracles, and it listed several dimensions of consistency by which we might find or describe problems in the product. However, it’s been a long time since the article was published, and since then, James Bach and I have observed testers using other oracle principles, both to anticipate problems and to describe the problems they’ve found. Keep experts on tap, not on top Among the many qualities that distinguish successful leaders from millions of less-successful executives in the world is an awareness of the limits of their knowledge. They know what they know, they appreciate what they don't know, and they have a healthy respect for what they don't know they don't know. In short, they have great meta-knowledge. In my experience, experts are among the least successful predictors in times of massive uncertainty. Valve Handbook for New Employees Tracking Testing on the Scrum Taskboard ”How do you treat bugs on the taskboard that are found during testing? Create a new test for each bug, and put the test task back in ToDo? Or create a bug, and a bug-follow-up testing task? ” Two problems with Context Driven Testing The most interesting thing about this articles is actually the comments by Cem Kaner: “As I see it, my responsibility (as a tester) is to serve the project and its stakeholders, in ways that advance their goals for the project, and if I can’t focus on that and do that well, I shouldn’t be on that project.” Test Estimation Tips
  • 23. ”Estimating the time it takes to test a feature is an important, yet often overlooked, skill. If your estimate is too low, it can affect your product’s quality and release date. If your estimate is too high, it can lead to an inefficient use of resources. The topic of project estimation is complex enough topic to fill entire books. But here are a few simple tips that can quickly improve your estimation skills, not to mention your work/life balance and general happiness.” The Growing Queue at the IT Morgue ”For one, hardware and user-experience companies like Apple, who make their money selling products and apps, are more valuable than web companies like Google who make their money advertising web content. This value gap is only going to get wider. As the web recedes into the background, the opportunity shifts from making money on ads to providing actual value to users monetized through direct sales and subscriptions. Devices and data centers are the future and it is a very compelling future.” Estimates are often helpful, Estimates are often not ”Estimating helps when the process of estimating builds shared understanding among the people who want the work done and the people doing the work.. Collaborative estimating gives the best results. Diverse experience yields a broader range of perspectives and questions. Questions and perspectives build understanding of the what, why, and who related to the request. That’s helpful. Group estimating reveals differences in knowledge and understanding. Finding those gaps early is helpful. Group estimating surfaces assumptions. When we are aware of our assumptions, we can verify–or debunk– them.” The Hallmark of a Great Test Team “One hallmark of a great test team is how they behave in the final days before a release. I’m not referring to working late to automate the last few tests, validate bug fixes, and run a final test pass. These long hours are an expected part of the job, and when you think about it, these tasks are really just bookkeeping. I’m talking about something that actually affects the product: does your test team still try to find bugs just before a release?” The elements of Improvement (Three pre-requisites for improvement) Information. People need information about the context and how their work fits into the big picture. They need information from the work so they can self-correct. Without this information, systematic improvement is impossible. A desire to improve. Most people want to do their best and learn to do better–until that impulse is squashed. One-sided evaluations, organizational hurdles, relentless pressure strangle the desire to improve. Time to reflect and learn. People need time to design and implement new processes, and practice new skills. Relentless deadline pressure makes this impossible. People under pressure are less likely to try something new, or think clearly about anything. What do middle managers do in Agile?
  • 24. ”Organizations moving to agile still need management, and often still need people in management roles, especially in large complex organizations. In traditional hierarchies, middle managers look up the hierarchy for direction, and focus down the hierarchy to accomplish cascading goals. When teams pull work from queues and self-organize to meet goals, the real opportunity for middle managers is to look across the organization to improve the system and develop people and teams.” Test Plans & Test Cases in Agile/Scrum ”First, yes, the “where’s the testing in Agile?” question is a good one. Many of the people involved in defining Agile methodologies seemed very focused on automated unit testing, and perhaps missed the fact that unit testing by itself is not enough. According to Capers Jones’ studies, even the best unit testing removes only 50% of defects, so testing needs to be added to the methodology.” Personal Kanban Tool (Free for personal use or for small teams, costs money for more users) Can TDD Help Reduce Integration Time? ”It has been known for decades that the sooner you find a problem the easier it is to fix. TDD takes that idea and applies it to the minute to minute activity of programming. Not that TDD is only about defect prevention, but each defect prevented can potentially save considerable time in the future, including time spent in integration.” Manual Test is Unsustainable ”Creating automated tests can be very difficult, especially when the code has gotten long in the tooth and was not created with automated tests to begin with. Many product development teams don’t invest in automated tests. They think they cannot afford them. They think their product is different and can’t be manually tested. This thinking is flawed.” Management 3.0 Is a Way of Work ”My core belief is that organizations are complex adaptive systems, and that managers must care for the system, instead of bossing people around.” Ethics of Testing (James Bach) ”I have an ethical code. Actually, several! The Association for Software Testing adopted the ACM Code of Ethics, verbatim. It’s okay. I would have preferred the simpler IEEE Code, personally. I also almost completely follow Jerry Weinberg’s code. I take comfort from all of them, plus I have my own.” Microsoft vs. Google Part 1: Innovation Their grassroots efforts at innovation clearly show they believe (correctly) that creativity isn't governed by a person’s title or the org in which they sit. Creative people can literally be anyone and are often those without advanced degrees (Gates, Jobs, Zuckerberg …) and without an entrepreneurial track record (many top innovators have been one hit wonders). However, they take a very different approach to encouraging the masses to innovate. The bottom line: with 20% time
  • 25. Google lowered the bar for innovators and thereby watered down the value of the results; with its Garage, Microsoft raised the bar and created a more sustainable model. Sleepy Automated Tests “One of the most common problems with automated tests is synchronizing the automated test and the application or service under test. Automated tests sometimes race through their instruction set faster than the application or service can respond leading to false positive (test reports a failure but there is no bug). Often times testers use a Band-Aid approach to solve the problem by sprinkling Sleep() methods throughout their test code in an attempt to synchronize the test code and the application or service under test. Unfortunately, these Sleep() methods may negatively impact the performance of an automated test and artificially increase the time of the automated test pass.” The Goal and the Path ”I’ve seen many people with “visions” fail to succeed because they spent their time looking for shortcuts through the system rather than understanding that navigating the system is the key to achieving a vision (conversely, I’ve seen many others fail because they focused on navigating the system rather than their initial goals).” Management 2.0 is the right thing done wrong ”There’s also nothing wrong with the idea behind balanced scorecards. The problem with measurements is that one metric will lead to sub-optimization, and therefore you need multiple metrics to have a holistic view of the organization. It is useful to have multiple perspectives on complex problems. Unfortunately… managers see the organization as a collection of parts. They try to optimize departments. But in a complex system the performance is in the relationships between the parts. And thus you need metrics that cover perspectives of multiple stakeholders, or “measure from-the- outside-in”.” Proving the Embedded Software Test Team ”I am afraid that I do not have a lot of quantitative data, as most software engineering metrics lack construct validity. (Or, in different words, "All bugs are not created equal.") Three arguments I have for embedded teams: 1) Track the amount of time testers spent blocked on different clients over a period of years, I see a correlation between tester-is-expensive-paperweight waiting for a build and external test teams. Of course, there are too many independent variables to track here; the improved teams didn't just embed the testers, they also generally moved toward one-piece flow, pair programming, etc. 2) Track the number of handoffs that occur, from dev-to-test-to-dev. When people sit next to each other, it is easier to show the dev your test methods, so the programm can 'poke test' the app before handing it back - not just fix the first blocking bug you found as a tester. Also, allowing mentions-in- passing and desk pairing increses the chance the bug will get fixed-fixed on the first go round. 3) Track the costs of hand-offs. If testers have to file a bug report that needs to get triaged before it
  • 26. gets fixed, then costs increase over a conversation and a fix. The most project management layers on top of this, the more the fixes cost.” Using Project Artefacts to their full potential (The same thoughts are applicable to many test artefacts as well)!/2012/05/hello-world-today-i-realized-its-been.html “Problem: People in projects are very seldom using the basic artefacts to it's full potential. Which artefacts? Time plans. Resource committments. Budget plans. Processes and delivery plans.” Visualizing Stakeholder relationships!/2012/05/thursday-evening.html ”Just a thought about stakeholder maps. If you're into 6 Sigma, you know what I mean. Quite possibly you know about them anyway. Four quadrants, Interest or Support on the x-axis, power or influence on the y-axis. Map people according their interest in the project and their power in the organization. Anyway, when I did my 6 Sigma Green Belt training, we were advised not to show the stakeholder map to the people actually listed there. What if we actually did the opposite and took the stakeholder map to all concerned parties and showed it to them and asked them to comment it?” A sense of ownership ”A friend of mine recently told me that his off-shore team was producing terrible results, and slowly too. When he traveled there to discuss the quality and productivity problems, he found out that the same people had their own little startup on the side, working on it in their spare time. And the product they were making as a startup was amazing. While the product they were making as off- shore employees was crap. I think these people had an ownership problem.” We Cannot Choose Between Management And Leadership leadership.html “I subscribe to a number of services that look for pithy quotes from Big Names, authors, and other people who are looking for publicity. I saw one about moving from manager to leader. Ok, so these are writers or reporters, and they may not know. Choosing to be a manager without being a leader is like choosing to drive across the country without a map.” Metrics & Ethics They (management) understood that the metrics were imperfect. But they felt that they needed ways to summarize what the organization knew about projects. They felt they needed ways to compare progress, costs, priorities, and risks. They felt they needed ways to organize the information so that they could compare several projects or groups at the same time. And they felt they needed to compare what was happening now to what had happened in the past. Hung then made three points: 1. These are perfectly legitimate management goals. 2. Quantification (metrics) is probably necessary to achieve these goals. 3. The fact that there is no collection of metrics that will do this perfectly (or even terribly well) doesn’t eliminate the need. Without a better alternative, managers will do the best they can with what they’ve got. .
  • 27. It is important (I think very important) to pay attention to the validity of our metrics. It is important to improve them, to find ways to mitigate the risks of using them, and to advise our clients about the characteristics and risks of the data/statistics we supply to them. More adventures in magic automation numbers ‘When will our *functional+ automation remove the need for our extensive manual regression testing?’. This is of course a trap. There is no way to determine this number with any degree of certainty. But what if you really needed to provide an argument around the inability to predict this? Wherever You Go: Testing Mobile Applications Joey McAllister: What are some particular concerns you’ve encountered with popular mobile products like games, social applications, and mapping software? Jonathan Kohl: A couple of the big things that are difficult are security and privacy. The devices themselves aren’t that secure with their network connections, so it’s easy for people to snoop in, crack your accounts, and find out personal information. But, privacy is a huge issue. Some of the big social applications get in trouble in Canada because they violate our privacy laws and they have to clean things up. Two of the most dangerous terms in Agile; Done and Whole Team I have come to realize in the last two weeks that there are two words in the Agile lexicon that really bug me. Not because they are bad per se, but that they are so commonly misused and abused. Not to mention being insanely context sensitive. The Secrets of Faking a Test Project This presentation is satirical – we challenge people to think about how they would approach a project where the goal is to release bad software, but you make it look as if you really tried to test it well. It didn’t take much effort on our part, we just looked to typical, all too common practices that are often the staple of how people approach testing projects, and presented them from a different angle. Why 100% Test Pass Rates are bad When you change your mindset and write tests to prove the product doesn’t work, rather than to prove that it does work, you’ll be surprised how many more bugs you expose. This simple change will make you a better tester, and your products will be better for it. Testing the Limits with Zynga’s Galina Kramer (Localization Testing) uTest: Aside from being challenging, localization is arguably one of the most misunderstood testing types. We don’t expect you to disclose any specific bugs, but what are some of the types of issues
  • 28. that are uncovered during the course of a localization test? Misspellings? Poor translations? Others? And while we’re at it, how do you define localization testing? GK: Localization testing covers all of the above and more. We definitely look for misspellings, poor translations, truncation issues… We need to make our games equally enjoyable in every language. This means keeping in mind culture differences, religious beliefs, language intricacies – all of it. Testing with surrogate code points Surrogate pairs are often problematic in string parsing algorithms. Unlike “typical” 16-bit Unicode characters in the base multilingual plane (BMP or Plane 0) surrogate pairs are composed of 2 16-bit Unicode code points that are mapped to represent a single character (glyph). (See definition D75 Section 3.8, Surrogates in The Unicode Standard Version 6.1) Surrogate code points typically cause problems because many string parsing algorithms assume 1 character/glyph is 1 code point value which can lead to character (data) corruption or string buffer miscounts (which can sometime lead to buffer overflow errors). As an example of string buffer miscounting let’s take a look at Twitter. Seven Ways to Find Software Defects Before They Hit Production This article provides a quick list of test ideas in seven techniques, along with hints of where to go for more: 1. Quick Attacks 2. Equivalence and Boundary Conditions 3. Common Failure Modes 4. State-Transition Diagrams 5. Use Cases and Soap Opera Tests 6. Code-Based Coverage Models 7. Regression and High-Volume Test Techniques Quality Assurance is a Process, not a Department an-ipo/ “Of course every great development company will have a final step in the process called Quality Control or Quality Assurance, but it is my sense that the QA formal group is there to be the standard- bearer for Quality and rally the company around it, putting a final go or no-go procedure in place before the world gets its hands on a product, but not accepting proxy status for an otherwise poor process. To ignore or be ignorant of a showstopper in one’s own product is a reflection of a process that needs to be re-engineered. When you’re working with world-class engineers, it is much easier and far more fruitful to make sure the process is engineered correctly before the products go through it. The material cost of discovering a bug early in development is a tiny fraction of what it can cost you in the hands of the public. Give your engineers a voice and they will save you every time.” The Embedded Tester ”I hear it a lot at conferences, work meetings, interviews, discussions with managers, etc., the idea of “embedded testers.” The utterances are along the lines of, “Our testers are embedded on development teams!” or “How do we embed our testers with our developers?” It unfailingly puts me
  • 29. in the mind of a reporter from say, CNN wearing an ill-fitting helmet and flak jacket*, standing there as well-drilled troops go about their business. In Agile, testers aren’t “embedded” on teams any more than programmers are, or analysts, or any other role that is needed on the team. To say that they are suggests that this is an option, or a particular strategy you might employ to help with Agile development. It’s not! It’s an essential part of it!” Eliminate boring testing: Automating visual comparison ”It was a pretty straightforward process to set up an automated script that would import a campaign and show the preview screen. Then I just needed a way to capture the previews in each environment as a screenshot for easy comparison. Visually comparing two screenshots was still pretty annoying and time-consuming. So I added a step to automatically compare the two images pixel by pixel, and tell me if they’re identical. If they’re not identical, then I want a new screenshot showing me how they’re different.” The Secrets of why Testers are valued and proud at Microsoft “The short answer is, hire great people, give them challenging work, and give them a vision for growth.” Testing Dojo Example ”The dojo started with the introduction of the rules and our mission statement. The attendees were going to deal with touring heuristics FCC CUTS VIDS and pass as many of the tours as possible.” Why internet search as we know it today will disappear ”The problem with Internet search is that being stupid about it is profitable. The more ugly blue links you serve up, the more time users have to click on ads. Serve up bad results and the user must search again and this doubles the number of sponsored links you get paid for. Why be part of the solution when being part of the problem pays so damn well?” Misusing Software Process Metrics as People Metrics “I wish I could say that I never saw organizations make this kind of mistake with process metrics (i.e., mistaking a software process metric for a people metric), but unfortunately it is all too common.” Failing Fast “MacCready took a different approach. He knew that no matter how good his designs, the likely result would be failure, just like all the others before him. As a result, he expected failure and built his prototypes with that in mind. He use common and cheap materials that were easy to work with and easy to repair and replace. As a result he could test new designs in a matter of weeks rather than years.” Indicators, Testing & Wine Tasting
  • 30. ”Essentially, it was about how to judge feedback from a continuous integration system next to the other information that the testers in the team were producing. The reason I'd started asking questions was that with the myriad of information available there was a risk (as I saw it) to ignore some information - or cherry-pick the information to use...” Everything You Wanted to Know About SCRUM, But Were Afraid to Ask ask/2012/04/ Want to learn about SCRUM, but don’t have 10 minutes of spare time? Then checkout this 9-minute video on the basics of this popular methodology. A nice introduction to a very important subject. Enjoy! Test doesn’t understand the customer ”On the first day of our mentorship, I asked his opinion around something I had casually observed at Microsoft. What I did not expect was how fundamentally his answer would shake my world. Me: “…In addition, given your experience, there’s another pattern I’ve observed that I’d love your opinion on. I’ve noticed that around 15 out of 20 folks in your role (Product Business Manager) got there via Program Management, about 4 via development, and around 1 out of 20 via test.” Mr. R: “I don’t know if that’s totally accurate, but seems about right.” Me: “So why do you think test so rarely gets the role?” Mr. R: “Oh, That’s easy! It’s because Test doesn’t understand the customer.” ” What is “leadership” anyway? “It's easy to see the absence of leadership - but how do you define it's presence?” “We've replaced leadership with management, and we suffer from it. This replacement pushes us, to a great extent, to many of the maladies you've seen on this blog: The desire to institutionalize process, to have easy answers, to manage to a template or a spreadsheet or process instead of getting out there, understanding the problem, and, you know, fixing it!” Why I joined Microsoft by James Whittaker “I suppose that pointing out that the Google to Microsoft transition is not particularly rare won't suffice as an explanation so what follows is a more long winded account. For those uninterested in the finer points, here's the short version: I think much of what is happening in the mobile and web space is broken and is trending toward becoming more broken. Users have their privacy under assault and are losing control of their identity and personal data. Independent developers are facing walled gardens around the data and services necessary to move the web forward. Solving this problem is going to require a diverse set of intellectual property, technical and information assets and an inclusive attitude toward software developers. In my opinion, Microsoft is among the best positioned companies to lead such an effort.” We're Not So Different, You And I ”No matter where I go, people have the same problems with management & leadership, the same problems with self-organizing teams, and the same problems with organizations & departments.” “In my experience, cultural differences between countries pale in comparison to the cultural differences between industries.”
  • 31. 97% ”Our test cases shouldn't be written to prove there are bugs in our code. They should be written to prove there are no bugs in our code. So having bugs associated with 97% of our test cases in one of our projects means there are way too many bugs in our code and we need to push quality upstream. ” Look to the Data “I was recently asked to investigate my team’s automated daily test cases. It was taking more than 24 hours to execute our “daily” test run. My job was to find why the tests were taking so long to complete, and speed them up when possible.” Exploring Test Roles ”Roles that testers play on teams vary. They vary a lot. You can’t compare them. That’s ok, and (IMO) part of the growth of the role. I find it disturbing and detrimental when testers not only assume that their version of testing is “the way”, or that some roles (or people in those roles) do not qualify as testing.” Every Test Failure Means Something ”The point of this story is simple. Every test failure means something. The failure may mean a product failure. It may mean you have flaky tests. When you start to assume flaky tests or environments, you’re heading into the land of broken windows and product failures you could have found earlier (actually, you probably did – you just ignored them).” “Relying on test automation for any part of your testing is pointless if you don’t care about the results and look at failed tests every time they fail.” The Robots are Taking Over “Normally, I don’t blog about news stories, but on the way to work this morning I heard an interesting discussion on a local talk radio station. It turns out that there are people up in arms over the purchase because using robots in the warehouse puts people out of jobs. I laughed out loud in my car as I realized that these people calling into the radio station were having the same discussion many people have about test automation.” Testing the Limits with Gerald Weinberg Bug Title Crash Course * As short as possible, but no shorter than that. * Start the sentence with the most important, to capture the reader’s interest. * Don’t overdo “externalization” to capture interest, rather describe the dry facts, and let the readers draw conclusions
  • 32. * If it’s difficult, try many times, or write the title after everything else is done (you might find the right words on the way.) * Include a brief description of what happens. * Include where the problem happens. * Describe observations rather than (presumed) facts. * Use a fair description, don’t exaggerate or understate the consequence of the problem The Linear Filter Trap ”People like to solve problems that they can see - which means we have a tendency to "fix" the problem we see in front of us. This occurs more often if the problem appears to have a "straightforward" fix -> I think of this as a form of cognitive ease in action. Digging for root causes is a challenging activity, and we sometimes want to believe that the cause we identify is good enough to fix.” Delegation does not mean Negation ”With delegation of work you do not delegate your accountability. Instead, you share it.” Context Driven Testing Sleeping with the enemy - Gojko Adzic describes why independent testing should be a thing of the past. Traditional software delivery models are based on a lack of trust. Because the business doesn't trust developers, testers are asked to provide independent validation. Because developers don't trust testers, everyone wastes a lot of time arguing about whether a problem is in the code or in the tests. And testers are taught not to trust anyone! All of this distrust even though we share the same end- goal—delivering a product that satisfies our customers. Gojko Adzic describes why independent testing should be a thing of the past. Focusing Testing on Business Needs Testers often forget that they are service providers whose role is to provide critical information to the project’s stakeholders. Testing must focus on business needs to add the most value and gain respect. Artful Testing In this presentation I will investigate what happens when we infuse testing with aesthetics. Can the fine arts in any way support or complement our testing efforts? How I wish users knew how I help them through context driven testing
  • 33. testing This talk will be about the journey of being a context driven tester and the opposition I faced and how I paddled through it to win in most of my assignments. Context-driven Testing is not a religion Over the past few years, several people have gained visibility in the testing community who express ideas and values that sound context-driven to me. Some call themselves context-driven, some don’t. My impression is that some are being told they are not welcome. Others are uncomfortable with a perceived orthodoxy. They like the approach but not the school. They like the ideas, but not the politics. Contexts Differ We don’t manage the project managers. We don’t decide what information they have to give to the people they report to. Context-driven testing at a crossroads (James Bach and Cem Kaner part ways) ”Cem Kaner, who controls, has announced an interesting change in his view of the Context-Driven School. He says he prefers to think of it in terms of the Context-Driven approach, not a school of thought. This is a significant change from his original view, which was that CDT is a different paradigm.” Lessons Learned from Context-driven testing (Follow up to the above article – with comments from Cem Kaner which are interesting) There has been some fluff and rumor around context-driven testing yesterday. Some folks even talked about the death of context-driven testing. Most of it was issued by the about page from Cem Kaner. If you haven’t read it yet, go ahead, read it now, I will wait here. The Hindsight Trap Root cause analysis (RCA) activities are an important part of many retrospectives - whether as part of a team activity, analysis of test reports, analysis of fault (bug) reports or as standalone activities to investigate organizational and process problems. The most common form of this trap in testing circles (where documentation is involved) usually crops up as a question, "why didn't you/someone spot this problem earlier (as it's specified in document X)", or statement, "this requirement should been tested because it's specified in document X". 60% of Users Expect Mobile sites to load in 3 Seconds or Less (Importance of Load testing for mobile web) That’s right, if your mobile site doesn’t load within THREE seconds 60% of visitors will abandon it. And, seeing how we’re not a very patient society anymore, 57% of mobile users will only give your site an extra two seconds (bring the grand total up to five) to load before aborting the mission.
  • 34. Test Reporting by Michael Bolton Calculating a ratio of passing tests to failing tests is a relatively easy task. If it is used as a means of estimating the state of a development project, though, the ratio is invalid, irrelevant, and misleading. At best, if everyone ignores it entirely, it’s simply playing with numbers. Otherwise, producing a pass/fail ratio is irresponsible, unethical, and unprofessional. Testing is like investigative journalism, researching and delivering stories to people. The newspaper business knows how to direct attention efficiently to the stories in which we’re interested, such that we get the level of detail that we seek. As testers (or, heaven help us, quality assurance) people, we tend to be chartered to look for problems, and to investigate them in ways that are most helpful to programmers, managers, and our other clients. A failing test on its own tells us little, and a failing check even less; a failing test is only an allegation of a problem. Investigation and study of a failing test is likely to inform us of something more useful: whether someone will perceive a problem that threatens the value of the product. What Metrics do you Use in Agile? agile/ This is a great question. Too many organizations transitioning to Agile don’t ask it. They continue using the same metrics they were using for traditional projects even though the process is fundamentally different. The problem is that traditional measures don’t work in Agile. Sometimes they’re actively harmful. First, I only use metrics to get a 50,000 foot view on what’s going on. Second, I do not use metrics to compare teams or individuals. Ever. Third, I am much more focused on qualitative than quantitative measures. Where do bad managers come from? "Real" executives take the long view. They are the kind of people who, at age 60, can be found planting trees. IT managers who think that fast moving technology requires short-term quick fixes are stuck in a middle management mentality. They will never become real executives. How to Measure Productivity That’s why it’s also a bad idea to rely on one metric. Use a combination of tools to focus on areas of concern and improve them. If velocity is going up, but cycle time is staying the same, there may be some problems with story estimation – maybe one voice has started to dominate the process. If output is up, but velocity is down, there may be a problem with prioritization (we’re only working on easy stories). Take the Crappy Path
  • 35. Developers take the Happy Path. Their Unit Tests often consist of using their code in the obvious way in order to produce the expected results. They instinctively want to see their new creations succeed so they inevitably focus on success. See how nicely behaved my code is! Happy, happy. Product Managers take the Happy Path. Put nice values in, take a nice marketing-oriented screenshot of the results. Show it to the Sales Team. Happiness all around. End Users typically take the Happy Path during User Acceptance Testing. A little bit of typical clean input in, typical expected output out. Now let me stop wasting time with this new system and get back to my real work, so I can leave on time and be Happy. Cost-Benefit Analysis of Exploratory Testing in Comparison with Scripted Testing$file/BTH2011Pang.pdf 7.2 Scenarios in Which SBTM Can Be More Cost-Beneficial Most of the survey respondents and interviewees were of opinion that SBTM is adaptable in any situation of the project. And it could always be cost-beneficial if it is employed in a thoughtful way that is complementary to the testing activities of the development team. But SBTM could be more cost-beneficial when compared with TCBT in the following scenarios. Requirements are vague or insufficient, and not testable They do not have every bit of information about the system Cost constrained projects, for example, when testing is time critical Short iterations Rapid feedback required to the developers High pace There is rapid and constantly change (in requirements, design, etc.) Manual regression testing 7.3 Scenarios in Which TCBT Can Be More Cost-Beneficial Interviewees stated that, in certain scenarios, TCBT is required. When test execution is too expensive or too difficult and what we have is a lot of cheap time to think. If we are working with a test platform that is quite expensive or is available less frequently, it is more cost-effective and makes sense to think about the tests in advance, because what we have is a lot of cheap time to think and expensive time to execute. For example, when we have to use very complex SQL queries to test reports, it is better to prepare it before execution. When there are regulatory requirements Large project or long term project Large project always involves complex settings where the test data needs to be prepared in detail. When they do not have confidence that testers can do good testing without detailed test cases. For example, new testers In traditional mode, there can be a requirement for repeatable test execution When verification of the requirements is required When test cases for automated regression testing are required Management 3.0 in China In Europe I have asked many people for their favorite management practices. But I never expected to find the most inspiring example of Management 3.0 practices in China. 37 Sources for Test Ideas
  • 36. It is often stated, with right, that you should use many, different information sources in order to come up with good test ideas. Learning is Dead Perhaps I’m just being an old fart, but the serendipity of these three events launched a thought that scared the crap out of me. In these days of search engines and Wikipedia, I wonder if anyone knows how to think anymore. Knowledge is much more than learning or regurgitating facts. Many Models – Better Test Ideas ”Henrik Emilsson has convinced me that skilled software testing is based on invisible mental models that help us see what can be tested. If we can make these visible, we can sharpen our skills, and also teach testing more effectively.” User Interface Test Automation and its Challenges in an Industrial Scenario “The growing demand for UI test automation has triggered the development of many tools. Researchers and developers have been continuously working to further improvise the existing approaches. If we look at GUI test evolution we can observe a clear progress from manual testing towards complete automation.” utest Success Story (Interesting story about a stay-at-home dad who makes money as a freelance tester through utest) ”I am a Gold rated uTester who, after 10 years as a stay-at-home dad, has gone from no income to earning legitimate monthly pay and tremendous professional satisfaction in 3 short months. I felt my story was one worth sharing and hope it both inspires and encourages those in the uTest Community, those considering joining and those involved in creating it.” Word on The Street: 75% of Apps Not Tested “There’s an obvious reason why that app you just downloaded doesn’t work: it wasn’t tested! In fact, according to a recent article on The Street, that’s the case for 75% of all mobile applications.” Exploring Test Automation ”I’m all for saving time and money, but I have concerns with an automation approach based entirely (or largely) on automating a bunch of manual tests. Good test design considers manual and computer assisted testing as two different attributes – not sequential tasks.” The Secrets of Leadership ”When people get to “Senior” roles at Microsoft, one of the things we expect is an ability to influence without authority, work through others, and basically make those around you better. For our introverted tech geeks, this is often a big wall to climb. Inevitably, one of these Senior non—manager
  • 37. folks tracks me down and asks, “How can I get better at influence?”, or “How can I get others to accept my ideas?”.” Something I did not expect ”Over the past few years, I have been an ardent, arguably minority voice stating my skepticism about the value of GUI-driving test automation, developed by someone other than the developers, after the code is "complete." At the same time, I've admitted that the reality of an increasingly technological society will mean that, while we may not all need to be SDETs, we'll need to be more technical simply to participate. Then something happened that threw me for a loop.” The Skeptic's Dilemma ”But it’s easy to go overboard with skepticism. Time and time again, I hear good testers apply their skepticism broadly and liberally. Some (paraphrased) quotes I’ve heard recently include: “Model-based testing is interesting, but it doesn’t work in a lot of places” “I’m skeptical of static analysis tools – sometimes they have false positives” “Metrics are evil, because someone may use them to measure people” I agree whole heartedly with each of these quotes. However, I worry that folks are throwing the baby out with the bathwater.” "In-The-Wild Testing" (ITWT) QA-Chain/Test-and-QA-Software-Test-Professionals-Conference "In-The-Wild Testing" (ITWT) is an effort to educate tech leaders about how to help QA teams and organizations launch higher quality software, quicker, faster, and cheaper. The idea of in-the-wild testing is about providing organizations with the real-world testing data necessary to make informed decisions about releasing products to market. What percentage of testing should be done in a lab, and how much of our testing should be pushed into the wild, to catch issues before the users do?” 8 Things you ought to know if you do not know anything about hiring a software tester hiring-a-software-tester/ ”I believe there are some unique skills I would look for in a software tester that I would not necessarily look for in a programmer. So, here it is.” Maybe it is like building a house? ”A few years ago, I overheard part of a conversation between an academic department on how to teach project management at the graduate level. One professor thought that project management for IT projects is "just like building houses" -- you break down the work into very small chunks, add the chunks up, make a plan, and manage to conformance to plan. The second thought it was more ...
  • 38. nuanced than that. The argument split the department so much that the department chair ended up appointing a third person to teach the slot - one of the few people in the department with a couple of decades of relevant experience. Which brings us to today's question: Is Project Management for software the same as it is for building a house?” That’s a nice theory – How to get people to accept a new approach such as agile ” In many organizations agile suggestions fly in the face of their accepted “best practices.” Such ideas also tread on political toes. So one response I hear a lot is: “That’s a nice theory, but it won’t work here.” … The simple reframe from “Why not?” to “What would have to change?” opens up possibilities. What could have become an argument becomes instead a brainstorming session. The result is a chain of steps we can take to go from where we are now to where we want to be.” You are not a complexity thinker when … “You’re not a complexity thinker when… you complain your model is misunderstood or misrepresented by many people because the point of your model should be to enable sense-making.” In Search of Code Quality (there is also a link to a fun comic in here ”Quality and care are two different ways of looking at the same thing. Quality comes from care, so perhaps what I want to measure is the care put into coding. The fact that a developer wanted to compile at a higher warning level or turn on more static analysis checks, or that they cared enough to get the testers input on testability, or a peer’s advice on maintainability may be the best ways to measure code quality of all.” The Value of Testing ”I received the following message from Helen Huang: ”I am tester .I has work this job of 5 years on China. Now I meet some problem in my working. 1. In china , I find many company always talk the tester value ,what is the tester value ? . eg: Find the number of bug Or find a bug problem ?” This is a common problem.” Testing the Limits with Anne Marie Charrett – Part 1 ”uTest: In terms of money, time and effort, what is the biggest waste of resources you see in testing today? Certifications? Test reports? Automated tools? You tell us. AMC: Behind this waste lies the concept that there is a simple solution to a complex problem. The above solutions are wasteful when they aim to easily resolve a complex problem. If we could stop looking for the silver bullet, then perhaps these wasteful solutions wouldn’t look so tempting.” Visualising Quality
  • 39. But let’s deal first with the two nihilistic categories. I don’t believe that visualising quality is impossible – it is just difficult. If it were easy we’d all be doing it already, but it is hardly in the same category as flying through space faster than the speed of light. As for the other category, with comments like “it’s between the user and the system, we need to put something in front of the user” etc I disagree. Quality isn’t only between a user and a system, it is between those two and also the people who are charged with delivering software. The Brutal Difficulty of Transformation ”2012 has not gotten off to a great start for Eastman Kodak. Three of the company's directors quit near the end of last year, and word recently emerged that the company was on the brink of filing for Chapter 11 bankruptcy protection. The easy narrative is that Kodak is a classic case of a company blind to the disruptive changes in its marketplace. Like many easy narratives, this one is not quite right.” On Test Design “Testing is the dominating method for quality assurance of industrial software. Despite its importance and the vast amount of resources invested, there are surprisingly limited efforts spent on testing research, and the few industrially applicable results that emerge are rarely adopted by industry. At the same time, the software industry is in dire need of better support for testing its software within the limited time available. Our aim is to provide a better understanding of how test cases are created and applied, and what factors really impact the quality of the actual test.” Software Testing in Agile Development : Technological and Organisational Challenges ”The contemporary industrial trend towards agile software development brings forth new concerns, challenges as well as opportunities. One of the main issues concerns the quality of the final product, for which testing is the well-known assurance mechanism. However, how to perform testing using existing expertise in an agile environment presents a challenging issue for the software industry.” “This result was interpreted as “Respondents would like to use TDD to a significantly higher extent than they actually do presently”.” ”… identified and listed seven potential limiting factors of industrial adoption of TDD: increased development time, insufficient TDD experience/knowledge, lack of upfront design, domain and tool specific issues, lack of developer skill in writing test cases, insufficient adherence to TDD protocol, and legacy code.” Scripted vs. Exploratory Testing ”I have been using the Scripted – Exploratory Testing Continuum in classes to explain how scripted testing and exploratory testing intertwines; and to explain that most of our testing is somewhere in the middle of the continuum.” Model-based Testing – Some Assumptions and Traps
  • 40. ”In December I had a very interesting discussion - about model based testing - I was there as a 'customer' and model based testing was being 'sold'. I made my skepticism known before the meeting, by my assertion that all testing is based on models - and that some people are better at observing, constructing, expressing and examining them than others. I also stated I'd seen the power of modeling techniques that allowed testing earlier but that they are far from the complete picture.” Looking for the Great Thought about Management ”This Friday and Saturday some 20+ management thinkers and practitioners will gather in Stoos (Switzerland) and talk about management. In particular, we will discuss why management isn’t changing fast enough.” Seven ways to make testing irrelevant on your team ”Wouldn't it be great if everyone on your team loved testing and working with testers? When developers and testers have a respectful relationship, it's easier for developers to see the value and take an interest in testing. Instead, many testers pick up a knack of alienating themselves—and testing in general—from their team. The following is a list of ways in which testers may make themselves and testing irrelevant to others on their team, along with tips for avoiding getting stuck in these situations.” Top Ten Reasons Why Large Companies Fail To Keep Their Best Talent keep-their-best-talent/ “Whether it’s a high-profile tech company like Yahoo!, or a more established conglomerate like GE or Home Depot, large companies have a hard time keeping their best and brightest in house.” A Christmas Gift From a Tester ”A while back I received a gift from Matt Heusser, my early mentor in the Miagi-Do school. It asked me to test it. So, I decided to make a series of blog entries over the holidays out of it.” Testing the Limits with Richard Stiennon (Security expert) uTest: You’ve said before that mobile will not require its own anti-virus systems. That said, it seems that mobile threats are multiplying by the hour. In your view, what’s the biggest security challenge in terms of mobile? RS: Apps, apps, apps. VPNs, firewalls, and carrier filtering are going to impede network based attacks. Containing and vetting applications is the biggest security challenge for the platform vendors.
  • 41. By 2015 the overall threatspace will be ten times worse than it is today. Think about that. There will be TEN breaches as critical as the RSA attack. There will be TEN Google Aurora’s. There will be TEN Stuxnets. There will be 300 thousand new malware variants a day. Management Models, Values and Principles For example, Peter F. Drucker said there are five tasks for managers: 1. Set objectives 2. Organize 3. Motivate and communicate 4. Measure 5. Develop people Wilful Ignorance on Parade What a senior tester does at Microsoft Decision Making – senior testers make confident decisions, and do not rely on consensus for making (most) decisions. Influence – Dwight Eisenhower said, “Leadership is the art of getting someone else to do something you want done because he wants to do it.” Senior testers make their team better through influence rather than mandates – they are the point guard of the test team. Network – senior testers use their network to find answers (or knowledge), and connect their teammates with their network (i.e. tester matchmaking) Credibility – senior testers don’t proclaim they’re a leader, and no title demands respect. Leaders know that respect as a leader comes through establishing credibility with their team and peers. For me, the moment you demand respect as a leader is the moment you lose my respect as a leader. Test Levels in Agile Testing Test levels and Test quadrants. How to classify different types of testing. Worthwhile Documentation Would you hire a completely incompetent tester who needed to be told absolutely everything, in painful detail?” “We wouldn’t hire anyone like that.” “Fair enough, and I’d hope not. So, why do you insist that people write instructions for them that way?” What Exploratory Testing is Not else-testing/
  • 42. Michael Bolton tries to define exploratory testing further/more practically by stating what is not Exploratory testing. Status Reporting ”Status reporting of testing activities is extremely project-dependent. The needs of when and how and what will differ every time. Maybe that’s why there’s so little good writing about status communication; you have to make it up every time.” Why scripted testing is not for novices ”Scripted test execution is quite a bit more difficult than exploratory testing, unless you are assuming that the tester following the script has exactly the same knowledge and skill as the test designer (even then it is a qualitatively different sort of cognitive process than designing). An exploratory tester is following (indeed forming as he goes) his own intentions and ideas. But, a scripted tester, to do well, must apprehend the intent of the one who wrote the script. Moreover, the scripted tester must go beyond the stated intent and honor the tacit intent, as well– otherwise it’s just shallow, bad testing.” Ten Must-know tips for Performance Testing ”More than other automation, bad performance test automation leads to: Undetectably incorrect results Good release decisions, based on bad data Surprising, catastrophic failures in production Incorrect hardware purchases Extended down-time Significant media coverage and brand erosion More than other automation, performance test automation demands: Clear objectives (not pass/fail requirements) Valid application usage models Detailed knowledge of the system and the business External test monitoring Cross-team collaboration Unfortunately, bad performance test automation is: Very easy to create, Difficult to detect, and More difficult to correct.” How do I deal with my crappy organization? ”It is one of the questions I get most often… How do I deal with my crappy organization? I like my work but I don’t like what management is doing. How do I deal with it? Well, that’s easy…” Agile Alure ”I got to visit another company’s location this week – Allure Global. They make dynamic signage for movie theaters – concessions and ticketing, etc. They are an Extreme Programming shop, which
  • 43. means they use the XP practices like TDD, collective code ownership, and others. I took a couple of pictures.” The Death of Quality ”“The truth is, bad software makes more money” – Goranka Bjedov, STANZ 2011. If quality is dead, what are we doing here? If we are adding value, what is it and how do we measure it?” Selecting Automation Tools If you want a tool to do functional test automation (as opposed to unit testing), you will probably need both a framework and a driver. The framework is responsible for defining the format of the tests, making the connection between the tests and test automation code, executing the tests, and reporting results. The driver is responsible for manipulating the interface. 10 Things about testing that should die ”Face reality testers, neither the product nor the business revolve around you. Think about it. No business, no product, no developers => no need for testers. In fact, if developers wrote perfect code, there'd be no need for you. You are a service provider and your primary clients are the managers, developers, and/or executives. Your secondary clients are product users and investors.” Don’t Forsake Testing in the Rush to Market The question if it is better to get an application out early but bugged, and then fix it later, or making good software that is late to the market. Feature Phones still dominate World Market With all the talk of iPhone vs. Android these days, it’s easy to forget how the majority of the world’s mobile users still make calls and access data: via feature phones. A recently released report (download) from mobile strategy firm VisionMobile takes a look at today’s mobile marketplace finding that, despite the sharp rise in smartphone shipments over 2010 and 2011, global smartphone penetration (by OS) is at just 27% API Testing: How can it help Key benefits of API testing include: Reduced testing costs Improved productivity Higher functional (business logic) quality Just Fix It!
  • 44. Chris McMahon’s latest post (Just Fix It) proposes that as far as bug tracking goes, the best course of action is to skip the “tracking” part of the workflow and “Just Fix It.” I’m a huge fan of this approach and think that for the most part, tracking a large number of bugs in a big fat bug system (and often overemphasizing the church of the bug curve) pretty much encourage a test-last / test-quality-in workflow. The Future is Mobile Technology ”Technology is moving toward a combination of mobility, social connections, and gaming and entertainment. That means we need to be aware of combined technologies, combined software and tools, and combined activities. This requires that we solve new problems with new technology, or different combinations of technology we are familiar with.” Disposable Programs ”In many IT installations today, the number one problem is program maintenance. Although the total problem is far from simple, there are a number of relatively simple ideas that can be applied immediately to furnish "prompt relief." One such idea is the disposable program.” Shape of Actions ”In The Shape of Actions, Collins and Kusch describe key differences between two kinds of intentional human actions that they call mimeomorphic and polimorphic. In both words, “morph” refers to shape, or form. “Mimeo-” refers to copying. That is: software development is a polimorphic activity, and if that’s true, testing needs to respond accordingly.” Test is Dead (Test is Dead was the topic of the Keynote of the Google Test Automation Conference this year: On the alleged death of testing The problem here, at least in my opinion, is simple. Testers have become so focused on legitimizing their discipline and their role that they've forgotten that their role and their discipline exists to serve the companies that employ them. Conversely, the managers and executives of companies that employ testers are regularly directing testers to do these things they don't really understand rumored to be "best practices"-- and of course, then blame the testers when those "best practices" don't provide sufficient value. There is no "basic training" for testers, and there is no "manager basic training" that covers testing and test management as it relates to value to the business -- and even if you believe the training exists, you can't deny that: a) many folks wouldn't agree that training is of value b) even more folks who "should" have received that training haven't even heard of it New Ways of Thinking in Software Testing (Read the comment from Alan Page at the bottom of this blog post)
  • 45. Software-Test-and-QA-Technology-Strategy-Process-Editorial To put it a different way: If your company makes it's money off of giving away free software and selling advertisements, then, perhaps, if your customers are forgiving, and if you can respond to defects quickly, you can be successful by hiring automated-test-writers (selenium, etc) and crowdsourcing your testing with a product like uTest. The idea here is to have humans find bugs fast - and cheap - perhaps over the weekend - while the automated tools prevent major regressions. And hey, that might just work for google, or for Yahoo, or Mozilla. Would you want your bank, your credit card company, your automobile transmission, or the medical device company that created your pacemaker to use such a strategy? --- Alan Pages comment: I know I don't always play nice with others, but I sort of feel that the test-o-sphere is overreacting to the test is dead story. *If* you're shipping a web service, *and* the web service is mostly non-critical to day-to-day mission critical application, would you pay a tester to test it? Likely, you'd want deep functional testing and some perf load testing, and who knows what else, but it's certainly ok for developers to write this stuff. But I don't think we should take the message so literally. IMO, many testers have been doing developers work (unit & functional) testing for years - and of course it makes sense for developers to do that work. If you have the right set of diagnostic and monitoring tools in place, by all means, get rid of testers. Of course, as you say, if you're testing a medical device, or desktop software, you probably have to modify your story a bit - but to me, I still think there are aspects of the story (e.g. have developers do more testing; or collect customer usage patterns) that can apply to any software project. The reactions (this, and others) are interesting to watch - it's almost as if testers feel that they have to defend their job - but I see the topic as one we can all take something away from. Hell, I've seen some pretty insane ideas come from testers, and I've found some good in nearly all of them. Looking forward for when we can get to Test is Dead (is dead)? Testing the Limits with Noah Sussman uTest: What types of traits/qualities/skills does Etsy look for when hiring testers? NS: On my team there are two roles for which I hire. These aren’t formal roles and we switch them up sometimes. I hire software engineers who value quality and who are interested in how complex systems fail. These are the people who customize our mocks and fixtures, manage our coding standards, build
  • 46. static analysis tools, develop Jenkins plugins, design the CI system in general other things of that nature. … Here I look for talented QA analysts who are longtime Etsy users and deeply involved in the Etsy community. These are the people who develop functional testing tools, manage Selenium integration with CI and work with others in the organization to formulate test plans and resolve bugs. They also design and improve the process by which we triage bugs found in production. Complexity vs Lean vs Agile vs Me System thinking in a bar If we want to decide whether or not something must be treated as a system, we first need to consider what part of the world we want to look at, and what questions we want to answer. API Testing – Functional Testing Below the UI interface/ ”Some people assume that API testing is a ‘white-box’ testing activity in which the tester has access to the product source code. But in reality, API testing is truly black-box testing in the truest sense of the testing approach. API testers make no assumptions about how the functionality is implemented, and are not limited by constraints or distracting behaviors of a graphical user interface. ” A tester’s commitments ”This is the latest version of the commitments I make when I work with a programmer. 1. I provide a service. You are an important client of that service. I am not satisfied unless you are satisfied. 2. I am not the gatekeeper of quality. I don’t “own” quality. Shipping a good product is a goal shared by all of us.” RPF Google’s Record Playback Framework ”At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF. The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect.” Activities and Roles ”I’ve been thinking about testing activities and tester roles lately, or more lately, since it seems to be a recurring topic on this blog. I have to admit that I remain a bit surprised how often testers argue about what is and what isn’t testing.”
  • 47. Iterative Development: Some History computer.pdf ”We were doing incremental development as early as 1957, in Los Angeles, under the direction of Bernie Dimsdale [at IBM's Service Bureau Corporation]. He was a colleague of John von Neumann, so perhaps he learned it there, or assumed it as totally natural. I do remember Herb Jacobs (primarily, though we all participated) developing a large simulation for Motorola, where the technique used was, as far as I can tell, indistinguishable from XP.” Test Innovation ”Many years ago, test systems at MS reached a level of maturity and scale that enabled teams to easily run millions of tests. These systems[1] were the culmination of several innovations, but eventually led to other problems. The prime example of a problem created by the test system innovation is the challenge of what to do with all of those failed tests. Think for a moment. If you run a million tests and have a 99% pass rate (I know, but this is just an example), you have ten-thousand failures to investigate. Automatic failure analysis is an innovation that greatly reduces the time testers need to investigate automation failures (my stance is that automation is worthless unless every aspect of automation beyond test authoring is not also automated). We’ve also made great strides in test selection (let’s say you only had time to run five thousand of those million tests – which would you run?).” Smoke Test vs. Sanity Check ”The distinction between the smoke and sanity testing is not generally important. In fact, it’s one of the most trivial aspects of testing that I can think of, offhand. Yet it does point to something that is important.” Build in the Cloud: Distributing Build Outputs (Part 4)!/2011/10/build-in-cloud-distributing-build.html ”In the previous post, we saw how distributing build load across many machines and reusing the results of previous build actions can produce extremely fast build times. This distribution and reuse itself exposed other performance bottlenecks. Specifically, a full build of a large project may produce several gigabytes of output files, all of which need to be shipped from the cloud back to the developer’s local machine. This taxes both our networks and the developer’s local disk, the limited latencies and bandwidth of which slow down the build.” Build in the Cloud: Distributing Build Steps (Part 3)!/2011/09/build-in-cloud-distributing-build-steps.html “This post builds on these to describe a system for efficiently distributing build steps across a large number of machines. As you will see, details of the source file system and the build system are important to how we achieve fast and efficient distributed builds.” Build in the Cloud: How the build system works (Part 2)!/2011/08/build-in-cloud-how-build-system-works.html “In this article we will go into a little more detail in how we tell the build system about the source code and its dependencies. In a later article we will explain how we take advantage of having this
  • 48. exact knowledge to distribute the builds at Google across a large cluster of machines and share build results between developers.” Build in the Cloud: Accessing Source Code (Part 1)!/2011/06/build-in-cloud-accessing-source-code.html ”Much of our day-to-day activities as software engineers involves source code. When we join a project one of the first things we do is look at the source. We want to build it, run it, experiment with changes, test it, and challenge our assumptions about how it works. For most of us this means we start by “checking out” the source from version control. For small to moderately sized projects almost any reasonable version control system is adequate. But as the number of engineers increases and the code base grows, this can put a strain on the version control system and decrease engineer productivity.” Tools for Continuous Integration at Google Scale (The youtube video that was the foundation for the articles above) Five orders of Ignorance and Software Testing ”0OI – lack of ignorance. It’s when you know something. I know, for example, that the first track on the Rolling Stone’s Sticky Fingers album is Brown Sugar. 1OI – lack of knowledge. I don’t know (or remember) what the track is following Brown Sugar, but I could find the answer quickly. 2OI – lack of awareness. You have 2OI when you don’t know what you don’t know. I know that there are Stones tunes that I’ve never heard before, but it would be impossible for me to make a list of Stones songs I’ve never heard before. 3OI – lack of process. You have 3OI when you don’t have a suitable method for discovering 2OI (for discovering what you don’t know you don’t know). 4OI – meta-ignorance. You have 4OI when you don’t know about the five levels of ignorance.” ”When we test to ensure requirements were implemented, or the stories function as expected, we are testing for 0OI.” ”Testers also perform a lot of 2OI testing. We explore or run dynamic tests on a system to help discover what we don’t know we don’t know.” Checking Alignment “How do we know that our Intentions matched the Actual Need, that the Implementation matched our Intentions, and ultimately that the Implementation matched the Actual Need?” ”Finally, if we want to be sure our Implementation matches the Actual Need we have to watch customer behavior carefully. That means monitoring usage statistics …” Testing the Limits with Michael Bolton (Interview)
  • 49. ”uTest: You recently wrote a blog saying that testers should learn to program. Conversely, should programmers learn to test? Why or why not? MB: Programmers should learn to test. They do learn to test. Most programmers attempt to run the code they’ve written and look at it critically to some degree. Great programmers do that testing in a really diversified and yet focused way. What I’d like to get rid of is the idea “I program, I test, and therefore I know all I need to know about testing. I’m good.” To test really well requires study of many ways that things can go wrong, and many ways in which we can be fooled, and many approaches for defending ourselves against them. The best programmers I’ve ever worked with were themselves excellent testers, but they were also aware of their own limitations. They were really respectful of good testers, and, of course, good testers were respectful of them. It was a virtuous cycle.” Should testers play planning poker? “Having observed Planning Poker in action, I’m conflicted. Estimating anything is always a bit of a dodgy business, even at the best of times. That’s especially true for investigation and in particular for discovery.” The 3 Criteria for Agile Leadership Practices ”On Twitter I have claimed that a leadership practice is Agile when: 1. It supports people and their interactions; 2. It helps to deliver value to stakeholders; 3. And it does this by improving the system. ” Google Test Analytics - Now in Open Source (James Whittakers ideas of replacing the test plan) “A group of us at Google set about creating a methodology that can replace a test plan -- it needed to be comprehensive, quick, actionable, and have sustained value to a project.” Behavior-Driven Development vs. Testing (The comments are more interesting than the article) ”This is that BDD scenario turned into testing: +Scenario 1: Account is in credit+ Given the account is in credit And the card is valid And the dispenser contains cash When the customer requests cash Then check that the account is debited And check that cash is dispensed And check that the card is returned And check that nothing happens that shouldn’t happen and everything else happens that should happen for all variations of this scenario and all possible states of the ATM and all possible states of the customer’s account and all possible states of the rest of the database and all possible states of the system as a whole, and anything happening in the cloud that should not matter but might matter.”
  • 50. Book Review: Perfect Software and Other Illusions About Testing “Some common questions from the preface of the book: Why do we have to bother testing when it just seems to slow us down? Why can't people just build software right, so it doesn't need testing? Do we have to test everything? Why not just test everything? What is it that makes testing so hard? Why does testing take so long? Is perfect software even possible? Why can't we just accept a few bugs?” The Liberation of Testing ”In the keynote, Julian talked about how Dr. Barry Boehm has moved on from saying that bugs are found more cheaply earlier in the software process. Instead, Boehm has been writing that it is now just as cheap to fix bugs in production. This was the perfect setup for Keith Stobie’s presentation, “Testing Services in Production”. Keith drove home the point that he is currently much more comfortable testing in production than testing in any type of “staging” environment. Aside from moving testing into a production environment, there is some testing that Keith doesn’t even call testing anymore. He calls it “monitoring.” This is where the wall between testing and production really crumbles away. He talked about moving asserts from tests into the production code. I’m sure that this is nothing especially new. What’s new is that now there are people relying on this code as an indication of system health. The focus is undoubtedly shifting away from looking for bugs and presuming to know what customer’s want to making sure real data is available about system health and that we know how to interpret it.” Complexity thinking “Here's my latest presentation (that I'm still working on). It is called Complexity Thinking. I did it for the first time this week at the Lean Kanban Central Europe conference, and I will repeat it (with an improved version) at LESS2011.” Take a BITE out of Bugs and Redundant Labor This is one way to empower the manual tester when it comes to browser testing of Crome. Tool- supported manual testing. This was one of the tools James Whittaker talked about when we said that we cannot rely on quantity when it comes to manual testing, we need expert manual testers – and they need support from tools to be effective. Standardized Risks and Charters for Exploratory Testing This presentation is about standardizing how we work with exploratory testing to make it easier for junior testers to work with it, and to make the results more easily accessible to management and project leaders. Adaptive Software Testing
  • 51. This presentation is about how to use automated tests together with scripted tests and exploratory testing in one methodology. Using automated tests to explore the risk space at low cost, and then use this information as input to your manual and exploratory testing. Change the environment, not the people ”People's behaviors depend on the environment. If you manipulate the environment, you manipulate behaviors.” Work the network ”Behaviors spread through a social network like viruses. You must work the network to change a whole organization.” Fixing bugs should be the highest priority ”I wanted to share a short but interesting video essay from Scrum co-founder Jeff McKenna on the subject of software bugs – mainly, how we think of them and how we act on them. Here a few items I found particularly intriguing, see if you agree: There’s virtually no difference between high or low priority bugs Fixing bugs is most often based on business factors (i.e. a small bug impacting a big customer) Bug triage is waste of time in terms of learning how to write less bugs Fixing bugs should be the highest priority Unit tests help, but they are not a cure-all Better to find bugs ASAP, to improve the learning” Testing as a whole team activity ”Testing is an activity. Testers happen to be really good at it. We need testers on Agile teams. But if we want real agility, we need to see that completing testing as part of the sprint is the responsibility of the whole team, not just the testers.” And some interesting exercises that could be valuable to try out for any team “I invented a new exercise which allows people to reflect on their motivation, and how this is being affected by organizational change.” “The exercise allows players to visualize and discuss organizational structures.” “The exercise allows players to discuss how to delegate decisions to a team.” The Dangers of Delegating Discovery – Very interesting article on the dangers of delegating where there is opportunity for unexpected innovation
  • 52. ”Delegation is a necessary survival skill for senior executives. But when executives delegate their discovery-related innovation tasks, the odds of them finding the surprising insights that often spur transformative-growth businesses decrease dramatically. ” You Can't Analyze Your Way to Growth – Interesting article on that change often cannot be predicted by using historic data ”But in the majority of businesses, if the available data are crunched, it shows a slowly growing industry — one growing with GDP or population. That generally convinces the company in question that there aren't really opportunities for top-line growth, and that in turn becomes a self-fulfilling prophecy.” Four Ways to Lead a Successful Transformation ” 1. Make the transformation meaningful 2. Be the change you want to see the mindsets and behavior you want to see 3. Build a strong and committed top team 4. Relentlessly pursue impact Managing an organizational transformation, executives tell us, is like trying to change the wheels on a bike while you're riding it. You have to take your organization apart and put it back together in a new way, but you have to keep the business running at the same time. It's a lot to ask, and the senior leader bears much of the burden. ” At least three reasons for testers to learn to program program/ ”Yet there are at least three very good reasons why it might be a good idea for a tester to learn to program. Tooling Insight Humility” Testing Speeds Development “But the reason I write this post is something Joakim Holm said at his talk, that should be told more often: if you do a lot of testing, development goes faster” The Little Black Book on Test Design “It contains collections of knowledge, and generalizations of my ten years of testing the same product suite. I think it can be useful for ambitious testers that want to find any problems that might be important.” Open Lecture by James Bach on Software Testing
  • 53. Google: The 10 Minute Test Plan ”Every time I look at any of the dozens of test plans my teams have written, I see dead test plans. Plans written, reviewed, referred to a few times and then cast aside as the project moves in directions not documented in the plan. This begs the question: if a plan isn’t worth bothering to update, is it worth creating in the first place? To combat this, I came up with a simple task for my teams: write a test plan in 10 minutes. The idea is simple, if test plans have any value at all then let’s get to that value as quickly as possible.” Microsoft: Design for GUI automation ”First off, let me state my main points for disliking GUI automation: It’s (typically) fragile – tests tend to break / stop working / work unsuccessfully often It rarely lasts through multiple versions of a project (another aspect of fragility) It’s freakin’ hard to automate UI (and keep track of state, verify, etc.) Available tools are weak to moderate (this is arguable, depending on what you want to do with the tools).” Test Design for Automation ”A common variation of this is the automation of manual tests. It pains me to hear that some testers design scripted test cases, and then automate those actions. This tells me two things: the manual test cases suck, and the automation (probably) sucks. A good human brain-engaged test case never makes a good automated test case, and automating a scripted scenario is rarely a good automated test (although occasionally, it’s a start). Some teams even separate the test “writers” from the “automators” – which, to me, is a perfect recipe for crap automation.” More Thoughts on Leadership ”Leadership is much more than management. A successful leader manages projects by articulating a clear vision, guiding people towards achieving goals, and motivates people by helping them grow. When folks ask me what my first priority is as a Test Lead I say it is the people on my team. But, what does that mean?” Titles for Testers ”I’ve had a few conversations on twitter over the past few weeks about tester titles. My conclusion is that test titles are meaningless, but let’s look at why I think that, and why it doesn’t matter.” Changing People’s Barriers ”The crucial parts of a social system are the people in it. And since all people are different, there’s no one-approach-fits-all solution for change. If you need an organization to improve, you have to work with people’s individual needs, and the various barriers people put up in their minds…” Tomorrow through the Past
  • 54. ”This month, on the ASQ blog Paul Borawski notes that perhaps thirty percent, or less, of the attendees at a typical American Society for Quality (ASQ) have heard of W. Edwards Deming, the champion of quality. He asks, in essence "Are we forgetting our history?" I see similar issues in my own tiny sub-field of software testing, mostly in the area of test automation. I have seen organizations implement test automation, expecting test cost to go down ... only to see it actually increase.” Dealing with locale/language specific static test data data/ ”Many regular readers know that I am a strong proponent of pseudo-random test data generation in conjunction with automated testing to increase the variability of test data used in each test iteration and generally improve test coverage. But I also understand the value of static test data in providing a solid baseline, and in some cases enabling access to specific test data in different locales or languages.” In the Middle ”Should you automate everything, or nothing? Should you test everything, or nothing? How about leadership – should you dictate every detail of what your team should do, or give them no guidance at all. The answer for all of these questions – as you’d expect, is “somewhere in the middle”.” Managing Multiple Bosses ”In the movie Office Space — a comedy about work life in a typical 1990s software company — the protagonist, Peter Gibbons, has eight different bosses. All of them, seemingly unaware of each other, pass by his desk and tell him what to do. While the film is most certainly a satire, for some, it is not far from the truth. More and more people report to more than one boss and learning to handle multiple managers is an essential skill in today's complex organizations. ” Why You Should Automate Parts of Your Job to Save It ”What is the most important thing you do on your job? What portion of that could be turned into an app that anyone in your organization could effectively use? What portion of that could be automated and fed directly into the larger system with only minimal review by you? What's the least valuable but essential part of your job? Why aren't you figuring out ways to automate it on your iPad or Android?” Exploratory Testing in an Agile Context Actually a very good presentation on exploratory testing, and especially chapter 2 about writing charters is very interesting. Recommended read. Technique: Paired Exploratory Survey
  • 55. ”Paired exploratory survey is a process whereby two testers confront one product at the same time for the purpose of learning a product, preparing for formal testing, and/or characterizing its quality as rapidly as possible, whereby one tester (the “driver”) is responsible for open-ended play and all direct interaction with the product while the other tester (the “navigator” or “leader”) acts as documentarian, mission-minder, and co-test-designer.” Re-thinking thinking and planning ”In the tutorial, Hooray We’re Agile Testers! What’s Next?, Janet Gregory apologized a couple of times for saying upfront thinking or planning. I know Janet wanted to let the audience know that she isn’t a fan of massive test plans or documents written way ahead…But her remarks got me wondering. Why in the agile community is it a taboo to recommend or admit to doing any upfront thinking or planning?” Who says exploratory testing is good for medical devices? The FDA! ”This is what I’ve been arguing for a couple of years, now. If you want to test a medical device very well, then you have to test it in an exploratory way. This prepares the way for what the FDA here calls the “pivotal study”, which in software terms is basically a scripted demonstration of the product." Pretotyping: A Different Type of Testing ”Testing typically revolves around making sure that we have built something right. Testing activities can be roughly described as “verifying that something works as intended, or as specified.” This is critical. However, before we take steps and invest time and effort to make sure that something built right, we should make sure that the thing we are testing, whether its a new feature or a whole new product, is the right thing to build in the first place.” Criticism against Test Tool Vendors ”Test tool vendors that bug me: Any vendor who designs tools by guessing what will impress top managers in large companies who know nothing about testing. In other words: tools to support ceremonial software testing.” Effective, Efficient, and Elegant Software Testing ”In chapter 8 *of the Advanced syllabus], I was reading about Standards and Test process improvements. There you talk about Efficiency and Effectiveness. I have one definition about these terms, but what do I have in mind about these terms in order to be prepared for the advanced exam?” The test process and important testing techniques ”As software development gets more complex and high paced, the pressure on testing becomes greater to find business critical defects in the software as fast as possible. But testing that is only introduced in the last stages of development will most likely not give any valuable information. The fact is that testing needs to be an active part of development even at the early stages. Requirements need to be written with the
  • 56. input from the testers and testers need informed of what requirements matter, in order to start writing test cases early on. By making the process more effective testers have the chance to increase the quality of both the testing and software.” Testing dojos ”A Testing Dojo is a meeting where testers come together to work on a testing challenge. The testing challenge can consist of testing a product, or generate test ideas for a particular software, or even exercise bug reporting. Mainly the testing challenges will use Exploratory Testing.” Metrics ”Testing provides information, so you should find out what is important to different people, and give them what they need, not necessarily what they want. Metrics will hide importance, and also skew the development efforts. Try to take them out, with surgical precision. Then you are left with one final catch: Qualitative information is hard to aggregate, trust is needed.” How Pradeep teaches software testing How we tested Google instant pages ”How did this testing get organized? As with many things in testing at Google, it came down to people chatting and realizing their work can be helpful for other engineers. This was bottom up, not top down.” // “Anytime test automation scales, is repeatable, quantified, and developers can validate the results without us is a good thing!” Interview with Michael Cooper of T-Mobile ”uTest: What’s the best way for them to overcome the unique challenges of mobile app testing? MC: I have found that mobile app testing requires a more technical type of testing, and automation is key. We are also looking into crowdsource testing to augment our current testing handset application testing efforts. uTest: What are the biggest pitfalls they should try to avoid? MC: Don’t forget about performance and security testing when working with a partner, or a vendor of mobile testing. uTest: You’ve managed some pretty large testing teams throughout your career. What advice do you have for QA managers whose testing teams are growing beyond 3-5 person teams? MC: I recommend standardizing your testing deliverables and keeping track of metrics so you can manage by facts. Be independent – by that, I mean do everything in your power to not have to report to development. Speak your mind…without whining and complaining. Learn from your mistakes! If you or your team make a mistake or miss a defect, take responsibility and do whatever it
  • 57. takes to ensure that it doesn’t happen again. And, finally, treat your team and your peers with the utmost respect and dignity.” Software Design Engineer in Test vs. Software Test Engineer (Microsoft) ”Testers today face many challenges, and hiring great testers (regardless of the job title) is about finding people who not only have a passion and drive to help improve our customer’s experience and satisfaction, but can also solve tough technical challenges to advance the craft and help improve the company’s business.” Software G Forces: The Effects of Acceleration (Suggested to me as very interesting by Magnus Ernstsson, former DA at PLD SW) ”As deployment cycles shrink, what constitutes effective software engineering changes radically. Practices that bring improvement to a quarterly release cycle can be fatal with an hourly release cycle. This talk outlines the changes required of software engineering and organization at different cycle times: quarterly, monthly, weekly, daily, and hourly.” Designing Mobile Apps Using Old School Tools for New School Technology Designing-Mobile-Apps-Using-Old-School-Tools-for-New-School-Technology.aspx ”If we can figure out what we are building and why early on, and support programming by exploring the design quickly, that can save a lot of painful re-work later on. Some old school tools in my Business Analyst design toolbox have turned out to be surprisingly appropriate for mobile application design. I’ll focus on two of them: context-free questions, and state transition networks.” Seven Guidelines For Designing High-Performance Mobile User Experiences mobile-user-experiences/ ”We’ve gone over seven guidelines that address performance relative to different aspects of mobile app design. Design choices affect performance, and so performance should be considered a key factor in the design process. Unfortunately, it tends to be looked at too late in the process, which ends up impairing the user experience significantly.” Testing Tours ”Tours help the tester gain insight into the multidimensional nature of this complex, infinite space. They help us imagine, envision, and as we gain experience on the tour, prioritize the different sampling strategies we could use when we do more thorough, more intense testing after finishing the tours. So which tour is best? The ones that give the testers more insight and that achieve a greater stretch of the testers’ imagination. For this, some tours will work better for me, others for you.” Testing the Limits with Jim Sivak (Senior QA Manager at McAfee) ”uTest: In your opinion, is there a law of diminishing returns when it comes to software testing? In other words, after a certain amount of testing, does it really make a difference?
  • 58. JS: Ah, probably the most asked question—are you done testing? There are diminishing returns although my definition might be a bit different than some others. I stop testing when the development team will no longer fix problems because the business has decided that the release must happen. In other words, if bugs that are found are just all deferred, then it is time to stop. uTest: You’ve hired your fair share of software testers over the last 30+ years. What sorts of skills and attributes do you look for when filling out your roster? JS: The best people I hire are the ones who have three key attributes—they love to learn, they constantly look to do things better (not satisfied with the status quo) and they love puzzles. To be a great QA engineer, you have to continuously learn to make yourself and the team better. The best teams continually try new approaches and processes. And testing is solving puzzles—how do you make sure it works, how do you determine what can break and when it does, doing initial looks at what actually happened. I feel that the other skills—programming, product expertise, etc. can be learned on the job.” The Purpose of a Business is NOT customer value ”The aim proposed here for any organization is for everybody to gain - stockholders, employees, suppliers, customers, community, the environment - over the long term. W. Edwards Deming – New Economics” Dance with the System ”PDCA, which stands for Plan, Do, Check, Act, is an iterative four-step management process. Any change management initiative consists of a sequence of interventions in a system (or its environment), where you check the effects the interventions have on the system. When your actions work as intended, you move on to the next step. When they don’t work, you try some other ones. It is a never-ending cycle of setting goals (Plan), implementing ideas (Do), measuring feedback (Check) and analyzing results (Act).” Do software testers need a college education Best practice for Mobile Application Development ”In order to develop feature-rich mobile applications, there are some key development points which need to be followed upon at the very early development stage. These points will surely enable an IT outsourcing software development to achieve an optimal consumer experience as well as satisfaction.” Failing Fast, 20% Time and Project Mobility ”Failing Fast. Nothing destroys morale more than a death march. Projects going nowhere should do so with the utmost haste. The ability of a company to implode pet projects quickly correlates directly to a great place to work. Engineers working on these project gain not only valuable engineering experience, they experience first-hand the company's perception of what is important (and, in the case of their project, what is not important). It's a built-in lesson on company priorities and it ensures good engineers don't get monopolized by purposeless projects. You gotta like a company willing to experiment. You have to love a company willing to laugh at itself when the experiments don't pan out.
  • 59. 20% Time. Any company worth working for has any number of projects that are worth working on. It's frustrating for many super-sharding engineers to see cool work going on down the hall or in the next building and not being part of it. A day job that takes all day is tiresome. Enter 20% time, a concept meant to send a strong message to all engineers: you always have a spare day. Use it wisely. Project Mobility. Staying fresh by changing projects is part of mobility. Continuous cycling of fresh ideas from new project members to existing projects is another part. The downside here is obviously projects with a steep learning curve but I scoff in the general direction of this idea. Whose fault is it when a wicked smart engineer can't learn the system fast enough to be useful in some (even a small) context? Only the weakest organization with the poorest documentation can use that excuse. The only good reason for keeping people on a project is because they have no desire to leave.” Levels of Testing ”When I teach fundamental testing concepts I often use the ‘levels of testing’ as a model to help students conceptualize different types of testing activities and potentially different stages of testing processes during the software development lifecycle (SDLC).” Automatic Measurement of Source Code Complexity ”The aim of this master thesis is to explore the area of software metrics and to identify software metrics related to the code complexity. In this thesis, thorough study is made to determine whether or not the automatic measurement of source code complexity is possible.” Change Management 3.0 ”No matter how big the CEO’s desk is, an organization is not a hierarchy. It is a social network. Sometimes you would like to change something in that social network. Sometimes you want people to be nicer, more mindful, or better disciplined. Sometime you just want them to fill out a form, or give you some money. When you want to change behaviors in something as complex as a social system, you have to understand the four aspects of change management.” Fully Automatic Software Testing possible? ”One of the fads that seems to be reappearing is the idea of automating away humans from the software development work at all. This fad came up with the rise of UML, and the most recent fad appears to be model-based testing. One of the interesting things I noticed is the ignorance of other past movements. It seems that universities keep on bringing up new talents from time to time that claim to save the world because universities favor a particular competition-based learning model in which everyone wants to be the next hero.” Beyond Agile Programming Testing at the Speed and Scale of Google
  • 60. ”Continuous integration systems play a crucial role in keeping software working while it is being developed. The basic steps most continuous integration systems follow are: 1. Get the latest copy of the code. 2. Run all tests. 3. Report results. 4. Repeat 1-3. This works great while the codebase is small, code flux is reasonable and tests are fast. As a codebase grows over time, the effectiveness of such a system decreases. As more code is added, each clean run takes much longer and more changes gets crammed into a single run. If something breaks, finding and backing out the bad change is a tedious and error prone task for development teams.” Basics Revisited: Test Strategy ”I see many test strategy documents in my work with clients. Almost without exception, they are big—fifty or more pages—and stuffed with soporific boilerplate copied from one project to the next. Defect management process, test suspension and resumption criteria, entry and exit criteria, meaningless generic risks ... yawn ... again. One frequently used model even parrots textbook test definitions, from unit to systems integration to load.” How do you decide what to test? ”The “Test This” game pops up frequently in testing circles. Sometimes it pops up in the “How would you test a …” variety – e.g. How would you test a stapler. There are better flavors of the Test This game. Sometimes, testers play the Test This game with real software. This form has huge potential, but never really delivers, as the software under test is typically a bug-infested piece of crap that anyone with a pulse could find bugs. Bugs are part of my problem with this form of Test This – the goal of this game is to find bugs – not to test or learn testing. That’s an important distinction for me – the goal of testing isn’t to find bugs. Finding bugs is merely a side-effect of the testing process. I fear that these exercises encourage testers to dive into bug finding before thinking about how to test. To be clear, I’m not talking about big-upfront-test-design – but a holistic approach to testing.” Triple Loop Learning ”The first thing you should do is to gather requirements. What is it that users want your product to be capable of? Do they want a long battery life? A high quality GUI? Automatic updates? A smooth grip? A sturdy case in the color pink? You go out of your way to get a decent list of features to increase your understanding of what is necessary to build the product. It also reduces the risk of welding the foobar to the plingle when users prefer to attach it to a vringle. (I just made that up.)” Tester Respect at Microsoft ”Here’s a tip – you don’t "get" respect, you earn it. In my experience, testers "earn" respect by being credible, showing good results, and making good decisions. You don’t "get" respect from your "level", or by demanding it. Seriously, stop whining and figure out how to earn respect.” Testing the Limits with James Whittaker
  • 61. ”uTest: The world of testing moves incredibly fast at times. That said, is it possible to “future-proof” the theories and methods you write about in your books? Talk about some of the challenges of trying to stay current on the latest testing trends. James: That’s easy. You stay current by inventing the latest trends not following them. Invention is less about being able to read the future and more about a willingness to experiment and be wrong. You should see our drawing room floor at Google, it’s a graveyard of failed experiments. We purposefully try to do things differently to make sure we’re not missing some great new idea or technique. We’ll try almost anything once and we push hard to fail fast. Staying open minded is the key. If you are willing to experiment and expect everything you try to fail, when something actually works, well, you know its something special.” The Illusion of Control ”Your eyes capture only a tiny percentage of what happens around you. Your memories are largely just a figment of your imagination. And your “free will” is little more than an elaborate illusion created by your brain. (The Grand Delusion - NewScientist, 14 May 2011)” The Life of a Test Engineer at Google ”Obviously, the work of a TE varies greatly depending on the project. Some TE’s spend much of their time programming, much like an SET, but with more of a focus on end-to-end user scenarios. Other TE's take existing code and designs determine failure modes and look for errors that will cause those failures. In such a role a TE might modify code but not create it from scratch. TE's must be more systematic and thorough in their test planning and completeness with a focus on the actual usage and system experience. TE's excel at dealing with ambiguity in requirements and at reasoning and communicating about fuzzy problems.” Test Automation – Beyond rudimentary script-lets “In the crud example we basically entered a static “hard-coded” string into a textbox and used a simplistic oracle to make sure the “test data” appeared in a message box. To me, this is about the same as retesting 1 + 1 = 2 over and over again to verify that a calculator program’s addition functionality is computationally correct. IMHO, a “functional test” that verifies a hard-coded string gives great confidence that that string works, but not other strings.” Test Framing ”Framing is the network of logical statements and connectives that link the three story elements and their details. To frame a test means to create an unbroken line of logic from the mission—the information objective within some context—through the product story and our activities, down to the observed outcome of the test and our interpretation of it.” Binary Disease (lots of comments on this article – discussions regarding pass/fail and how to present test results) ”We make software for computers, and use computers for planning, execution and reporting.
  • 62. Our theories reflect this much more, way much more than the fact that each piece of software is unique, made for humans, by humans. Take a look at these examples: * Pass/Fail hysteria; ranging from the need of expected results, to the necessity of oracles. * Coverage obsession; percentage-wise covered/not-covered reports without elaborations on what is important. * Metrics tumor; quantitative numbers in a world of qualitative feelings. * Sick test design techniques, all made to fit computers; algorithms and classifications disregarding what is important, common, risky, error-prone.” Software Requirements Quality ”I’m running an ISTQB Advanced Test Analyst course this week in DC. This course includes a hands- on set of exercises based on a realistic requirements specification.” How Google Tests Software – Part Six ”SETs are Software Engineers in Test. They are software engineers who happen to write testing functionality. First and foremost, SETs are developers and the role is touted as a 100% coding role in our recruiting literature and internal job promotion ladders. When SET candidates are interviewed, the “coding bar” is nearly identical to the SWE role with more emphasis that SETs know how to test the code they create. In other words, both SWEs and SETs answer coding questions. SETs are expected to nail a set of testing questions as well.” How Google Tests Software – Q&A ”SETs and SWEs are on the same pay scale and virtually the same job ladder. Both roles are essentially 100% coding roles with the former writing test code and the latter doing feature development. From a coding perspective the skill set is a dead match. From a testing perspective we expect a lot more from SETs. But the overlap on coding makes SETs a great fit for SWE positions and vice versa.” Automation isn’t bad – Bad automation is bad ”The first problem in almost any discussion about automation is the lack of context. Automated tests are different and when done well can be effective at helping us identify some types of functional issues. An automated unit test suite can be a powerful tool for developers during refactoring. Of course some bugs escape unit testing because unit tests are not intended to be comprehensive. I have long been a proponent of functional testing at the API layer and this is where my team plays today. I see first hand the value of our automated tests in the early identification of potentially critical functional issues during our continuous integration process. While this level of testing is effective in helping prevent build breaks, it is not comprehensive and does not test the behavior of the application through the user interface. And then, comes GUI automation.” More on the Coding Tester ”It’s important to note that the role of a coder-tester is NOT to automate everything (everyone knows that you should automate 100% of the tests that should be automated). Writing automation is one task of a coding tester, but certainly not the primary focus (yes, I know that in some companies,
  • 63. automation “experts” only write automated tests. In my opinion, these people are not really testers, so they don’t count).” To automate or not to automate Good testers test first – or at the very least they think of tests first. I think great testers (or at least the testers I consider great) first think about how they’re going to approach a testing problem, then figure out what’s suitable for automation, and what’s not suitable. I say that like it’s easy – it’s not. My stock automation phrase is "You should automate 100% of the tests that should be automated". nearly every tester knows that, but finding the automation line can be tricky business.” Management – The following 30 days project/ ”By day I find my time split between people management (team meetings, 1:1’s, etc.) and project management (triage meetings, status meetings, external partner sync-ups, etc.) At night I read through the API specs and brush up on my C++.” Working on Lync ”The "how" part of our work is (to me) much more interesting (and a big reason why I love working on this team). We strive to have a culture high in trust. We have few (none that I can think of, but I’m sure I’m missing something) top-down mandates. We believe that the people closest to the work are in the best place to make decisions on what the work is that they need to do. We obviously give more guidance to employees new to the team and managers are involved in coaching, but for the most part, we trust the people on the team to do what they think is best. For example, we have no requirements on test case counts, bug find rates, code coverage rates, or anything else like that. If someone decides that working on a side project for a few days will help them get their job done better, they don’t need to clear it with anyone – they just do it. Failing is ok (and expected – if you’re not failing, you’re not trying hard enough). ” Ensuring software quality (the first link is a review of the second link)  Management – the first 30 days (Microsoft) ”It has been more than 8 years since I directly managed a team. In that time some ‘operational’ things have changed, but most importantly I have changed. Personally, I think my responsibilities as a lead include helping the people on my team grow their career, providing a vision and guidance towards clear goals, protecting the team from political fallout and pressure, managing the features(s) I am responsible for, preventing project and personal ‘fires,’ and also actively participating in the testing effort. There are skills, experiences, and values that I have learned in the past that will help me in my transition, there are some adaptations to make, and there is a whole lot to learn.” Management 3.0
  • 64. ”Jurgen continued to develop his management model, which consist of six parts – and that formed most of the remainder of the course: Energize people Empower teams Align constraints Develop competence Grow structure Improve everything ” Structure in explortatory testing “When someone says “Exploratory testing is unstructured,” I immediately hear, “I have a very limited view of what ‘structure’ means.”” It’s a software crisis! “At least that’s what they were saying in the late 1960s. By chance, I stumbled across a brief Wikipedia entry today on that precise term and suddenly realized that very little has changed in 50 odd years.” Software testing in the development lifecycle (V- W- model) ”I think an important thing to remember about these lifecycle models–whether V-model, W-model, Agile model, iterative model, etc.–is encapsulated in a witticism attributed to W.E. Deming: “All models are wrong; some are useful.”” Questions to ask during debriefs ”What questions do you ask during an Exploratory Testing session debriefing?” People decide, models inform ”Reducing variety is often not the right approach when trying to handle the complexity of the world.” Summary of Swedish Exploratory Testing Workshop 2 Does pair programming obviate the need for code review? code-review/# ”If you're working together with another person to write software, surely you don't need someone else to look over the code. That's part of the process. Or... is it?” Three years of Scrum ”More than an article about Agile transition, this is about actually doing Scrum, with a focus on the daily standup meeting and how it changed over the years.”
  • 65. Software Engineer in Test career path at Google ”SETs are developers who write test code and automation as their primary task. They are in every sense of the word a developer. When we interview SETs, SWEs are on the interview loop and SWE questions are asked. They are not all of the interview, but they are part of it.” My job as a tester at Microsoft “I have somewhat of a non-traditional test role at Microsoft. I’m a tester (that’s what my business card says), but I don’t own any specific components or features. I test, but mostly where I want to discover something for myself, experiment with an idea, or when I’m coaching others. The bulk of my time, in fact, is spent coaching and mentoring other testers on the team, as well as working with the test managers on the team and our test director to make sure we’re doing the "right thing" with both strategic and tactical decisions and goals.” It takes complexity to handle complexity ”In other words, in order to survive a system must have an internal model that reflects the variety it encounters in the world outside.” Questioning test cases: Part 2 ”I often hear this kind of justification for test cases: “We need test cases so that testers can learn quickly how to test the product.” To me, that’s like saying that we need driving cases so that people can learn quickly how to drive in a new city. In a new city, a map or a general set of ideas might be helpful. Driving cases won’t, because to drive through a new city well, you have to know how to drive. In the same way, test cases are unlikely to be anything more than negligbly helpful. In order to learn quickly how to test a new program well, you have to know how to test.” Questioning Test Cases: Part 1 More of what testers find: Part 2 Inspectional testing Dissecting the V-model What is test control Testing the Untestable “Robert Scoble recently interviewed Trey Ratcliff (one of my favorite photographers) about his new photo editing app for the iPad. Trey remarked that because there is no camera on the iPad 1, they
  • 66. had to “blindly” add the feature for taking pictures using the built-in camera – that is, without testing. That’s because when they wrote the feature, the iPad 2 (which includes a camera) didn’t yet exist. Kind of scary, but understandable. No amount of testing in the world would validate that feature until the iPad 2 went on sale.” A follow up by Michael Bolton on James Bach’s article on what testers find ”How do we know that something threatens the value of the product? The fact is, we don’t know for sure. Quality is value to some person, and different people will have different perceptions of value. Since we don’t own the product, the project, or the business, we can’t make absolute declarations of whether something is a bug or whether it’s worth fixing. The programmers, the managers, and the project owner will make those determinations, and often they’re running in different directions. Some will see a problem as a bug; some won’t. Some won’t even see a problem. It seems like the only certain thing here is uncertainty. So what can we testers do?” It was a big deal – an article about unintended consequences ”Look, I've been at companies before with a "save the world" mission, I understand the pace of development, and I know how hard it can be to take a few minutes to look around, think, and say something like "Wait ... this feature has some unintended consequences."” Why information is important (Microsoft) ”In the absence of information, people will make stuff up.” ”I was thinking about some of the thoughts and rumors about testing at Microsoft – particularly some of the musings about why MS only hires testers with coding knowledge for full time positions. There are many reasons for this move, and most of them are good reasons – but the role shift hasn’t been without its problems.” What testers find ”While testing at eBay, recently, it occurred to me that we need a deeper account of what testers find. It’s not just bugs. Here’s my experimental list: Testers find bugs. Testers also find risks. Testers find issues. Testers find testability problems. Testers find artifacts, Testers find curios.” Testing the Limits with Jon Bach ”uTest: Part of your new role at eBay will be to hire and recruit a top-flight team of testers (in addition to the ones already there). What sort of traits/skills/attributes will you be looking for in particular? JB: The ability to come up with ideas – either old or new – and execute them in a way that helps us improve notions of Search. For years, I used the triangle program in test auditions. Now I use
  • 67. something more simple. I draw a long horizontal rectangle on the whiteboard with a little “Submit” button below that. I say “this is a text input field for Search, just like the one you see on the eBay site. Help me create a test plan for it.” I’m hoping that instead of an interview, it comes across more like an invitation to a real collaboration.” Bugs that automated tests aren’t good at finding ”Sure, some automated tests can help find bugs, but more importantly it is one approach I might use for defect prevention (esp. computational logic problems), earlier identification of key integration issues, potential degradation of critical areas (battery, performance, memory), efficient execution of redundant ‘checks’ (if necessary or for confidence) more effective/precise ‘oracles’ as compared to humans cost reduction in long term sustained engineering” If you think the primary goal of an automated testing effort is to find lots of bugs, or the same types of bugs as a person then you probably don’t know very much about test automation.” Advanced software testing: Bug isolation ”Reader Gianni Pucciani has a good question about a question in the Advanced Software Testing: Volume 2 book (regarding bug isolation)” How Google Tests Software – Part 5 ”Instead of distinguishing between code, integration and system testing, Google uses the language of small, medium and large tests emphasizing scope over form. Small tests cover small amounts of code and so on. Each of the three engineering roles may execute any of these types of tests and they may be performed as automated or manual tests.” Testing with Code ”However – if your team only writes regression-ish automation – or if your view of the tester- developer is someone who only writes automated regression tests, you are completely missing the point of coding skills for testers. I know (and fear) that this is the case for way-too-many people in the profession today, and I find it a bit disheartening.” Innovation at Google ”Patrick Copeland presents the first three principles of the eXtreme innovation approach based on the Pretotyping Manifesto: Innovators Beat Ideas, Pretotypes Beat Productypes, and Data Beats Opinion.” Calculating Defect Detection Effectiveness
  • 68. ”You are a test manager in charge of system testing on a project to update a cruise-control module for a new model of a car. The goal of the cruise-control software update is to make the car more fuel efficient. Assume that management has granted you the time, people, and resources required for your test effort, based on your estimate. Which of the following is an example of a motivational technique for testers that will work properly and is based on the concept of adequate rewards as discussed in the Advanced syllabus? A. Bonuses for the test team based on improving fuel efficiency by 20% or more B. Bonuses for the test team based on detecting 90% of defects prior to release C. Bonuses for individual testers based on finding the largest number of defects D. Criticism of individual testers at team meetings when someone makes a mistake” Are testers made or born? ”So, is having a team of really good testers and turning them loose without any guidance a good strategy for testing? No.” Technical Debt and Accidental Complexity code.html ”Introducing technical debt increases accidental complexity, and as a side-effect invalidates previous estimates and increases the likelihood that future estimates will differ in size drastically.” Does Quality come from Testing? ”Do QA professionals truly assure the quality of a product, or do they assist in delivering high quality products?” Simulating your way out of regression testing ”Julian spoke about alternative testing – or better put alternatives to testing – arguing that time to market is a key competitive advantage for Internet businesses and that exhaustive testing damages that. He cited examples of Facebook, Flickr and Google as companies that do not perform exhaustive regression testing before releases but deal with problems effectively when they appear.” Delegation Poker ”I came up with the idea for Delegation Poker when I tried to create a game that would teach people the following ideas (learning objectives): Delegation is not a binary thing. There are plenty of “shades of grey” between being a dictator and being an anarchist. Delegation is a step-by-step process. You hand over accountability to other people, in a controlled and gradual way. Delegation is context-dependent. You want to delegate as much as possible; but if you go too far, chaos might unfold.” Triangle Test Revisited Implementing Lean and Agile software development in Industry
  • 69.$file/Petersen_diss.pdf You need testers because … “… testers have a different mindset as compared to developers, and we have different perspectives. Some testers may write, or partner with developers for component level or API tests, but most testers generally focus on integration and system levels of testing. Using various approaches in our craft we often expose functionality issues as well as behavioral or usability issues that might adversely your customers.” Beyond regression tests “In my opinion, an automation strategy that only performs regression testing is short-sighted and incomplete.” “Regression tests, for all the value they bring, tend to train the system to pass.” Battling the Bias ”There seems to be a particular word that crops up again and again in blogs, papers and presentations about software testing and beyond: Bias” How Google Tests Software Part 4 ”One of the key ways Google achieves good results with fewer testers than many companies is that we rarely attempt to ship a large set of features at once. In fact, the exact opposite is often the goal: build the core of a product and release it the moment it is useful to as large a crowd as feasible, then get their feedback and iterate.” Tips on Android Usability Testing “… it is important for app developers and testers to focus on usability from the very beginning. An app that is is easy-to-use and intuitive, and similar to industry-accepted interfaces will tend to do well. Usability for mobile apps is not the same for desktop apps: For mobile apps especially the input method, screen size and connection speed differs.” Articles about ISTQB test certificates Certification is evil? We don’t need no education – follow up on the above articles about certification The Optimal Fallacy
  • 70. “There exists a Lean principle called “Optimize the Whole”, which is too often interpreted as an instruction to only optimize “the whole of the system” and to refrain from optimizing subsystems in an organization.” Test Automation: Checking for Bit-ness “So, rather than having separate automated tests for 32 and 64 bit operating systems a better test design approach might be to simply detect the bit-ness and auto-magically set variables or redirect the control flow path through your test for the appropriate operating system environment.” Goals, Values, and stuff that makes me mad ”The problem many testers face is that they’re too busy with their "day job" to look up and think about the future. This is as true for looking at the big picture of testing as it is for thinking about their career. I call this "The Tester Treadmill". What I do often in my mentoring sessions is try to help people look up and see a little ways into their future.” Selecting Software Testing Tools ”As software engineers, we want to build useful things, and tools can make us more effective and efficient in doing so. Before we start to use a tool, we should understand the business objectives the tool will promote. Understanding the business case will allow us to properly select a tool. With the tool selected we can then go through one or more pilot projects with the tool, followed by a wider deployment of the tool. As we deploy—and after we deploy—we should plan to measure the return on investment, based on the business case. By following this simple process, you can not only achieve success with tools—you can prove it, using solid ROI numbers.” Problems and Future solutions – Keynote from an Indian test conference Acceptance Test Driven Development Acceptance Test Driven Development brings teams together together/ Testing the Limits with Elizabeth Hendrickson Management 3.0 State Transition Testing: Thinking in models
  • 71. ”Of course, some people found 1 or 2 issues using an exploratory approach without first creating a state diagram. But, after viewing the state diagram those folks were also able to realize paths they had not considered or traversed and were able to find more issues. The single biggest epiphany that most people have after participating in this exercise is how much more they understand the feature they are modeling compared to what they thought they knew.” The Dual Nature of Context Driven Testing (read the comments for criticism of the article and James’ response) “Five schools of testing Context Driven: Because testing (and any engineering activity) is a solution to a very difficult problem, it must be tailored to the context of the project, and therefore testing is a human activity that requires a great deal of skill to do well. That’s why we must study it seriously. We must practice our craft. Context-driven testers strive to become the Jedi knights of testing. Agile (the only other school that names itself): sees testing as an adjunct to programming, preferably automated. They accept that exploratory testing is a good thing, but don't feel that there are any special testing skills that deserve or require development. Quality Control: sees testing as an admission of process failure. Someone in the Quality Control school of testing keeps reminding everyone that bugs must be prevented, rather than found. They are testers but don't want to be testers. Analytical (mainly academics): see testing as a fascinating mathematical problem that they are content to solve in well-bounded research exercises, and then hope to scale up. Factory (the biggest school): sees testing as a matter of systematically manufacturing testing artifacts. They focus on documentation, metrics, and detailed instructions intended to control tester behavior. They love automation. T-Map and TPI are great examples of the factory school in action.+” Risk-Based Testing: How fine grained should we be ”… the question of granularity of the risk analysis is also important. The granularity must be fine- grained enough to allow unambiguous assignment of likelihood and impact scores. However, if you get too fine-grained then the effort goes up to an unacceptable level. A proper balance must be struck.” 4 Ways to Make the Ship No-ship decision ”In the last decade, I’ve seen a few ways to make the call. I present a four ways I’ve seen for “when to ship” decisions, along with some of the tradeoffs involved with each technique. (And, unlike that textbook, I do it without the big words or fancy symbolic notation, thank you very much.)” Productivity and Projects ”We have a lot to draw on for planning and getting productive, but the concept of staying productive is often lost on us. How a team can sustain value creation over the long haul can be fascinating, and we overlook it at our peril.”
  • 72. This Code if CRAP ” As the CRAP acronym suggests, there are several possible patterns that make a piece of code CRAPpy, but we had to start somewhere. Here is the first version of the (in)famous formula to help detect CRAPpy Java methods. Let’s call it CRAP1, to make clear that this covers just one of the many interesting anti-patterns and that there are more to come. CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m) Where CRAP1(m) is the CRAP1 score for a method m, comp(m) is the cyclomatic complexity of m, and cov(m) is the basis path code coverage from automated tests for m.” The One That Got Away “Many testers have been involved in post-ship decisions about bugs that "got away"–bugs that escaped testing and found their way into customer's hands. Often, these post-mortem discussions end up with finger pointing and threats, but with the right focus, these discussions are a wonderful opportunity for learning and growth.” How to test with little to no requirements ”To wrap things up, software testers need to be agile enough to test under both “planned” and “free” testing environments. And under relatively free environments, testers must draw on creativity, past experience, common sense, and generally accepted practices from similar applications.” – If only our complex reality was this easy … App malware on the rise ”If installed, the Trojan gathers the IMEI and IMSI numbers of compromised devices, uploading this information to a remote server, before generating counterfeit queries against particular search results. The malware specifically generated fraudulent clicks on the Baidu ad network, according to anti-virus firm AVG, which reckons the Trojan is the work of a group also producing malware targeting Symbian smartphone.” W. Edwards Deming Some quotes: “It is not necessary to change. Survival is not mandatory.” “Quality is everyone’s responsibility.” "The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them." "There is no substitute for knowledge." “Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.”
  • 73. W. Edwards Deming The Wisdom of W. Edwards Deming And there is also a short youtube documentary about him The Five Deadly Diseases Deming’s 14 Points of Management How Google Tests Software “The SWE or Software Engineer is the traditional developer role. SWEs write functional code that ships to users.” “The SET or Software Engineer in Test is also a developer role except their focus is on testability.” “The TE or Test Engineer is the exact reverse of the SET. It is a a role that puts testing first and development second.” “… stop treating development and test as separate disciplines. Testing and development go hand in hand. Code a little and test what you built. Then code some more and test some more. Better yet, plan the tests while you code or even before. Test isn’t a separate practice, it’s part and parcel of the development process itself. Quality is not equal to test; it is achieved by putting development and testing into a blender and mixing them until one is indistinguishable from the other.” How we test at Microsoft – The most complete book review of all time Agile Testing Best Practices Android surpasses Symbian and Blackberry Judgement in testing – Part II Why do some testers find the critical problems
  • 74. Judgement in Testing Mobile Challenges for Project Management, Part 1 Mobile Challenges for Project Management, Part 2 How Google Tests Software More on Test Design Globlization Testing by BJ Rollison testing/ A blog post from Michael Bolton discussing that it is a testers job to not only report bugs, but also provide information regarding risk Fuzzing Through the Side Door by Jonthan Kohl Authority & Delegation Technical Debt Don’t start a project with scarcity Career Paths for Testers Using Personas When a problem isn’t really fixed Agile Management - Complexity Thinking
  • 75. New Year’s Resolutions by James Whittaker (Google) Globalization Testing by BJ Rollison (Microsoft) Testing Clichés: Testing needs a coverage model (Interesting to think about how his criticism against test coverage models maps against the test coverage model we use) Effectiveness of Exploratory Testing$file/MSE-2010- 24%20(ET%20Thesis).pdf “Using exploratory testing approach testers found far more defects than test case based testing. Although, there was no statistical significance difference between the two approaches for false defects.” “The results drawn from experimental and survey data shows that exploratory testing proved effective and efficient in finding more critical bugs in limited time.” Investigating Exploratory Testing in Industrial Practice$file/MSE%20THESIS% 202010-01%20Final%20Version.pdf “One of the biggest misconceptions of exploratory testing from the customer’s perspective is that it does not provide complete coverage of the system. According to the testers in the study it is possible to completely cover the system testing however due to little or no reliance on structured documentation and simultaneous design and exertion of test cases it is easy to forget the test case or commit mistakes.” “According to the customer perspective the biggest benefit of ET is the ability of providing focused testing. ET facilitates the customers to perform selective but intensive testing according to the project requirements.” Google Test Automation Conference 2010: Videos of all talks Strategic Problem Solving with the 5 Whys (Toyota root cause analysis) Testing Challenges & Testing Dojos Manage yourself with measures ”How We Test Software at Microsoft” Review
  • 76. More bang for your testing buck Test Design Technique Name Competition How to deal with unknown unknowns – Black Swans No Silos – Combining roles (read the comments as well) Basic International Sufficiency Testing in Action The Agile Acid Test Book Review: Exploring Requirements Testing the Limits with Matt Evans The biggest difference between a web product and mobile device is the amount of testing and certification that must precede the launch of a mobile product. A smartphone such as the Pre is an incredibly complex and highly integrated piece of technology–much more so than a typical web application. First off, a smartphone contains a fully-functional OS, usually based on some variant of Linux running on very constrained hardware. It must perform all of its concurrent services utilizing limited memory and limited CPU horsepower. The smartphone must also respond correctly to the multitude of many current events, from those generated from the environment–like switching from wifi to a WAN internet connection–to handling data input from the user, as well as handling events from the onboard applications. Launching a mobile product requires exhaustive certification of individual hardware components such as the CPU, modems, codecs, and displays. Even then, the finished product is really launched by the carrier and must go through their exhaustive certification tests as well. Testing an onboard mobile application is also a much harder testing task. There are so many conditions and constraints that are involved in testing a mobile application. A typical mobile application is nearly functionally equivalent to any counterpart desktop client-side
  • 77. web application. Take, for example, a mobile email application. It must behave and interact with the server-side application in nearly the same way as a desktop web client. The established protocols were designed for a stable communications environment, but this is just not the case in a mobile environment. The internet connection may be lost and reconnected very rapidly. The connection may even be lost for long periods of time. The application may, at any moment, be swapped out of memory. The system may be shut down abruptly. Lots of system conditions happen in a smartphone that would rarely or never happen in the context of a desktop web client application. However, a mobile application must perform its main functional operations of retrieving and sending messages flawlessly with no loss of data and full operational integrity. Testing mobile applications under these environmental scenarios is a huge challenge. In short, testing a web application is no easy task, but mobile applications and products represent a much tougher and larger testing challenge. Classification of different tests (integration tests, unit tests, E2E tests, etc) at Google Exploratory Test Management Generating random dates (Un)common testing insights – A list that according to the author presents “truths” that he thinks are sometimes disregarded Agile transition and employee retention Story of a doomed software business Why is software quality so unmeasured (read the comments as well) Coding Dojo – Simple instructions on how a Coding Dojo works Testing Dojo – Applying the coding dojo technique on verification Combinatorial testing with BJ Rollison Leadership by Allan Page from Microsoft
  • 78. The Drake Equaition of Software Testing Synthesizing Test Ideas Testing the limits with Rex Black The Future of Test Management How outsourcing affects testing Return of Investment on Software Testing Testers: Is it time to reinvent the wheel? – So why are testers expendable? Because we’re not perceived as driving value to the company. And that’s the problem. Next question: How do we go about fixing this? Peering into the White Box: A testers approach to Code Reviews Testers doing Code Review Areas of Improvement in SW Testing Manage People, Not Roles Engineering Management (Old article from 2009 from a manager at Facebook Engineering) Hiring Technical People: How long does it take you to hire a candidate
  • 79. Combinatorial testing: Testing highly probable combinations combinations/ So, up to now we have discussed: A key to understanding how to use combinatorial or pairwise testing depends largely on our ability to effectively model the input parameters that affect a common output condition We can modify the model to deal with mutually exclusive input value combinations using a simple built in syntax for conditional or invariant constraints We can increase the likelihood of testing different values using random data generation techniques We can increase the test coverage of different combinations by using PICT to randomize the output of the baseline set of tests We can increase the potential to use certain values with greater frequency by weighting important values We can increase the thoroughness of test coverage between specific input combinations with sub-models We can force the tool to include specific input value combinations in a complete n-way output of other combinations using a seed file that specifies those combinations Discussions around the content of a test proposal Software Quality Characteristics 1.0 Turning the Tide of Bad Testing Combinatorial testing: Invalid Combinations & Output Condition then-we-are-just-guessing/ Forward thinking in Software testing – Read the comments as well, as there is some discussion around the post Combinatorial Testing: Selecting the right values – Part 2 part-2/ Who owns quality?
  • 80. How to win a utest bug battle Combinatorial Testing: Selecting the right values – Part 1 part-1/ Complexity versus Lean – An interesting 90 page presentation to look through A Mnemonic for Mobile App Testing – Article by Jonathan Kohl Putting testing to the test – Using crowsourcing for application testing Tester Personas at Microsoft Quality in the Test Automation Review Process and Design Review Template Test Automation Code Review Guidelines Rapid Test Preparation Testing the Limits with Jeff Papows Estimations Part 3 from Michael Bolton James Bach’s experience report from SWET 1 Interesting experience report from an test automation conference Thoughts on Test Strategy
  • 81. Software Test Estimations VII Two new articles by Michael Bolton on project estimation A tool for managing agile projects – interesting to look through how it works Combinatorial testing Quality Assurance vs. Test Old Test Plan Template 1 million $ worth of bugs study-finds.htm Comparing the various crowdsourced testing companies About Software Engineering – A very interesting lecture regarding software engineering Test Framing – Michael Bolton and James Bach introduce a new concept Exploratory Testing Best Practices Perception of Testing – Future of testing Trust and Testing Managers new to agile may not know what to do
  • 82. Why exploratory? Isn’t it all just testing? Can exploratory testing be automated Testing the limits with James Bach – Part 2 Encouraging programmers to be testers Exploratory Testing and Review Test Estimations – part six Pradeep blogs about what aspiring testers should think about in India Automated Test Case Design – First step Competence needed for testing – Part 4 What is a Test Strategy? Testing the limits with James Bach – Part 1 Exploratory Testing – The Learning Part Managing focus when doing exploratory testing – An article summarizing a workshop, with links to several presentations regarding exploratory testing – some quite interesting and practical Agile Backlash – Elizabeth discusses criticism against agile Definition of Done The Complete List of Testing Inspiration
  • 83. On test estimations – 2 more articles on test estimation The challenge with developers writing automated tests: QA’s perspective qa%E2%80%99s-perspective/ - Forum discussion regarding the above article Automation problems – Discussions around that automation often costs more than you think On test estimations The Role of Leadership in Software Development by Mary Poppendieck Automating Screen Captures A picture is not worth a thousand words – This article is mentioned in the above article. Talking about how a picture can not replace a good bug report. Putting the pieces together – An interesting discussion regarding testing complex systems Ingredients for testing – Part Three The Motive for Metaphor – A discussion regarding the value of using metaphors such as black box and white box testing 50 Ways to ... Improve Test Automation Will we survive the future of software
  • 84. Are we innovative enough when it comes to software testing? Are we improving how we work as software becomes more complex and harder to test? Are we discussing the same problems today that we were discussing 5 years ago? Introducing Thread-based Test Management – New test management ideas by James Bach How can I tell users what testers did? Statistician or Journalist – Blog post by Michael Bolton as answer to the above post Status of software testing professionals Code Coverage: Unreachable Code and Hard to reach code reach-code/ The code coverage number is not really useful information to anyone. It is the analysis of the code coverage results that can help us decide whether we need to design additional tests, identify areas of the code that can’t be executed without even more expensive testing such as fault injection and/or code mutation, or refactor the code to improve testability (which often increases the code coverage measure). Ingredient List for testing – Part two Testing the limits with Ben Simo Why the metric? All testing is confirmatory - Reply to the above post by Michael Bolton Code Coverage – Finding missing tests Vilken typ av kompetens en testare bör ha – del 1 Why testers need interpersonal skills
  • 85. Relationships matter – Tester & Programmer Exploratory Testing at Microsoft Code coverage with manual testing How do I create value with my testing? The Testing Planet Magazine No test is also a test Careers in test (Microsoft) Code Coverage – Did we run the right tests? (BJ Rollison) An approach to code coverage (Microsoft) How to handle regression testing (Bolton) testing/ Exploring regression testing and product line testing Search-based approaches to software fault prediction and software testing Structural System-level Testing of Embedded real time systems Test Driven Code Review
  • 86. Kaner on testing Model-based testing for non functional requirements QA activities in Agile Let’s change the tune – Agile terminology Let’s change the title too – Uppföljning på ovanstående artikel av Michael Bolton Redefining Done – Definition of Done i Agile 9/2010/redefiningDone&devLanguage=Java What should done mean How short can your iterations be Testing lessons from a glass factory Model-based testing: An evaluation Investigating Exploratory Testing in Industrial Practice Testing the limits with Cem Kaner Why test automation costs too much Random thoughts on record-and-playback
  • 87. State modeling An analysis of session-based test management Behind the scenes of the test group Quality is value to some person at some time There, but for the grace of testing, go I How many errors are left to find? Testign stories from developing MacPaint To Cert of not to Cert – That is the question In-the-lab-testing vs In-the-wild-testing Testability Combinatorial tests combinatorial-tests/ An analysis of session-based test management Testing the limits: James Whittaker Agile is not a process; Agile is a mindset Is unit testing automated?
  • 88. Challanging each other helps the craft Transition to Agile Testing Some test teams may be stumped on how to transition to agile. If you're in such a team, you probably have manual tests for regression either because you never have had the time to automate them or because you are testing from the GUI and it doesn't make sense to automate them. You probably have great exploratory testers who can find problems inside complex applications, yet they tend not to automate their testing and need a final product before they start testing. You know how to plan the testing for a release, but now everything has to be done inside a two-, three-, or four-week iteration. How do you make it work? How do you keep up with development? Women in Agile The List is not enough UI automation beneath the presentation layer using-nets-reflection/ 76&Itemid=122 Experience report: Testing vs. Checking Usability Testing Are testers going away? Technical Agile: An experience report
  • 89. We can’t find all the important bugs? Introduction to Test Driven Design Test Driven Development It is better with no model than one model Heuristics & Leadership Shipping software is a team effort Crowdsourcing community/2010/05/ A new brand of snake oil for software testing – Cem Kaner The truth about testing – Jon Bach Stuart Reid’s Bizarre Plea by James Bach Handling inexperienced testers testers/ What is a good test case Testing the limits with Lanette Creamer
  • 90. Improve Your Testing and Your Testers with Paired Testing (Test) Automatic Automation (Test) Measure progress in Scrum by acceptance tests instead of completed tasks (Scrum/Agile) Without support you will fall (Ledarskap) The nonsense of leadership (Ledarskap) Reconnaissance – A testing story (Test) Shio or No Ship Testers: Get out of the Quality Assurance Business av Michael Bolton How Introverts and Extroverts Perceive Each Other Testing the limits with Scott Barber uTest: Hypothetical: All testers are barred from practicing their craft for an entire year. What does the software industry – nay, the modern world – look like without their services? Please be as dramatic and hyperbolic as possible. SB: Sadly, I think most of the world wouldn’t even notice. I think for the period of a year, non-testers would pick up enough of the slack to keep things moving, and enough individuals would perform enough “heroics” to keep the software world running. Sure, more bugs would make it into production, more software would ship late, and there would be more evening news reports of software failures, but I’m not sure the general public would even notice.
  • 91. In fact, if software producers were smart, they’d pass some of the savings they were realizing by not having to pay testers on to the general public, making them even less likely to mind a few extra bugs. But eventually, some kind of tipping point would be reached and the public would demand less buggy software. That is when the disaster would actually happen. By the time companies re-engaged with testers, re-integrated them into their new (clearly, more streamlined, process), started getting results, realizing just how much re-work needed to be done and how long it would take to get software back up to the “old” standard, the public would be so furious that the “old” standard would no longer suffice. Of course, achieving the new, higher, quality standard would take more time, and make the software cost more, and thus make the public demand yet *more* for their money. Google's Innovation Factory (and how testing adapts) Bättre testdesign ger lägre kostnader Expect the unexpected Software Testers: A costly necessity Exploratory Testing - Is this as good as it gets? BEhavioral Regression Testing Five top causes of nasty embedded software bugs. Complex != Better Working on my first playbook Bug Reporting Lessons From Toyota: Are Your Brakes Show Stoppers? stoppers/2010/03/#more-4706
  • 92. Journey of a passionate tester Boundary Bugs Management Mistakes Demystifying Exploratory Testing How to reduce the cost of testing When Extreme Programming (XP) started in 1999, the rhetoric was that XP teams did not need “QA People” because we were going to automate all the tests. Today, even the founders of XP (Kent Beck, Ward Cunningham, Ron Jeffries) would admit there can be a role for a tester in an XP shop. Now we’re hearing the rhetoric again. How are we supposed to respond? Automated Checks Exhaustive testing Meaningful Measures Interview with Jon Bach JIRA Defect Tracking System James Bach : Toyota : Indian Testers
  • 93. Joel on Software goes Offline The Customer is always right Do I really need to automate this test? Lessons Learned from Toyota Interview with James Bach Interview with Michael Bolton Interview with James Whittaker Interview with Patrick Copeland Scary stories and GUI automation
  • 94. Lines of Code and Tests in Balance The Role of the Test Manager in an Agile Organization Burning Issues of the Day av Michael Bolton An Overview of “A Visual Approach to Risk-Based Integration Testing av Neil Pandit Logging: Exploratory Tester’s Friend Exploratory Testing is accountable Code Coverage – More than just a number proportional-to-the-criticality-of-the-information-it-provides/ ”there is no correlation between code coverage and quality, and code coverage measures don’t tell us “how well” the code was tested”
  • 95. How to create test cases from bugs Critical Mass of Software Limitations of TDD My experiences in testing with an exploratory mindset and methodology Be an explorer Structures of Exploratory Testing
  • 96. Best Bug or Bugs pract.html?cm_mmc=npv-_-MANAGEMENT_TIP-_-NOV_2009-_-MTOD1119 _____________________________________________ Moving to an Agile Testing Environment: What Went Right, What Went Wrong ”Why is testing taking so long” Weekend Testing Agile, Testing, and Quality: Looking Back, Moving Forward av Elizabeth Hendriksson Test Management in an Agile organization av Elizabeth Hendriksson Fedex Tour ”How to get started with TDD” _____________________________________________ James Whittaker on the Future of Testing
  • 97. James Bach som länkar till content/uploads/2009/10/et-dynamics22.pdf Michael Bolton Aaron Evans Scott Barber _____________________________________________ Exploratory Testing Dispute