Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LaunchCoderGirl transcript, 5/20/2015 talk on Agile QA


Published on

Transcript from 5/20/2015 talk for LaunchCoderGirl

Published in: Technology
  • Be the first to comment

LaunchCoderGirl transcript, 5/20/2015 talk on Agile QA

  1. 1. [Slide: Blue Screen Of Death, Fitbit error, BBC 500 error] Let's start with a quick survey. How many of you have had a program crash on your laptop? Who's had an app crash on their phone? Now put your hand up if you had an app crash and then you tried to make it crash again. You may be QA people. We go by a lot of names. Quality Assurance, Quality Engineer, Tester. At Asynchrony, we're called Quality Advocates. Quality is everyone's job on the team, and we're the point person. We're the conscience, we're the scout, we're the bridge between dev-speak and customer, we're the most evil hacker your software will probably encounter this side of a public release, and we're the dumbest new user seeing it for the first time. Good QA people are the best ally a developer has, we make them look good and they rely on us to put their work through the wringer. We don't do it because we're mean, or to prove they did it wrong, we do it to make the app better. [Slide: QA walks into a bar] For instance, one of my teams routinely puts code in their web app to prevent the user from pasting large chunks of text in. Why? Because if you get the plain text public domain edition of Tolstoy's "War and Peace", you've got yourself three megabytes of text, which is about five hundred thousand words. The first time I dumped that into a field on a web app and hit submit, the web service call rejected it and the app got stuck. So they fixed that and I tried it again, and this time the database rejected it because it couldn’t fit that much text into the field it was aiming at. So they fixed it again. Then I put War and Peace directly into the database and had the app try and pull it out, which was another set of bugs. So what do Agile QAs bring to a project? Let me show you three different scenarios. The first project ran with no QA at all for the first 12 weeks. Then they got QA, and the five developers spent two weeks on nothing but fixing the bugs, and at the end of the project, we got this: [Slide: Case Tracker story type graph] This is a chart showing what kind of things we worked on, and nearly a third of our time was spent fixing bugs. User stories are the features, tech debt is stuff you have to fix but it doesn't provide a new feature to the user, technical stories are things you have to do that aren't related to a user story, like fixing memory leaks or database queries, design is wiring in the new mock-up from the UX team. We defined a defect/bug as anything that was in the main branch of the code. The main branch is supposed to be something you can release at any point, you never put bad code into the main branch. So this was not good. Really not good. And it’s not that this team was full of sloppy coders, these are smart people who write good code. But developers aren’t good at testing their own stuff. No-one wants to kick their own baby. That’s what we’re there for. The next project is one that's had a QA for the whole lifetime of the project, and we’re still adding new features [Slide: TMS story type graph]
  2. 2. So this is better, the bug rate is down to about 10%, so the team can spend more time working on features. Ten percent of the time, developers were fixing bugs instead of writing new features. This is a piece of software that's been released to the client several times, it's been in the field for well over a year. That's a much better graph, but we can do better than that. [Slide: Combinator story type graph] This is from a project where QA was on the team from day one, and I was writing automated user acceptance tests in the first week, using C# and NUnit inside Visual Studio. This app was installed at the client site, and in a year of use by a company of over 2000 people, it's not had a single bug reported. We're pretty proud of this one. We repeated this process on the next two projects with similar stats and it's a model that's spreading. What it requires is a QA that can write code . A good QA who can read and write code is an invaluable person on a team. Imagine you're testing a car, but the hood has been welded shut. You can drive the car, you can see the dials, you can mess with the wheels, but if you could only lift that hood, there's a whole world of testing opportunities that you've never seen before. As a QA, if you can code, you understand how the app is put together, what it depends on, and you know where to shove a screwdriver into the works to cause an error. You know where the weak spots are. You also understand it when your team starts talking about something and they haven't taken into account problems like internationalisation. Or accessibility. Or security, encryption, network load, performance, database access on a shared cluster, any one of a hundred things. So what’s the difference between a developer and a coding QA? Not a huge amount, but the biggest difference is we’re much more rare. Asynchrony has three, and those three are training another six, and five people are teaching themselves using Code School and books. That's fourteen people out of a group of twenty four. I think QA has the best job in the house. Developers are nice people who want to build things that work. QA gets to play, and burn things down, and make a mess, and we don’t even have to fix the problems, just find them and fight for the ones that will cause the users the most pain. Who saw the movie Tron? The original one and Tron Legacy both had the same line: “I fight for the Users!” That’s what QA does. We work with the users in mind. Smart ones, not so smart ones, malicious ones, ones who speak other languages, ones who are colour-blind, ones who can’t use a keyboard, ones who can’t use a mouse, partially-sighted ones, chronically busy ones, we fight for all the users. What do we do on an Agile team? We start working on day one of the first iteration, even before a line of code has been written. That’s when we start thinking about how we’re going to test the application, how the team will work together, what the definition of a “bug” is for this project. We might do some investigation and learn a new tool we’ll need to know, or pull together some requirements for what we’ll need from the developers. One thing my team lead builds for me is a data loader to put several thousand records into the database so I can test under more realistic production conditions. And it’s where we start thinking about roles. Who plays D&D? You should recognize this:
  3. 3. [Slide: D&D alignment chart] We cycle through these roles all the time. Up in the corner, the Lawful Good square represents all the “happy path” tests. These are the tests that the application does what it’s supposed to under ideal conditions. Developers spend most of their time thinking about the Lawful Good users. Chaotic Neutral is a fun role to play. There is a tool for Android developers called the Monkey. Monkey will click everything clickable in the user interface, and in the Android system, as fast as it can, over and over until something breaks or you get to the end of the test. Monkey has no rhyme or reason, it’s totally random. For Lawful Evil, you’re staying within the bounds of the application, but playing everything as a worst case scenario. They give you an address field? You paste a photo of a kitten into the zip code field. You’re testing a mobile app? You step into a Faraday Cage which cuts off all network signal, or you call the test phone from your cell phone. Chaotic Evil is where you really get to play. Anything goes here. Go into the database and put in deliberately bad data, pull the network cable out of the test machine, poke at the API with a sharp stick and see what squeaks. Change your system language, run something to make the network really slow and unreliable, intercept the network traffic and tweak it to introduce errors, run a piece of JavaScript that hits the submit button seven times in a row as fast as possible and see what gets through to the database. There are no limits but time and your imagination. A good QA has this sense of diabolical creative glee at putting an app through the wringer. When we’re done with it, it will be stronger and ready to face the users. [Slide: Little Bobby Tables] This is what's called a SQL Injection attack. If you don't sanitize your inputs by treating them as a block of text and stripping out stuff that would look to the database like a command to run, bad things can happen. This is from a web cartoon called xkcd, which doesn't stand for anything, but has a lot of geek humour. You can also inject JavaScript to mess with a web app, or PHP script. What makes a good QA? QA is the one job where you are paid to tell the truth. You never sugar-coat it, you never tell it like it isn’t, and your team is depending on you to be honest with them. How many jobs can you say that about? We would never do this: [Slide: Dilbert performance test failure] The most important thing that a good QA does is ask questions. Lots of questions, even when we know the answer. Especially when you know the answer will be “I have no idea”, or “I didn’t think of that” or “Oh crap!” Those are the good questions. Sometimes we play dumb to get our team to explain things to us. When you’re breaking things down to tell someone, you get it straight in your own head, and sometimes that process shakes loose something you forgot to include, or a flaw in your logic, or a
  4. 4. requirement that was never added but should have been there, like the ability to print something in a particular format. We ask a lot of questions, and we listen to everything. [Slide: Stuff QA says] If we don’t ask, no-one else will. We fight for the users, and the client, and our teams. Our QA apprentices have two jobs: Listen, and ask questions about everything you don’t understand. There are no dumb questions, because the right dumb question will make your team think about things differently. We just don’t know which question is the right one is until we’ve asked all of them. QAs are detail-oriented people. We see the painting that’s crooked on the wall, the typo in the help files, the colour that's one shade off the rest of the screen system. We are creative in finding every possible way to stress a system, bust a database, and blow out a web request. It’s a good day when we find a bug, or a glitch, or a missed acceptance criteria. Those things are like buried treasure, and the more effort it takes to track down a bug, the bigger the treasure. QAs are fearless. We’ll switch the language on our systems to Mandarin to see how the app responds to switching locale. We’ll delete the database out from under a running application. We’ll put something in the app that blows up the web service, never makes it to the database, and overloads the server error-reporting tool. That was a good day. The tools in our arsenal are many and varied. One of my favorites is War and Peace. Another is a tool which can throw thousands of HTTP requests at a server to simulate a crowd of users hitting the app all at once. There’s a tool that lets you spy on the network traffic going in and out of your machine. There are tools that can make your network connection slow and unreliable. Firebug is a tool for the Firefox browser that lets you inject JavaScript into a page, check on variables, and see the inner workings of a site. And then there’s these: [Slide: Foreign text] From top to bottom, that’s Arabic at the top, which is a non-Roman alphabet that you read from right to left, not left to right. There aren’t any characters in common with English. Next up is Cyrillic, which is used in Russia. Some letters are the same, some are very different. And finally we have Greek, a third set of unique characters. If you’re testing an application that will be used in a non-English speaking country, you need to test in those languages. Right-to-left text looks odd when it’s on a left-to-right layout. The German work for “Cancel” is “Stornieren”, and that’s not going to fit on a button designed for the word Cancel. A good QA is always learning. If you always test in a certain way, you’ll find a certain set of bugs. And when those bugs are fixed, you won’t find others until you change up your process. One way to do that is to call in another QA and hand them our app. No explanations, just hand it over and watch what they do with it. Every QA is different. We all see different things. For example, can anyone see what’s wrong with this app? [Slide: LDI app]
  5. 5. We found a glitch in the layout that only affected Motorola devices running Android 4.1.2. Didn’t see this on the Android emulator, didn’t see it on the HTC phones, or the Samsung ones. [Slide: LDI app with red circles] Right under the C of account, a piece of the header is shifted down by one pixel. Not all the way across, it bumps back up further along. “Please move it three pixels to the left” is a valid thing for QA to ask for, if those three pixels make the app look more polished, more professional. Asynchrony QA apprentices I mentioned QA apprentices earlier. In April of 2014 we hired someone with no software background at all. He worked side by side with me every day for three months and worked hard. He was the sole QA on that team for about 9 months before it merged with another team. Right now, I’d rate him as one of our better QAs and he’s still learning. He’s also teaching two other QAs that have joined that team. When we hire an apprentice, they are assigned to a senior QA as their mentor. They work side by side, the senior explains their process, and the apprentice listens and asks questions. When we hire an apprentice we’re looking for people with a strong desire to learn, someone with curiosity about how things work and how to break them, someone that can think on their feet, someone that’s not afraid to ask for help and ask for people to explain things until they get it. We look for people that want to be part of a team. At Asynchrony, the team succeeds together, and fails together. We discovered that having an apprentice around was good for the team, and good for the person mentoring them. When you have to explain why you do everything you normally do by reflex, it makes you pause, review how you work, and spot the gaps in your process. The job of an apprentice is to listen and ask questions. The job of the mentor is to answer all the questions, and teach their apprentice how to be a good QA. Asynchrony is currently training our fourth QA apprentice. She’s on an iOS team with one of our coding QAs learning how to test an iPad app that lives in an ecosystem of connected applications. Wrap up I hope I've given you some idea of the life of a QA in a software company. As a QA, you develop a superpower you can't turn off. There are typos everywhere. Software may just break around you. And this stuff is funny to us. We giggle when we see the mall display screen that's showing an error, because someone didn't test it fully. We critique every app on our phones and try to break things just for fun. We see broken software, it's everywhere. And somewhere behind the software you use every day there was a QA. [Slide: I see broken software]