Delivering Quality: OneWeekend at a TimeLessons Learned from Weekend TestingMichael Larsen
Who am I?Michael Larsen • Senior Tester, SideReel.com (Rovi Corp.) • Chair, Education SIG, Association for Software Testing • Facilitator/Co-Founder, Weekend Testing Americas• TESTHEAD (http://mkltesthead.com)• Twitter: @mkltesthead
How do we learn about testing?• Books• Online Forums• Web Sites• Conferences• Sometimes other testers • (if we have access to them)
What if we could get help?• From an organization that can: • Practice the craft of testing • Learn from mentors from all over the world • Tackle interesting problems • Encourage both novices and experienced testers to participate • Be available for no cost
Weekend Testing• It already exists• - http://www.weekendtesting.com/
Weekend Testers• Who are we? • A loose confederation of testers from around the world • Anchored by facilitators that lead testing sessions • A library of past session experiences and transcripts • A model for short and focused testing sessions/learning opportunities
Benefits • No default “schools” • Learn and practice in a fail-safe environment • Have an opportunity to give back to the community • Learn through collaboration • Share experiences • Learn about new tools and techniques • Share and receive peer feedback
."Software development advanceswhen the cool kids get bored”-- Michael Feathers (@mfeathers, 02/04/2012).• Same can be said for testers and testing.
History• Formed in Bangalore, India • Testers looking for a better way to learn and practice • Breaking free of scripted methods • Looking to learn from and teach each other
History• First session held August 15th, 2009 • (Indian Independence Day) • Founders liked that symbolic link. • Felt they were giving "freedom" to Indian testers.
Growth• Over 2000 members and participants since its inception.• More than 135 sessions conducted.• Chapters span the globe: • India • Europe • Australia/New Zealand • Americas • United Kingdom (Weeknight Testing)
FacilitatingKey role in all sessions is the facilitator:• Determines when sessions held.• Announcement of sessions • main Weekend Testing site • Mailing list • Twitter• Chooses the topic of the session and application to test• Announce charter and mission• Guide testing and discussions that follow.
Initiating a SessionFor Weekend Testing Americas sessions:• The Facilitator announces a session (Email, Twitter, Skype, etc.)• The Tester adds the SkypeID of the Weekend Testing facilitator (“weekendtestersamericas”) to their contact list.• Tester sends an email or Skype message to let us know they want to participate.• Facilitator adds participants to a group chat through Skype.
Directing A Session• Each session has similar steps • Introductions • Announce Mission and Charter(s) • Open testing (pairing encouraged) • Regroup and discussion • Closing• Sessions are time bound at two hours • Active testing during the 1st hour • Discussion/debrief during the 2nd hour• Facilitators provide an experience report • Discussion of session and objectives, lessons learned • Provide a full chat transcript
Challenges• Limited time and focus • Missions must be focused to fit the time.• Standalone sessions • Rare for topics to span multiple sessions • Continuity is an issue• Limits on certain topics • Automation can only be covered at a fundamental level • Advanced topics that require prior knowledge
Lessons LearnedLesson 1: Geography MattersSession: WTA-01: Let’s DanceProduct: Stepmania (PC Based Dance/Rhythm game)Issues: • Application files were easily downloaded by American testers • Testers in India not able to download the application at all during the course of the session.Discoveries: • Different geography areas have different levels of Internet capacity • If downloading an application, announce well in advance of the session
Lessons LearnedLesson 2: The Value of Session Based TestingSession: WTA-08: Quoth the Developer, Nevermore!Product: ribbit.cc (Software Developer Quote Generator)• Explore the product and plan testing missions – charters.• Each pair or team split off and worked on their own testing charters• Each group then broke down the areas further and created specific charters in those individual areas.• At the end of the session, all charters and missions combined into one document.Through this, the session attendees decided to optimize their reportingdirectly into the session notes by creating hash-tags:• Bugs used the hash-tag #bug,• Issues used the hash-tag #issue• Charters were added with the hash-tag #charter.• Focus areas within the charter were added with the hash-tag #area.
Lessons LearnedLesson 3: Let Divide and Conquer Be Your AllySession: WTA-04: Put it on the BoardApplication: Corkboard.meGoal:“Employ domain testing techniques (“Divide and Conquer The Data”) on the sticky notes incork-board application to find bugs.”As a reference, we provided the participants with the following documents:www.testingeducation.org/a/DTD&C.pdfhttp://www.kaner.com/pdfs/GoodTest.pdf• Attendees split up into small groups and each took a chunk of the testing puzzle.• Smaller pairs or teams were able to effectively “Divide and Conquer” the application.Discoveries:• By divvying up the testing into slices, the potential for effective coverage goes way up• While not complete coverage, it gets the group closer to that goal.
Lessons LearnedLesson 4: Make Sure the Testers Understand the MissionSession WTA-03: Rapid Reporter, Mapping the Testing ToolApplication: Rapid ReporterGoal:The stated mission was “Produce either a list or a mind map of the features ofRapid Reporter with the goal of guiding a future session of Weekend Testing“.• The mission could be interpreted two ways: • 1. Understand the features of Rapid Reporter so that we could test Rapid Reporter more thoroughly (i.e. directly testing Rapid Reporter). • 2. Understand the features of Rapid Reporter so that we could test another application and utilize Rapid Reporter in the process.Discoveries:• Make sure that all testers understand clearly the intention of the mission.• Ask questions to help clarify the mission.
Lessons LearnedLesson 5: The Challenges of Understanding Cultural NormsSession: EWT-39: Let’s DateApplication: Calendar Conversion FunctionalityGoal:The application must support Nepali dates in combination to the standardGregorian calendar.• Question: What makes the Nepalese calendar different?• Nepalese calendar based off of the Bikram Samwat Calendar (lunar-solar calendar), which is decided by a committee every decade.• Dates twenty or 30 years in the future are impossible to guarantee.Dicoveries:• Realization that there was no discernable repeatable pattern to the dates• Requirements of any tests may be based on erroneous information• Promise of quality might be made that could not be delivered upon.• Reminder of the value of “testing the requirements”.
Lessons LearnedLesson 6: Usability Testing and the Value of the “Persona”Session: WTA-12: What Are You Watching?Product: Sidereel.comGoal:Have a look at the aspects of “user experience”.• Different interpretations of what feels right in an application and what doesnt.Discoveries:• Trying to get into the mind of who might be the primary user of the SideReel product,• Developing “Persona” roles of different users. • A middle aged man & a teen-aged girl will have different expectations of what’s acceptable and what’s not.
Lessons LearnedLesson 7: Clear Communication of Issues and the Value of a Close ReadSession: WTANZ-11: The Critical Thinking Skill of Close ReadingApplication: Bug DatabaseGoal:Help testers recognize the value of a close reading and applying criticalthinking skills to the process.• You have two bugs that have been merged • Which of the two bugs should have been used? • Which should have been marked duplicate? • Should a third bug have been written?Discoveries:• We have conditioned ourselves to focus on particular areas.• We often neglect to do a close read of the real issue,• Reading the same issue at different times, we can get more information on each read through.
Conclusion• Weekend Testing has the power of harnessing and utilizing the minds and efforts of passionate testers all over the world.• The methods and processes used in Weekend Testing can also provide a jump-start to any organizations testing efforts.• Teams can develop skills, transfer knowledge train newer testers with greater confidence and have fun in the process.• For those who are curious about this approach, by all means consider joining a session of Weekend Testing.• Consider joining one of the chapters in your area, or even starting one of your own.