What You Need to Know about Beta Management


Published on

This is the first chapter of our upcoming book on beta test management. What You Need To Know | Beta Management offers an introductory overview of beta test management as a whole, including topics like recruiting and feedback management.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

What You Need to Know about Beta Management

  1. 1. What You Need to KnowBeta Management Brought to you by the experts at Centercode v1.1
  2. 2. WHAT YOU NEED TO KNOW • BETA MANAGEMENTIntroductionAbout the SeriesCentercode’s “What You Need to Know” series is designed to walk readers through the basicsof different kinds of beta tests. This chapter will provide a foundation on beta testing, whilefuture chapters will focus on specific types of beta tests such as hardware or mobile tests.AudienceThis series is intended for quality, marketing, project, and product managers who’ve beentasked with running their first beta test. If you’re an experienced beta manager, it mayprovide useful tips to help further improve your beta process.GoalsThis chapter will explore the core issues and terms found in beta test management, providingyou with a basic understanding of how to manage a beta test. It will also provide theframework for future chapters in the series. Download a simpler, print-friendly version of this chapter. WWW.CENTERCODE.COM 2
  3. 3. WHAT YOU NEED TO KNOW • BETA MANAGEMENTAbout Us• We’ve been focused on delivering great Beta Test Management Solutions since 2001.• Our offerings include Centercode Connect™, a hosted beta test management platform that’s been used to run thousands of beta tests globally, fully outsourced Managed Beta Tests, and Onlinebeta.com, a community of more than 60,000 highly profiled test candidates.• Feel free to contact us with feedback or questions at info@centercode.com.• Our customers run the gamut of products, markets, and company size, including: WWW.CENTERCODE.COM 3
  4. 4. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTable of ContentsIntroduction 2 Planning 21 Managing 39About Centercode 3 The Beta Plan 22 Beta Activities 40 Writing a Beta Plan 23 Bug Reports 41Definition of a Beta Test 6 FAQ: How long to beta? 25 Feature Requests and Forums 42Other Terms for Beta 7 Tasks and Journals 43Beta Details 8 Recruiting 26 Surveys and Testimonials 44Types of Beta Testing 9 Tester Recruitment 101 27 Wikis, Calls, and Visits 45Benefits of Beta Testing 10 Why Do People Beta Test? 28 Tips for Tester Management 46The Beta Message 11 Friends, Family, and Employees 29 FAQ: Tester Time Dedicated? 47Major Beta Challenges 12 Advertising For Testers 30Top 10 Beta Mistakes 13 Selecting Great Testers 31 Closing 48 FAQ: How Many Testers? 32 Keys to Closing a Beta Test 49The Tools of Beta Testing 15 Incentives! 50Productivity Tools 16 Kickoff 33 Developing a Closure Report 51Non-Beta Software & Services 17 Getting Ready for Beta 34In-House Beta Systems 18 Product Readiness Checklist 35 Improve 52Commercial Beta Management Tools 19 Tester Readiness Checklist 36 Elements for Beta Success 53Other Useful Tools for Beta 20 Team Readiness Checklist 37 Next Steps 54 Legal Agreements 38 Contact Us 55 WWW.CENTERCODE.COM 4
  5. 5. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 5
  6. 6. WHAT YOU NEED TO KNOW • BETA MANAGEMENTDefinition of a Beta Test“A test of a product performed by real users in real environments”(basically your customers testing instead of your QA team) WWW.CENTERCODE.COM 6
  7. 7. WHAT YOU NEED TO KNOW • BETA MANAGEMENTAlso Known As...Beta testing goes by many names. Depending on the company,sometimes they have slightly different applications, but they all focus oncustomer feedback prior to product release. Here are a few: • User Acceptance Testing (UAT) • Customer Acceptance Testing (CAT) • Customer Validation (CV) • Field Trials (common term in Europe) • Pre-ReleaseNote: While we use the term “beta” throughout this series, the sameprinciples, processes, and benefits apply, no matter what you call it. WWW.CENTERCODE.COM 7
  8. 8. WHAT YOU NEED TO KNOW • BETA MANAGEMENTA Beta Test... (the details) • Is the final test prior to shipping a product • Generally begins after an internal alpha test • Begins when the product is at or near feature complete • Includes existing and/or targeted prospective customers • Commonly prioritizes quality improvement (i.e. squashing bugs) • Establishes a final assessment of the complete product prior to release • Extends QA testing (but should never replace it)And Most Importantly... • Differs greatly in challenges, execution, and benefits based on tools, product type, target market, beta goals, and development phase WWW.CENTERCODE.COM 8
  9. 9. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTypes of Beta TestingThere are many different types of beta tests, each with its own set of challenges, goals, andbenefits. Any individual product test is likely comprised of a number of these, based on the typeof product, its target market, the goal the test is attempting to achieve, and the phase at whichit’s happening. For example, the following illustrates a traditional performance-based hardwarebeta test aimed at the business market. 1. Product Type 2. Target Market 3. Primary Goal 4. Test Phase FF Software FF Consumer FF Requirements FF Pre-Alpha FF Hardware FF Children FF Performance FF Alpha FF Video Game FF Business FF Focused FF Pre-Release FF Mobile FF Enterprise FF Stress FF Agile FF Web FF Government FF Awareness FF Post-ReleaseEach of these defining attributes (hardware, enterprise, etc.) will be covered in individualchapters of this series, focusing on their unique challenges, goals, and benefits. WWW.CENTERCODE.COM 9
  10. 10. WHAT YOU NEED TO KNOW • BETA MANAGEMENTBenefits of Beta Testing • Beta testing provides true customer validation and improves product quality beyond the inherent constraints of the quality testing phase, resulting in: 1. Reduced Development Costs 2. Increased Sales 3. Reduced Support Costs 4. Reduced Returns 5. Increased Customer Satisfaction 6. Improved Product Planning • Other benefits are unique to product type, market, beta goals, and phase (each will be covered in further chapters in this series) • Incredible return based on minimal investment (see our Beta Test ROI Kit for details) WWW.CENTERCODE.COM 10
  11. 11. WHAT YOU NEED TO KNOW • BETA MANAGEMENTThe Beta MessageEveryone will see the value of beta differently. To gain buy-in, it helps to communicate howbeta will help achieve each group’s unique goals. • Product Management • Beta testing uniquely covers every aspect of a product, including support and documentation. • Marketing • Beta testing is an opportunity to gain initial impressions from your target market. This feedback can be used to improve the launch of the product. • Engineering • Beta testing ensures the product is performing exactly as expected in the hands of real customers. • Executive Level • Beta testing provides a high level view of the entire product. This information can be used to make crucial decisions about the launch of the product. • Quality • Beta directly extends quality efforts to reach real-world environments that are difficult or impossible to simulate in a test lab. • Sales • Beta testing can generate testimonials and other assests useful in early sales efforts. In addition, beta quality enhancements can lead to higher reviews resulting in more sales. WWW.CENTERCODE.COM 11
  12. 12. WHAT YOU NEED TO KNOW • BETA MANAGEMENTMajor Beta Challenges 1. Dedicating Qualified Resources • Beta testing generally happens at the least opportune time, right when the product team is frantically trying to close up every loose end, and has little available time or energy to focus on beta. 2. Recruiting Appropriate Beta Testers • Not every interested applicant is the right beta tester for your product. Finding an adequate pool of targeted but unbiased testers is a difficult task. 3. Maintaining User Participation • The single most common challenge in beta is achieving high participation levels from beta testers. The industry average is lower than 30%, meaning that 70% or more will fail to provide any meaningful feedback. 4. Collecting Relevant Feedback • Not all feedback is valuable. Relevant feedback is any feedback that either results in a direct improvement in the quality of your product, or meets another specific goal set for your beta test. 5. Organizing and Distributing Feedback • Successful beta tests generate an enormous amount of feedback. Distributing the right feedback to the right people is a complicated and time consuming task. WWW.CENTERCODE.COM 12
  13. 13. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTop 10 Beta Mistakes 1 Lack of universal beta program support 2 No well defined plan 3 Underestimating ramp-up time requirements 4 Releasing a non-viable product to beta test 5 Too few or too many beta testers 6 Test period is too short or too long 7 Poorly motivated and/or managed testers 8 Ineffective tools resulting in lost time and feedback 9 Failing to manage feedback effectively 10 Badly managed incentives WWW.CENTERCODE.COM 13
  14. 14. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 14
  15. 15. WHAT YOU NEED TO KNOW • BETA MANAGEMENTThe Tools of Beta TestingBeta testing is a highly unique process that includes individuals ranging a wide variety of disciplines,including your customers. Effective tools can contribute greatly to the success of a beta test. 1. Shorten beta cycles • A pre-profiled pool of candidates can reduce your recruiting and ramp- up period substantially, allowing you to achieve more, faster. 2. Reduce time investment • Great tools can cut the time you spend coordinating, communicating, and reviewing feedback of your beta test by more than half. 3. Increase tester participation • Providing a clear path for feedback reduces the friction of providing feedback, resulting in higher participation rates. 4. Impove feedback quality • Dedicated tools can help produce structured feedback that greatly improves its clarity, providing much more actionable data. 5. Increase visibility • Great beta tools will integrate with your existing systems, ensuring that your team is aware of feedback and able to react accordingly — fixing bugs and improving plans. WWW.CENTERCODE.COM 15
  16. 16. WHAT YOU NEED TO KNOW • BETA MANAGEMENTProductivity ToolsExamples • Email, spreadsheets, word processorsPros • Extremely inexpensive (you probably already have them) • No learning curveCons • Can’t scale (more than ~10 participants is extremely difficult to manage) • No means for communication among beta testers • Not sufficient for a team environment (little to no visibility or accountability) WWW.CENTERCODE.COM 16
  17. 17. WHAT YOU NEED TO KNOW • BETA MANAGEMENTNon-Beta Software & ServicesExamples • BugZilla, Jira, Survey Monkey, Get Satisfaction, phpBB, WordPress, etc.Pros • Inexpensive • Easily accessible • Many choicesCons • Utilizing multiple systems is disjointed for testers (reducing participation) • Difficult to manage / Produces fragmented data • Participation is difficult to track WWW.CENTERCODE.COM 17
  18. 18. WHAT YOU NEED TO KNOW • BETA MANAGEMENTIn-House Beta SystemsExamples • A tool developed within an organization for managing beta testsPros • Built for beta, customized for specific processes and needs • Dedicated experience for testers reduces friction and increases feedback • May include other custom system integrationsCons • Immensely expensive to develop and maintain • The platform doesn’t evolve without ongoing investment • Risks going end-of-life if internal support wanes WWW.CENTERCODE.COM 18
  19. 19. WHAT YOU NEED TO KNOW • BETA MANAGEMENTCommercial Beta Management ToolsExamples • Centercode Connect, VOConline FMS, BetaEasyPros • Built for beta (community, agreements, CMS, surveys, feedback, forums, reporting) • Low learning curve for customers/testers • Flexibility to meet evolving requirements, integration with existing toolsCons • Niche offering with few vendor options • Likely more costly than single-function tools • New application to adopt WWW.CENTERCODE.COM 19
  20. 20. WHAT YOU NEED TO KNOW • BETA MANAGEMENTOther Useful Tools for BetaProject Planning Tools (ex. MS Project) – A beta is a project like anyother, with milestones, activities, and resources.Spreadsheets (ex. Excel, Google Docs) – Spreadsheets containeverything needed to extract actionable data from any beta test.The Social Web (Portals, Social Networks, Forums) – Mined correctly, theInternet offers an endless collection of great beta candidates.The Centercode Library – We’ve produced a number of educationalresources and useful kits to help run successful beta tests, all availablefor free at www.centercode.com/library. WWW.CENTERCODE.COM 20
  21. 21. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 21
  22. 22. WHAT YOU NEED TO KNOW • BETA MANAGEMENTThe Beta Plan • Every good project begins with a plan. In the case of Levantra Confidential Aurora 2.1 Beta Project beta, the plan ensures that the project parameters, goals, milestones, activities, and participant demographics are EXAMPLE HARDWARE BETA TEST PLAN TEMPLATE well-defined and communicated to everyone involved. • We’ve developed two kits which include everything Mobile Phone needed to develop a comprehensive beta plan. Each kit Beta Test Plan includes a complete walkthrough, a detailed sample, Levantra, Inc. and a template to create your own plan. Aurora 2.1 Beta Test Prepared by Austin StevensDownload the free kits: ALL DATA IS FICTITIOUS • Hardware Beta Test Planning Kit BETA TEST PLAN EXAMPLE 1 • Software Beta Test Planning Kit WWW.CENTERCODE.COM 22
  23. 23. WHAT YOU NEED TO KNOW • BETA MANAGEMENTWriting a Beta Plan (1)While there are countless ways to structure a beta plan, at a very minimum any plan shouldinclude the following components.1. Budget • Identify any costs which will be associated with the beta, including tools, beta units, shipping logistics, and incentives.2. Product Definition • Construct an outline of the product itself, including information regarding its current state (alpha, beta, beta2, etc.), high-level modules, etc.3. Test Parameters • Outline the basic scope of the project, including total number of beta testers, project schedule, and milestones.4. Target Market Definition • Detail the target market for the product. If the market is segmented, break down the types of users into profile pools. Include the tester requirements (time, hardware, etc.) in this definition. WWW.CENTERCODE.COM 23
  24. 24. WHAT YOU NEED TO KNOW • BETA MANAGEMENTWriting a Beta Plan (2)5. Test Goals • List the unique goals of the project, such as general quality improvement, interface acceptance, testing real-world performance, testing support infrastructure, collecting customer suggestions, and recording testimonials. Also include the areas (modules) of the product that will be tested.6. Project Stakeholders • Define the responsibilities of those involved in the project. Define how feedback will be managed as it changes hands and progresses.7. Initial Testing Activities • Plan out the initial activities which will be performed throughout this beta (activities will likely change in response to the project progression). This may include items such as bug reporting, feature requests, forum discussions, survey completion, and assigned tasks.8. Incentives • Plan out the awards for participation, as well as the participation levels which they will be based on. WWW.CENTERCODE.COM 24
  25. 25. WHAT YOU NEED TO KNOW • BETA MANAGEMENT Frequently Asked QuestionQ How long should my beta last?A Beta tests generally run anywhere from 4-8 weeks of test time, often split between multiple phases with short breaks in between. With effective processes and tools, betas can produce great results in as little as 2 weeks. Beta length should be determined by the set of goals you are intending to accomplish. Depending on those goals you may also be able to reduce your test length by increasing your tester pool. WWW.CENTERCODE.COM 25
  26. 26. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 26
  27. 27. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTester Recruitment 101Your tester team is the most valuable resource in your beta test. This team can make or break yourentire project. Therefore, finding the right testers is absolutely crucial.Recruitment is generally split into two phases: 1. Advertise via various means to recruit a large pool of interested candidates. Candidates are generally asked to complete a qualification survey. If possible, it’s best for this pool to be 5-10x the size of your intended tester team. 2. Use whatever data you have available to select the most qualified best testers from the pool of candidates. The better job you do selecting effective testers, the fewer total participants you’ll actually need in your test. WWW.CENTERCODE.COM 27
  28. 28. WHAT YOU NEED TO KNOW • BETA MANAGEMENTWhy Do People Beta Test?Knowing why people become testers will help both recruit great participants and keep them engaged. Here aresome of the primary reasons people test:Problem Solvers • Many products (or new revisions of previous products) exist to solve a problem or fill a voidof some sort. People feeling the pain of those problems are often eager to be the first to have them solved.Early Adopters • Some people are enthusiasts who enjoy experiencing early versions of products. These testerslove technology and are excited for the opportunity to carry even a minor role in its development process.Raving Fans • Many people are passionate about a brand or product line, and are interested in having a chanceto influence where those products go. Beta provides a rare opportunity to do so.Career Minded • Business focused beta tests offer beta testers an opportunity to expand their skillset toinclude knowledge and use of products that aren’t even on the market, gaining them a competitive edge.Similarly, betas can provide an opportunity to network with peers.Free Product • The opportunity to earn free products is one the most commonly recognized reasons toparticipate (but generally not what you’ll use for recruitment). WWW.CENTERCODE.COM 28
  29. 29. WHAT YOU NEED TO KNOW • BETA MANAGEMENTFriends, Family, and EmployeesThe most readily available beta testers are often those closest to you. Unfortunately, these aregenerally not the best choices.EmployeesYour co-workers are generally a bad choice for testers as they either have too much insight intothe product itself or are impartial because of their relationship with the company.FriendsUsually friends make bad testers because they are inclined to feel that their insider relationshipnegates the requirement to participate at the same level as other testers.FamilyFamily members are also not the best choice for testers as their connection to you or yourteam may inhibit them from being objective. WWW.CENTERCODE.COM 29
  30. 30. WHAT YOU NEED TO KNOW • BETA MANAGEMENTAdvertising For Testers• The Internet opens the door to an endless supply of beta testers. It’s important to manage this source effectively to collect the right ones.• Find portals/networks that interest the target market (every market has them). Ask an administrator to help advertise for you.• Clearly indicate the technical requirements for each tester. Wading through unqualified candidates is a waste of time.• Incentives deserve mention in a tester call, but be vague. Testers are most effective when selected based on excitement rather than reward.• It is not necessary to advertise specific features of the product (especially anything confidential). Focus on the product category being tested.• Be very clear and honest about the time requirements to participate in the project. WWW.CENTERCODE.COM 30
  31. 31. WHAT YOU NEED TO KNOW • BETA MANAGEMENTSelecting Great Testers• Always keep your target market in mind. Selecting testers from the wrong market is the easiest way to produce useless results.• Look for responsive and communicative behaviors. These traits will likely remain constant in the actual beta phase.• Look for detail-oriented applications. Select candidates whose applications resemble the style that would ultimately be most valuable in feedback.• Recognize those who are excited by your product.• Avoid those who refer to or question test incentives immediately.• If possible, include a few previously successful beta testers. They’ll appreciate the recognition and positively influence other testers. WWW.CENTERCODE.COM 31
  32. 32. WHAT YOU NEED TO KNOW • BETA MANAGEMENT Frequently Asked QuestionQ How many testers should I use?A It varies. The size of your tester team should depend on your product, target market, prior experience, and the tools available. As a baseline, consumer products should generally be tested with anywhere from 50-200 people, while business products commonly include only 10-50. Many variables can change these numbers. Keep in mind that internal resources should drive tester team size as well. WWW.CENTERCODE.COM 32
  33. 33. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 33
  34. 34. WHAT YOU NEED TO KNOW • BETA MANAGEMENTGetting Ready for BetaKicking off a successful beta test requires alignment of the following three key components: • The product is in a state that is ready to be tested. The product is stable and near feature complete. General functionality should work, with the majority of testers not discovering unknown bugs in their first hour or so of testing. GETTING READY FOR • Testers are both ready and able to test your product. Testers have been selected, understand the schedule and what’s BETA TESTING expected of them, and meet any technical or knowledge requirements to successfully test. For many companies, beta test preparation is fraught with uncertainty. • The team is ready to engage and react in real-time. Even knowing when you’ll start can be complicated. You might want to base the decision on QA milestones, ensuring the product is stable enough for users. But if you’re facing a product release window that can’t be missed, your beta schedule might be set by counting backwards on Stakeholders have been identified and educated, and are ready to the calendar. The goal of this whitepaper is to establish a set of best practices for beta readiness (private beta tests, specifically). That way, you can be confident that you’re ready to launch an effective beta test regardless of any looming uncertainty. interact with testers and feedback. In our experience, the best way to get ready for beta testing is to establish concrete guidelines in three areas: product readiness, team readiness, and tester readiness. If these areas are out of sync or underprepared, your beta will most likely suffer. But when they come together, they enable higher participation, easier management, and better results. WWW.CENTERCODE.COM 1Failing to align any of these areas will greatly reduce feedback quantityand quality, ultimately reducing the value and accuracy of the beta test. Free Resource: The following checklists are summarized from our Getting Ready for Beta Testing whitepaper. WWW.CENTERCODE.COM 34
  35. 35. WHAT YOU NEED TO KNOW • BETA MANAGEMENTProduct Readiness Checklist FF The engineering team has verified that all components of the product are ready to begin beta testing. FF Auxiliary components (documentation, etc.) have been assembled into a single package which represents exactly what will be sent to testers. FF The out of the box experience has been successfully reviewed, including setup, installation, and documentation. FF Basic product functionality has been successfully reviewed (all key features are working) by product management. FF Known bugs which could not be addressed prior to beta have been clearly documented and communicated to testers. FF The uninstall process (if applicable) has been successfully verified. WWW.CENTERCODE.COM 35
  36. 36. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTester Readiness ChecklistFF A sufficient number of testers meeting each required target market segment have been selected and notified.FF Accurate contact information and addresses (if testing hardware) have been verified for all testers.FF Non-disclosure and beta participation agreements have been explained in plain English, signed, and returned by all testers.FF Responsibilities and the project schedule have been clearly communicated to the tester team.FF Testers understand how to use the systems provided for feedback. If they aren’t dead simple, training or documentation has been provided.FF All resources needed by testers to carry out their responsibilities are easily accessible and easy to understand. WWW.CENTERCODE.COM 36
  37. 37. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTeam Readiness Checklist FF Core parameters and processes (e.g. testing goals, strategies and mechanisms for collecting feedback, categories for bugs, incentives, etc.) have been defined and communicated to all stakeholders and contributors. FF Milestones and deadlines have been communicated and understood and all necessary resources are readily available. FF Stakeholders have delivered all pre-test deliverables (e.g. tools, documentation, surveys, packaging, product keys, NDAs, beta units, etc.) FF Contingency plans have been defined for any stakeholders with limited availability (e.g. planned vacation, potential birth of a child, etc.) FF Any infrastructure that will be relied upon during the beta (beta test management tools, customer support, bug tracking, content delivery, servers, etc.) is accessible and has been tested. WWW.CENTERCODE.COM 37
  38. 38. WHAT YOU NEED TO KNOW • BETA MANAGEMENTLegal AgreementsGiven the pre-release nature of beta products, most tests requirea Non-Disclosure Agreement (NDA) forbidding participantsfrom openly discussing their experiences with the product. BetaNDAs are similar to traditional NDAs, but generally include a fewattributes unique to beta. Beta Test Agreement KitIn addition to NDAs, it is common to execute a separate Presented by Centercode(or combined) agreement known as a Beta ParticipationAgreement (BPA). This agreement outlines the expectations forparticipating in the project.Free Resource: We’ve developed a comprehensive Beta Test Agreement Kit, which includesNDA and BPA templates, along with an explanation of their contents. WWW.CENTERCODE.COM 38
  39. 39. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 39
  40. 40. WHAT YOU NEED TO KNOW • BETA MANAGEMENTBeta ActivitiesThere are a wide variety of activities your beta testers would be more than willing to participate inwith the goal of improving your product. The activities most relevant to your product will vary basedon the objectives outlined in the beta plan. Some of the most common activities are: • Bug Reports • Feature Requests • Forum Discussions • Assigned Tasks • Daily Journals • Surveys and Polls • Testimonials • Wiki Collaboration • Tester Calls • Site Visits Over the next few pages we’ll cover each of these in more detail. WWW.CENTERCODE.COM 40
  41. 41. WHAT YOU NEED TO KNOW • BETA MANAGEMENTBug ReportsBug reports are collected in nearly all beta tests. This is generally considered to be the most valuable aspect ofbeta testing, as correcting defects will lead to higher quality. The following are tips for collecting beta bugs:• Bug report forms for beta testers are generally simpler than those completed by trained QA staff, but at a minimum should include a title, description and/or steps to reproduce, and severity (the tester’s view of severity may not match your own definition, but it’s important to capture the impression).• Bug reports should allow testers to attach files (for screen shots, logs, etc.). These files often describe the bug more clearly or concisely than the tester may be able to (saving a great deal of time).• If details about the tester’s environment (operating system, hardware configuration, etc.) are not available, this information should be collected. Note that collecting this information on every bug report will require repetition, ultimately reducing feedback (i.e. if possible, collect it up front and reference it).• A comment log should be used to communicate back and forth with the tester (to collect other details).• Testers should be encouraged to report bugs not only about the product, but also the documentation and any other auxiliary systems or services they are experiencing as part of the beta (support system, etc.). This feedback should be distributed to the appropriate stakeholders.• Bug reports should flow into your development team’s defect tracking system (Jira, BugZilla, etc.). Automating this process saves a great deal of time, allows engineering to use familiar tools, and reduces the opportunity for crucial feedback to be lost. WWW.CENTERCODE.COM 41
  42. 42. WHAT YOU NEED TO KNOW • BETA MANAGEMENTFeature Requests and ForumsFeature Requests • Given the near feature complete state of most beta tests, the goal of feature requests is to capture early insight into customer requirements for future revisions of the product. In this sense feature requests are most useful for product planning. • If an overwhelming majority of your testers are asking for a specific feature, it may be worth considering including that feature in the final version of the product. This is especially true in business product tests. • It’s a good idea to include a disclaimer on a feature request form which claims ownership of any ideas generated by testers as part of the beta test.Forum Discussions • Social forums in beta tests (1) Increase participation by providing testers with a sense of community, and a reason to keep using and discussing the product; (2) Reduce beta management and support requirements by allowing testers to support each other; (3) Provide a simple mechanism for hosting directed discussions (e.g. an online focus group); and (4) Help reduce confidentiality liabilities by providing testers with a controlled outlet to express their excitement, reducing their urge to do so on the public Internet. • While forums are a great tool in beta testing, they’re a very inefficient means of managing bugs and feature requests. It’s important to educate testers on the proper channels for different types of feedback. WWW.CENTERCODE.COM 42
  43. 43. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTasks and JournalsAssigned Tasks • Assigning specific tasks to groups of testers is an excellent way to both achieve goals and increase participation. Tasks can initially kickstart a project and encourage exploration. This can ensure coverage while giving testers an initial push to get started. Once the project is up and running, assigned tasks can be used for focused regression testing or to assess the impact of an issue. • If used incorrectly, tasks can backfire and stifle exploration, more closely emulating QA testing as opposed to true customer beta testing. Early tasks should be focused on things that are valuable for all users to do, while for regression, tasks should be assigned to as few testers as possible to get the job done.Daily Journals • Daily journals are simple forms that testers are expected to complete each day. Each one provides a short overview of the tester’s recent experience with the product. If possible these should include a rating scale that allows you to focus on the highly negative or positive experiences. • Daily journals are a great way to increase participation, as they give less social users an outlet to discuss their experiences with your product. They also provide a reason to return to your beta site, which will keep the tester engaged with updates, builds, etc. WWW.CENTERCODE.COM 43
  44. 44. WHAT YOU NEED TO KNOW • BETA MANAGEMENTSurveys and TestimonialsSurveys and Polls • Surveys and polls are a great way to both collect feedback and keep testers engaged throughout your beta test. These are most commonly used to capture opinions (about features, functionality, or performance), or to collect specific feedback about issues (bugs, etc.) identified throughout the project. • Generally surveys should be limited to once per week, and less than 20 questions. If essay questions require a great deal of effort this should further limit the number of questions. Polls are simpler and may run multiple times per week.Testimonials • Most beta testers are more than willing to share their excitement for your product. These forms should include instructions encouraging the tester to be open and honest, followed by a freeform text box. If you’re looking for a specific length, it’s important to make this clear, as some testers will naturally be very short, while others will be overly verbose. • Testimonials are generally collected toward the end of a beta test (possibly part of a final closure survey) and should include a disclaimer that describes how the testimonial will be used. Depending on your audience, it may also be useful to include a checkbox confirming use of the users name and/or company when referencing the testimonial. WWW.CENTERCODE.COM 44
  45. 45. WHAT YOU NEED TO KNOW • BETA MANAGEMENTWikis, Calls, and VisitsWikis • Collaborative wikis are a very contemporary beta activity where testers are allowed to either directly edit or suggest improvements for future customer facing assets, most commonly including support (documentation, FAQs, etc.) and marketing materials. These are an excellent way to give customers another means of feedback, while directly contributing to the improvement of your product.Tester Calls • Customer conference calls (including either single or multiple testers) offer direct real-time feedback, similar to a focus group. These are generally scheduled both early and late in a beta test, offering the product team the ability to converse directly with customers prior to release. Customer calls also increase participation rates by demonstrating the high value of the beta tester and their feedback.Site Visits • When possible, visiting a beta customer is a great way gain a first-hand understanding of the customer experience. Beyond the natural benefits of a face-to-face conversation, customer visits allow product teams to watch customers perform tasks in their natural environment, providing valuable insight into real-world usage. Similar to customer calls, customer visits can increase tester participation. WWW.CENTERCODE.COM 45
  46. 46. WHAT YOU NEED TO KNOW • BETA MANAGEMENTTips for Tester ManagementAchieving great beta tester participation rates is widely considered to be the most frustratingchallenge when running a beta test. The following tips will help raise your participation rates. 1. Clearly express expectations early on and throughout 2. Consistently encourage confidentiality (require NDAs, etc.) 3. Balance gratitude with expectations Reaching 90% Beta Participation 4. Keep testers involved with ongoing activities A Best Practices eBook for Getting the Most 5. Keep testers current regarding project progress Out of Your Beta Program by Centercode 6. Offer a simple method for testers to provide feedback 7. Allow testers to communicate amongst themselves www.centercode.com v1.1 8. Respond quickly to all issues and requests 9. Contact inactive testers directly (by phone if possible)Free Resource: Check out our eBook on Reaching 90% Beta Participation. WWW.CENTERCODE.COM 46
  47. 47. WHAT YOU NEED TO KNOW • BETA MANAGEMENT Frequently Asked QuestionQ How much time should testers dedicate?A This depends highly on the complexity of the product and goals of the project. Simple products may only require a few hours a week, while more complex products can require 10-20 hours a week or more. Keep in mind that a properly targeted beta tester should be fitting this product into their daily life, therefore testing shouldn’t feel like “work”. WWW.CENTERCODE.COM 47
  48. 48. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 48
  49. 49. WHAT YOU NEED TO KNOW • BETA MANAGEMENTKeys to Closing a Beta TestWe’re in the home stretch now. The following steps will help close out your test. 1. Give your testers time to submit final issues. Generally it’s best to give your testers at least three days’ notice before the end of a project. 2. Cut-off tester access to submit issues. Once your end date has been reached it’s important to cut off feedback. 3. Close all open issues. Review and close any remaining feedback. This should be done quickly in case additional information is needed before you lose touch with your testers. 4. Offer testers a simple means to return the product. If the product is hardware based, be sure to provide return instructions (testers should never bear any expense). WWW.CENTERCODE.COM 49
  50. 50. WHAT YOU NEED TO KNOW • BETA MANAGEMENTIncentives!Incentives are incredibly important to your beta test. You want to make sure the testersthat provided you with all that amazing and critical feedback are well rewarded. Doing so isnot only fair, but also sets a good example for the next time you need their help. • Include multiple levels of incentives based on performance. Participation will likely vary heavily between your beta testers. It’s important to recognize and thank those who went above and beyond. • Ensure incentives match participation requirements. If you expected a great deal of time out of your participants, the incentive should match that effort. • Reward all who meet those levels. Generally a “top 10” approach is not as successful as levels. If everyone provided incredible feedback, everyone deserves an award. • Distribute incentives quickly. The golden rule applies here: You expected your testers to participate immediately, so it’s only fair for you to do the same. If possible, have your incentives sent within a week of test closure. • If possible, thank testers individually. A personal touch goes a long way. WWW.CENTERCODE.COM 50
  51. 51. WHAT YOU NEED TO KNOW • BETA MANAGEMENTDeveloping a Closure ReportNow that your testers are happy and on their way, it’s time to go back to your stakeholders. Aclosure report provides an overview of the results of the beta test. • Develop an executive summary of the project • Document key issues found • Document frequently requested features • Document survey results (with pretty charts) • Document top testers & incentives rewarded • Ensure all departments obtain a report WWW.CENTERCODE.COM 51
  52. 52. WHAT YOU NEED TO KNOW • BETA MANAGEMENT {Beta Management Basics { Tools 1. Planning 2. Recruiting Process 3. Kicking Off 4. Managing Improve 5. Closing WWW.CENTERCODE.COM 52
  53. 53. WHAT YOU NEED TO KNOW • BETA MANAGEMENT8 Elements for Beta Success1. A Detailed Plan • A solid plan provides a clear roadmap for everyone involved.2. Universal Buy-In • A team that believes in beta will be much more willing to help.3. Great Testers • The wrong testers can produce useless results.4. Communication • Provide regular updates on timelines, responsibilities, and progress.5. Responsiveness • Make testers feel involved as key members of the team.6. Effective Tools • The right tools increase the efficiency of everyone involved.7. Organization • Effectively manage the enormous amount of data generated in beta.8. Great Incentives • Great testers deserve great rewards.Free Resource: Check out our eBook, 100 Tips for Better Beta Tests, for a comprehensivecollection of practical tips from Centercode and beta managers at top technology companies. WWW.CENTERCODE.COM 53
  54. 54. WHAT YOU NEED TO KNOW • BETA MANAGEMENTNext StepsWe hope this resource has helped you on your quest to understanding beta. If you’d liketo improve your betas further, we’d recommend taking a free trial run of our completebeta test management platform, Centercode Connect™. • Recruit great candidates and select the best beta testers who meet your own specific target market. • Collect and manage the feedback you need (including bugs, suggestions, tasks, surveys, forums, and wikis). • Generate and automatically distribute custom reports based on your own beta objectives.We offer a fully-featured free trial, available at www.centercode.com/trial WWW.CENTERCODE.COM 54
  55. 55. WHAT YOU NEED TO KNOW • BETA MANAGEMENT Banner image courtesy of Flickr user xjrlokix (Thanks!)www.centercode.com • 800-705-6540 • info@centercode.com • @centercode WWW.CENTERCODE.COM 55