Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Setting the right goals and objectives for your beta test

1,846 views

Published on

Beta testing produces enormous amounts of information on your product. But how do you know you're collecting the right feedback and data? Here our research team shows you how to set targeted goals for your next beta test and what methods you can use to reach them. Click this link to access the on-demand webinar: https://www.centercode.com/webinar/2016/september/

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/XgGPm ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The program designed by Jeevan summarise what you need to know for your exam by focusing on the key concepts. The resources are well written and take the complexity out of the topics covered. The shortcuts and tips are especially useful and I'm already reaping the benefits; I can answer exam questions far quicker than I previously did. I achieved a 'C' grade in the foundation-tier paper this summer (2015). However, I am not satisfied with just a 'C' grade so I've entered the higher-tier paper in November 2015, where I am aiming to get an 'A' grade overall. Before I came across Jeevan's program, I would have been really happy with just a 'C' grade at GCSE. I now want to aim for the best grade possible because I know I can do it. Thanks Jeevan for giving me the belief that I can do well in in this subject. I'm even considering A-Level Maths! That's quite phenomenal for someone who failed their GCSE maths before!➤➤ http://t.cn/AirrSv7D
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Setting the right goals and objectives for your beta test

  1. 1. 1 Setting the Right Goals for Your Beta
  2. 2. 2 Meet the Speaker John S. Little Director of Research @ Centercode john@centercode.com
  3. 3. Agenda 1. What are goals? 2. How to set good goals for your beta test 3. How to maintain your goals throughout the beta phase 4. How to connect the insights from your test to your goals 5. Q&A 3
  4. 4. 4 Who is Centercode? Services Fully managed tests and programs Software Complete beta program management platform Testers Access to 150k+ deeply profiled global candidates Content Free tools and best practices Customer validation is our core competency We've been collecting customer feedback for tech companies since 2001
  5. 5. Goals Introduction 5
  6. 6. Webinar Goals Establish good goals for beta tests Accomplish goals of the beta test Leverage information to make a better product 6
  7. 7. 7 Goals Refresher
  8. 8. Business Definition An observable and measurable end result having one or more objectives to be achieved within a more or less fixed timeframe. http://www.businessdictionary.com/definition/goal.html#ixzz4KBQiUDi6 8
  9. 9. Goals for betas “Goal is something that you want to achieve” How am I going to do it? - Method Why am I going to do it? - Return Useful tools ● S.M.A.R.T. ● Definition of Done ● Commander’s intent 9
  10. 10. Quote If one does not know to which port one is sailing, no wind is favorable. – Seneca 10
  11. 11. 11 Good Goals
  12. 12. Beta Goal Types 1. Overall Test Goal 2. Feature Set Level(s) 3. Attributes 12
  13. 13. Overall Test Goals - Finding a Focus What is the main goal of the beta test? ● Natural usage ● Coverage of features ● Opinion of features ● Partial, focused coverage ● Unique environments and use cases 13
  14. 14. Real vs. Complete Natural usage Coverage of features 14
  15. 15. Feature Set Goals - Finding a Focus Where do I find the areas of the product that need focus? ● Product/Market Requirements Document (PRD/MRD) ● Marketing Messaging ● Packaging Design ● Key Stakeholder Requests 15
  16. 16. Priorities for Constraints 16 Constraints ● Time ● Products ● Cost ● Tester Availability Feature Sets ● Setup ● Documentation ● Website ● App ● Settings ● Out of box ● WiFi ● Feature 1 - Feature n VS
  17. 17. Attributes 17 Overall Satisfaction Perceived Performance Ease of Use Usefulness Preference Comparative Self Reported / Attitudinal
  18. 18. Communicate Goals Document, distribute, and follow-up ● Key Stakeholders ● Immediate team members ● Extended team members (build awareness) ● Testers (if appropriate) 18
  19. 19. Goals for Feature Sets 19 Bad Good Collect feedback on feature Measure user’s satisfaction with ease of use of feature Measure usability of the product Evaluate customer acceptance of product Measure the performance of the product Measure perceived performance while initiating application How long did it take to setup the product? Evaluate satisfaction with time to set up Will they pay $150 for it? Conjoint analysis After Scenario Questionnaire (ASQ)
  20. 20. 20 Example
  21. 21. Product Example Internet of Things Light ● Mobile App ● Preset Colors ● Manual Color Selection ● Timer ● Alarm ● Auto-dimmer 21
  22. 22. Beta Goals ● Evaluate total customer experience ● Assess MVP readiness ● Roadmap validation ● Competitive analysis ● Identify bugs ● Real-world impact of known issues ● Regression testing ● Analyze real-world performance ● Test compatibility ● Study user experience ● Assess new user experience ● Test documentation ● Test support processes ● Collect mock reviews ● Increase product confidence ● Collect case studies 22
  23. 23. Objectives Quality Customer Acceptance Natural Use ● Test compatibility ● Assess MVP readiness ● Competitive analysis ● Identify bugs ● Regression testing ● Analyze real-world performance ● Evaluate total customer experience ● Assess new user experience ● Collect mock reviews ● Increase confidence ● Real-world impact of known issues ● Study user experience ● Test documentation ● Test support processes ● Collect case studies ● Roadmap validation 23
  24. 24. SMART Beta SMART Criteria Specific Customer acceptance of product Measurable Quantifiable by survey(s) & severity of issues Achievable Product state / Tester availability / Tool Relative Feedback will go into this version or next Timebound 2 weeks of test -> 2 weeks regression -> GM 24
  25. 25. Definition of Done ❏ All new feature in the product tested by customers ❏ Measurable feedback on the customer acceptance of the product ❏ All discovered bugs prioritized and sent to QA ❏ Tester suggestions provided to product management ❏ Support alerted of specific pain points that are not being addressed 25
  26. 26. Commander’s Intent Complete a beta study of the customer acceptance of the product in order to understand and make critical improvements prior to release or general improvements in the next revision 26
  27. 27. Feature Set vs. Constraints Features ● Packaging ● Easy Setup ● Preset Colors ● Manual Color Selection ● Timer and Alarm ● Auto-dimmer Research Attributes ● Ease of Use ● Stickiness ● Usefulness ● Feature Satisfaction Constraints Units = 32 Time = 2 Weeks Tools = Centercode Testers = Betabound Test Team = ½ Time (me) 27
  28. 28. Feature Set vs. Constraints Features ● Packaging ● Easy Setup ● Preset Colors ● Manual Color Selection ● Timer and Alarm ● Auto-dimmer Research Attributes ● Ease of Use ● Stickiness ● Usefulness ● Feature Satisfaction Constraints Units = 32 Time = 2 Weeks Tools = Centercode Testers = Betabound Test Team = ½ Time (me) 28
  29. 29. Establish Goals Primary Test Goal = Customer Acceptance Feature Sets = Easy Setup; Preset Colors; Manual Color Selection; Timer; Alarm Attributes/Feedback ● Satisfaction, Ease of use, etc. (Surveys) ● Bugs (Issue reports) Week 1 ● Setup; Manual Colors; Timer Week 2 ● Alarm; Preset Colors; Timer 29
  30. 30. Centercode Test Plan 30
  31. 31. One more time 1. Identify test objectives 2. Group and provide reasonable options 3. Ensured that the goal is well crafted 4. Identified feature set goals & attributes 5. Prioritized based on constraints 6. Document and send out for review Start Finish 31
  32. 32. 32 Maintaining Goals
  33. 33. Surveys Goal: Measure Attributes Pros Pointer to areas that might have problems Very simple to send to large numbers of people Cons Qualitative aspects difficult to analyze Attitudinal satisfaction with attributes of items 33
  34. 34. Bugs and Suggestions Goals: Unique Environments and Use Cases Pros Facts & opinions from testers Build both - let it sort for you Cons Difficult to quantify without tools Can require back-and-forth for clarity 34
  35. 35. Journals Goal: Natural Use Information Pros Diary type study Requires limited guidance to testers Cons Requires more time for analysis Could be limited 35
  36. 36. 36 Goal Dangers
  37. 37. Type Mismatch Article - https://www.nngroup.com/articles/which-ux-research-methods/ Study Type Results Attitudinal vs. Behavioral What they say vs. what they do Qualitative vs. Quantitative Why/how to fix vs. how many / how much 37
  38. 38. Miscommunication Goals not agreed upon Feedback not solicited New or unaware stakeholders Misguided testers Wrong methodology 38 ¯_(ツ)_/¯
  39. 39. 39 Goals to Insight
  40. 40. Connect Feedback Types Surveys and Bugs Bugs and Suggestions Surveys and Suggestions Journals and … Marketing, UX, Support, Sales, QA 40
  41. 41. Focus on Product Goals Focus results on original goals - they were goals for a reason Bring in updates to the information that helped you define the goals Only highlight other areas if they are significant 41
  42. 42. Generate Top X List Information overload happens to everyone Prioritize for your own sake Short list will get more attention than long list Bring in the “good/promotes” from the test Everyone likes to have a couple wins, and so does the customer 42
  43. 43. Problem Elevator Pitch 20 - 30 second rundown Customer’s point of view Focused on selling the item to the team One for each item on “Top X” list 43
  44. 44. 44 Key Takeaways
  45. 45. If you remember nothing else... 1. Clearly define your goals - always; and associate the best method to achieve them before signing off on the project. 2. Stay focused on goals, unless significant change 3. Beware of research myopia 4. If you could use help, contact us! 45
  46. 46. 46 Q&A John S. Little Director of Research @ Centercode john@centercode.com Sign up for our newsletter and our upcoming webinars For more resources, visit our library
  47. 47. 47 Thank You!

×