After weathering recessions with a wide range of iconic customers, CEO and Product Manager Luke Freiler has seen first hand the impact the Voice of the Customer has had in making or breaking tech companies during hard times. He's going to walk you through an agile process for continuous customer testing that saves you time and gives you full confidence in your products — no matter how many you're sending out the door this year.

1. Using Continuous Customer Testing for
Pandemic-Proof Product Success
Luke Freiler Rebecca Komathy
With: Moderated by:
Candid Conversations with Product People
Webinar Series
TO USE YOUR COMPUTER'S AUDIO:
When the webinar begins, you will be connected to audio using
your computer's microphone and speakers (VoIP). A headset is
recommended.
Webinar will begin:
11:00 am, PDT
TO USE YOUR TELEPHONE:
If you prefer to use your phone, you must select "Use Telephone"
after joining the webinar and call in using the numbers below.
United States: +1 (562) 247-8422
Access Code: 478-943-628
Audio PIN: Shown after joining the webinar
OR

2. Logi Analytics empowers the world’s software teams with the most intuitive, developer-grade
embedded analytics solutions and a team of dedicated people invested in your success. Logi
leverages your existing tech stack, so you can quickly build, manage and deploy your application.
And because Logi supports unlimited customization and white-labeling, you have total control to
make the application uniquely your own. Over 2,100 application teams have trusted the Logi
platform to help power their businesses with sophisticated analytics capabilities. Logi Analytics is
headquartered in McLean, Virginia, with offices in Ireland, England, Ukraine, and China.
Learn more about what’s possible with Logi at LogiAnalytics.com.

3. Click on the Questions panel to
interact with the presenters
https://www.productmanagementtoday.com/webinar-series/11062/candid-conversations-with-product-people

4. 4Candid Conversations
With Product People
4
About Luke Freiler
Luke Freiler is the CEO and co-founder of Centercode, a beta and delta testing solutions provider that helps hundreds of
enterprises and high-growth tech companies deliver flawless products. A self-identified Tech Idealist, his mission is to
create "less" technology by turning cutting-edge tech into products that fit seamlessly into the everyday lives of
consumers. With a background in user experience and software development, Luke enthusiastically continues to serve
as Product Manager for the Centercode Platform, a comprehensive SaaS solution that enables companies to
continuously engage with their audiences and collect more product feedback with less time invested. Prior to founding
Centercode, Luke worked for Samsung, Ericsson, and Native Instruments, where he developed, tested, and helped
launch and support a variety of B2B and B2C products.
About Rebecca Komathy
Rebecca attended California State University, Long Beach, where she earned her M.F.A. in Creative Writing with a
concentration in Fiction. After working in the publishing and English education fields, she became more interested in the
information science and business aspects. She is attending SJSU for her second Master’s, this time in Library and
Information Science so she can learn more about digital curation. She currently works at Aggregage as a Webinar
Coordinator. When she’s not working, she is probably snuggling with her French bulldog named Kira.
Using Continuous Customer Testing for
Pandemic-Proof Product Success

5. Continuous Customer Testing
for Pandemic-Proof
Product Success
September 2020

6. 1
2
3
4
5
Who is Centercode
Product Strategy in a Recession
Intro to Delta Testing
How it Works
Q&A

7. Centercode Overview
Continuous customer-driven product success via real-world user testing
Platform
SaaS-based Tester recruitment,
engagement, feedback, and insight
1 Testers
Extensible global network of
enthusiastic test candidates
2 Services
Fully managed customer tests
and continuous programs
3

8. Centercode Customers
WE HELP HIGH-GROWTH TECHNOLOGY AND MODERN ENTERPRISES MAKE GOOD TECHNOLOGY GREAT

9. 1
2
3
4
5
Who is Centercode
Pandemic Product Strategy
Intro to Delta Testing
How it Works
Q&A

10. A More Economic Approach
Budgets Tighten
Revisit portfolios and plans
Prune the roadmap
Take fewer risks - Focus on what works
Every product needs to be great

11. 1
2
3
4
5
Who is Centercode
Product Strategy in a Recession
Intro to Delta Testing
How it Works
Q&A

12. Delta Testing?
What is
The last-mile of development that gathers and
prioritizes actionable real-world target market
feedback in order to perfect your release
i.e. next generation beta testing methodology aligned with modern development practices

13. Customer Testing GapCustomer Testing Gap
Legacy Product Delivery Model
Waterfall + Beta Testing
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
New Product Development
Product 1
v0.9v0.x
Beta
New Product Development
Product 2
v0.9v0.x
Beta
This gap resulted in sparse resources,
processes, and tools

14. Legacy Customer Testing Process (e.g. beta testing)
Pray nothing fell through the cracks
Shotgun distribution of results via spreadsheets
Plead with testers for feedback via email or other disparate tools
Figure out how to distribute the beta product
Scavenge for a handful of people willing to be testers
Product Managers beg engineering for a beta build or units

15. Technology has changed…
Deeply
Connected
Alexa, Homekit, SSO, APIs, etc.
More
Sophisticated
Software + Hardware + Services
In addition, there's increased competition and decreased customer patience...
Continuously
Evolving
Products have become services

16. Modern Product Delivery Model
Agile + Delta Testing
Continuous and Increasing
Customer Testing
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
New Product Development
Product 1 v0.7 v0.8 v0.9v0.x v0.6
Delta
v1.1 v1.2 v1.3 v1.4 v1.5 v1.6 v1.7 v1.8 v1.9v1.0
Iterative Development
Delta
New Product Development
Product 2 v0.7 v0.8 v0.9v0.x v1.1v1.0
Iterative
v0.6
Delta
This change creates a need for
resources, processes, and tools

17. Delta Testing Delivers
Rapid design and scoping
Ideal testers
Persistent user engagement
Useful feedback
Actionable recommendations
Optimized resources

18. Fix
Issues
Engineering, QA,
& Support
Improve
Ideas
Product Management
& UX/CX
Delta Testing
Results
Promote
Praise
Marketing, Sales,
& Executives

19. Delta Testing Impact
DECREASE
Quality Cost
Dev Time
Internal Bias
Time to Market
Refunds
Support Cost
INCREASE
Quality
CX / UX
Reviews
Sales
Customer Sat
Planning

20. 1
2
3
4
5
Who is Centercode
Product Strategy in a Recession
Intro to Delta Testing
How it Works
Q&A

21. Repeated Each Active Test Week
Tester Recruiting or ReplenishmentPrep
Plan Phase Scoping and Feature Mapping
Directed EngagementExecute
Ongoing Feedback
Delta Testing Process Overview
Share Ranked Results with Stakeholders

22. Phase Scoping and Feature Mapping
Features represent the individual features, functions, and experiences you want tested
Software
Configuration
The process of configuring the
software for first time use.
Navigate to the "Settings" menu and
explore the options available to you. H M
User Account
Creation
The process of creating a personal
account to customize the product.
Log into www.acme.com and create
a new user account. M S
Feature
Software
Installing
Description
The initial experience of installing the
application
Activity
After downloading, doubler click the
icon and follow the instructions.
Value
VH
Size
S

23. Tester Recruiting and Replenishment
US Resident, No Experience with Veridian Products
Android iOS .
Smart Home Interest + Experience
Age 18-25 Age 26-35 Age 36-50 . Targeted
Enthusiastic
Strangers
Core Requirements
Technographic Markets
Additional Traits
Demographic Markets
Executed via a 5-10 question "Qualification Survey" - Ideally auto-qualifying
Customer Base +

24. Share Ranked Results with Stakeholders
1 2 3 4
Weekly Engagement Process
Announce
Features &
Activities
Support
& Review
Feedback
Feature
Satisfaction
Survey
Collect
Prioritized
Feedback

25. Hi Beta Team,
This week we're focusing on the new customer experience of our product.
With that in mind we'd like you to focus on these areas:
● Software Installation - install the software for use
● Software Configuration - configure the software for use
● User Account Creation - create your own personal user account
Please submit your feedback including Issues, Ideas, and Praise.
If you have any questions, don't hesitate to ask!
Thanks!
Chris Rader
Veridian Beta Team
Weekly Engagement Example

26. Feature Satisfaction Survey

27. Continuous Feedback Collection
All Feedback should be associated with scoped Features
Fix
Test quality, interoperability, and real-world performanceIssues1
Improve
Measure acceptance, prioritize backlog, new ideasIdeas2
Promote
Gather delights, strengths, real-world usage insightsPraise3 M
H
VH

28. 1. Please rate your satisfaction of installing the software
It sounds like you ran into issues that lead to a negative
experience. If you haven't already done so, please take a
moment to report any of these Issues right now.

29. 2. Please rate your satisfaction of configuring the software
It looks like you see room for improvement in this area.
If you haven't already done so, please take a moment to
share any Ideas that would lead to a higher score.

30. 3. Please rate your satisfaction of creating your user account
Awesome! Would you please take a moment to
share your Praise regarding whatever specifically
earned the 5th star in your rating. Thanks!

31. Prioritizing Results

32. Impact Scoring is a prioritization system
which considers the relative Weight and
Popularity of each piece of tester feedback,
based specifically your Features.
Feedback Impact Scoring
i.e. Severity * Frequency

33. Every Issue or Idea DiscoveredImpactScore
Why Impact Scoring?
1. Not all feedback is created equal 2. You'll never be able to address everything
Widespread
Security Issue
Cosmetic
Pet Peeve
200
Total
Feedback
40
Top
Feedback
80%
Addressed
Impact
Combined
Impact Score
1,144
1,430
20%

34. Example Use Case
Consider a software product, where a
crash is happening during the installation
process, which results in issues
reported by multiple testers.

35. Determining Feedback Weight
Type Issue 2.0
Feature Installation 1.8
Severity Critical 2.0
Status Reproduced 1.5
Weighted Average
1.8

36. Determining Feedback Popularity Score
Duplicates
Votes
Comments
Views
2.0
1.0
0.1
.02
Weight
4
9
18
60
Count
8
9
1.8
1.2
Value
*
*
*
*
=
=
=
=
Sum
20

37. Determining Feedback Impact Score
Weight
1.8
Popularity
* 20
Impact Score
= 36

38. Leveraging Impact Score
Prioritize all feedback
Identify product problem areas
Identify areas of improvement
Identify praise-worthy feature sets
Benchmark products against each other
Identify most impactful testers

39. Presenting Results

40. Fix Recommendation
Feedback
Created June 6, 2019
Severity Critical
Status Logged Internally
Liveview Doesn't Initialize When App First Opens
The liveview window displays a buffer icon indefinitely if you open the
app and stay on the home page. The liveview instantly displays if you
open the app a second time, or go to a menu page and return to the
liveview.
#01
ISSUE RANK
167
IMPACT SCORE
DESCRIPTION
TITLE
Live Video Streaming 1.8
FEATURE
ISS-0001
ISSUE ID
WEIGHT
Popularity
Occurrences 17
Comments 13
Viewers 25
Issue Details

41. Improve Recommendation
Feedback
Created June 6, 2019
Sentiment Must have
Status Logged Internally
Web Browser Based Viewing
I think it'd be a huge value add if we could review footage we've
recorded via a web browser.
I prefer to handle things on my PC, and using a web browser is a much
better and more precise experience for me when compared to using a
mobile app.
#01
IDEA RANK
116
IMPACT SCORE
DESCRIPTION
TITLE
Live Video Streaming 1.8
FEATURE
IDE-0002
IDEA ID
WEIGHT
Popularity
Occurrences 12
Comments 14
Viewers 22
Idea Details

42. Promote Recommendation
Feedback
Created June 10, 2019
Sentiment Exceeded Expectations
Status Logged Internally
Convenient Camera Controls
I’ve had some great luck with the camera controls. It’s really nice to
be able to zoom in and out during the live stream to get a better idea
of what’s going on when I’m not home.
#01
PRAISE RANK
90
IMPACT SCORE
DESCRIPTION
TITLE
Live Video Streaming 1.8
FEATURE
PRA-0030
PRAISE ID
WEIGHT
Popularity
Occurrences 14
Comments 9
Viewers 18
Praise Details

43. 1
2
3
4
5
Who is Centercode
Product Strategy in a Recession
Intro to Delta Testing
How it Works
Q&A

44. Q&A
CEO & Co-Founder,
Centercode
Linkedin page: /in/freiler/
Email: luke@centercode.com
Twitter ID: @LukeFreiler
Website: centercode.com
Webinar Coordinator,
Product Management Today
Linkedin page: /in/rebeccakomathy/
Twitter ID: @ProdMgmtToday
Website: productmanagementtoday.com
https://www.productmanagementtoday.com/webinar-series/11062/candid-conversations-with-product-people
Luke Freiler Rebecca Komathy
With: Moderated by:

45. Remember when we all had offices? (800) 705-6540 info@centercode.com