Introducing a New UX Maturity Metric Team Engagement Score (TES) During Usability Testing
•Download as PPTX, PDF•
2 likes•265 views
The document introduces a new metric called the Team Engagement Score (TES) to measure team engagement during usability testing. TES is calculated based on three components: observation of usability sessions, collaboration during testing, and engagement level while observing. Several scenarios are provided to demonstrate how TES is calculated in different situations. The document discusses how TES could be used to track and improve team engagement over time, but also notes limitations since engagement is difficult to fully quantify. The presenters seek feedback on whether TES is a useful metric and how team engagement could best be measured.
Recommended
Recommended
More Related Content
What's hot
What's hot (7)
Similar to Introducing a New UX Maturity Metric Team Engagement Score (TES) During Usability Testing
Similar to Introducing a New UX Maturity Metric Team Engagement Score (TES) During Usability Testing (20)
More from UXPA International
More from UXPA International (20)
Recently uploaded
Recently uploaded (20)
Introducing a New UX Maturity Metric Team Engagement Score (TES) During Usability Testing
- 1. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Introducing a New UX Maturity Metric: Team Engagement Score (TES) During Usability Testing Bill Albert Executive Director, Bentley University User Experience Center Josh Rosenberg User Experience Researcher, DonorsChoose.org June 6, 2017 @BentleyUXC | @UXMetrics
- 2. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Two Short Stories @BentleyUXC | @UXMetrics
- 3. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Why Team Engagement Matters • Adoption of UX recommendations • Efficiency in the design process and product development • Culture of UX throughout the organization • Sparking innovation • Value to co-workers @BentleyUXC | @UXMetrics
- 4. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Our Motivation • Formalize our thinking through development of a team engagement metric • Launch discussions internally – explore the value of a team engagement metric • Share with others in the UX community • Promote team engagement
- 5. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! TES: Team Engagement Score TES = Observation + Collaboration + Behavior • Observation = Observing usability sessions • Collaboration = Being part of the team during testing • Behavior = Being actively engaged when observing @BentleyUXC | @UXMetrics
- 6. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Component 1: Observation @BentleyUXC | @UXMetrics # Actual Observed Sessions # Total Observable Sessions
- 7. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! 25 Actual Observed Sessions100 Total Observable Sessions Component 1: Observation @BentleyUXC | @UXMetrics 0.25=
- 8. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Component 2: Collaboration # Feasible In-Person Observations # In-Person Observations
- 9. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Component 2: Collaboration 2 In-Person Observers 5 Possible Observers = 0.40
- 10. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Component 3: Engagement @BentleyUXC | @UXMetrics 10 1-10 Rating
- 11. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Component 3: Engagement @BentleyUXC | @UXMetrics Took Notes Follow-up Questions Paid Attention Brainstormed Solutions Etc. Rating: 8 ✓ ✓ ✓ Rating: 8 10 = 0.80
- 12. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Component 3: Engagement @BentleyUXC | @UXMetrics Component 3: Engagement @BentleyUXC | @UXMetrics • Attention (0 -2) • Low = meetings, multi-tasking; High = undivided attention • Conversation (0-2) • Low = none; High = creative brainstorming based on insights • Notes (0-2) • Low = no notes; High = took detailed notes • Attitude (0-2) • Low = discredits participants; High = empathizes with participants • Questions (0-2) • Low = no questions; High = a lot of questions
- 13. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Calculating TES • TES = Observation + Collaboration + Behavior TES = ((# actual observations/ # total possible observations) + (# in-person observers / # feasible in-person) + (engagement score/10)) / 3 • Average the three numbers • TES = (0.25 + 0.4 + 0.8) = 1.45, divided by 3 = 0.48 • TES runs from 0 – 1 (higher the better) • All three dimensions weighted equally, but can apply different weights • TES works for large or small teams @BentleyUXC | @UXMetrics
- 14. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Scenario 1: High Engagement @BentleyUXC | @UXMetrics • Observation = 10 sessions (2 days), 8 attend all 10 sessions, 2 attend the first day only (5) = 90/100 = 0.90 • Collaboration = 5 people local, all attend in person = 5/5 = 1.0 • Behavior = highly engaged = 9/10 = 0.9 • TES = 0.9 + 1.0 + 0.9 = 2.8/3.0 = .93
- 15. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Scenario 2: Low Engagement @BentleyUXC | @UXMetrics • Observation = 10 sessions (2 days), 4 attend day 1 only, 6 don’t attend any sessions = 20/100 = 0.20 • Collaboration = 5 people local, 2 attend in person = 2/5 = 0.4 • Behavior = low engagement = 3/10 = 0.3 • TES = 0.2 + 0.4 + 0.3 = 0.9/3.0 = .30
- 16. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Scenario 3: Remote Only Studies @BentleyUXC | @UXMetrics • Observation = 10 sessions (2 days) with 10 invited - 3 attend both days, 5 attend day 1 only, 2 don’t attend any sessions = 45/100 = 0.45 • Collaboration = all remote, so this component is not considered • Behavior = moderate engagement = 6/10 = 0.6 • TES = 0.45 + 0.6 = 1.05/2.0 = .525
- 17. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Changing Weights @BentleyUXC | @UXMetrics • Change weights if you value one component more than another • Observation = 0.5 x 2 = 1.0 (weighted twice) • Collaboration = 0.4 x 0.5 = 0.2 (weighted half) • Behavior = moderate engagement = 0.6 (no change) • TES = 1.0 + 0.2 + 0.6 = 1.8/3.0 = 0.6
- 18. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Interpreting TES @BentleyUXC | @UXMetrics 0.1 0.2 0.3 0.4 0.60.5 0.7 0.90.8 Big Problem Fully Engaged Opportunity to ImproveWork to Do Take it Seriously
- 19. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Improving TES • Formalize the process • Bake into design schedule • Senior-level support • Correlate metrics to individual/team performance reviews • Free food is no longer viable! @BentleyUXC | @UXMetrics
- 20. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! How Would You Measure Team Engagement? @BentleyUXC | @UXMetrics
- 21. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Value of TES • Easy to collect and measure • Track team engagement over time • Report with other UX metrics • Roll into larger UX maturity score • Is this something you would use? @BentleyUXC | @UXMetrics
- 22. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Limitations of TES • Not a validated metric, no normative data • Engagement is more than participating in usability testing • Not all observers are the same, not all sessions are the same • Potential political ramifications • Other dimensions of engagement are hard to measure @BentleyUXC | @UXMetrics
- 23. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Delivering Value @BentleyUXC | @UXMetrics
- 24. #UXPA2017 www.uxpa2017.org Download our mobile app UXPA.org/2017app to fill out our conference survey! Questions? Bill Albert Executive Director Bentley University User Experience Center walbert@bentley.edu www.bentley.edu/uxc @BentleyUXC | @UXMetrics
Editor's Notes
- Commitment = # actual observations / # total possible observations
- Example: Total of 10 usability sessions 10 people are invited Number of possible observations = 10 x 10 = 100 5 people watch the first five sessions, and the other 5 people don’t watch any session Number of actual observations = 5 x 5 = 25 Commitment = 25/100 = 0.25
- Define: In-person means that the stakeholders are present for research activities, regardless of the location of the participant.
- Example: Total of 10 usability sessions taking place at your company in Toronto 10 people are invited 5 are in Toronto 5 are in Winnipeg Of the five people that are local, only two watch the sessions in-person Number of in-person observers = 2 Number of possible in person observers = 5 Priority = 2/5 = 0.40 Alternatively, can use the same formula as commitment
- Engagement: Subjective rating (1-10) given by usability lead, the higher the rating, the more engaged Engagement: observing, asking questions, brainstorming, etc. For example: Score = 8 (pretty engaged) Divide by 10 to be on the same scale 8 / 10 = 0.8
- Engagement: Subjective rating (1-10) given by usability lead, the higher the rating, the more engaged Engagement: observing, asking questions, brainstorming, etc. For example: Score = 8 (pretty engaged) Divide by 10 to be on the same scale 8 / 10 = 0.8