Presented by Karl Melder

1. UXPA 2018
Karl Melder, Senior Design Researcher
the art of
scaling
observational research
as a
team sport

2. Agenda
Story of PerfTips
Dig into the practitioner’s role
Ways we scaled
Principles, guidance, tips,..
Confront, share, challenge & beer

3. My World: The
Developer
Division
Our customers are developers
Several teams & tools we brand Visual Studio
15 person UxR team + ~30 Design team
190+ Program Managers
5 years ago aggressively transitioned to Agile
3 years ago shifted to a highly data-driven culture
Today, customer and product insights and data
are table stakes for the business

4. Me…
Love the discipline of Ethnography …
I am a New Yorker by birth, but not by attitude
Masters in CS from VT with focus on HCI
Attended UPA in July in Arizona (116 in shade)
Swapped Manager and IC roles 2x
I absolutely HATE doing studies by myself
Skiing (any form) is my thing.
I remember Microsoft back when…

5. My passion is to put empathy at the
center of everything I pursue –
from the products we launch, to the
new markets we enter, to the
employees, customers, and partners
we work with.
Satya Nadella
Hit Refresh

7. The Story of
PerfTips
Only a small percentage our
developers were using our advanced
diagnostic tools.
How do we get these advanced
diagnostic technologies adopted by
our customers into their day-to-day
work patterns?

8. Our
Challenge
A new team had formed and were
given little guidance
We had a very hard problem to solve
Our lab was being remodeled
Upper management already had a
feature idea that they really liked
Complicated stuff to code

9. I suggested…
Lets just bring in a few users each
week to try ‘stuff’
‘Stuff’ would be low investments
ideas - mocked
We decide early each week what we
should experiment
Had the team pick a day of the week
that works for them
Create a self-built lab next to their
team room

11. Epic Failure
“I am not sure what to do with it.”
“I would probably turn it off!”
We heard…
Devs only fix perf if users complain.
Yet another tool window - noise!
Don’t waste time on high fidelity UI.
We Learned (together)…

12. Along the
way we
found our
Anti-Persona
Ted and the
power of a
common
language
“Ted is the one guy in the
team who does diagnostics
our dark arts master.”
We heard about variations of
“Ted” from several of our
users.
Ted became our Anti-Persona.

14. We were on to something…
“What makes VS remarkable is the capabilities
when you are on a line of code!”
“Can this be policy?”
Week 4 - 5

15. Week 37
74 users in-Lab
110+ ‘mini’ experiments
Lots of ideas & features
explored
Internal Microsoft release

16. Agile Engineering + Lean Ux
= Agile Manifesto
Minimized time planning too much, writing
reports,…
‘Quick Pulse Fridays’ had fewest meetings
Issues that came in engineering got in the lab
that week
Team watching users real-time
Continuous and collaborative Formulating,
Experimenting, and Sensemaking – all
happening at once – not asynchronous.
Agile Manifesto
Individuals and Interactions over processes and tools
Working Software over comprehensive documentation
Customer Collaboration over contract negotiation
Responding to Change over following a plan

18. Lets Now talk about
Scaling
SCALING
Experiments
Participation
Skills
Customers

23. Lessons Learned for Daily and Weekly Stuff
1. Be a coach and example - not a dictator
2. Start from where the product team is: mindset, location, cycle
3. When things look bleak, there lies opportunity
4. Don’t let Perfection get in your way
5. Plan the plan, but not the details
6. Create gravity through inclusion
7. Be a methods Ninja
8. Your most dangerous person THINKS they know how to do UxR

25. Lets really ramp
things up
Conference Floor
Developer Day

28. Lets Now Really Scale!
SCALING
Experiments
Participation
Skills
Customers

29. 100 Traditional interviews, Concept value tests, participatory design, Cognitive walkthrough, and usability tests
x 6 = 600 engagements

31. PrinciplesLets circle back

32. Build Bridges, Not Walls
Everyone is responsible for the
customer’s experience
Share expertise, don’t safeguard it
Promote continuous collaboration
Partner with other disciplines

33. Use a Common Language
Identify your assumptions and
formulate them into hypotheses
Hypotheses can be tested,
assumptions cannot
Hypotheses are the lingua franca of
DevDiv – from the board room to the
team room

34. Build Leaders that Build Culture
Microsoft is Hitting Refresh
Customer Connections are the new
Currency
Leaders are exceptional customer
advocates

35. Meet Teams Where They Are
Make it Your Own Make it Theirs
Leverage Lightweight Tools
Think Globally, Act Locally

36. Make Data Representative and Relatable
Speak to the head and the heart
Share Real Stories, Not “Personas”
Capture VIVID stories

37. My Big Lessons Making it a Team Sport
Don’t let perfection get in the way
Letting failures happen
Give expertise away and grow your powers
Share credit, don’t hoard
Enable failing forward faster
Be with your team: physically, process, cycle,
and culturally

38. Reading List
Lean UX: Applying Lean Principles to Improve User
Experience: Jeff Gothelf and Josh Seiden
Lean Customer Development: Building Products Your
Customers Will Buy: Cindy Alvarez
Sketching User Experiences: Getting the Design Right
and the Right Design: Bill Buxton
The Customer-Driven Playbook: Converting Customer
Feedback into Successful Products: Travis Lowdermilk
and Jessica Rich
Spencer, R. (2000) The Streamlined Cognitive
Walkthrough Method, Working Around Social Constraints
Encountered in a Software Development Company CHI
2000 vol.2 issue 1 pp353–359.
Medlock, M.C., Wixon, D., Terrano, M., Romero, R., and
Fulton, B. UPA (2002). Using the RITE method to
improve products: A definition and a case study.