Our research program was going great – happy clients, engaged team – and we were feeling good. Then, overnight, our passionate new leadership challenged us to enact user testing for ALL external facing products, services, and even policy proposals. Unlike many execs, this one has a strong background in UX and described, in detail, the results they wanted to see: Test everything! Reinvent approvals! Have clients do their own testing! No new resources were given, and the deadline was always two weeks ago. While this change affected the entire organization, a lot of the details fell to our small team. This is the story of how we dealt with a seismic change over a few intense months, and the lessons we learned along the way. We had to quickly iterate on how we thought about work, evaluated it, staffed it and who our partners were.

1. Dealing with massive change
What we did when execs asked us to test EVERYTHING
Jonathan Rubin and Jessica Schafer | June 2023

2. About us
2

3. This presentation is being made by a Consumer Financial
Protection Bureau representative on behalf of the Bureau. It
does not constitute legal interpretation, guidance, or advice of
the Consumer Financial Protection Bureau. Any opinions or
views stated by the presenter are the presenter’s own and may
not represent the Bureau’s views.
Disclaimer
3

4. Setup
3 possible Ideas
1. Topical story to set stage, like the Uber CEO who never took an uber,
and then contrasting later with a hands on UX exec like Erie
2. This is a story of change and human resistance to it...
3. This is the story of every UXers dream – an exec who cares - that
turned into a boatload of stress, and what we learned from it

5. Setup

7. Our context
Working in the Federal government
7

9. We’re from the CFPB
The Consumer Financial Protection Bureau implements
and enforces Federal consumer financial law
▪ Aim to make consumer financial markets work for
consumers, responsible providers, and the overall economy.
▪ Protect consumers from unfair, deceptive, or abusive
practices and take action against companies that
break the law.
▪ Arm people with the information, steps, and tools that they
need to make smart financial decisions.
9

10. We work on the in-house design & dev team
▪ Design & Development: digital product development,
human-centered design, creative services, software
engineering, software delivery and platforms
▪ Financial products are increasingly technology-based, and
having an in-house team of designers and developers
enables us to keep pace with today’s market and develop new
tools that help consumers succeed in their financial lives.
10

11. Example: consumerfinance.gov
11

12. User experience culture: building steadily
▪ User experience thinking has been in the mix since the
beginning and we’ve steadily built our domain expertise and
relationships within the organization
▪ User research team established in 2015 and started
negotiating a very risk-averse, regulated space
▪ By 2021, good understanding of the benefits of user
research, but actually including user research or usability
testing in most projects was often still aspirational
12

13. Fall 2021: New leadership
13

14. New mandate: Test everything!
▪ No public launches without usability testing
▪ Test every iteration, and keep doing it
14

15. Ack!
▪ Awesome! But….
▪ We couldn’t do it. Not the way we had been working.
▪ This meant a lot of change: for our tools, processes & people
▪ Disruptive to planned work
▪ Culture of “user research is great, but it takes too long”
15

16. How we responded
16

17. Change is scary!
▪ New challenge required most logistical action and change
management messaging
▪ Balancing both “this is an actual new requirement for the
initiatives your office is working on” as well as “this will be
helpful and isn’t very complicated!”
17

18. 1. Create a compliance gateway
2. Scale research operations
Two major tactical efforts
18

19. Task 1: Create a compliance pathway
▪ Address Federal Regulations regarding data collection,
privacy, and security through our established relationships
▪ Defined a “superhighway” pathway: user research studies
operating within specific parameters are pre-approved
▪ Opt-in recruiting
▪ Unstructured method (interviews, conversational discovery,
observation) OR less than 10 participants
▪ Using approved tools and with re-usable privacy statements
▪ Cut planning time down to days instead of weeks or months!
19

20. Task 2: Scale research
20
Research Operations Democratization
(aka Self-Service)

21. A definition
ResearchOps = the orchestration and optimization of people,
processes, and craft in order to amplify the value and impact
of research at scale
- Nielson Norman
Research Operations = Research Process Improvement (or
research support) offered as a service
- Me
21

22. Researcher Research Insights
Finding participants
Procurement
Budget
Compliance
Incentives
Templates
Tools
Knowledge
Management

23. Researcher Research Insights
Finding participants
Procurement
Budget
Compliance
Incentives
Templates
Tools
Knowledge
Management
Research Ops

24. (Some possible) Elements of ResearchOps
▪ Participant Recruitment
▪ Recruitment Panel
Management
▪ Procurement
▪ Research Spaces
▪ Data Governance
▪ Travel booking
▪ Budget Management
▪ Event coordination
▪ Research Library
Management
24
▪ UX Comms
▪ Ops Service Design
▪ Intakes
▪ Incentives
▪ Research Tools
▪ Delivering APIs
▪ Managing migrations
▪ Manuals and Guidelines
▪ Consent forms, NDAs
▪ Research Branding
▪ Onboarding
▪ Program Management

25. Our elements!
25
Category Examples
1. Knowledge Management
Box UX research, UX and Video, Insight
Management, Office of Civil rights preso, Hubcap
and UX repo
2. Tools and Technology Qualtrics set up, assess UX tools, Validately
3. Team Building / care Goal and client working sessions, Retros, surveys
4. Recruitment
Internal testing pool, recruitment guidance, audience
prototype, accessibility testing
5. Guidelines and templates
URT Research template, survey training, Research
standards, Hubcap
6. Governance DIG guidance, Research standards
7. Capability and Opportunity QRCA presentation, UXPA conference proposal
8. Asset Management UX Archive, UX Archive into cloud, UX tools
9. Event Management UX COP, Office Hours
10. Research Spaces Design toolkit
11. Finances NA
12. Communication and Branding Friday URT updates, team meetings, Hubcap

26. ResearchOps: Our Five elements
Participants Governance Knowledge
& Sharing
Tools Skill
Building
• Finding
• Screening
• Paying
• Approvals
• Privacy
• KM
• Insights
• Repository
• Acquire
• Liaising
• Admin
• Assessing
• Mentoring
• Training

27. Tools tools tools

28. Task 2: Research Democratization
Democratizing research can mean a lot of things…
28
Is it simply
exposing people to
research?
e.g., sharing research
findings and highlight
reels
Is it involving
people to research?
e.g., onsite visits,
synthesis workshops
Is it having non-
researchers leading
research?
e.g., allowing non-
researchers to define,
plan, and execute
research projects
Source: Meghan Wenzel, Unqork

29. ▪ Increasing awareness
▪ Bigger bench
▪ Increase user awareness
▪ Better research partners
▪ Becomes a core activity
▪ Researchers as guides, not
gatekeepers
▪ More perspectives
Cons
▪ Ignoring negative feedback
▪ Poor / biased data
▪ Losing control of research
▪ Cheapens practice
▪ Burnout / sad researchers
▪ Taking time away from more
important research
Pros
Is democratization a good idea?
29
Source: Meghan Wenzel, Unqork

30. Progression of democratization
30
Source: Meghan Wenzel, Unqork

31. Progression of democratization
31
Source: Meghan Wenzel, Unqork

32. 32
Our Democratization Framework

33. 33
ResearchOps
Democratization
“Yes, we can
test everything!”
Compliance Path
(Superhighway)

34. What we learned along
the way
34

35. Lesson 1: Prepare and breathe through it
▪ When change happens, prepare but stay flexible as the winds
can shift again
▪ Fight the impulse to over-plan
▪ Communicate, communicate, communicate: articulate value
and impact simply and directly
35

36. Lesson 2: Treat democratization as a first draft
36

37. Lesson 3: ReOps builds capacity for future change
37
User Research
Success
Research
Operations

38. Lesson 4: New tools, new issues
38
Source: @pilotganso

39. Lesson 5: Exec champion who truly “gets” UX is
secret weapon
▪ Leadership support is always critical, but even more so when
introducing a new way of working
▪ Leadership needs to be able to speak to the value of this
change without you (subject matter expert) around in order
to handle freak outs
39

40. In summary…
You got this.
40

41. Groundwork makes the dream work
▪ Always build relationships (especially during down times)
▪ Discover core needs of:
▪ Your user researchers
▪ Your compliance officers
▪ Your clients
▪ Build ReOps components (especially during down times)
▪ Tools and democratization can help scale UX
▪ Expect change to change
▪ Look to change management (“What’s in it for me”)
41

42. Building on our success!
▪ Now that we’re set up to quickly and iteratively test public
facing products, providing space to tackle other challenges:
▪ Exploratory and discovery research
▪ Design sprints
▪ Usability testing internal products
▪ Testing with users of accessibility technology
▪ Testing in more languages
42

43. Thank you!
Questions?
43
Jonathan Rubin, jonathan.rubin@cfpb.gov
Jessica Schafer, jessica.schafer@cfpb.gov