Talk given by Rebecca Buck, Lead UX Researcher at Salesforce, at Praxis Conference 2015 in October.
In this never before shared case study, we explain how our UX research team increased its scope of work from surface-level UI issues to a full portfolio of user-centered research. We use organizational ethnography and organizational change literature to develop a three phase model of research team growth. We then discuss the implications of our model for strengthening ethnographic praxis in cultures dominated by usability engineering. We conclude with a reflection on building internal bridges to facilitate change. Keywords: organizational change, UI, UX, research, ethnography, usability, lean.
“It is by logic that we prove, but by intuition that we discover.” (Poincaré 1914: 129)
5. Problem Framing
Design redefines the challenges facing an organization.
Feature, Form, Function
Design makes things work better than they did before.
Style
Design is the avenue to being hip and cool.
No Conscious Design
Design has no perceived value for the organization.
DESIGN MATURITY STAGES
Problem Solving
Design finds new opportunities by solving existing
problems.
Wu and McMullin, 2006
6.
7. Done by
Executives
Outsourced
Problem Framing
Design redefines the challenges facing an organization.
Feature, Form, Function
Design makes things work better than they did before.
Style
Design is the avenue to being hip and cool.
No Conscious Design
Design has no perceived value for the organization.
DESIGN MATURITY STAGES
Problem Solving
Design finds new opportunities by solving existing
problems.
8. Problem Framing
Design redefines the challenges facing an organization.
Feature, Form, Function
Design makes things work better than they did before.
Style
Design is the avenue to being hip and cool.
No Conscious Design
Design has no perceived value for the organization.
Problem Solving
Design finds new opportunities by solving existing
problems.
Research
Team’s Focus
Research
Team’s Focus
DESIGN MATURITY STAGES
9.
10. Problem Framing
Design redefines the challenges facing an organization.
Feature, Form, Function
Design makes things work better than they did before.
Style
Design is the avenue to being hip and cool.
No Conscious Design
Design has no perceived value for the organization.
Problem Solving
Design finds new opportunities by solving existing
problems.
Full portfolio
Goal
DESIGN MATURITY STAGES
11. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
Key Executives
Outsourced
Salesforce.com
Founded
12. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
Hired
UX
team
Usability
Analysis
Key Executives
Outsourced
UX Research
13. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
New
research
role
Usability
Analysis
UX
Researchers
Key Executives
Outsourced
UX Research
14. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
Usability
team
Key Executives
Outsourced
UX Research
Consolidated under
“Usability”
?
15. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
Usability
team
Key Executives
Outsourced
UX Research
Focus on
generative
?
UX
Researchers
?
?
22. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
Usability
team
Key Executives
Outsourced
UX Research
We
joined
?
?
23. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
Usability
team
Key Executives
Outsourced
UX Research
We
joined
?
?
?
What info sources do you trust?
What business challenges do you wish
you had better insight on?
24. What info sources do you trust?
What business challenges do you
wish you had more or better
insights on?
25. research = large sample size + usage data
What info sources do you trust?
What business challenges do you
wish you had more or better
insights on?
31. Problem Framing
Design redefines the challenges facing an organization.
Feature, Form, Function
Design makes things work better than they did before.
Style
Design is the avenue to being hip and cool.
No Conscious Design
Design has no perceived value for the organization.
Problem Solving
Design finds new opportunities by solving existing
problems.
EXAMPLES
DESIGN MATURITY STAGES
41. UX:
Right tool for the job
https://www.flickr.com/photos/petermartinhall/2905767476/
42. 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Problem
Framing
Feature,
Function
Style
Problem
Solving
Key Executives
Outsourced
UX Research
Right
tool for
the job
Today
43.
44. Problem Framing
Design redefines the challenges facing an organization.
Feature, Form, Function
Design makes things work better than they did before.
Style
Design is the avenue to being hip and cool.
No Conscious Design
Design has no perceived value for the organization.
Problem Solving
Design finds new opportunities by solving existing
problems.
DESIGN MATURITY STAGES
Lightning Design
System
45. Problem Framing
Feature, Form,
Function
Style
Problem Solving
Today research impacts every level
IoT
Cloud
S1
Mobile App
Lighting
Design System
Declarative
App Builder
Behavioral maps,
competitive benchmarking,
ethnographies
Researchers enable new use cases
through existing products
Researcher help translate existing
functionality to wider audiences
Optimizing UI affordances,
page layouts, click targets
We’re visiting from San Francisco, home of the Golden Gate Bridge.
Recently released a redesign of our core sales product. This has been such a huge success for the company, and this work all started with ethnographic field work.
But it was a long road to get there.
Lots of companies have gone through the growing pains of embracing design thinking. If you haven't yet you should read the fast company article on “how Google got Desing.” These are important stories.
This is our story of how we our research team evolved from the team who created the product on the left, and became the UX team who created the product on the right.
So, what changed? How’d we get here? You’re probably familiar with the stories of “how Google got design”, and how Intuit grew to have X# of Chief Design officers. Today, we’re proud to share the success story of how the scope of research, and UX in general, has grown in the past few years.
To describe that cultural shift, we find it helpful to reference the design maturity stages model. (explain levels)
Our founders respected great UX at each of those levels from the start. Even when they were working out of this studio apartment, and couldn’t afford to have researchers or designers on staff, they contracted that work out to make sure it was high quality.
So using the design maturity model, this is what our research portfolio looked like when the company was founded.
Where, executives were doing site visits to understand what customers wanted and how they were using our products. Then the UI design and usability tests were outsources to top notch agencies.
But, as the company grew, UX research had become focused on incremental feature improvements, and researchers were becoming typecast at “the usability people.”
Research had become synonymous with usability testing. People on the team tended to have HCI backgrounds, and were using many evaluative methods, like cog-tool and eye tracking. These are all important and powerful tools, but are limited in the types of problem they can help you solve. This mindset of evaluating and optimizing concepts falls short when applied to undeveloped products, or new markets, or users whose expectations are unclear, or unknown technical constraints. This is what we’re calling the “fixing bugs mindset”: limiting to role of research to only evaluative methods.
Our goal was to balance our research portfolio, to have research inform each level of design strategy
Understanding the history of the team.
In 2004, a former usability analyst was hired as the founding member of the UX team. The role of the “usability analyst” was introduced.
Team staffing dynamics were actually supporting that model. PMs are usually staffed to a particular feature for years, unlike user experience (UX) researchers who are more commonly staffed to a feature for a weeks or months. Because of this was abnormal for these subject matter experts, our PMs to seek out UX researchers for insights on user behaviors or industry trends.
In 2008, the role bifurcated. “UX researchers” using generative methods like site visits, and diary studies.
But, this model only lasted a few years because the org structure and processes didn’t match this model. PMs are usually staffed to a particular feature for years, unlike user experience (UX) researchers who are more commonly staffed to a feature for a weeks or months. Because of this was abnormal for these subject matter experts, our PMs to seek out UX researchers for insights on user behaviors or industry trends.
In 2010, the company hired the first director of UX research and consolidated the team under the name “Usability Team.” Not everyone changed their title. Not everyone stopped doing ethnographies and reporting on user behaviors. But, during this time, members of the team were spending most of their time in usability labs.
Early in 2012, the pendulum swung in the opposite direction. An executive requested for the team to provide insights to inform future releases,and the Usability Team was rebranded the User Research Team, and IC’s titles were updated to reflect the change. This change to the team’s mandate was really the beginning of the growing pains that help get us where we are today.
It was in that time of uncertainty that we joined the team and our began our work. You can see this tension of describing the scope of the work and who should play where has a long history. These were all natural shifts in the process of growing as a company and as a research team.
But, this was the context we walked into. We know that if we were going to contribute at a “problem framing” level we’d need to understand the needs and challenges of the stakeholders working in that space.
At the time, it felt like frustrating uncertainty.
operational changes that we needed to
Would the team get more headcount to cover this additional work? Should researchers deny work requests from the scrum teams they’d been supporting to prioritize time for future-oriented work?
This also happened to be when we joined the team.
To begin our process of organizational change we pulled lessons from the work of
Armenakis and Bedeian who break down organizational change into the issues of content, context, process, and criterion.
And we pulled from the work of other practitioners like Jump associates and Second Road, who have published great models for understanding cultural context and providing proof of the value of design thinking.
And we pulled from the work of other practitioners like Jump associates and Second Road, who have published great models for understanding cultural context and providing proof of the value of design thinking.
Tsoukas and Chia work on organizational becoming.
Before we could start making changes, needed to understand the terrain, and start from solid ground. to do that, we needed a shared understanding of the goals of the team, (which had to be informed/decided by the team and our stakeholders).
turned tools of research on ourselves....
history of research at salesforce
stakeholder interviews
team card sorting
visualizing gaps in research (circle product development process image)
Next, we turned our attention to executive stakeholders, and again turned the tools of research on ourselves. After conducting about 25 interviews with the company's most senior executives to learn what information sources they trust, and the types of insights they wished they had, we found that what this audience wanted most was usage data based on large sample sizes.
This was problematics for a few reasons:
(1). the team’s new mandate was to provide generative, future facing insights, a research problem that ethnography is well suited to inform, but wasn’t at all the information what this group of stakeholders was eager for
(2) our SLAs make usage data had to get, (we take customer privacy very seriously), so history weren’t collecting much usage data
(3). even though our SLAs had evolved to allow collecting anonymized usage data, most features didn’t have the instrumentation to collect that data.
So, in some ways, the new mandate of our team was even more ambiguous than when we started! But, in analyzing these interviews, we also learned that we as a team we didn’t have shared language or expectations about our teams work.
Next, we turned our attention to executive stakeholders, and again turned the tools of research on ourselves. After conducting about 25 interviews with the company's most senior executives to learn what information sources they trust, and the types of insights they wished they had, we found that what this audience wanted most was usage data based on large sample sizes.
This was problematics for a few reasons:
(1). the team’s new mandate was to provide generative, future facing insights, a research problem that ethnography is well suited to inform, but wasn’t at all the information what this group of stakeholders was eager for
(2) our SLAs make usage data had to get, (we take customer privacy very seriously), so history weren’t collecting much usage data
(3). even though our SLAs had evolved to allow collecting anonymized usage data, most features didn’t have the instrumentation to collect that data.
So, in some ways, the new mandate of our team was even more ambiguous than when we started! But, in analyzing these interviews, we also learned that we as a team we didn’t have shared language or expectations about our teams work.
So that became our next step, to align as a team about the scope of our work. To to that, we setup a workshop and had the team basically use a card sort method to categorize all the different the types of reports and types of insights our team was producing.
We tacitly knew there were lots of different ideas about what “good” research was and this workshop created the tools and forum for us to discuss those differences.
The outcome of that work was this framework, which unified language and explanation for how vastly different approaches to research fit together. Gave us a way to have conversations. For example, we started discussing career paths, and explaining that it’s fine to specialize within areas or to play the whole field.
Most importantly, that framework gave us, a shared language to have more effective conversations about the scope of our work and the goals of our team.
From
- Undocumented tribal knowledge at the outset of projects (domain expertise was tacit knowledge of AEs and Execs, hard for UX teams to make use of),
- Blurry shifts from Ideation through Optimization phases (we work fast, and these lines were very blurry, that’s hard for people just starting to work together).
- Limited usage data for Evaluation
To
- Documented Problem Framing and project briefs
- Structured time for Ideation, Experimentation and Optimization
- Usage data to evaluate success
This was the model that allowed us to align as a team about the research services we wanted to provide as a team.
Shared language so that we could have productive conversation about what our team goals were. Set consistent expectations with stakeholders. shared goals to “build” as a team
with that shared ground, we could start building.....
To start the process of building our foundation, we needed a proof of concept, an example of how this type of research would be valued within the current company context. We were looking for a project with these characteristics:
succeed at a team - collaborate across a variety of UX roles
qual and quant - to satisfy stakeholder expectations for large sample sizes, and highlight the qualitative research skills of our team
visual presentation (company has high standards for marketing :)
executive champions who were very invested in using the research for strategy
So… SUPER DUPER ADMIN SURVEY
Our product managers often want to know whether we’re building the right things for the right people. (development teams often had cyclical, unproductive conversations).
So we wanted to know more about how Salesforce administrators do their jobs: are we building the right administrative tools for our administrators?
We used a combination of quantitative and qualitative work to answer this--and then we involved designers and product managers and executives in the creation of our deliverable.
The organizational culture we’re working in today is hungry and eager for all types of research. For example, this is from a segmentation report that went viral in the company. The day after we first presented it there were literally hundreds of people in the dec and we really did get comments on 113…
This work sparked strategic conversations with teams we’d never even heard of.
(we’ve blogged about this). Great example of problem framing project because new language and tools for approaching the problem. for product dev teams to use to work together more effectively. After we presented our results, product managers were asking, “Wait! My feature wasn’t included! How do I measure my sadmin/gladmin ratio?!?!?”
We’re pretty stoked that our UX team invented the sadmin/gladmin ratio. The designers we brought in to help build and tell our story actually coined the terms sadmin and gladmin. An accessibility engineer who had gone to grad school for cartooning drew our illustrations.
This was such an important project for us because it changed the conversation about what our team “does.” This work didn’t “fixed any bug.” It gave teams new tools for decision making and empowered them to make more informed decisions.
Key features like the opportunity workspace, came from our researchers going into the field, and seeing that sales reps were creating kanban boards to work their leads.
within under a year, we had two foundational projects to demonstrate how research could change the way we approach problems and inspire better solutions.
Again, the way to choose a good foundational project is to make sure it has:
UTILITY: immediately useful for both tactical and strategic decisions
PARTNERS: succeed through partnerships with people across the organization. designers aren’t there just to make your slides pretty--they are there to help you build the arc of the story, too.
VISIBILITY: visible enough to build traction across the organization. market however you need to: email, chatter, brown bags, whatever. get everyone involved in the project to spread the project through their networks.
we are here.
Expanding the scope “research deliverables” to include things like behavioral models and needs statements.
We’re still experimenting. Expanding the idea of “research deliverables” to include things like workshop and journey maps.
When things work, formalize the process so more people can repeat that success.
Continue to focus on outcomes, highlighting the decisions that can be made the product improvements.
at this point you can start to Experiment. You don’t want that first proof of concept project to be an experiment! once you have a track record of success, it’s much easier to develop new processes with your partners in the organization
that brings us back to where we are today, where research is an integral part of the earliest phases of new product development.
“Design Maturity” implies Style and Form are “less mature,” that stylization is the avenue to being hip and cool. We argue that there are strategic opportunities at each level. Our goal was never to lose rigor in researching the form and function of products. Ways to contribute strategically at each level.
(Jan Schmiedgen)
As a UX team we’re very proud of our recently released style guide and think it is an example of design strategy at it best. Besides empowering the ecosystem of independent software vendors who build products on our platform, the style guide improved design operations making it easier for teams to work quickly, it provided the standards for designers to follow so that ux researchers don’t have to police consistency.
Today, we’re proud to have a more balanced research portfolio… where research is.
Today, there’s not just one way to get across the bridge. There is a lot of movement happening in both directions, where our executives are getting more involved in nuanced UI decisions, and our UX team is becoming more involved with long-term product strategy.
Remember, when you want to build a strong bridge, take the time to excavate the land and lay a foundation.
ethnography--on ourselves and on our clients--helped us build the foundational bridge to go from a fixing bugs mindset to a building businesses mindset.
What will your bridge look like?