Olark - SupConf Fall 2017 - Live Chat Gender Experiment Results
•Download as PPTX, PDF•
1 like•7,070 views
For three months in 2017, Sarah Betts of Olark conducted a gender experiment while providing customer service on live chat. Sarah measured responses to three different chat agent avatars - Samuel (male), Sarah (female), and Sam (neutral). This presentation summarizes her experiment results and shows how customers interacted with agents of different gender types.
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Olark - SupConf Fall 2017 - Live Chat Gender Experiment Results
- 2. Hi Samuel!
- 3. Hi Sam!
- 4. Hi Sarah!
- 5. What would you do? Overall Chat CSAT: 4.52 Escalated Cases: 1 Blocked chats (trolls): 12 Overall Chat CSAT: 4.8 Escalated Cases: 12 Blocked chats (trolls): 32 Overall Chat CSAT: 4.8 Escalated Cases: 9 Blocked chats (trolls): ?
- 7. The thing is.. It’s all me
- 8. Only One Difference ● Same knowledge ● Same voice ● Same approach to customers What the customer saw
- 9. Why?
- 11. Sexism is a real thing ● Anecdata isn’t enough ● With great data, comes great responsibility ● With data, we can make bigger impacts
- 12. Creating an experiment You can do this!
- 13. Hypotheses ● Sarah would take more abuse on chat ● Samuel would have higher CSAT ● Sarah would escalate more cases
- 14. How I did it 1. One month as Sarah 2. One month as Samuel 3. One month as Sam 4. Changed avatar and name in chat 5. Collected data
- 15. Sarah brought all the trolls to the yard Sam ~3% of chats Sarah ~7.5%of chats
- 16. Data Doesn’t Play Fair ● Overall team escalations plunged ● Components were fixed ● Resist temptation to elevate your “anecdata”
- 17. CSAT (aka how outsiders can make career decisions for your team)
- 18. What would you do? Overall Chat CSAT: 4.52 Escalated Cases: 1 Blocked chats (trolls): 12 Overall Chat CSAT: 4.8 Escalated Cases: 12 Blocked chats (trolls): 32 Overall Chat CSAT: 4.8 Escalated Cases: 9 Blocked chats (trolls): ?
- 19. A word of caution Data doesn’t exist in a vacuum
- 20. We live in a data-obsessed time. Just having data is not enough. We have to know what it means, and we need to know the implications of using it.
- 21. What are you measuring? How might this data be used? Goals v Impact
- 22. What I did not measure ● Anything financial ● Race, age, ethnic, religious, or cultural biases ● Other support channels
- 23. Personally... The sun emoji stays No more abuse tolerance Higher self- confidence Easier than I expected
- 24. Who is scoring your team? How’s that working for you?
- 25. Talk to ME!
Editor's Notes
- At Olark we <3 experiments. So much so we have an entire Jira filter for them, and a template for setting them up. I wanted take at some support issues, especially around how folks perform with CSAT, Escalations, and the like
- I enlisted some folks from my Olark team to help Let’s meet my co-conspirators!
- Stop--as a manager, how are you going to view these? I might want to work with Samuel on his communication and empathy. Sarah and Sam may need some help skill building to reduce load on the escalation team
- 5 years on frontline support, concern about burnout levels for unsupported teams Work in tech Flashy articles aren’t enough, wanted numbers NickO’s monthlong experiment was inspiring Because I wanted to know how my perceived gender affected my interactions with customers
- Also, seeing this every day gets quite old
- Sexism is very real and flashy stories aren’t enough But with data collection comes responsibility The flip side of that is with data, we have the power to rethink how we do things and make a change
- So let’s backtrack to how I set this up Because experiments are FUN and give you powerful information
- Any experiment starts first with a hypothesis. If you’re thinking an experiment starts with data, no Why did I choose these? What we’ve seen is Women are less respected, less valued. Treated poorly. I wanted to see if I could quantify that. And talked to our data ops, Ben Schultz who helped me find what we can collect and think through the experiment
- Then make a plan. Time, who is involved, resourcing, etc
- Trolls SUCK, Facebook example, proposition with apology Explain !block feature & prob density Again though, as a manager, I might worry about Sarah’s quick banhammer
- Data doesn’t play fair because life is real, yo Story of “I’ll have my team take a look, thanks!”
- Let’s talk about customer satisfaction ratings Overall, all 3 reps scored very well. 4.52-4.8/5
- Stop--as a manager, how are you going to view these? I might want to work with Samuel on his communication and empathy. Sarah and Sam may need some help skill building to reduce load on the escalation team
- So let’s have the responsibility talk
- We’re data obsessed. We think about responsibility with customer data, privacy acts, COPA, GDPR But it goes further. We USE data all the time, HOW?
- Determine what you want to know, then design the experiment not the other way round With piles of data at our fingertips, that just appears through our tools, do we know what the data really measures? If you measure ticket escalation, are you measuring knowledge? OR your customers willingness to listen to your team member?
- Because of this, I stopped short of measuring quite a few things No financial, cultural appropriation (personal comfort) or other channels
- My anecdata-ey takeaways
- In the coming month, I’ll be sharing some resources on how you can set up a similar experiment. I think we have a responsibility to collect data on issues we see and share that. With enough data, support can be a leader in fair practices, impacting the broader world of tech