URF2010: Research Blunders

956 views

Published on

This casual presentation was given at Bolt|Peters User Research Friday conference on Nov. 17, 2010.

The purpose of this preso was to not only to provide lessons learned, but also to encourage a culture where the eff up is celebrated.

**If you download the preso, I put a rough outline of the script in the notes field.**

Published in: Business, Technology
3 Comments
1 Like
Statistics
Notes
No Downloads
Views
Total views
956
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
3
Likes
1
Embeds 0
No embeds

No notes for slide
  • Lighthearted look at me. And my eff ups.
    Maybe you can see yourself – your past, immature, naïve self – in me.
  • I work for a not-for-profit. We support online academic research.
    I’m part of the User Experience group.
    We conduct research, design, and test for usability.
  • My team and I spend a lot of time trying to understand how researchers interact with the technology our organization builds.
    We’re small, and we get to pick whatever tools we want to use to get the job done right.
  • Using a variety of tools in conventional and unconventional ways allows us not only to understand the limits of technology but to understand our limits as a researcher.


    In trying to narrow this talk, I had to reflect on all the eff-ups I’ve made during the years. I also gathered eff-up stories from my colleagues.
    Lots of war stories on tools and logistics…
    What I came up with as a theme…
  • Tools can only get you so far. One of my biggest lessons was in the art of the setup. We worry so much about learning how to use the tool, the content of our script, the wording of our interview questions, that the activity is appropriate for getting a useful answer, and we forget the one thing that can usurp our best laid plans: CONTEXT.

    Environment
    I’m going to describe a few things that happened when we didn’t pay close enough/appropriate attention to setup and the context in which the tools were being used.
  • How many here have used Morae before?

    We use Morae heavily so a lot of my war stories revolve around this tool.
    --conventional: online/onscreen
    --unconventional: as a recording device in the field or with paper prototypes

    We love and hate Morae. Love it because we can’t live without it. Hate it, because it frequently makes us feel stupid.
  • How many of you have captured something gross on audio?

    Microphone placement is key. I worried a lot about syncing the mike, about normalizing audio, and I didn’t think much about where I put the microphone other than “It needs to be near their mouth”.

    One time a colleague, a very pregnant colleague, was helping me tag some video in morae. She came storming into my office one day and told me that if I was going to put the microphone so close to participant’s heads, then I could tag my own damn video. Apparently the noises were not helping with her nausea.

    Lesson learned:
    People will bring drink and eat while doing online research. Slurping and swallowing makes for gross noises.
    I live in AA, MI. Winters in the midwest can be brutal. Cold weather makes for gross noises so keep a box of tissues on hand. Unless you want to listen to sniffling and horking.

    Don’t put the microphone near the throat unless you want to listen to hours of gross noises.

    The real problem was that our video became difficult to work with.
    These noises can be not only difficult/time consuming to have to work around, but can be distracting for anyone viewing video. We usually show video clips to stakeholders, developers, etc. You don’t want the point to be lost because you were obsessed with a gross noise.
  • How many of you have captured too much cleavage on camera?

    Placement of the camera is key. While Morae provides a preview of the shot, it does not warn you when the shot becomes inappropriate after someone removes their sweater in a hot room. You also cannot easily edit the video censorship style.

    Morae – repurposing tool to record, tag, video for a paper prototype.
    Lessons learned:
    Always test the camera angle with real people. Simulate height, weight, clothing, etc.
    We tested camera angle with me, in by business casual, non-slutty outfit. My outfit is not reflective of two things: 1) Summers in the midwest can be HOT…lots of tank tops; 2) when you’re a student: if you’ve got it, flaunt it.

    Unusable video because it became R-rated.
    Can’t edit/manipulate video because it becomes “What are you hiding???” for stakeholders/clients.



  • How many of you have had notices, IMs, texts, popups, system messages, etc. interrupt a test?

    When you’re running a study in a lab or controlled environment, you can try to prevent embarrassing technological interruptions.
    Email notifications
    System update messages – Reboot computer before a study to ensure all updates are installed and won’t interrupt.

    But be prepared when you’re out in the wild to take these things in stride. I’ve conducted observations where some rather personal information has popped up on screen via IM or email…
  • How many of you have had a tool fail during research?

    Tools are awesome, but technology ultimately fails.
    Never rely on technology as a sole means for executing research.

    Recently, a colleague and I were going out into the field to do some research. When we were packing and preparing, I packed every recording device we owned. When my colleague questioned me about necessity of all that redundancy, I just smiled knowingly.
    During the session we had several blunders: the flip video cam got jostled and ended up pointing at the ceiling, the computer Morae was running on died because, while it was plugged into the power bar, the power bar was not plugged into the wall. The digital recorder worked and my notes and descriptions were the only capture of the environment, the body language, the facial expressions etc. While I may have seemed anal about all the recording devices, my OCD was vindicated.

    Always have an analog backup.

  • It can change our behaviour.
    It can change culture.
    It can build relationships and careers.
    It can change the way we interact or the way a community socializes.

    I’ve been building a safe place in my org for people to eff-up. My group regularly talks about eff-ups and FTWs. What I noticed is that over time, a lot of our eff-ups are now also coupled with a win, something we learned and leveraged to make progress, improve a product, or build a trust relationship.

    Everyone on my team knows I screw up on a regular basis. I like to think it is endearing. Really, my screw ups make it ok for them to risk, to try, and to fail. The other day one of my team walked into my office and frantically said to me: “What are the top three reasons Morae Observer won’t start?” She asked me not because I’m the most adept with the tool, but because I had screwed up most. I gave her my top three eff-ups and sure enough, one of them provided her the solution.

  • One of my team actually quoted this motivational poster at me.

    Research blunders are a way of life for me and my team. We don’t strive to make them, but when we do, we make the best of them.
    Eff ups allow us to become better researchers faster.


    Right now, you might be thinking about how amateurish my mistakes are. You’d be right. I’ve been doing this 9 years and I still feel like an amateur every day. My work has nothing to do with routine, or predictability. I work with people. People are CRAZY. Do you know the only time I don’t feel like an amateur? When I hear about other people’s fuck ups. Then, I feel like I’m learning.
  • Anyone tell me an eff-up you’ve fallen in love with?
  • URF2010: Research Blunders

    1. 1. Research Blunders or: How I Learned to Stop Worrying and Love the Fuck Up Christina York (aka Xtina) @UXtina Presented at User Research Friday, November 19, 2010
    2. 2. My job
    3. 3. My support system
    4. 4. Some tools I have used to eff up research data:
    5. 5. What won’t tell you...
    6. 6. Tools won’t tell you when your audio is full of gross noises.
    7. 7. Tools won’t tell you when your video is X-rated. Censored
    8. 8. Tools won’t filter out embarrassing interruptions to your test.
    9. 9. Tools won’t tell you that they suck.
    10. 10. The power of the eff-up.
    11. 11. The power of the eff-up.
    12. 12. And that’s why I’ve learned to stop worrying and love the fuck-up. Xtina York cyork.mlis@gmail.com @UXtina

    ×