Failing gracefully: Lessons learned from Catan for XBLA


Published on

Presented at Casual Connect 2009.

Jason Schklar covers how Big Huge Games did Rapid Iterative Testing and Evaluation (RITE) user-testing on Catan.

We identified and fixed usability issues with core mechanics (trade, building) and the “learn as you play” tutorial mode in real time while conducting user-testing studies.

The key takeaway is that instead of getting a list of action items a week after the study is done, you end up with a dramatically improved game before the study is even over.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Catan as a board game has been enjoyed by millions and millions of people. It’s best played with 3-4 players sitting around a table. The idea is to build up your civilization by harvesting the resources found on the island and trading with opponents to improve your situation. Our goal was to bring Catan to Xbox Live Arcade Players.
  • Some of the key reasons for its appeal... Easy to teach and learn in a social setting The core mechanic involves wheeling and dealing with your friends in real time to move yourself ahead without giving your opponents the upper hand.
  • So, when we approached the Xbox Live Arcade version of Settlers of Catan, we started out with a couple of key user experience goals we wanted to carry over from the original board game.
  • I’m going to talk about a few key lessons we learned... But first, our approach: Best Guesses: Visuals Functional Spec Iterative user-testing: Get it in front of target users as quickly as possible (we stacked the deck against ourselves: people who hadn’t played Catan the board game) needed to learn the game needed to learn our UI - 1 lead designer/developer and 1 usability guy on site; other coders and artists available by IM/email. - Several sessions per day with the goal of new session = new build with fixes addressing previously discovered issues.
  • Now I’ll talk about 3 of our key learnings from our work on Catan Live! Common early error in the game: Top Left: Player discovers needs wood and brick to build road. Top Right: Player WANTS wood, can give away wool (sheep) Bot Left: Player focuses on “wanting wood” and forgets that still needs brick; trades his brick for Sun Tzu’s wood. Bot Right: Player realizes his mistake when tries to build road Penalty: Player would need to wait until his next turn to try again... Is forced to proceed before has mastered a core concept Feels “stupid”
  • Finding common early mistakes allowed us to bullet proof the tutorials...
  • We “fixed” the AI so that it would allow players to make common mistakes and not have to suffer unduly for it. Sun Tzu gladly accepts a trade that on higher levels of difficulty (or with more competitive players) wouldn’t likely fly.
  • Remember: Your familiarity and expertise with the game make you experience it very differently than the player trying it out for the first time. Just because we’re suffering due to “lack of challenge” or “repetition” doesn’t mean that the first-time player is.
  • Sometimes you just need to try something radically different to get out of a user experience rut Q: Which of the two do you think work best in conveying what opponent players wanted to “Give” vs. “Receive”? give/want arrows from opposing players. started with basic necktie, added arrows within neckties, animated the arrows. Nothing worked. we could try a radically different visual design – and validate it – because we were capturing issues in real time and trying fixes in real time (no need to wait weeks)
  • Being able to identify issues in real time and try solutions before the next participant meant that we could experiment when we got into a rut without serious risk to our ship schedule.
  • Failing gracefully: Lessons learned from Catan for XBLA

    1. 1. Failing gracefully: Lessons from Catan for XBLA Jason Schklar / Initial Experience Consulting e: [email_address] / t: @jackalshorns July 21-23 2009 Casual Connect Seattle 2009
    2. 2. Settlers of Catan has been playtested...
    3. 3. As a board game, it’s popular and fun <ul><li>Easy to teach new players in a social setting </li></ul><ul><li>Core mechanic: Trading is fluid and engaging </li></ul>Creative Commons Credit
    4. 4. User experience challenges for Catan Live <ul><li>Easy to learn solo? </li></ul><ul><ul><li>Trial version is Single Player Only </li></ul></ul><ul><li>Trade still fluid & fun? </li></ul><ul><ul><li>Trading around a table is easy </li></ul></ul><ul><ul><li>Trading via a UI presents challenges </li></ul></ul>
    5. 5. Our approach: Rapid Iterative User-testing <ul><li>Start with our best guesses </li></ul><ul><li>Validate and improve via rapid iterative user-testing </li></ul>
    6. 6. Tip #1: Watch users struggle and fail...
    7. 7. It’s OK to torture your users <ul><li>... so you can design the game to allow customers to learn by failing gracefully </li></ul><ul><li>Allowed us to bullet-proof the tutorial </li></ul><ul><ul><li>Stacking the deck </li></ul></ul><ul><ul><li>Tweaking opponent AI to be forgiving </li></ul></ul>
    8. 8. It’s all good... Sun Tzu plays nice
    9. 9. Tip #2: Calibrate assumptions with users ... <ul><li>Watching cards dealt slowly, one at a time </li></ul><ul><li>Watching the 14 th person in 3 days “Try for a Road” </li></ul>
    10. 10. It’s OK to torture your self <ul><li>... because users experience the game very differently than the developers </li></ul><ul><li>Prevented us from “overfixing” the game </li></ul><ul><ul><li>Default vs. Advanced settings </li></ul></ul><ul><ul><li>Novice vs. Expert learning curve </li></ul></ul>
    11. 11. Tip #3: Be willing and able to experiment... <ul><li>Is it neckties with animations? </li></ul><ul><li>Or asymmetrical arrows pointing to and from avatars? </li></ul>
    12. 12. Find the difference that makes a “difference” <ul><li>... because sometimes even silly experiments can yield amazing results </li></ul><ul><li>If you are set up correctly, you can experiment </li></ul><ul><ul><li>Lead designer/developer coding while we observed </li></ul></ul><ul><ul><li>Lead UI artist just an IM/phone convo away </li></ul></ul><ul><li>The results speak for themselves </li></ul><ul><ul><li>Who could have predicted Arrows vs. Neckties? </li></ul></ul>
    13. 13. User-testing Takeaways <ul><li>Set crisp user experience goals </li></ul><ul><ul><li>Identify core components that MUST be fun </li></ul></ul><ul><ul><li>Identify challenges that MUST be overcome </li></ul></ul><ul><li>Set users up to fail: You’re paying them! </li></ul><ul><ul><li>Don’t wait until you have a scripted tutorial </li></ul></ul><ul><ul><li>Don’t give them instructions or hand-holding </li></ul></ul><ul><li>Watch and learn from how users fail </li></ul><ul><ul><li>Note whether they realize they’re failing </li></ul></ul><ul><ul><li>Note what they try to do when they’re failing </li></ul></ul><ul><li>Design the experience so users fail gracefully </li></ul><ul><ul><li>Allow them to recover from honest mistakes </li></ul></ul><ul><ul><li>Ensure failure states are informative and encouraging </li></ul></ul>