On April 4th, 2005, citizens with disabilities from all over the state tested voting systems and completed surveys documenting their reactions to each system. A total of 496 surveys were completed.The survey results on the following page will be one of several factors used to help the Oregon Secretary of State decide which voting system or systems to purchase to give citizens with disabilities the opportunity to vote privately and independently. Survey was a modified SUS – 12 questionsI would like to use this voting system in an election. There were too many steps in using this voting system.I thought this voting system was easy to use. The instructions for this voting system were difficult to understand. The buttons or touch screen were easy to use. It was hard to move around the ballot with this system. I think that most people could learn to use this voting system very quickly.I found this voting system awkward or difficult to use. I felt very confident that my vote was cast correctly with this voting system. I would need help each time I used this voting system. The ballot text was easy to read or hear. Casting my vote was easy.
Need a little usability?
Need a little usability?What you can learn from usability testingWhitney QuesenberyDana ChisnellUsability in Civic Life
3Usability testing answers questions, so we have tounderstand the question we are askingIf we want to know: Which ballot design helps voters be more accurate?We are asking a question that is Comparative, quantitative (errors, time) (and why they made those mistakes)If we want to know: Do voters understand how to mark their ballot?We are asking a question that is abut Mental models, variability, qualitative insights (and how prevalent each model is)
4Helpful models for thinking about usabilityA’s Ability, aptitude, attitude (what voters bring to the election)E’s Efficient, effective, engaging, error-tolerant, easy to learn (dimensions of usability)Use Useful, usable (operable), desirable (how we define success)
5The recipe for planning a usability testLocation and context: formal informalRecruiting: defined opportunisticActivities: instructed tasks free tasksQuestions: structured unstructuredData collection: observation task/data onlyResults quantitative qualitative
7Which instructions work better?Traditional ballot instructions vs. plain language version About this project 45 participants 3 lab setting, 8 days 2 moderators Instructions for voting Careful observation Satisfaction survey We learned about Voter accuracy Order effects Voter preference Results Statistical analysis Participant preference
8Which layout works better?Left aligned vs. right aligned names? About this project 100 participants 2 locations, 1 day 4 moderators + 6 others Instructions for voting Minimal observation We learned about Voter preference Time to vote Number of errors Results Statistical analysis of marked ballots
9What do voters do with unusual options?How do voters react to “double-vote” options? About this project 200 participants 5 locations, 1 day 12 moderators + 5 others Self-defined tasks Minimal observation We learned about Voter awareness of party Types of marks and errors Need for voter ed. Results Inspection of ballots for marking patterns
10What problems might voters have with a new ballot?What problems might voters have with a new ballot design? About this project 1 of 5 similar tests 10 participants 1 locations, 1 day 2 moderators + 1 others Self-defined tasks Observation Post-task questionnaire We learned about Navigation on the ballot Instructions Results Found pattern of errors for one interaction Time on task
11What do voters think about different systems?Which accessible voting system do voters prefer? About this project 496 surveys; multi-disabilit 11 systems Multiday “voting fair” They learned about Voter assessment of usability Voter comments about accessibility Differences by disability Results Statistical analysis survey results Qualitative analysis of comments
12Do voters understand how their votes are counted?Can voters explain and act on that understanding? About this project 20 participants 4 locations, 1 day 8 moderators Self-defined tasks Minimal observation We learned about Navigation on the ballot Instructions Mental models of counting Whether there’s a problem Results Qualitative insights
13Do voters understand a new voting system?What is their reaction to new procedures? About this project Focus group Mock election 1 moderators They learned about Attitudes and reactions Results Qualitative insights
14Things to watch out for whenplanning a usability test
15Treating participants with respectLanguage and cultural issuesManaging contextParticipant protections and informed consent
16Avoiding biasLeading questionsClosed vs. open questionsSampling biasModerator influenceObserver (or moderator reactions)