The document discusses the CUTGroup (Civic User Testing Group), which was created by Smart Chicago to test civic apps and websites. The CUTGroup provides user experience testing and helps develop digital skills in the community. Testers are paid Chicago residents of varying technical abilities. The CUTGroup has tested several apps, providing useful feedback to developers. Its goal is to ensure technology works for all residents.

1. The CUTGroup
Daniel X. O’Neil
Open Gov Hack Night
December 9, 2014
1
@danxoneil @smartchicago #CUTGroup

2. Agenda-cize
• My personal journey
• Smart Chicago
• Origins
• Components
• Tools
• Methods
• Examples
2
@smartchicago #CUTGroup

3. My personal journey
• Web worker since June 15, 1998
3
@smartchicago #CUTGroup

4. First
• I made my first civic app in 1999
• Text-based WAP systems were just
coming into general use on phones
• I tried to think of the most compelling use
case for this technology. In other words,
what can’t wait for you to get back to your
desk?
• The answer: Killer on the Loose
December 10, 2014 @danxoneil 4

5. Most compelling wireless utility,
1999
December 10, 2014 @danxoneil 5

6. I’ve worked with some developers
December 10, 2014 @danxoneil 6

7. I’ve worked with some developers
December 10, 2014 @danxoneil 7

8. December 10, 2014 @danxoneil 8

9. December 10, 2014 @danxoneil 9

10. December 10, 2014 @danxoneil 10

11. Our principles
11
@smartchicago #CUTGroup

12. The Strategy
@smartchicago 12

13. Access, Skills, and Data
• Access to the Internet for all
– Broadband at home
– Public computer centers and Community technology
centers
– Next generation stuff
• Skills once you’re on the Internet
– All on ramps
• Data so that there’s something worth looking at
once you’re on and know how to work it
– Apps
– Infrastructure
13
@smartchicago #CUTGroup

14. The Founders
@smartchicago 14

15. Three Chicago Institutions
• The John D. and Catherine T. MacArthur Foundation
– A center for ideation and conception
– 2007 report, “The City that NETWorks”
– The primary source of funding
• Chicago Community Trust
– Provides housing for the organization and is fiscal agent
– At the center of funding and knowledge for the region
– Critical financial, human resources, and other support
• The City of Chicago
– The most important policy lever in the region
– Departmental relationships
– Getting things done
15
@smartchicago #CUTGroup

16. CUTGroup
16
@smartchicago #CUTGroup
• The Civic User Testing Group is a set of
regular Chicago residents who get paid to test
civic apps
– http://www.cutgroup.org/
• The CUTGroup Book: Civic User Testing
Group as a New Model for UX Testing, Digital
Skills Development, and Community
Engagement in Civic Tech
– http://www.cutgroupbook.org/

17. Origins
17
@smartchicago #CUTGroup
• Stemmed from Illinois Open Tech Challenge
(IOTC)
– Very successful in recruiting tech talent
– Not so successful in recruiting residents
– My experiences
• We basically came to the thought: “If it doesn’t
work for you, it doesn’t work.”

18. Three Main Components
18
@smartchicago #CUTGroup
• UX Testing
• Digital Skills
• Community
Engagement

19. Components: UX Testing
19
@smartchicago #CUTGroup
• Legitimate UX testing -- the developers who
work with us get specific, actionable feedback
from relevantly situated users
• Both quantitative and qualitative info
• We’ve found glaring bugs, discovered unique
insights, and helped people plan new product
releases.
• We stray from UX test design principles,
however, in a very key way: by requiring the
developer to participate in the test.

20. Components: Digital Skills
20
@smartchicago #CUTGroup
• Each test consists of a
mix of 10-20 people of
widely varying digital
skills, all learning and
teaching each other, in
the same room, out in
plain public view
• Developers are
learning how to
conduct UX tests and
to make their sites
better

21. Components: Community Engagement
21
@smartchicago #CUTGroup
• Biggest surprise to us
• Conduct tests in Connect Chicago locations
– More than 250 places in Chicago where you can
use a computer for free
– http://weconnectchicago.org/
• We were surprised to see how much people
like the CUTGroup.
– Same last two questions on every CUTGroup test
form:
• “Did you like this CUTGroup test?”
• “Anything else to add?”

22. Components: Community Engagement
22
@smartchicago #CUTGroup
• Diverse group of testers from different
backgrounds and neighborhoods

23. Tools
23
@smartchicago #CUTGroup

24. Tools
24
@smartchicago #CUTGroup
• Wufoo forms: http://www.wufoo.com/
– Collect information
– Recruitment e-mails
– Tests
• Mailchimp: http://mailchimp.com/
– E-mail campaigns
• Excel
• Patterns: software that helps collect data from
different systems; assists with segmenting

25. More Tools
25
@smartchicago #CUTGroup
• Devices
– Laptops & iPads
– Encourage testers to use their own devices
– We can segment based on tester devices
• Recording equipment:
– http://www.smartchicagocollaborative.org/hardware-and-
software-for-cutgroup-and-civic-hacker-events/
• Gift cards:
– http://www.smartchicagocollaborative.org/gift-cards-for-cutgroup/
– $5 gift card for sign up, $20 gift for a test
• Candy!

26. Methods: Pre-Test
26
@smartchicago #CUTGroup
• Developer meeting
– Target testers, Type of test, Questions to ask
– What do we want to learn?
• Recruitment
– 1st e-mail campaign
– The more we ask of testers, the smarter we get about
testing!
• Segmenting
– 2nd e-mail campaign asking for time availability
• Scouting for test locations
– We want to be all over the city

27. Methods: During Test
27
@smartchicago #CUTGroup
• Proctoring
– Different test types:
direct proctoring, focus
groups, remote, self-driven/
in-person
– Depends on developer
meeting and what we
want to learn from the
test
• We use Wufoo forms
to ask test questions
with or without a
proctor

28. Methods: Post-Test
28
@smartchicago #CUTGroup
• Analysis
– In-depth analysis of different questions
• Publishing results for every test
• Follow Up
– Github Issues; example from CUTGroup #9:

29. Tests
29
@smartchicago #CUTGroup
We have conducted 9 tests to date:
• #1 – FreedomPop Router Test
• #2 – Go2School User Application Test
• #3 – Chicago Health Atlas
• #4 – EatSafe.co
• #5 – ChicagoWorksforYou.com
• #6 – OpenStreetMap Editor
• #7 – EveryBlock iPhone App
• #8 – Waitbot App
• #9 – Foodborne Chicago

30. Examples: EatSafe.co
• EatSafe.co Tester #8
– https://www.youtube.com/watch?v=rGjMhz3TIkI
30
@smartchicago #CUTGroup

31. Examples: OpenStreetMap
• OpenStreetMap Tester #5
– http://youtu.be/BWNZxVhV7Kk
31
@smartchicago #CUTGroup

32. Foodborne Chicago
• An app that
searches Twitter for
tweets related to
food poisoning and
helps report these
incidents to the
Chicago
Department of
Public Health.
32
@smartchicago @foodbornechi #CUTGroup

33. Foodborne Questions
• Here is a list of questions we wanted to answer through
this test
• How do users feel about responding to a random tweet?
• Does the form require too much information?
• Would users feel better or worse if there was more
Chicago Department of Public Health or 311 visibility?
• Why might residents on the South side of Chicago not
respond as often as residents from other Chicago
neighborhoods?
33
@smartchicago @foodbornechi #CUTGroup

34. Foodborne Results
• How do users feel about responding to a
random tweet?
- Twitter is used as a private communications
network for the majority of our testers.
- Twitter handle & logo helps
- Tweets should have three elements:
compassion, question, and an official element.
• Does the form require too much information?
- Surprised that form wasn’t too long per testers
34
@smartchicago @foodbornechi #CUTGroup

35. Foodborne Results
• Would users feel better or worse if there was more
Chicago Department of Public Health or 311 visibility?
- Wanted stronger connection to City of Chicago
• Why do residents on the South side of Chicago not
respond as often as residents from other Chicago
neighborhoods?
- Twitter users see Twitter as a way to connect with
people they know, or are within in their circles
whether that be their friends, their work, or their
community.
35
@smartchicago @foodbornechi #CUTGroup

36. Improvements
• Change the Tweet
• URL Update
• Be more Official
• More about Process
• Being part of more Networks
36
@smartchicago @foodbornechi #CUTGroup

37. Foodborne and the “News”
• Foodborne has received much media
attention
- Twitter since it uses a Twitter platform
- Various Blogs
- Fox News
- Mortality and Morbidity Weekly
- Others
37
@smartchicago @foodbornechi #CUTGroup

38. The CUTGroup Book
http://www.cutgroupbook.org
38

39. Be a Tester!
• Are you a Chicago Resident?
• Interested in testing?
• Sign Up Here  http://www.cutgroup.org/
39
@smartchicago #CUTGroup

40. Sign up to test yer stuff!
• Are you a developer?
• Want to do this?
• Sign Up Here 
http://www.cutgroup.org/developers.html
40
@smartchicago #CUTGroup

41. So ya thanks
• @SmartChicago
• @DanXONeil
• @SmartChgoKyla
• @SSMarziano
• www.SmartChicagoCollaborative.org
41