Slides from my phd defence of my dissertation, Open Government Communities: Does Design Affect Participation?

1. OPEN GOVERNMENT
COMMUNITIES
DOES DESIGN AFFECT
PARTICIPATION?
!1

2. ABOUT ME
M.Sc. Political Science, University of Copenhagen (2009)
!
Partner & Chief Strategy Officer in Better Collective
- Start-up (est. 2004) focusing on online communities in
the igaming industry.
- Industrial PhD focuses on increasing participation to
online communities in open government.
!2

3. PRESENTATION AGENDA
- PART I: WHY STUDY OPEN GOVERNMENT?
- PART II: RESEARCH QUESTION
- PART III: K10: AN OPEN GOVERNMENT COMMUNITY
- PART IV: EXPERIMENTS, DESIGN & FINDINGS
- PART V: LEAN EXPERIMENTATION
!3

4. PART I
WHY STUDY OPEN
GOVERNMENT?
!4

5. WHAT IS IT? OPEN
GOVERNMENT DEFINED
Open government is a global movement to make
government more open and increase participation
& collaboration through the means of technology.
!
(Obama 2009; Lathrop & Ruma 2010b; Open
Government Partnership 2013)
!5

6. OPEN GOVERNMENT CAN
BE MANY THINGS
Open data
Public service delivery
Transparency eRulemaking
Anti-corruption Innovation
Accountability Access
Citizen participation
!6

7. EXPECTED BENEFITS OF
CITIZEN PARTICIPATION
- Improve democracy (Harrison et al. 2011)
- Better decisions (Janssen 2011)
- Increased efficiency (Lakhani et al. 2010)
- Foster Innovation (Lakhani & Panetta 2007)
!7

8. Growing, global phenomenon: Over 60 countries have signed
the Open Government Partnership since 2011
!8

9. WE ARE INVESTING A LOT
IN OPEN GOVERNMENT
The Danish government is
currently implementing 33
different open government
initiatives.
!9

10. THERE IS A PROBLEM:
PARTICIPATION
-
Citizen participation initiatives will only succeed if
the citizens actively participate.
-
But only 25 % of online communities gather more
than 1,000 users in their lifetime (Farzan et al.
2011).
-
1,000 users is hardly enough to fulfill the aims and
ambitions of open government.
!10

11. kbh.dk: Social network for Copenhagen Citizens (2008).
Price: DKK 7 million.
Aim: 125,000 users. Only got 1,800 users (1,44% of the goal).
!11

12. Worldclimatecommunity.org (2011).
Price: DKK 5 million.
Was quickly closed due to lack of participation.
!12

13. Borger.dk e-democracy forum.
Closed due to lack of participation.
!13

14. THE STATE OF OPEN
GOVERNMENT
- Open government offers great potential
and many promises.
!
- But efforts often fail to meet expectations
due to lack of participation.
!14

15. EXISTING RESEARCH CAN
HELP WITH THE PROBLEM
- Community design affects participation.
(Wasko, Faraj and Teigland 2004; Ma and Agarwal 2007;
Venters and Wood 2007; Zhang and Watts 2008; Moon
and Sproull 2008; Shen and Khalifa 2009).
- Social psychology theory can be used to generate
ideas and hypotheses about community design.
(Beenen et al. 2005; Cosley et al. 2006; Chen et al.
2010; Burke, Marlow, and Lento 2009; Kraut and
Resnick 2012).
!15

16. PART II
RESEARCH QUESTION
!16

17. RESEARCH QUESTION
“Can social psychology theory help increase
participation in open government communities?”
!17

18. RESEARCH COMMUNITIES
!18

19. MAIN CONTRIBUTIONS
- Substantial knowledge about increasing
participation to open government communities
through social psychology inspired design.
!
- Lean Experimentation Framework: Make it
cheaper and faster to test design ideas in the
future.
!19

20. PART III
K10: AN OPEN GOVERNMENT
COMMUNITY
!20

21. K10 is an open government community for people involved
with “early retirement pension” and “flexjob”.
!21

22. ABOUT K10
- Purpose: “K10 is created with the aim of sick
people, who are trapped in the system during their
case handling with the municipalities, have a place
where they can get advice and support from other
people who are in the same situation. This support
can be of statutory or moral nature…”
!
- Registered users: 9,500
- Contributing users: 4,000
!22

23. PART IV
EXPERIMENTS, RESEARCH
DESIGN & FINDINGS
!23

24. RESEARCH APPROACH
!24

25. Experiments
Experiments are used to test different design patterns
Experiment
Experiment 1
Experiment 2
Experiment 3
Experiment 4
Theory
Social comparison theory
Goal setting
theory
Self-efficacy
theory
Social identity
theory
Proxy
Email
Survey
Email & Message
Google
advertisements
Effect
Measure effects
on participation
Measure effects
on participation
Measure effects
on participation
Measure effects
on participation
!25

26. EXPERIMENT 1
Research Question: How does exposure to social comparison information affect users’
participation in K10?
GROUP 1
(n = 107)
T0
GROUP 2
(n = 111)
Measure participation
Receives an email with
T1
Receives an email with
comparative
non-comparative
participation
participation
information
Intervention
GROUP 3
(n = 1,265)
information
Measure participation
!26
Receives no email

27. EXPERIMENT 1 FINDINGS
MAIN FINDINGS
- Below-means users participate significantly more.
- No significant difference between comparative and noncomparative information is detected.
P-VALUES
EFFECT SIZE
IMPLICATIONS FOR
DESIGN
VALIDITY CONCERNS
- Ranging from p < 0.026 to p < 0.269
- Medium-large
- Participatory information can alter participation patterns.
- Participatory information might upset users.
- Can be implemented through stats, leader boards or emails.
- Sending emails per se might have an effect.
- Extrapolation unknown
- Dependent variable might not be most salient comparison
metric.
!27

28. EXPERIMENT 2
Research Question: How does a self-assigned goal to a number of contributions to K10
affect subsequent participation?
GROUP 1
(n = 51)
T0
Intervention
T1
GROUP 2
(n = 33)
GROUP 3
(n = 3,998)
Measure participation
Subjects are asked to
Subjects are answering
set a participation goal
a survey without
in a survey.
participation goals.
Measure participation
!28
Subjects are not
answering a survey.

29. EXPERIMENT 2 FINDINGS
MAIN FINDINGS
P-VALUES
EFFECT SIZE
- Goal assignment to high goals might increase participation.
- Only significant effect for subjects committing to a higher goal.
- Ranging from p < 0.003* to p < 0.057*
- Large
- Simply asking users how much they will contribute might
IMPLICATIONS FOR
DESIGN
increase participation.
- Goal-setting can be implemented in several ways: in sign-up
process, in surveys and through direct outreach.
- * Significant change in control group, indicating potential
VALIDITY CONCERNS spurious variable - no extrapolation should be made.
- Extrapolation unknown
!29

30. EXPERIMENT 3
Research Question: How does knowledge of other users’ gratitude for previous
contributions affect future contributions?
GROUP 1
(n = 23)
T0
Intervention
GROUP 2
(n = 23)
Receives an email and message at
K10 containing a thank you note.
Receives no email or message.
Measure participation
T1
Intervention
Receives no email or message.
Receives an email and message at
K10 containing a thank you note.
Measure participation
!30

31. EXPERIMENT 3 FINDINGS
MAIN FINDINGS
- Thanking subjects increase subsequent participation
- As a medium, internal message system is more effective than
email
P-VALUE
EFFECT SIZE
IMPLICATIONS FOR
DESIGN
- P < 0.040
- Medium
- Expressing gratitude can be used to attain desired behavior.
- Recognition can be implemented in different ways including
automated recognition, peer-to-peer recognition and expert
recognition.
- Potential carryover effect. Compromises effect size, not logic
VALIDITY CONCERNS
- Non-random sampling makes it challenging to extrapolate
- Possibly unclear who is thanking the subjects
-Extrapolation unknown.
!31

32. EXPERIMENT 4
Research Question: How does benefit of contribution affect subsequent behavior on
K10?
GROUP 1
(n = 185)
GROUP 2
(n = 86)
GROUP 3
(n = 70)
Subjects see an
Subjects see an
advertisement
highlighting
highlighting
benefit to similar
benefit to self by
others (small
joining K10.
Intervention
advertisement
group) by joining
K10.
T1
GROUP 4
(n = 112)
Subjects see an
Subjects see an
advertisement
advertisement
highlighting
highlighting no
benefit to others
benefit by joining
(large group) by
K10.
joining K10.
Measure participation
!32

33. EXPERIMENT 4 FINDINGS
MAIN FINDINGS
P-VALUES
EFFECT SIZE
Identity constructs do not affect subjects’ participation. But it
does affect engagement.
- Ranging from p < 0.000 to p < 0.906
- Small
- First impressions matter - it affects the subjects’ subsequent
IMPLICATIONS FOR
DESIGN
behavior or attracts certain types of subjects.
- Advertisement platforms are useful for recruiting and
assigning subjects
VALIDITY CONCERNS
-The two-week time span might have been too short.
-The different advertisements might attract different people to
begin with.
- Poor group constructs.
!33

34. OVERALL FINDINGS
INSIGHTS ACROSS
EXPERIMENTS
SHARED
METHODOLOGICAL
CHARACTERISTICS
- Design can influence participation behavior.
- Effects not unambiguous as expected.
-Participation problem can be addressed.
- True experiments
- Theory-based
- Conducted in the field
- Use Proxies to test design ideas
- External validity: Unknown to which degree findings apply to
KEY CHALLENGES
ACROSS
EXPERIMENTS
other open government communities.
- Between different demographic groups in Denmark
- Even more so between different cultures such as East / West
-Temporal validity: Unknown how findings apply in 1, 5 or 10 years.
!34

35. PART V
LEAN EXPERIMENTATION: AN
ANSWER TO THE VALIDITY
CONCERNS
!35

36. LEAN EXPERIMENTATION
!36

37. LEAN EXPERIMENTATION &
BETTER COLLECTIVE
- Conduct experiments early on before investing in
products / features / changes.
- Using proxies and website to conduct tests on.
- Many tests fail to improve status quo.
!37

38. SUMMARY
- Open government has a potential to improve
democracy, decisions, efficiency and innovation.
- Success requires sustained participation.
- Social psychology can help inspire design.
- Experimental methodology can help identify
successful design patterns.
- Much more experimentation is necessary.
- Lean Experimentation might help on scalability.
!38