The document discusses unintended consequences of decisions from a complex systems perspective. It presents an overview of topics including the ubiquity of unintended consequences, a complex systems approach focusing on interactions, and results on wisdom of crowds in online communities. Specific results analyzed include how social influence affects crowd wisdom, designing wise crowds through mechanisms like ranking and competition, and factors influencing resilience in online social networks. The document examines unintended consequences across different study systems like humans and social animals using methods like experiments, modeling and data analysis.
1. Decisions and their unintended consequences
Pavlin Mavrodiev
Doctoral examination – D-MTEC
Chair of Systems Design
www.sg.ethz.ch
2. Overview
1 Motivation
The ubiquity of unintended consequences
Chair of Systems Design Decisions and their unintended consequences Overview
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 2 / 20
3. Overview
1 Motivation
The ubiquity of unintended consequences
2 A complex systems approach
Designing and inferring interaction mechanisms
Study systems and challenges
Chair of Systems Design Decisions and their unintended consequences Overview
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 2 / 20
4. Overview
1 Motivation
The ubiquity of unintended consequences
2 A complex systems approach
Designing and inferring interaction mechanisms
Study systems and challenges
3 Outline of results
Wisdom of crowds, resilience and collapse of online communities
Mitigating negative consequences in animal systems
Chair of Systems Design Decisions and their unintended consequences Overview
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 2 / 20
5. Overview
1 Motivation
The ubiquity of unintended consequences
2 A complex systems approach
Designing and inferring interaction mechanisms
Study systems and challenges
3 Outline of results
Wisdom of crowds, resilience and collapse of online communities
Mitigating negative consequences in animal systems
4 Conclusions
Summary
Impact
Chair of Systems Design Decisions and their unintended consequences Overview
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 2 / 20
6. The road to hell is paved with good intentions
NYPD starts a public Twitter
campaign to improve image
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 3 / 20
7. The road to hell is paved with good intentions
NYPD starts a public Twitter
campaign to improve image
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 3 / 20
8. The road to hell is paved with good intentions
NYPD starts a public Twitter
campaign to improve image
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 3 / 20
9. The road to hell is paved with good intentions
NYPD starts a public Twitter
campaign to improve image
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 3 / 20
10. The road to hell is paved with good intentions
NYPD starts a public Twitter
campaign to improve image
Campaign ends in huge fiasco
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 3 / 20
11. The road to hell is paved with good intentions
NYPD starts a public Twitter
campaign to improve image
Campaign ends in huge fiasco
In colloquial language:
Streisand effect, Cobra effect,
Boomerang effect, ...
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 3 / 20
12. Causes
[Leibniz G., 1710]: Theological (theodicy, predestination). Provide
justification for the existence of evil
[Smith A., 1790]: Moral philosophy and accountability
[Merton R., 1936]: Cognitive limitations, misaligned incentives,
self-defeating prophecies, value systems
[Boudon R., 1981]: Introduce compositional effects - focus on
interactions
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 4 / 20
13. Causes
[Leibniz G., 1710]: Theological (theodicy, predestination). Provide
justification for the existence of evil
[Smith A., 1790]: Moral philosophy and accountability
[Merton R., 1936]: Cognitive limitations, misaligned incentives,
self-defeating prophecies, value systems
[Boudon R., 1981]: Introduce compositional effects - focus on
interactions
Challenges in sociology [Boudon 1989, Portes 2000]
Diversity of context
Linear thinking: Cause ⇒ Effect
Chair of Systems Design Decisions and their unintended consequences Motivation
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 4 / 20
14. Social complexity: the micro-macro link
“Time and again, we sociologists are surprised by the limitations of our
conceptual blueprints in comparison with the complexities of empirical
phenomena”
Alejandro Portes
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 5 / 20
15. Social complexity: the micro-macro link
“Time and again, we sociologists are surprised by the limitations of our
conceptual blueprints in comparison with the complexities of empirical
phenomena”
Alejandro Portes
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 5 / 20
16. Social complexity: the micro-macro link
“Time and again, we sociologists are surprised by the limitations of our
conceptual blueprints in comparison with the complexities of empirical
phenomena”
Alejandro Portes
¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Micro Level
⇒ ¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Macro Level
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 5 / 20
17. Social complexity: the micro-macro link
“Time and again, we sociologists are surprised by the limitations of our
conceptual blueprints in comparison with the complexities of empirical
phenomena”
Alejandro Portes
¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Micro Level
⇒ ¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Macro Level
Unintended consequences as emerging
phenomenon
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 5 / 20
18. Social complexity: the micro-macro link
“Time and again, we sociologists are surprised by the limitations of our
conceptual blueprints in comparison with the complexities of empirical
phenomena”
Alejandro Portes
¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Micro Level
⇒ ¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Macro Level
Unintended consequences as emerging
phenomenon
Focus on interactions (design, inference)
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 5 / 20
19. Social complexity: the micro-macro link
“Time and again, we sociologists are surprised by the limitations of our
conceptual blueprints in comparison with the complexities of empirical
phenomena”
Alejandro Portes
¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Micro Level
⇒ ¸ ¸
¸
¹
¹ ¹¹¸¸ ¹
¹
¸
Macro Level
Unintended consequences as emerging
phenomenon
Focus on interactions (design, inference)
Context-awareness vs. broad generalizations
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 5 / 20
20. Study systems and challenges
c DeSciL
Humans
Controlled experiments in humans –
design, conduct, noisy data
Agent-based modelling – KISS
Large datasets
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 6 / 20
21. Study systems and challenges
c DeSciL
Humans
Controlled experiments in humans –
design, conduct, noisy data
Agent-based modelling – KISS
Large datasets
c Gerald Kerth, Rene Janssen
Social animals (Bechstein’s bats)
Intrinsic interdisciplinarity
Data analysis – major underlying part
Develop systematic methods to infer
patterns
Chair of Systems Design Decisions and their unintended consequences A complex systems approach
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 6 / 20
22. Wisdom of crowds and social influence
when tho heads are not better
than one
social influence without
pressure to conform
↓
crowd wisdom still deteriorates
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 7 / 20
23. Wisdom of crowds and social influence
when tho heads are not better
than one
social influence without
pressure to conform
↓
crowd wisdom still deteriorates
opinion change of agent i ∝
F(xj, xi) + β(xi(0) − xi(t))
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 7 / 20
24. Wisdom of crowds and social influence
when tho heads are not better
than one
social influence without
pressure to conform
↓
crowd wisdom still deteriorates
opinion change of agent i ∝
F(xj, xi) + β(xi(0) − xi(t))
0.05
0.10
0.15
0.0 0.2 0.4 0.6 0.8 1.0
1.0
1.2
1.4
1.6
1.8
2.0
social influence
individualconviction
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
0.0 0.2 0.4 0.6 0.8
1.0
1.2
1.4
1.6
1.8
2.0
social influence
individualconviction
0.2
0.3
0.4
0.5
0.6
0.7
0.0 0.2 0.4 0.6 0.8 1.0
1.0
1.2
1.4
1.6
1.8
2.0
social influence
individualconviction
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 7 / 20
25. Wisdom of crowds and social influence
initial conditions matter
aggregate information
full information
0.005
0.010
0.015
0.020
0.025
0.0 0.2 0.4 0.6 0.8
1.0
1.2
1.4
1.6
1.8
2.0
social influence
individualconviction
0.4
0.5
0.6
0.7
0.8
0.0 0.2 0.4 0.6 0.8
1.0
1.2
1.4
1.6
1.8
2.0
social influence
individualconviction
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 8 / 20
26. Wisdom of crowds and social influence
initial conditions matter
aggregate information
full information
0.005
0.010
0.015
0.020
0.025
0.0 0.2 0.4 0.6 0.8
1.0
1.2
1.4
1.6
1.8
2.0
social influence
individualconviction
0.4
0.5
0.6
0.7
0.8
0.0 0.2 0.4 0.6 0.8
1.0
1.2
1.4
1.6
1.8
2.0
social influence
individualconviction
Effects of social influence modulated by initial conditions
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 8 / 20
27. Designing wise crowds
Recover the wisdom of crowds without individual expertise
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 9 / 20
28. Designing wise crowds
Recover the wisdom of crowds without individual expertise
c DeSciL
⇒
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 9 / 20
29. Designing wise crowds
Recover the wisdom of crowds without individual expertise
c DeSciL
⇒
Mechanisms: ranking and competition
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 9 / 20
30. Designing wise crowds
Groups consistently converge toward the right solution
⇒
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 10 / 20
31. Designing wise crowds
Groups consistently converge toward the right solution
⇒
Wisdom of crowds: low collective error + high group diversity
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 10 / 20
32. Designing wise crowds
competition drives micro-dynamics
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 11 / 20
33. Designing wise crowds
competition drives micro-dynamics
initial conditions affect speed of convergence
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 11 / 20
34. Designing wise crowds
relevant when objective truth
exists but is unknown
c Safecast
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 12 / 20
35. Designing wise crowds
relevant when objective truth
exists but is unknown
c Safecast
Mechanisms to harness WoC in unknown solution space
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 12 / 20
36. Resilience of online social networks
rational users
Θ[Bi(H) − c] ⇒ leave or stay
departure
cascades
⇒
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 13 / 20
37. Resilience of online social networks
rational users
Θ[Bi(H) − c] ⇒ leave or stay
departure
cascades
⇒
ks
LivejournalFriendster1
100
200
304
Reveal deeply connected
cores
Quantify resilience
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 13 / 20
38. Resilience of online social networks
Comparing resilience
100
101
102
ks
10-4
10-3
10-2
10-1
100
P(ks>K)
Friendster
Facebook
Orkut
MySpace
LiveJournal
Existence of tight communities
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 14 / 20
39. Resilience of online social networks
Comparing resilience
100
101
102
ks
10-4
10-3
10-2
10-1
100
P(ks>K)
Friendster
Facebook
Orkut
MySpace
LiveJournal
Existence of tight communities
Topology alone is not enough to
explain success or failure
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 14 / 20
40. Resilience of online social networks
Comparing resilience
100
101
102
ks
10-4
10-3
10-2
10-1
100
P(ks>K)
Friendster
Facebook
Orkut
MySpace
LiveJournal
Existence of tight communities
Topology alone is not enough to
explain success or failure
The collapse of Friendster
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 14 / 20
41. Mitigating negative consequences
Information transfer as mechanism for collective coordination
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 15 / 20
42. Mitigating negative consequences
Information transfer as mechanism for collective coordination
Novel application of network theory to study information transfer
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 15 / 20
43. Mitigating negative consequences
Information transfer as mechanism for collective coordination
Novel application of network theory to study information transfer
1. 02.06;1:00:00;b1f6
2. 02.06;1:00:20;1ac4
3. 02.06;1:01:00;1ac4
4. 02.06;1:01:01;ed81
⇒
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 15 / 20
44. Mitigating negative consequences
Information transfer as mechanism for collective coordination
Novel application of network theory to study information transfer
1. 02.06;1:00:00;b1f6
2. 02.06;1:00:20;1ac4
3. 02.06;1:01:00;1ac4
4. 02.06;1:01:01;ed81
⇒
Network generation processes to infer individual recruitment
mechanisms and to test for systemically important individuals
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 15 / 20
49. Summary
Understanding the interplay between social influence and wisdom
of crowds
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 17 / 20
50. Summary
Understanding the interplay between social influence and wisdom
of crowds
agent-based modelling and experimental verification
redeeming wisdom of crowds with practical relevance
Departure cascades as unintended effects of individual actions
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 17 / 20
51. Summary
Understanding the interplay between social influence and wisdom
of crowds
agent-based modelling and experimental verification
redeeming wisdom of crowds with practical relevance
Departure cascades as unintended effects of individual actions
Inferring individual rules in achieving collective coordination
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 17 / 20
52. Summary
Understanding the interplay between social influence and wisdom
of crowds
agent-based modelling and experimental verification
redeeming wisdom of crowds with practical relevance
Departure cascades as unintended effects of individual actions
Inferring individual rules in achieving collective coordination
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 17 / 20
53. Scientific contributions
1 Complexity as an irreducible component of social systems
simple individual rules can lead to unexpected collective outcomes
(collapse of OSNs, mental trap of more information, social influence)
bottom-up control of social systems: design interactions to steer
groups (ranking and competition)
cautious intervention vs. ideological dogmas in interfering with
complex systems
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 18 / 20
54. Scientific contributions
1 Complexity as an irreducible component of social systems
simple individual rules can lead to unexpected collective outcomes
(collapse of OSNs, mental trap of more information, social influence)
bottom-up control of social systems: design interactions to steer
groups (ranking and competition)
cautious intervention vs. ideological dogmas in interfering with
complex systems
2 One of the first attempts to study information transfer from a
network perspective
rigorous methods to infer patterns from inherently noisy data
inspire novel experimental set-ups to test mechanistic explanations
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 18 / 20
55. Scientific contributions
1 Complexity as an irreducible component of social systems
simple individual rules can lead to unexpected collective outcomes
(collapse of OSNs, mental trap of more information, social influence)
bottom-up control of social systems: design interactions to steer
groups (ranking and competition)
cautious intervention vs. ideological dogmas in interfering with
complex systems
2 One of the first attempts to study information transfer from a
network perspective
rigorous methods to infer patterns from inherently noisy data
inspire novel experimental set-ups to test mechanistic explanations
3 Scientific output
3 articles published, 2 submitted, 3 forthcoming
4 External collaborations (University of Greifswald)
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 18 / 20
56. Scientific contributions
1 Complexity as an irreducible component of social systems
simple individual rules can lead to unexpected collective outcomes
(collapse of OSNs, mental trap of more information, social influence)
bottom-up control of social systems: design interactions to steer
groups (ranking and competition)
cautious intervention vs. ideological dogmas in interfering with
complex systems
2 One of the first attempts to study information transfer from a
network perspective
rigorous methods to infer patterns from inherently noisy data
inspire novel experimental set-ups to test mechanistic explanations
3 Scientific output
3 articles published, 2 submitted, 3 forthcoming
4 External collaborations (University of Greifswald)
5 Media attention
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 18 / 20
57. Cooperation through social herding
cooperation is crucial in social
dilemmas
defection is strategically optimal
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 19 / 20
58. Cooperation through social herding
cooperation is crucial in social
dilemmas
defection is strategically optimal
D ⇒ C
(1 − ζ)P(.) + ζS(fC)
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 19 / 20
59. Cooperation through social herding
cooperation is crucial in social
dilemmas
defection is strategically optimal
D ⇒ C
(1 − ζ)P(.) + ζS(fC)
herding - evolved imitation
process
rational under uncertainty
presumes less information
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 19 / 20
60. Cooperation through social herding
DC C
D
D
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 20 / 20
61. Cooperation through social herding
DC C
D
D
⇒
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 20 / 20
62. Cooperation through social herding
DC C
D
D
⇒
efficient mechanism: herding matters most in tie-situations
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 20 / 20
63. Cooperation through social herding
DC C
D
D
⇒
efficient mechanism: herding matters most in tie-situations
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 20 / 20
64. Cooperation through social herding
DC C
D
D
⇒
efficient mechanism: herding matters most in tie-situations
limit information → induce herding → cooperation
Chair of Systems Design Decisions and their unintended consequences Results
www.sg.ethz.ch Pavlin Mavrodiev Doctoral examination D-MTEC June 10, 2014 20 / 20