A summary of feedback received on the AEA Systems in Evaluation Topical Interest Group's 2015 Digital Unconference.

2. Coordinators Bethany Laursen
Kylie Hutchinson
Meg Hargreaves
Jan Noga
Erin Whalen
Ginger Fitzhugh
SETIG Leadership

6. Registered)
Registered)
Registration was much higher than attendance.
Possibly because it was free.
Registered
51#
80#
127#
188#
220#
249#
Day#3#
Day#2#
Day#1#
A/ended#
Registered

7. Most participants attended for one day only.
A"ended&1&day&
67%&
A"ended&2&
days&
24%&
A"ended&
&all&3&days&
9%&

8. Participants found the break-out sessions the “most useful”.
(N=13)
0%#
7%#
14%#
79%#
Cha$ng'with'other'par/cipants'in'the'
main'lobby'
Watching'the'live'broadcast'panel'
discussion'in'the'lobby'
Other'
Par/cipa/ng'in'the'small'break=out'
session'I'a?ended'
0%

9. People participated as they felt comfortable.
(N=14)
7%#
21%#
29%#
43%#
I"didn't"par+cipate"in"a"breakout"
session"
I"listened"
I"mostly"listened,"but"occasionally"
spoke"
I"listened"and"spoke"

10. Yes$
85%$
No$
15%$
Yes$
100%$
The unconference met most people’s expectations.
Day 1
(N=33)
Day 2
(N=24)
Day 3
(N=19)
Yes$
63%$
No$
37%$

11. 4.8$
4.8$
3.8$
5.0$
4.4$
4.0$
4.4$
4.4$
3.6$
3.4$
3.3$
3.0$
0.0# 1.0# 2.0# 3.0# 4.0# 5.0#
Approaches for Capturing High-level Systems Patterns and Outcomes
Support for a Systems Approach: Strategies for Promoting Buy-in
Using a Developmental Evaluation Approach: Real-life Examples
Role of Accountability in Systems Evaluation
Defining and Distinguishing Systems Evaluation: Where are the Boundaries?
Using a Complex Adaptive Systems Approach: Real-life Examples
Identifying System Leverage Points: Helpful Approaches
Link between Culturally Responsive Evaluation & Systems-Oriented Evaluation
Using a Soft Systems Approach: Real-life Examples
Defining System Boundaries: Building Stakeholder Consensus
Visualizing Complex Systems: Tools and Strategies
Incorporating a Systems Approach: Challenges and Strategies
Participants rated some sessions higher on average than
others.
Over all three days on a scale of 1=poor and 5=excellent.

12. People were generally satisfied with most aspects of the
unconference.
With the exception of becoming more connected to colleagues.
9%#
10%#
5%#
9%#
9%#
11%#
15%#
5%#
10%#
37%#
10%#
5%#
36%#
41%#
50%#
65%#
32%#
50%#
40%#
59%#
50%#
32%#
15%#
16%#
25%#
55%#
The$registra+on$process$was$straigh1orward.$
I$was$provided$with$the$informa+on$I$needed$prior$to$
the$unconference.$
I$experienced$no,$or$rela+vely$few$problems,$using$the$
MIT$Unhangout$technology.$
I$increased$my$knowledge$of$this$topic.$
I$feel$more$connected$to$my$colleagues$aAer$
par+cipa+ng$in$the$unconference.$
The$unconference$provided$me$with$good$value$for$the$
+me$spent.$
I$would$aBend$another$online$unconference$hosted$by$
the$Systems$in$Evalua+on$TIG.$
Strongly$disagree$ Disagree$ Neutral$ Agree$ Strongly$Agree$

13. • software used (MIT’s Unhangout)
• having participants visible
• ability to chat and share resources in sidebar
• facilitators for sessions

14. “I am so impressed with this experience!
What a low cost, low resource way to meet
with like-minded colleagues, and have deep
discussions about relevant things! I am
totally jazzed up to use what I learned in my
work and to connect with people I’ve met in
breakout chats.”

15. “This was a great experience - well
organized, technology worked perfectly
and the conversation produced was very
fruitful.”
“This was an excellent event. Thought
the format was just perfect for fitting into
my life, but also brought deep and
meaningful conversation about an
important topic.”

16. • tech problems
• accessing Unhangout
• accessing breakout room
• lack of testing and preparation
• no video or mic capability
• the usual technical glitches
• low attendance in some sessions
• low participation in some sessions
• expectations for a traditional webinar
Challenges

17. would attend another
unconference by SETIG
95%

18. would recommend
a similar-style event
to a friend or colleague
79%

19. Breakout facilitators similarly enjoyed the experience.
And would do it again if asked.
8%#
18%#
17%#
9%#
33%#
42%#
50%#
27%#
36%#
25%#
50%$
50%$
50%$
55%$
55%$
75%$
Enjoyed$the$experience$of$facilita7ng$
Few$technical$problems$while$facilita7ng$
Training$session$for$facilitators$was$helpful$
Able$to$upload$summary$points$easily$
Would$volunteer$again$if$asked$
My$role$and$expecta7ons$were$made$clear$
Disagree$ Neutral$ Agree$ Strongly$Agree$

20. “It was a great idea to do this. I hope
you’ll do it again!”
“I enjoyed the opportunity a lot.”
“Great modelling of risk taking with grace.”

21. Next Time
• no last minute registrations
• mandatory orientation session
• test connection ahead of time
• make participant expectations and
facilitator role more explicit
• shorten unconference to 2 days
• fewer sessions to fill breakout rooms
• crowd source session topics ahead of time
• shorten sessions to 45-50 minutes
• more extended descriptions of sessions

22. • designate beginner vs. more intermediate
sessions
• one facilitator and one note-taker per session
• share discussion prompts with participants
ahead of time
• shrink Participants Guide to one page cheat
sheet
• strongly recommend use of headset with mic
• display session title while in breakout room
• ask for 3 key take-aways from each session
• accept that some will always want to lurk
Next Time (cont’d)

23. 3 things you learned
2things you want to learn more about
1thing you’ll use going forward

24. Smaller more regular unconferences that:
• focus on one specific question or area
• focus on one problem from a systems lens
Other SETIG Event Ideas