2. HuskyHunt 1
Contents
Title ................................................................................................................................................. 2
Instructional Overview.................................................................................................................... 3
Purpose/Objectives.......................................................................................................................... 4
Formative Assessment .................................................................................................................... 4
Data Section.................................................................................................................................... 6
Player Overview and Demographic Information ........................................................................ 6
Engagement Data ........................................................................................................................ 8
Knowledge Transfer Data ......................................................................................................... 13
Pre-Survey Data ........................................................................................................................ 15
Post-Survey Data....................................................................................................................... 17
Analysis and Results Section........................................................................................................ 18
Reflections Section ....................................................................................................................... 24
Future Recommendations Section ................................................................................................ 26
Appendices.................................................................................................................................... 28
Appendix A - Social Network Reach........................................................................................ 28
Appendix B - Game Schedule and Module Format .................................................................. 29
Appendix C - Sample Player Badge Profile Page ..................................................................... 30
Appendix D - List of All Available Badges.............................................................................. 31
Appendix E - Introductory Video Script................................................................................... 35
Appendix F - Pre-Game and Post-Game Survey Questions ..................................................... 36
Appendix G - Awards Ceremony Agenda and Interview Questions ........................................ 38
Appendix H - Awards Dinner Interviews Transcriptions ......................................................... 39
3. HuskyHunt 2
Title
As the Chief Information Security Officer for the University of Connecticut one of my
responsibilities is providing security awareness training to faculty, staff and students. Because
the University has chosen not mandate this training I need to seek out opportunities to educate
and am required to find ways to encourage people to take training. This challenge led me to
create HuskyHunt.
HuskyHunt is an information security awareness training game and is intended to meet
the following specific goals:
Teach students practical lessons about Internet and information security awareness
topics that they can relate to personally
Utilize gaming mechanics to increase engagement and participation
Leverage Social media, specifically Facebook and Twitter to reinforce learning that
occurred in the game modules.
Leverage social media to increase the likelihood that UConn students not
participating in HuskyHunt are exposed to the security training.
The technology integration project was designed to use the HuskyHunt framework to
measure the impact that digital badges (“Digital badges - Wikipedia, the free encyclopedia,”
n.d.) have on player engagement, learning transfer and players attitudes regarding technology
and security.
The game was originally designed without digital badges and was redesigned for this
project. 17 digital badges were designed for the game and the game was configured to randomly
distribute badges to players at registration time. Badges and descriptions can be seen in
Appendix D – List of All Available Badges.
The game was developed to be both desktop and mobile friendly which proved important
in supporting both the learning modules and scavenger hunt.
Social networks, Facebook and Twitter, were used for three reasons:
Communicate of game events to players in a medium they prefer
Provide us with a longer term capability of communicating other information to students
Expand the reach of security information to friends of players not playing the game
(Appendix A – Social Network Reach)
Briefly HuskyHunt is a game consisting of 2 primary components, online learning modules
and real world scavenger hunt modules. The game has incentives structured throughout the
game, with grand prizes of $500, $100, and $50 to the UConn CO-OP and periodic incentives
provided by local business sponsors. The game was originally designed solely as a security
training tool and is now partnered with the OneCard Office and includes content related to their
functions.
4. HuskyHunt 3
HuskyHunt started on October, 6th 2014 and ran through October, 26th 2014. The game
was broken down into 3 weekly challenges. Each week included 6 daily online learning modules
and culminated with a physical scavenger hunt. Players were alerted to new content via postings
to social networks, specifically Facebook and Twitter.
HuskyHunt, as a learning tool leveraging gaming mechanics and digital badging, is
directly in line with the 2014 K-12 Horizon Report that suggests that the next three years will
show a significant increase in the amount of game based learning tools.
Instructional Overview
I don’t have a classroom, but have an assigned and ethical responsibility to try and teach
greater than 20,000 students about basic technology security concepts. HuskyHunt is designed to
be delivered at a large scale and has no technical limitations to the number of individuals that can
participate in an available and active course. The framework aligns most closely to an xMOOC,
which is geared towards an individual seeking an achievement, is likely offered in addition to
other complementary education and is designed to be repeated (“elearnspace › MOOCs are really
a platform,” n.d.).
HuskyHunt was developed to appeal, primarily, to University students. There are a
variety of challenges connecting with this demographic.
Students are geographically dispersed across seven campuses
Semester timeframes and varying course loads make getting sustained attention
difficult
Marketing/Advertising challenges
HuskyHunt was developed and implemented to deliver “the entities, relations and
attributes that the learner must ‘know’” (Duffy & Jonassen, 1992 location 192). It also shares
similarities with a current standards based curriculum concepts (Carr-Chellman, 2011)
leveraging directed instruction and assessments. HuskyHunt establishes clearly communicated
learning objectives and leverages online multiple choice assessments designed to test specifically
what was taught in the lesson.
The game was designed to collect assessment and behavioral data throughout the game,
for the purpose of aligning with the ISTE Standards for teachers to “provide students with
multiple and varied formative and summative assessments aligned with content and technology
standards, and use resulting data to inform learning and teaching.” (“ISTE Standards for
Teachers,” n.d.) The whole intention of the game is to educate and improve reach and quality for
each semester.
The curriculum was intentionally re-designed to meet the following criteria outlined in
the ISTE Standards for Teachers (“ISTE Standards for Teachers,” n.d.):
1. “Design and develop digital age learning experiences and assessments.
Communicate relevant information and ideas effectively to students....using a
5. HuskyHunt 4
variety of digital age media and formats.” HuskyHunt utilized videos, mobile
devices, online content, social media, online surveys, digital badges and QR codes
to enable players to interact and engage with the game.
2. Promote and model digital citizenship and responsibility “Teachers understand
local and global societal issues and responsibilities in an evolving digital
culture…”
a. Advocate, model, and teach safe, legal, and ethical use of digital
information and technology, including respect for copyright, intellectual
property, and the appropriate documentation of sources
b. promote and model digital etiquette and responsible social interactions
related the use of technology and information
The game learning modules, listed in Appendix B – Game Schedule and Module Format,
were designed to address both items a. and b. above.
Purpose/Objectives
The purpose of this study was to evaluate the affect that awarding digital badges to
players would have on game completion, task completion and engagement. Training materials,
such as information security awareness often has low participation and completion rates but are
considered useful tools for ensuring that students or employees are taught relevant information. I
assert that these tools can actually be made to be effective if design elements, such as digital
badges, are designed and incorporated to increase a participants motivation to complete the
training. I believe that learning transfer can be increased through more active participation with
the content. P.R. Subramanium Says that ‘Task design certainly plays a major role in impacting
situational interest” and that “By increasing the interestingness of the learning task, even students
with different individual dispositions are more likely to exhibit interest and be actively engaged.”
(Subramaniam, 2009). It is this specific idea that I sought to support with this implementation of
HuskyHunt.
As additional support for the idea that digital badges can increase engagement I
considered this New York City program where schools implemented a digital badge award
program that is described as having positive results (“Digital Badges for Microcredentials,
Student Engagement & Persistence | Online Learning Consortium, Inc,” 2012). I designed this
version of HuskyHunt such that approximately half of the players would receive digital badges
for certain achievements and other players would not. It is assumed that greater engagement
would lead to higher completion rates and correspondingly higher learning transfer.
Formative Assessment
The following qualitative data were collected:
Pre-Game Survey
There was a short, six question, optional survey provided as the first online activity of the
game. All players had the option of taking the survey and it was made available at the start of
the game, on October, 6th immediately following the introductory video. The survey was
available to any new player starting on or after October 6th.
6. HuskyHunt 5
Post-Game Survey
There was an eight question optional survey provided as the final activity of the game.
The survey was only available to players that completed all of the modules and was presented
only on October 26th, the last day of the game.
The complete list of all Pre-Game and Post-Game survey questions are listed in
Appendix F – Pre-Game and Post-Game Survey Questions. An analysis of survey results are
described in the analysis and results section below.
Awards Dinner Discussions
There was an awards dinner held on October 29th. The agenda and format of this event is
described in Appendix G – Awards Ceremony Agenda and Interview Questions. I took
advantage of the time players were arriving to engage in informal discussions to get a general
sense of their experiences. I also arranged for a group of players to remain after the dinner and
engaged in a more structured discussion. A summarized transcription is listed in Appendix H –
Awards Dinner Interviews Transcriptions . This conversation was captured in both audio and
video format and the transcription was made from the recorded conversation.
The following quantitative data elements were collected:
From game interaction and authentication sources:
NetId, Name, University Affiliation, University Status, Badges Enabled, Academic Year,
Gender, At least one module completed, Furthest Module Completed, Pre-Game Survey
Completed, Post-Game Survey Completed, Scavenger Hunt Participation, Created
Player Alias, Posted to Social Media, Completed Every Module, Completed Final
Assessment, Entered Credentials in Final Assessment.
The source data listed above was utilized to generate the following statistics. These
statistics were designed to be collected and compared to determine engagement, skills
development and attitude changes:
% of players completing each module
% of players remaining after each module
% of players completing all modules
% of players abandoning game
% of players abandoning by module
Total players receiving all badges
Total players receiving at least 1 badge
Total players that checked the badge description page
How far average player with badges got
How far average player without badges got
Total unique scavenger hunt participants
% of players completing post-game survey
% of players completing pre-game survey
% players changing profile name
% of players sharing in social media
7. HuskyHunt 6
Average times needed to answer an assessment question correctly
Average Final Assessment Score
% of people providing username and password (final assessment criteria)
Player’s thoughts and knowledge about Internet and information security before and after
the game.
Player’s thoughts on how badges influenced their desire to play the game.
Players attitudes towards the technologies and concepts used in the game
Data Section
Player Overview and Demographic Information
The data in this section is meant to describe player statistics and demographics. The
intention with gathering this data was ensure a complete understanding of overall player
breakdown so that any inferences from the data could be compared with the general player
information.
Figure 1
Figure 1. Illustrates the total game registrants. This data was generated from all players that
registered, but may not have played, the game.
Total
Registered Players 336
Male Registered 153
Female Registered 183
Male Registered % 45.54%
Female Registered % 54.46%
0
50
100
150
200
250
300
350
400
Total Game Registrations
8. HuskyHunt 7
Figure 2
Figure 2. Illustrates only the active players. This data was generated from all players who
participated in at least one game module. This is the population from which all future inferences
will be derived.
Figure 3
Figure 3. Illustrates the breakdown of active players with and without badges. It is important to
recognize that nearly 70% of players had badges and 30% did not.
Total
Active Players 192
Male 85
Female 107
Male Active % 44.27%
Female Active % 55.73%
0
50
100
150
200
250
Total Active Players
Total
Active Players 192
With Badges 134
W/O Badges 58
With Badges % 69.79%
W/O Badges % 30.21%
0
50
100
150
200
250
Total Active Players BadgeStatus
9. HuskyHunt 8
Figure 4
Figure 4. Illustrates the breakdown of active players by University affiliation. The intention of
gathering this data was primarily to gain a more thorough understanding of players and help
guide future player recruiting efforts.
Players interacted with the game using both mobile devices such as cell phones (Android
and iOS primarily), iPods and full computers (Mac, PC). There was no effort made to
distinguish between laptops or desktops. Game modules produced a Mobile/Computer ratio
of: 33.02%:66.98% while the scavenger hunt produced a Mobile/ Computer ratio of
65.09%:34.9. The ratio for the scavenger hunts is expected, but the ratio for the online modules
is skewed somewhat unexpectedly towards computers as we had assumed that most people
would interact with the game using their mobile devices.
Engagement Data
The following data is designed to describe how players interacted with the game, and
provide clarity regarding the breakdown of players with badges and players without badges. All
of the following data is based on active players (players playing at least one module) not
registered players. Recalling the data above, 69% of players had badges enabled and 31% of
players did not.
192
20 10.42% 71 36.98% 28 14.58% 64 33.33% 7 3.65% 2 1.04%
0
500
Total
Active Players By University Affiliation
Active Players Freshman Freshman % Sophomore
Sophomore % Junior Junior % Senior
Senior % Graduate Student Graduate Student % Other
Other %
10. HuskyHunt 9
Figure 5
Figure 5. Illustrates the active players by module. This clearly shows a steady decrease of player
activity over the course of the game. The decrease, however, is statistically consistent between
players with and without badges throughout the game.
Figure 6
Figure 6. Illustrates the players that participated in the optional pre-game survey. These
numbers are excellent with an 82.2% completion rate and likely influenced by the fact that the
survey immediately followed the introductory video in which I specifically request they fill out
the survey.
0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Total Active Players 205 152 134 124 120 106 104 96 90 86 85 77 77 74 70 68 65 24 35
Active Players with Badges 130 104 93 84 82 70 69 63 61 58 57 51 51 48 45 44 41 18 23
Active Players without Badges 75 48 41 40 38 36 35 33 29 28 28 26 26 26 25 24 24 6 12
ActivePlayers
Game Level
Active Players by Module
Total Active Players Active Players with Badges Active Players without Badges
Total
Total Players 158
With Badges 109
With Badges % 68.99%
W/O Badges 49
W/O Badges % 31.01%
0
20
40
60
80
100
120
140
160
180
Pre-GameSurvey Participants
11. HuskyHunt 10
Figure 7
Figure 7. Illustrates the players that participated in the optional post-game survey. The response
rate of 93% is excellent, and is even higher than the response rate for the pre-survey questions.
This is one of the more positive indicators that players had a good game experience and were
engaged.
Figure 8
Figure 8. describes players attitudes regarding the digital badges, after game completion.
Total
Total Players 27
With Badges 19
With Badges % 70.37%
W/O Badges 8
W/O Badges % 29.63%
0
5
10
15
20
25
30
Post-GameSurvey Participants
0
1
2
3
4
0 5 10 15 20 25 30
Post-Survey Question 8
Do you feel that having badges inthegame made you
more likely to play?
1 - Yes
2 - No
3 - There wereno badges
12. HuskyHunt 11
Figure 9
Figure 9. Illustrates the players that participated in the scavenger hunt.
Figure 10
Figure 10. Illustrates the players that created in-game nicknames (aliases). This data represents
another positive engagement metric as 35% of players explored game settings and identified that
they could create aliases.
Total
Total Players 26
With Badges 15
With Badges % 57.69%
W/O Badges 11
W/O Badges % 42.31%
0
5
10
15
20
25
30
Scavenger HuntStatistics
Total
Total Players 62
With Badges 43
With Badges % 69.35%
W/O Badges 19
W/O Badges % 30.65%
0
10
20
30
40
50
60
70
Alias Creation Statistics
13. HuskyHunt 12
Figure 11
Figure 11. Illustrates the players that shared content to their social media networks. This data
represents another positive engagement metric as 38% of players were willing to share
preconfigured posts to their community.
Figure 12
Figure 12. Illustrates the players that completed all modules. This total, 13 people (6%) is much
lower than expected and clearly lower than the expectations (as indicated by Pre-Survey
Question 6) of players prior to the start of the game.
Total
Total Players 73
With Badges 53
With Badges % 72.60%
W/O Badges 20
W/O Badges % 27.40%
0
10
20
30
40
50
60
70
80
Social Media Sharing Statistics
Total
Total Players 13
With Badges 8
With Badges % 61.54%
W/O Badges 5
W/O Badges % 38.46%
0
2
4
6
8
10
12
14
Players Completing Entire Game
Total Players
With Badges
With Badges %
W/O Badges
W/O Badges %
14. HuskyHunt 13
Figure 13
Figure 13. Illustrates the players that completed the final assessment.
Figure 14
Figure 14. Describes player’s thoughts, entering the game, regarding their expectation of
completing the game.
Finally, an example of the scoreboard, showing the current badge status of all players is
available in Appendix C - Sample Player Badge Profile Page. This was designed to provide a
current status of earned and unearned badges and it was hoped that it would increase players
motivation.
Knowledge Transfer Data
Total
Total Players 29
With Badges 20
With Badges % 68.97%
W/O Badges 9
W/O Badges % 31.03%
0
5
10
15
20
25
30
35
Players Completing Final Assessment
0
0.5
1
1.5
2
2.5
3
3.5
0 50 100 150 200 250
Pre-Survey Question 6
Do you expect to completethewholegame?
1 -Yes
2 -No
3 -Maybe
15. HuskyHunt 14
Figure 15
Figure 15. Illustrates the average levels completed. The game was unable to keep players
engaged for at least 50% of the game and players with badges actually abandoned 1.5 modules
earlier than those without.
Players answered 80.28% of the module assessment question correctly on the first
attempt. There were 5188 total question attempts and 4165 correct attempts. Players were
allowed as many chances as necessary to answer questions correctly, as the primary goal of
HuskyHunt is to train rather than assess.
The average final assessment score was 74.8, which represents an average of 82.33 points
out of a possible 110. Players were not allowed to answer questions more than one time for the
final assessment, as the intention here was to provide score differentiation and produce a clear
winner.
The average time to answer a question was eight seconds indicating that the question difficulty
was too easy.
Figure 16
Total
Total levels 18
Average Completion 7.78
Average Completion With Badges 7.32
Average Completion W/O Badges 8.8
0
5
10
15
20
AverageLevelCompleted
Total
Total Players 11
With Badges 6
With Badges % 54.55%
W/O Badges 5
W/O Badges % 45.45%
0
2
4
6
8
10
12
Players Providing Credentials
16. HuskyHunt 15
Figure 16. Illustrates the players that completed the final assessment, but provided their
username and password when asked.
Pre-Survey Data
Figure 17
Figure 17. Shows players that completed pre-game survey: 244/338. There were 192 active
players, but 336 registrants that could have participated in the surveys.
Figure 18
0
1
2
3
4
0 50 100 150 200 250
Pre-Survey Question 1
0
0.5
1
1.5
2
2.5
3
3.5
0 50 100 150 200 250
Pre-Survey Question 3
Why are you playing thegame?
1 - To learn about information securityandprivacy
2 - To win prizes
3 - Saw it advertised anddecidedto tryit
How would you describeyour attitudetowards
security?
1 - I am very security conscious
2 - I know thereare risks but am not concerned
3 - I don't think about securityat all
17. HuskyHunt 16
Figure 19
Figure 20
Figure 20. Provides useful contextual information regarding players technology attitudes and can
be utilized to inform future implementation decisions.
0
0.5
1
1.5
2
2.5
0 50 100 150 200 250
Pre-Survey Question 4
0
0.5
1
1.5
2
2.5
3
3.5
0 50 100 150 200 250
Pre-Survey Question 5
How would you describeyour attitudetowards
personal privacy?
1 - I am very conscious ofmy privacy
2 - I know thereare risks but am not concerned
3 - I don't think about privacy atall
How would you describeyour attitudetowards
technology?
1 - I am a technology enthusiast
2 - I use technology
3 - I don't like technologybut am requiredto use it
18. HuskyHunt 17
Post-Survey Data
Figure 21
Figure 21. Shows Players that completed post-game survey: 27/29. There were 29 players that
completed the final assessment, the post-game survey was presented after the final assessment.
Figure 22
0
0.5
1
1.5
2
2.5
0 5 10 15 20 25 30
Post-Survey Question 2
0
0.5
1
1.5
2
2.5
3
3.5
0 5 10 15 20 25 30
Post-Survey Question 3
Did you learn something about security,privacy or
technology from the game?
1 - Yes, I learneda lot ofusefulinformation
2 - Yes, but I knew mostoftheinformationalready
3 - No
Did you attituderegarding security change?
1 - Yes
2 - Somewhat
3 - No
19. HuskyHunt 18
Figure 23
Figure 24
Analysis and Results Section
The data collected was a mix of quantitative data and qualitative data. The majority of
the quantitative data I collected and evaluated was stored in an Excel document called
‘HuskyHunt Demographic and Game Data’ which contained the following data elements:
NetId, Name, University Affiliation, University Status, Badges Enabled, Academic Year,
Gender, At least one module completed, Furthest Module Completed, Pre-Game Survey
Completed, Post-Game Survey Completed, Scavenger Hunt Participation, Created
Player Alias, Posted to Social Media, Completed Every Module, Completed Final
Assessment, Entered Credentials in Final Assessment.
0
0.5
1
1.5
2
2.5
3
3.5
0 5 10 15 20 25 30
Post-Survey Question 4
0
0.5
1
1.5
2
2.5
3
3.5
0 5 10 15 20 25 30
Post-Survey Question 5
Did your attituderegarding personalprivacy change?
1 - Yes
2 - Somewhat
3 - No
Did your attitudetowards technologychange?
1 - Yes
2 - Somewhat
3 - No
20. HuskyHunt 19
The data was structured in this format so that the filtering and graphing functions of
Excel could be utilized to produce most of the information provided in the Data Section and was
intentionally collected to ensure that the questions identified in the Formative Assessment
section could be answered.
The qualitative data, primarily the responses to the Pre and Post Game Surveys and the
interviews that occurred at the awards dinner were all collected electronically and stored in the
HuskyHunt Demographic and Game Data Excel file as the following elements:
Pre-Survey Question 1, Pre-Survey Question 2, Pre-Survey Question 3, Pre-Survey
Question 4, Pre-Survey Question 5, Pre-Survey Question 6, Post-Survey Question 1,
Post-Survey Question 2, Post-Survey Question 3, Post-Survey Question 4, Post-Survey
Question 5, Post-Survey Question 6, Post-Survey Question 7, Post-Survey Question 8
The survey results were collected in Excel spreadsheets and graphed. The interviews
were all recorded and I created transcriptions in Word for each conversation. I utilized these
interviews as an anecdotal account in an effort to get game feedback and see if those results align
with the quantitative data collected.
The online survey data, figures 16 through 23, provided some interesting information for
game improvements and some optimism that the game can provide real learning value. Figure
16 clearly indicates that players are joining the game because they are interested in winning
prizes and that the topic itself is not the motivating factor. The same respondents
overwhelmingly indicate that they are concerned about security and privacy, but that concern is
not enough to sustain long term interest in the game. The post-game surveys do suggest that the
game did provide value for those that played it to completion. The responses to questions 2-5
(figures 20-23) indicate that the majority of players are thinking differently about security and
privacy as a result of the game. These results are supported by the in person conversations I had
with players at the awards dinner.
The quantitative data shows very little difference in engagement or knowledge gain
between players that received badges and those that didn’t. A close examination of the data
shows that approximately two of every three players were configured to receive badges. That
ratio remains close, with any differences being insignificant, for every category measured as
illustrated in the following graphs:
22. HuskyHunt 21
Figure 29
Figure 30
Figure 31 Figure 32
72.60%
27.40%
Players Sharing on Social
Media
Ratio- 2.64:1
Players With Badges %
Players W/O Badges %
69.35%
30.65%
Players Creating an Alias
Ratio- 2.26:1
Players With Badges %
Players W/O Badges %
61.64%
38.46%
Players Completing All
Modules
Ratio- 1.6:1
Players With Badges %
Players W/O Badges %
68.97%
31.03%
Players Completing Final
Assessment
Ratio- 2.22:1
Players With Badges %
Players W/O Badges %
23. HuskyHunt 22
Figure 33
The data shows that badges did not have any positive impact in engagement. In fact most
data show players without badges progressed further and participated more. In all categories
players with badges participated at approximately a 2:1 ratio to those without badges, which is
consistent with the overall badge distribution. Furthermore, the categories of scavenger hunt
statistics, all modules completed actually show a ratio approaching 1:1 indicating that players
without badges had a slightly higher engagement level than those with badges. Finally, the data
shows that of the 18 possible game levels players with badges abandoned the game by level 7.32
while players without badges abandoned the game by level 8.8 indicating that players without
badges were more engaged in completing the game.
The only category where there is some evidence that players with badges performed
better than those without is ‘players providing credentials.’ Fewer players with badges provided
their username/password combination than those without, however there were only 11 total
players that provided their credentials so the comparison of this data is likely statistically
insignificant.
The quantitative data describes similar results. The players interviewed at the awards
dinner had mixed opinions regarding the influence badges had on gameplay with most players
indicating that the badges had no influence on their desire to play the game. Two players,
however, indicated strongly that they liked receiving the badges and felt some amount of reward
when they received a badge. These same players suggested that badges could have been
displayed more prominently and could have provided additional extrinsic motivation if other
awards were associated with them.
I collected demographic data related to student status (Freshman-Senior) and Sex.
Evaluating a small portion of data to compare game interactions between Male and Female
provided the following results:
Players Failing Final
Assessment
Ratio- 1.2:1
Players With Badges %
Players W/O Badges %
24. HuskyHunt 23
Figure 34
Figure 34. Illustrates the total overall player distribution by sex.
Figure 35
Figure 35. Illustrates the player distribution that participated in the optional pre-game survey.
Figure 36
Figure 36. Illustrates the player distribution that participated in the Post-Game Survey.
44.27%
55.73%
Active Player Distribution by Sex
Male Players % Female Players %
52.29%47.71%
Pre-GameSurvey Completion By Sex
Male Players With Badges % Female Players With Badges
52.63%47.37%
Post-GameSurvey Completion By Sex
Male Players With Badges % Female Players With Badges
25. HuskyHunt 24
Figure 37
Figure 37. Illustrates the players that created a game alias.
Because these sample results did not show any statistically significant differences I chose
not to pursue additional results using student status as I expect the results to be similar. I had
expected to see a difference between males and females creating aliases and was surprised to see
no difference.
The final assessment metric is the single most concerning because 3 of the 18 modules
were related to password management in some way and the 11 people that submitted their
credentials represents 37.9% of the players that took the final assessment
Reflections Section
It was my expectation prior to starting the game that digital badge awards would provide
significantly improved incentive to play the game, which would result in greater learning transfer
due to engagement. While the data indicates no significant difference between the players
populations I continue to believe that awarding digital badges would have a positive impact on
game engagement. More accurate data to support this theory could be gathered if the following
game changes were made:
1. The digital badges need to be displayed on the users score and profile more prominently.
2. There was a digital badge awarded at the end that was transportable outside the game.
This would increase badge value by enabling players to leverage this training experience
in places like LinkedIn
3. The game was run at multiple institutions so that a greater sample set of data could be
collected.
51.16%
48.84%
Players Creating Aliases By Sex
Male Players With Badges % Female Players With Badges
26. HuskyHunt 25
4. The game was run with entire player populations receiving, or not receiving badges. I
believe having a mixed population in the same game instance confuses the purpose and
significance of badges.
One of the challenges of the data collection was related to the fact that there was a coding
error that allowed players to register for the game, and become active, prior to the actual start of
the game. While this didn’t affect gameplay or scoring it caused the ratio between players with
badges and those without to be approximately 70:30 while it was intended that the ratio would be
50:50. We did not have the luxury of fixing this without unregistering players and we felt it was
more important to have players register than it was to restart and risk potentially losing players.
I also think that doing individualize interviews would have been more effective and
accurate than doing a roundtable interview. Individualized interviews would have provided
more data points and a greater ability to attempt to identify trends that I was able to do with the
data collected from a group.
This version of the game included an introductory video of me and Jonathan Moore
describing the game. I wanted players to be able to understand why the game was being run and
get a sense of who was running. However, I do think that this video may have caused a few of
the players to provide their credentials at the final assessment due to a certain established level of
trust created by the video. I debated, for years, whether a practical final assessment of asking
people for their username and password a good idea and continue to be concerned about it. It did
provide an excellent (and alarming) data point that I do believe would remain statistically
consistent across a larger player population, but I’m not actually sure that our implementation of
that component taught players anything about the importance of password secrecy and may
actually have been discouraging.
I underestimated the time required to create a high quality video, and game launch timing
forced me to use a video that I wasn’t particularly happy with. The video script content was
good (Appendix G – Awards Ceremony Agenda and Interview Questions), but the video setting
was stale and my demeanor was too formal. This s can be addressed in future versions now that
I better understand how to approach producing a video. In general I found collecting video for
this project to be challenging due to lack of good equipment and the ad-hoc nature of many of
the discussions regarding game planning and implementation.
I feel like the data collected through electronic processes was adequate and sufficient to
address the questions I had when initially considering the scope of the project and I think it could
have been especially interesting if the overall player population was larger. However as I review
the data I have I continue to identify questions that would be useful to aid in future game
development that cannot be answered by the data I’ve currently collected. I believe I’ve
approached some of the questions too narrowly and addressing that limitation for future
implementations of the game would produce more interesting data. I am also confident that I
would be able to provide much better data and draw more complete correlations, with the data I
have, if I had a stronger background in statistics.
I should have considered additional ways to collect more qualitative data. I placed a
greater emphasis on application generated data (and continue to believe this provides the best
data) however as I listened and transcribed the interviews I did I realized that actual personal
27. HuskyHunt 26
accounts are very helpful in making improvements. I missed opportunities, during scavenger
hunts, to be able to talk to game players and get additional anecdotal game information. I also
found running the formal focus group to be a different experience that I had expected. I tried to
keep the conversation structured, but informal, and am not sure that is the most effective
approach to obtaining high quality data. It would likely be more effective to interview each
person individually and it would enable more complete data collection across all questions and
prevent participant’s responses from being influenced by other people.
Overall, though, I feel that this project went extremely well and the data I collected does
provide me valuable insight as I look to do design and gameplay updates. While I did not obtain
the results I expected or hoped for I feel like I’ve identified a tool, in HuskyHunt, that can be
leveraged to continue pursuing my theory that digital badges can and do provide improved
engagement and learning transfer.
Future Recommendations Section
The data indicates HuskyHunt can be an effective training tool. In the current model of
MOOC’s it is rare to get a completion rate above 1% (although participant numbers are much
larger) and HuskyHunt, even with its modest active player numbers, obtained 15% completion. I
realize that the incentives contribute to that total, but it still significantly higher than an average
MOOC completion percentage. I’m confident that the participation could be increased if we
improve our scoring such that players can catch up if they fall behind. Making the scavenger
hunt occur online would also help, as the current requirement to be a Storrs student is limiting
participation.
However, the player numbers do have to be increased to justify the effort of running the
game. The advertising, coordination, content generation and scavenger hunts require significant
staff time and cannot continue if the player counts don’t increase. Partnering with the OneCard
office has shown that the game can easily be extended beyond information security content and
other departments have shown interest in using the tool as well. If we can tie the start of the
game into the start of the semester, especially the fall semester, and leverage the framework as a
training tool for other departmental activities that occur at that time I believe we could
significantly increase participation in the game and provide greater institutional value. My goal
is simply to expose as many students as possible to critical security concepts, but recognize that
this is not high on their list of concerns. I’m willing to explore significantly expanding content
beyond information security if it enables me to ultimately present fewer, but critical, information
security concepts to a greater number of students.
We have also had conversations with a variety of other Colleges and Universities
regarding an inter-university ‘HuskyHunt’ challenge, run during security awareness month in
October. This could provide the potential to leverage school pride to increase engagement, as
well as helping other institutions train their students. We are hoping that we can run this across
Connecticut’s four state universities next year.
I have also considered, and am exploring ways to implement, using HuskyHunt as a tool
to provide data access. Many data protection regulations, HIPAA for example, require that
28. HuskyHunt 27
people with access to certain types of data are trained. I think it would be possible to leverage
HuskyHunt as that training vehicle and utilize the badge awards as access tokens. This would
significantly strengthen the importance of badge awards, as they could then be tied directly to
system privileges and would reinforce to the users that the effort they extended to learn
information provided value.
One of the major challenges of digital badges remains their value for badges outside of
the ecosystem that awarded the badges and a long term desire of mine is to identify ways to
establish that value. Currently there is little incentive to display the ‘HuskyHero’ badge, which is
the final badge, on a LinkedIn page, as it has no meaning to people that did not participate in
HuskyHunt. I think that the ideas I’ve described above, which are all intended to increase game
participation and purpose, would significantly help increase its purpose and value and continue
to strengthen the idea that digital badges have value and can provide participation incentives.
36. HuskyHunt 35
Appendix E - Introductory Video Script
Jason:
Hi, I’m Jason Pufahl the CISO for the Univ. of Ct. If you have any questions regarding data
security or huskyhunt please email me at Jason.pfuahl@uconn.edu
Jon: Hi, I’m Joh Moore, I’m the huskytech manager, student technology services and you can
contact me at xxxx.
Jason:
HuskyHunt is a security awareness game that we designed to give you practical tips you can use
at school or home regarding computer security, technology or online privacy. Our goal is for
you to understand some of the most common risks and to provide you tools to better protect
yourself or your device.
Jon:
Like Jason said, huskyhunt is an online game that will start October 6 and run for 3 weeks. Each
day a new level will appear at a random time with content related to tech security. Be the first to
beat the level and you can win a number of prizes. Earn points as you progress through levels
and you will compete for the grand incentive, $500 from the UConn coop. The game is student
focused but faculty and staff are welcome to play, just not eligible for the incentives.
Jason:
This year we will be running the game in 2 versions concurrently. One will award badges for
activities and one won’t so don’t be surprised if you have a different version than one of your
friends. The information we collect will be used to improve future versions of the game.
Finally, there is a short survey immediately following this video. If you would take a few
minutes to complete that I would appreciate it. Thanks and enjoy the game.
Jon:
Join the hunt at huskyhunt.uconn.edu.
37. HuskyHunt 36
Appendix F - Pre-Game and Post-Game Survey Questions
Pre-Game Survey
Why are you playing the game?
o To learn about security and privacy
o To win prizes
o Saw it advertised and just decided to try it
Have you played the game before?
o Yes
o No
How would you describe your attitude towards security
o I am very security conscious
o I know there are risks but am not concerned
o I don’t think about security at all
How would you describe your attitude towards personal privacy
o I am very privacy conscious
o I know there are risks but am not concerned
o I don’t think about privacy at all
How would you describe your attitude towards technology?
o I am a technology enthusiast
o I use technology
o I don’t like technology but am required to use it
Do you expect to complete the whole game?
o Yes
o No
o Maybe
Is there anything specific you hope to learn in this game?
o Open ended
Post-Game Survey
Did you enjoy the game
o Yes
o Somewhat
o No
Did the badge awards provide additional motivation to complete modules?
o Yes
o Somewhat
o No
Did you learn something about Security, Privacy or Technology from the game?
o Yes, I learned a lot of useful information
o Yes, but I knew most of the information already
o No
Did your attitude regarding security change?
38. HuskyHunt 37
o Yes
o Somewhat
o No
Did your attitude regarding personal privacy change?
o Yes
o Somewhat
o No
Did your attitude regarding technology change?
o Yes
o Somewhat
o No
How likely would you be to recommend HuskyHunt to a friend
o Very Likely
o Maybe
o Not Likely
o No way
Describe the game content
o It was interesting
o It was uninteresting
o It was OK
Do you have any suggestions for game improvement?
o Open Ended Question
39. HuskyHunt 38
Appendix G - Awards Ceremony Agenda and Interview Questions
Awards Ceremony Notes and Agenda
Awards gathering planning
Pizza/drinks/cookies provided
Invite a limited set of players (to be determined based on player pool. We will limit the
invites so that we assume approximately 75 attendees. We may select top 100 players, or
perhaps some percentage of users that have completed and will make this determination
at the end of the game.
Serve pizza immediately
Awards
Thank players for coming
Describe some major outcomes of the game (player numbers, unique awards)
Introduce desire for a conversation (below)
Award top 3 prizes
Q&A
This will be loosely structured and as conversational as possible. I am assuming that pre-
game and post-game surveys have gathered some consistent data. I will attempt to get the
information related to my key goals, but will also allow the conversation to evolve organically.
This data will likely not be associated to an individual but the outcome artifact will be an
aggregate compilation of the group.
Key goals for this discussion:
o Did they learn anything?
o Did they feel content was broad enough?
o Was there anything particularly useful?
o Why did they complete the whole game?
o What are the thoughts around badges?
o What would they like to see change?
o How did they find out about it?
o Why do they think the participation was low?
o
I expect the whole event will last between 30 and 45 minutes at most.
40. HuskyHunt 39
Appendix H - Awards Dinner Interviews Transcriptions
Awards Dinner Discussion
Informal Interview 1 – From Evernote Recording 20141029 17:06:05
Did you do it before: A few people did
Did you finish it: A few people did
Didn’t manage to finish it either time, because there was no opportunity to catch up when the
player fell behind.
Player didn’t understand the point decay. Ray, the developer, described the point decay and
solicited some feedback related to how to implement it better so that players could catch up.
Did anyone learn anything?
One person did – about HTTPS specifically
Another learned about UConn Software licensing options
Informal Interview 2 – From Evernote Recording 20141029 17:20:23
Did anyone watch the video? – Yes, people did.
Did anyone find the video helpful – No particular response.
Did you finish?
Rebecca Day - she finished it because she started and wanted to win.
Michael Lau - played because they were in a position to win.
2 people played the game, were off campus, and couldn’t really do the scavenger hunt.
Michael Lau - A person described a timing issue where the module announcements may come
after the module was actually released. Described the activity as a ‘mindless speed game.’
Informal Interview 3 – From Evernote Recording 20141029 17:27:15
Rebecca Day
Won because she timed the module releases
She felt it would have been more interesting if the game was more difficult and had a bit
more of a quest feel.
41. HuskyHunt 40
Girl says that it was annoying when it came out during class and she couldn’t respond quickly
enough
Formal Interview 4 – From Evernote Recording 20141029 17:41:56 – ~ 21 Minutes long. There
is an accompanying video as well, which is this exact discussion captured in video.
Question 1 – Did you learn anything?
Learned ‘stuff’- some of it stuck with him
Importance of using UConn Security vs UConn Guest.
How stuff gets stolen
Nobody felt the content wasn’t valuable, but perhaps not quite challenging enough
Question 2 – When would you like to pick up prizes?
Generally prefer prizes at end of week
1 person expressed desire not to have to go so far for scavenger hunt and to make
them more flexible for prize awards
Question 3 – How many of you completed the whole game
6
Question 4 – Why didn’t you finish the whole game?
Discouraged because of inability to catch up
Just didn’t have time to do the game with their coursework
Question 5 – Did the possibility of earning badges matter?
1 person clear that they didn’t care
1 person did think it was ‘cool to see them’ like a reward system. Thought it was
fun
Rebecca Day – liked earning them as well. Would have liked to also tie a point
incentive to the badges in some way. She liked collecting them all.
1 person said they just liked seeing points increase, didn’t care about the badges
Question 6 – How many of you play online games?
About Half
Question 7 – Did you have any questions from other players because you had badges and they
didn’t?
No, because people generally don’t know about the game
Question 8 – How did you hear about the event? (Asked again later by Jon Moore)
42. HuskyHunt 41
Flyer
People felt most people didn’t know about it.
People felt there was no reason to start after day 1
Word of mouth (3)
Daily Digest (3)
There were still dining table advertising from the last time the game was run
Posters in the Library
Question 9 – If we ran it again how many of you would play again?
All of them.
Michael Lau changed his mind….I think perhaps he liked the personal touch of
this discussion. It is something that would likely be worthwhile to do more of in
the future.
Question 10 – What would you like to see in the game?
Shorter time frame
Ability to catch up
Closer Scavenger hunts
Ability to get the regional campuses involved (scavenger hunt limits this)
Significant ancillary discussion about how to get people to know that the game is running.
43. HuskyHunt 42
Carr-Chellman, A. A. (2011). Instructional design forteachers: improving classroompractice. New
York: Routledge.
Digital Badges for Microcredentials, Student Engaement & Persistence | Online Learning Consortium,
Inc. (2012, June 20). Retrieved December 6,2014, from
http://olc.onlinelearningconsortium.org/effective_practices/digital-badges-microcredentials-
student-engagement-persistence
Digital badges - Wikipedia, the free encyclopedia. (n.d.). Retrieved August 9, 2014, from
http://en.wikipedia.org/wiki/Digital_badges
Duffy, T. M., & Jonassen,D. H. (Eds.). (1992). Constructivismand the technology of instruction: a
conversation.Hillsdale, N.J: Lawrence Erlbaum Associates Publishers.
elearnspace › MOOCs are really a platform. (n.d.). Retrieved from
http://www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform/
ISTE Standards for Teachers. (n.d.). Retrieved December 6,2014, from
http://www.iste.org/standards/standards-for-teachers
Subramaniam, P. R. (2009). Motivational effects of interest on student engagement and learning in
physical education: a review. International Journal of Physical Education, 46(2),11–19.
