Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FINAL_Tenorio_masters_thesis

76 views

Published on

  • Be the first to comment

  • Be the first to like this

FINAL_Tenorio_masters_thesis

  1. 1. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 1 Digital Badges as Micro-Credentials: Using Digital Badges in an Online Learning System by Lisa Tenorio A Field Project and Thesis Submitted to the DIGITAL MEDIA AND LEARNING PROGRAM In Partial Fulfillment of the Requirements For the Degree of MASTER OF EDUCATION University of San Francisco December 2014
  2. 2. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS i Table  of  Contents   List  of  Tables  .........................................................................................................................................  iii   List  of  Figures  .......................................................................................................................................  iv   Chapter  1  Introduction  .......................................................................................................................  1   Statement  of  the  Problem  ............................................................................................................................  2   Background  and  Need  ...................................................................................................................................  4   Purpose  of  the  Project  ...................................................................................................................................  7   Project  Objectives  ...........................................................................................................................................  7   Definition  of  Terms  ........................................................................................................................................  8   Summary  ..........................................................................................................................................................  10   Chapter  2  Literature  Review  ..........................................................................................................  11   Student  Perceptions  of  the  Value  of  Credentials  ................................................................................  12   Student  Perceptions  of  Digital  Badges  and  Alternative  Credentials.  .....................................................  14   Employer  Perceptions  of  the  Value  of  Credentials  ............................................................................  17   The  Impact  of  Credential  Source  on  Employer  Perceptions.  .....................................................................  19   The  Impact  of  the  Labor  Market  on  Employer  Perceptions.  ......................................................................  21   The  Impact  of  Employers’  Personal  Experience  and  Beliefs.  ....................................................................  22   Conclusion  .......................................................................................................................................................  23   Badge  Design  &  Implementation  .............................................................................................................  24   Case  Study  One:  IP&T  EdTec  Course.  ...................................................................................................................  25   Case  Study  Two:  Computer  Science  Student  Network  (CS2N).  ................................................................  27   Case  Study  Three:  TRAKLA2  and  Daechschen).  ..............................................................................................  31   Case  Study  Four:  RL  Hit  List.  ...................................................................................................................................  33   Conclusion  .......................................................................................................................................................  35  
  3. 3. DIGITAL BADGES AS MICRO-CREDENTIALS ii Chapter  3  Project  Development  ....................................................................................................  37   Background  of  the  Project  .........................................................................................................................  38   Project  Components  .....................................................................................................................................  39   Design  &  Implementation.  ........................................................................................................................................  40   Learning  Paths.  ..............................................................................................................................................................  41   Incentive  Framework.  ................................................................................................................................................  42   Badge  Design.  .................................................................................................................................................................  47   Conclusion  .......................................................................................................................................................  48   Chapter  4  Project  Outcome  .............................................................................................................  50   Issues  &  Challenges  ......................................................................................................................................  51   Future  Development  ....................................................................................................................................  52   Recommendations  for  Continued  Study  ...............................................................................................  54   Conclusion  .......................................................................................................................................................  55   Appendix  A  Digital  Badging  Platforms  ........................................................................................  56  
  4. 4. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS iii List of Tables Table 1. Points Systems in Learning Platforms. Points given to each lesson in popular online learning platforms. ................................................................................................................46   Table 2. Trailhead Point System. Final proposal for Trailhead point incentive system. ..............46   Table 3. Trailhead Usage Metrics. Use of Trailhead system since launch. ..................................50   Table 4. Trailhead Feedback Metrics. Social media posts since launch.......................................50    
  5. 5. DIGITAL BADGES AS MICRO-CREDENTIALS iv List of Figures Figure 1. IP&T EdTec Badge Hierarchy: Badge levels and types................................................26   Figure 2. CS2N Badging Hierarchy: Badge types and structure...................................................28   Figure 3. CS2N Badge Examples: Example of the CS2N badge design ......................................29   Figure 4. TRAKLA2 Badges: Example of badge types and design..............................................32   Figure 5. RL Hit List: Physical Pin Badges. .................................................................................34   Figure 6. Trailhead Incentive System: Initial design proposal.....................................................44   Figure 7. Final Trailhead Incentive System: Final design of the badging and award system for the Trailhead project. ..................................................................................................................45   Figure 8. Trailhead Profile and Badge Design. The Trailhead badges and user profile page where badges are stored...................................................................................................................48   Figure 9. Proposal for integrating projects into incentive framework. The updated Trailhead incentive framework..............................................................................................................53  
  6. 6. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 1 Chapter 1 Introduction Recent developments in technology-enhanced learning such as the Massive Online Open Courses (MOOCs), Khan Academy, and the availability of Open Education Resources (OER) have sparked questions about how these new learning experiences, which occur outside a traditional educational institution, can be assessed and credentialed (Conrad, 2013; Mazoué, 2012). At the same time gamification, or applying game mechanics to non-game activities, has become a popular topic in education as a way to motivate and engage learners (Lee & Hammer, 2011). Digital badges have emerged from the concepts of gamification not only as a motivational tool, but also as a potential form of credentialing and assessment that could provide value to students and employers as a way to recognize the specific skills and competencies acquired in both formal and informal learning environments (Watters, 2012). Certification and alternative forms of credentialing are not a new concept and many industries such as law, accounting, medicine, nursing, and teaching have used a system of certification and licensing in addition to traditional academic degrees. In fact, the Adult Education Training and Education Survey (ATES) Pilot Study conducted in 2012 found that 38% of adults in the US reported having some type of alternative credential such as a certification, certificate, or license, in addition to, or in lieu of a degree (Bielick et al., 2013). However, what makes badges unique is the ability to recognize individual skills and competencies and more discrete components of learning. Also, the official launch of the Open Badge Infrastructure (OBI) by the Mozilla Foundation in 2013 provides a standard framework for badges not available for other certification programs. This framework enables any individual or organization to create and issue these micro-credentials while providing a centralized platform
  7. 7. DIGITAL BADGES AS MICRO-CREDENTIALS 2 to communicate the metadata about each badge, including the assessment and evidence supplied by the badge earner (Rosewell, 2012). Having a mechanism to provide open access to information about the requirements that were fulfilled to earn a badge makes digital badges different from other forms of credentialing which are more difficult to validate or assess. Statement of the Problem Why do we even need new forms of credentialing? Why not leave credentialing to the traditional degree programs and universities? One problem is that traditional credentials, such as university degrees, may not adequately prepare graduates for the available job roles and the needs of industry. In fact, a 2011 report by ACT found that a higher level of educational attainment did not necessarily correlate to having specific work readiness skills (ACT, 2011). The unemployment rate of recent college graduates further validates this issue. The Economic Policy Institute’s report on the employment outlook for the class of 2012 reported that the unemployment for college graduates under age 25 was twice as high as the national average at 9.4% (Shierholz, Wething, & Sabadish, 2012). At the same time, employers struggle to find candidates for available job openings. A 2012 survey of employers conducted by McKinsey (Mourshed, Farrell, & Barton) found that only 43 percent of employers could find sufficient skilled entry-level workers. Even if job seekers have the skills that employers seek, most university degrees do not convey granular information about the specific skills and competencies that learners have acquired. Not all university students possess the same skills and competencies and not all university programs are equivalent, but employers lack an efficient way to gauge candidate’s skills in specific areas. Some states are working to fill that gap by creating a national work readiness credential that is separate from any specific educational credential. The work
  8. 8. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 3 readiness credential measures entry-level workplace skills such as reading comprehension, solving problems with math, situational judgment, and active listening through a web-based assessment tool (National Work Readiness Council, (2014)). Although not directly linked to digital badges, the work readiness credential is an example of how new forms of credentialing are emerging in response to this skills gap. In addition, some universities are developing competency-based degree programs which assess student performance on a set of core competencies defined with industry input (Fain, 2014). These competency-based programs can provide more granular information about graduate’s skill sets than a traditional degree. For example, Northern Arizona University, one university with a competency-based degree program, has developed a second transcript that their graduates can share with potential employers. This transcript provides information on the specific skills that students have mastered within the program such as working in a team, communicating with a diverse population or formulating and substantiating theses (Fain, 2013). The emergence of these competency-based degree programs points to a need for greater transparency about the outcomes of learning experiences as well as the need to better align educational programs with industry needs. Digital badges may also be useful in these competency-based programs as a way to signal the specific skills and competencies that students have developed. It is not just recent graduates that need to communicate their specific skill sets to employers, most adults will change jobs or careers multiple times in their lives. The median tenure for US workers with their current employer was 4.6 years in 2012 according to a report by the U.S. Bureau of Labor Statistics (Hipple & Sok, 2013). In fact, the education received in college and conveyed by a degree, may no longer be relevant for job roles attained 5 or even 10
  9. 9. DIGITAL BADGES AS MICRO-CREDENTIALS 4 years after graduation. In addition, the rapid evolution of technology has required employees to participate in ongoing training and development to keep pace with these changes. Lifelong learning and skill development has become a requirement for most careers, and much of this lifelong learning is occurring outside a traditional classroom. . With improvements in technology and Internet connectivity, many new forms of education and training have emerged. MOOC’s and online training courses offer ways to learn online, and new types of face-to-face instruction have emerged such as coding boot camps and intensive short-term classes through organizations like General Assembly. More and more learning is being done outside a traditional classroom, but individuals lack a clear way to communicate the learning and skills they have acquired through these channels. The cost of attending a four-year university and the high unemployment of university graduates has also raised the question about whether these lower cost educational offerings could replace a traditional university degree if there was a standard method to credential these learning experiences. Background and Need Badges as a form of recognition have a long history in the military where ribbons and medals have been used to recognize bravery and achievements (Ostashewski & Reid, 2015). The Boy Scouts have also used merit badges and embroidered patches to recognize achievements for over 100 years. More recently, video games have used badges, points, and leaderboards to track virtual achievements. It is from this intersection of merit badges and video games that digital badges have emerged as both a virtual representation of achievement and as a motivational element of gamification.
  10. 10. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 5 Erin Knight, one of the creators of the Mozilla Open Badge Infrastructure and a proponent of the credentialing aspect of digital badges, acknowledged the dual nature of badges in a 2011 interview: “There is an element of gamification in all of this in that we’ve all experienced badges or levels in games, and we know that they can be motivating. That’s important. Badges will range from smaller motivational badges, to larger certification-type badges, but as people are designing badge systems, many of the principles of game design do and should apply.”(Watters, 2011, p. n.p.) The fact that badges can be both a gamification tactic and a form of credentialing is important to note because some systems use badges purely for motivation and these badges may not be appropriate or viable as external credentials. For example, Foursquare (www.foursquare.com), which launched in 2009, rewarded users with playful badges and status levels for participation (Kincaid, 2009). Foursquare users checked in at physical locations such as restaurants or bars via their mobile phones and acquired badges and “Mayor” status that could be shared on social networks. Foursquare badges have now been retired while the company focuses their business strategy on search and local recommendations (Gayomali, 2014). However, Stack Overflow (www.stackoverflow.com), which launched in 2008, is another popular site that continues to use badges to onboard new users and encourage activities that benefit the community such as answering forum questions (Halavais, Kwon, Havener, & Striker, 2013). In these examples digital badges are used primarily to drive user behavior rather than validate skills or knowledge. One of the first implementations of digital badges in an online learning site was Khan Academy which announced the addition of badges in 2010 (Kamens, 2010). Peer 2 Peer
  11. 11. DIGITAL BADGES AS MICRO-CREDENTIALS 6 University (P2PU) was another early adopter, offering digital badges in their School of Webcraft (Einsenberg, 2011). New technical training providers such as Code Academy, Code School, and Treehouse launched their platforms and badging systems soon after. What is different about the implementation of digital badges in education is that badges are usually tied to some form of assessment which gives them more value as a credential and evidence of learning (Hickey, 2012). There are other benefits of digital badges in education such as making learning paths clear, scaffolding learners towards higher levels of certification, and providing motivation (Finkelstein & Knight, 2013). However, it is the potential to credential learning and skills that are not currently credentialed that is particularly interesting and disruptive. What is different about badges as a form of credential is that they function as a micro- credential of learning, which is a more granular and descriptive credential than a broader degree or certification. The Microcredentials Research Group at Arizona State University also point out that micro-credentials differ from other forms of credentialing because they can be issued by anyone and also can be more easily verified online than traditional credentials (“Microcredentials Research Group,” 2014). Although the concept of digital badges as a form of micro-credentialing is still relatively new, many organizations are now providing these credentials. Badges for Vets (www.badgesforvets.org), which launched in 2013, is an example of using digital badges to credential prior learning and experience (Fain, 2012). Veterans enter their military experience and training on the site and receive skill badges that translate their skills and competencies into non-military terms. Employers can then search the database by skill set to find qualified candidates for jobs. The official launch of the Mozilla OBI in March, 2013 provided a framework and infrastructure that organizations could use and was a step towards defining a standard format for
  12. 12. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 7 digital badges as credentials. As of mid-2013 Mozilla reported that there were already 700 badge issuers and over 75,000 badges issued through the site (Ostashewski & Reid, 2015). Digital badging platforms such as Cred.ly (www.credly.com) and Achievery (www.achievery.com) have also launched sites to make the process of creating and issuing badges easier by providing tools and hosting services for organizations that want to create digital badging systems. However, despite this progress, there is still a lack of clear guidance and best practices for organizations that are interested in developing their own digital badges. Also, because of the popularity of sites like Foursquare, badges are often thought of purely as a motivational or gamification tool. There is a lack of awareness about the potential value of digital badges as credentials, and how these credentials could be used and displayed effectively. Additional research is needed to understand how digital badges can best be designed and utilized to provide value to both learners and employers as micro-credentials. Purpose of the Project The purpose of this project is to gain practical experience designing and implementing a public-facing digital badge system and to gain a deeper understanding of how digital badges should be designed if they are to become viable forms of credentialing. Research on the factors that influence the value of credentials to both learners and employers as well as evaluation of existing badging implementations and badging platforms will inform the design of this new badging system. By documenting this project it may provide a useful artifact for others that wish to implement digital badges in a learning context. If the project is successful it will also provide a platform for ongoing research and study. Project Objectives
  13. 13. DIGITAL BADGES AS MICRO-CREDENTIALS 8 One of the objectives of this project is to review recent empirical studies to better understand: • What are students’ perceptions of the value of traditional academic credentials versus alternate forms of credentialing? • What are employers’ perceptions of traditional and alternate forms of credentials? • What factors influence the acceptance and value of credentials by employers? Case studies of digital badge implementations in educational environments will be reviewed to investigate how others have implemented digital badges in educational environments and if there are any best practices that could be established in designing a new digital badging system. Finally, the field project will examine the process of implementing a digital badging system in a new online learning environment from the initial design and development through implementation and launch. Initial feedback from users and recommendations for the future will also be explored. Definition of Terms Gamification: The use of game elements such as points, badges, and leaderboards in non-game activities. Digital Badge: Educause defines digital badges as “an image or symbol representing the acquisition of specific knowledge, skills or competencies” (Elkordy, 2012). More broadly, digital badges are virtual representations of achievements. Micro-credential: A form of credentialing that recognizes specific skills or competencies. Digital badges are a type of micro-credential. However, whereas badges must have some sort of icon or visual symbol, micro-credentials often have an icon, but that is not a required component (“Microcredentials Research Group,” 2014).
  14. 14. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 9 Mozilla Open Badge Infrastructure (OBI): A framework launched in 2013 that enables any individual or organization to create and issue these micro-credentials while providing a centralized platform for the metadata about each badge, including the assessment and evidence supplied by the badge earner (Rosewell, 2012). Skills Gap: The gap between the skills that employers need and the skills that job seekers and graduates have. Competency-based Education (CBE): Learning systems that are based on a clearly defined set of competencies. In these systems students progress at their own pace as they are able to demonstrate proficiency. Dr. Robert Mendenhall, the President of Western Governor’s University, makes the distinction that “the most important characteristic of competency-based education is that it measures learning rather than time” (Mendenhall, 2012). Technology is often utilized in CBE to provide personalized learning. ePortfolio: An electronic portfolio and collection of students’ work which provides evidence of achievements. These portfolios can be used to showcase accomplishments, share with prospective employers or document specific learning outcomes in a course (Lorenzo & Ittelson, 2005). MOOC: A term which stands for Massive Online Open Course. MOOC’s generally have no limit on enrollment and are open to anyone to participate. MOOC’s are typically structured similarly to university courses some and are offered in both synchronous and asynchronous formats (EDUCAUSE Learning Initiative (ELI), 2013).
  15. 15. DIGITAL BADGES AS MICRO-CREDENTIALS 10 Summary Current forms of credentialing do not necessarily meet the needs of employers, job seekers, and students. There are gaps in the types of learning activities and experiences that are credentialed and broader credentials, such as degrees, do not convey enough detailed information about the skills and competencies that students have acquired. In learning systems, digital badges provide a way to recognize achievements that can be used as a motivational tool and may also provide value as a credential of learning. These micro-credentials have the potential to fill some of the gaps in current forms of credentialing. This paper explores the factors that influence the acceptance and value of credentials by students and employers and may impact the acceptance of digital badges as a viable form of credentialing. This paper also examines the development of digital badging systems and how thinking about digital badges, not only as a form of gamification, but also as a form of credentialing, may impact the design and implementation of these systems.
  16. 16. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 11 Chapter 2 Literature Review As discussed in the previous chapter, digital badges can play multiple roles in educational contexts. Attention is often placed on the gamification and motivational aspect of badges, but digital badges have the potential to recognize and credential learning as well. In fact, for adult learners in particular, credentials, or the resulting career opportunities that these credentials provide, may provide more motivation for learning than the gamification aspect of badges (Finkelstein & Knight, 2013). The focus of this literature review is how digital badges can provide a valuable credential to both badge earners and employers. What factors determine whether a credential is meaningful? How do new credentials become accepted and what factors are most important to both students and employers? How can digital badges move beyond simply a gamification mechanism to provide value as a form of credentialing as well? In addition, to prepare for the field project and the design of a digital badge system, the existing literature was reviewed to identify case studies of digital badge implementations in educational contexts. The objective of this research was to explore how these systems were designed including: • What technology was used to create, issue and manage the badges? • How was the incentive structure designed in terms of the type and number of badges offered? • To what extent do these systems focus on the use of badges for gamification vs. the use of digital badges as micro-credentials? • What were the considerations in designing the graphical look and feel of the badges?
  17. 17. DIGITAL BADGES AS MICRO-CREDENTIALS 12 • What feedback or recommendations could be applied to future digital badge implementations? Because badges and micro-credentials are a relatively new concept there are few existing empirical studies on their use and perceptions. Of the handful of studies that do exist, most have focused on badges for learner motivation or reputation building and engagement within a community site. Given the limited number of studies on badges as external credentials, this literature review also includes studies of other forms of academic credentialing and assessment including ePortfolios and certification. In particular, studies on certifications within the IT industry are reviewed because of the relatively recent evolution of certifications in that field. The IT industry was also chosen because the rapid evolution of technology requires ongoing learning and training and a potentially higher need for ongoing certification or credentialing. Employer perceptions of online degrees were also considered given the recent emergence of online education and the similar perception other new forms of credentialing, such as badges, may encounter. Student Perceptions of the Value of Credentials Before looking at student perceptions of alternate forms of credentialing it is interesting to look at student perceptions of the value of traditional university degrees. A study by Griffin, Jones, and Spann (2008) focused specifically on Generation Y students and their motivations for pursuing an undergraduate degree and found that students could be characterized into two distinct groups they termed knowledge-seekers and certification-seekers. Students classified as knowledge-seekers reported more intrinsic motivations for learning while students classified as certification-seekers reported more extrinsic motivations for learning. A questionnaire was
  18. 18. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 13 administered to 167 students at the School of Business at A&M University in Alabama and included a mix of sophomores, juniors, and seniors born between the years of 1980 and 1988. Cluster analysis was used to separate the responses to the nine questions in the survey into two groups. The resulting analysis classified 76 students as knowledge-seekers based on their responses, and 82 students as certification-seekers. Respondents in the certification-seeker group agreed with the statements that it was their degrees and not their grades (4.73 on a seven-point scale) or knowledge (4.54) that would get them a good job after college whereas the respondents in the knowledge-seeker group agreed less frequently with those statements. Although this study focused on a narrow demographic of students within a particular region and age range, it does indicate that some students see the main purpose of education as attaining the credentials desired by employers. A study of higher education students in the United Kingdom examined students’ perceptions of their academic credentials and found that students felt a strong need to differentiate themselves in the job market beyond the credentials bestowed by their academic degree (Tomlinson, 2008). In this qualitative study Tomlinson conducted semi-structured interviews with 53 undergraduate students in their final year of study from a range of subject disciplines. Although students viewed their higher educational credentials as having value in the labor market, many expressed concerns about the intense competition for a limited supply of job opportunities and a difficulty in differentiating themselves from the large number of graduates with similar credentials and degrees. A business major in this study expressed his concern over the value of the degree this way: There are so many people with degrees now that it is becoming difficult for employers to tell them apart…with so many people with degrees it means far less
  19. 19. DIGITAL BADGES AS MICRO-CREDENTIALS 14 than it used to, like you are another person who has gone through the conveyor belt. I’d say you were at advantage having one but it won’t be enough on its own to get you a job. (Tomlinson, 2008, p. 55) These students expressed the need to add distinction to their academic credentials (e.g., through grades, awards and other activities) or to add additional post-graduate credentials in order to stand out from other candidates in the job market. This study also identified that a main concern among these students was the ultimate outcome of their studies and how they could market their skills to employers rather than “learning for its own sake” (Tomlinson, 2008, p. 56). The results identified by Tomlinson (2008), that students need to find ways to distinguish themselves beyond their degree, contradict the findings by Griffin et al. (2008) who found that many students believed the value of their education was in the degree itself, and not in their grades or other distinctions. However, both studies found that students are concerned about how to market themselves to employers and are sensitive to the importance employers place on credentials. Student Perceptions of Digital Badges and Alternative Credentials. In a recent quantitative and qualitative study researchers evaluated adult learner attitudes about badges in two open online courses (Cross, Whitelock, & Galley, 2014a). The Cloudworks platform was used to manage and award Open Badges in the courses and learners could display these badges in their Cloudwork’s profile and also add them to their Mozilla backpacks. There were approximately 220 active participants in each course and, of those, approximately 50% applied for one or more badges. Researchers evaluated learner attitudes about the badges through end of course surveys, badge statistics as well as blog, forum and Twitter posts. A total of 88 students responded to the course surveys between the two courses studied. Overall, the role of the badges as evidence of learning was reported as an important function by 42% of survey respondents. One
  20. 20. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 15 respondent indicated that their institution agreed to recognize the badges as evidence of professional development activity. However, many reported uncertainty about whether employers would value these credentials. One respondent expressed concern about the level of achievement the badge represented saying that if the MOOC was, “going to be one line of my CV then the badges were perhaps at too granular a level” (Cross et al., 2014a, p. 9). Another respondent indicated that they were “a little curious to see if there is any reaction to badges at my place of work when I list them as a part of my continuing education” while another commented, “I don't know if anyone in my institution will care, but in case they do, I have the badges” (Cross et al., 2014a, p. 10). Despite the uncertainty around the value of the badges as a credential, 66% of the survey respondents felt that the badges were a positive addition to the course and helped to guide and motivate study, monitor progress, and get feedback. Approximately 25% of respondents expressed a negative view of the badges and did not find them motivating or useful. In a qualitative study conducted by Rughinis and Matei (2013) students were asked about their perceptions of the external value of badges earned in their university courses. Semi- structured interviews were conducted with 26 students and 8 instructors to evaluate their perceptions of two different badge architecture systems implemented at the University Politehnica of Bucharest. Researchers observed that some students saw these badges as being only for fun rather than for any public significance or benefit. Other students did see a wider significance in the badges they earned, but were unsure how to use these badges on their resume or in conversations with employers (Rughiniş & Matei, 2013, p. 87). These results reflect the current early stage of badge implementation and a lack of general information and awareness that is available about how they can be used. Unfortunately, few details were provided regarding the substance of the interviews, how the results were analyzed, or about the badge systems
  21. 21. DIGITAL BADGES AS MICRO-CREDENTIALS 16 themselves so there is limited ability to analyze or assess other factors that impacted these results. Although not specifically tied to badges, Woodley and Sims (2011) studied student perceptions of ePortfolios and found similar mixed results in terms of students’ perceived value of this system in the job market. Similar to the Open Badges metadata, ePortfolios offer a way to display tangible evidence to an employer of the work completed towards a credential. In this study 60 students in the second year of a 3-year undergraduate program at the University of Victoria in Australia completed a web-based survey with a mix of quantitative and open-ended questions. Students were asked about the importance of their ePortfolio in showcasing their skills to employers and 38% felt that it was “important” or “very important” while 33% rated it as “not important” or “not at all important” (Woodley & Sims, 2011, p. 169). In the open-ended comments students were also divided in their views, with 50% of them making positive comments and 40% making negative comments about these ePortfolios. However, many of the criticisms that students gave related to the features of the platform used (PebblePad), rather than the portfolios themselves. Researchers noted that the students who had responded negatively also tended to have provided fewer assets or evidence in their ePortfolio so may not have used the platform as extensively as other students. Although not a focus of this study, four respondents reported that they had already shown their ePortfolio to an employer and three of those reported receiving positive interest from the employer. Overall these studies found that students value credentials when they believe that employers will value these credentials. With new forms of credentialing such as digital badges or ePortfolios students are unsure about how employers will perceive them and also how best to display or use these new credentials.
  22. 22. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 17 Employer Perceptions of the Value of Credentials The IT Industry provides an interesting example for studying employer perceptions of the value of credentials because of the rapid rate of change in this field, the lack of a single, unified industry standard for IT credentialing, and the diversity of IT worker’s educational backgrounds (T. Adams & Demaiter, 2008). Furthermore, a 2002 report on IT workers in the United States found that the majority of IT workers did not obtain a degree in an IT related field (Aspray & Freeman, 2002). Meanwhile the emergence of new technology such as virtualization, service- oriented architecture (SOA) and security and compliance of cloud-computing systems requires new categories of skill certification and training (O’Grady, 2011). Wierschem, Zhang and Johnston (2010) surveyed IT directors to identify the importance employers place on IT certifications and found that, although employers perceived these certifications as having value and providing some information about technical competence, the overall perception of the importance and accuracy of these certifications was mixed. Of the 141 respondents that completed this survey, 45% of employers either required or desired that their IT personnel obtain or possess certifications. The open-ended questions provided additional insight with the most often cited reason that employers found these certifications valuable was, “…providing a baseline of technical knowledge”(Wierschem et al., 2010, p. 99). However, when asked to rank the importance of certification in evaluating an applicant for a full-time position the average response was fairly neutral at 3.08 on a five-point scale. Respondents also ranked the accuracy of IT certifications as an overall indicator of competence more neutrally at 2.94 on a five-point scale. Experience was ranked as more important than certification at a weighting of approximately 60% to 40%. These findings were limited by the fact that the respondents were
  23. 23. DIGITAL BADGES AS MICRO-CREDENTIALS 18 all based in IT departments at universities. A wider sampling that includes respondents from the private sector would be beneficial to validate these results. A mixed-methods triangulation study conducted by Heise (2009) also evaluated the perceived value of technical certifications by employers and then compared those results to employee perceptions. This study overwhelming showed the perceived importance of these credentials in the hiring process, despite the perception that these credentials do not necessarily provide an accurate picture of an applicant’s skills. In this study approximately 150 IT hiring managers from northeast Indiana were surveyed using an open-ended questionnaire, with a total of 59 responses. Descriptive and thematic text analysis were used to identify themes from the employer surveys. From these themes a quantitative survey was designed and administered to 188 students that were near graduation or had graduated from an IT program in the last 5 years and had sought employment in the IT field. The results showed that both employers (75%) and employees (82%) believed IT certifications were valuable in the hiring process. The combination of credentials most preferred by employers (36%) was a degree, job experience, and some certification. However, despite the importance placed on these certifications, 84% of employers reported that they had hired IT certified staff that “did not perform technology tasks to their expectations” (Heise, 2009, p. 42) and 77% of employees said that they did not feel certifications accurately reflect an applicant’s IT skills. There were numerous discrepancies in this report between the data cited in the text and the supporting tables, which does raise some concerns about the validity and accuracy of the results. Also, the geographical location of participants was limited to Northern Indiana and should be validated with a wider sample. Mahatanankoon (2007) surveyed IT professionals in the mid west region of the United Status and found that formal professional development activities such as training and
  24. 24. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 19 certification made a significant contribution to the likelihood of promotion. On the other hand, informal and non-credentialed professional development activities were found to have no correlation to promotion. A quantitative survey was completed by 91 IT professionals who had been working in their current companies for an average of 4.8 years. Respondents were asked to enter the number of hours per month they spent on both formal and informal professional development activities as well as the number of times they had been promoted by their current company. Although it is not clear how many of these learning activities included an actual certification or credential, respondents who reported more hours spent on formal learning activities reported a higher rate of promotion. The study was limited to IT professionals in the early stages of their career and did not measure other factors influencing promotion such as work performance or motivation. It would be difficult to draw any cause and effect conclusions from this study, but the results do indicate a positive correlation between formal learning activities and career advancement. The Impact of Credential Source on Employer Perceptions. Another factor to be considered regarding the perception of value of a specific certification or credential is the influence of the issuing organization. A quantitative study by Gleghorn and Gordon (2012) evaluated the difference in employer perceptions of IT certification based on whether the certification was vendor-specific or vendor-neutral. Two certifications were compared: Cisco’s Certified Network Professional (CCNP) certification and the Certified Information Systems Security Professional (CISSP) certification offered by the Information Systems Security Consortium (ISC). Information security managers from the North Carolina Chapter of the Information Systems Audit and Control Association (ISACA) were surveyed and out of the 200 total members, 40 completed the survey. Participants were asked which certifications, if any,
  25. 25. DIGITAL BADGES AS MICRO-CREDENTIALS 20 they would like their IT staff to hold. The majority of employers participating (81.3%) favored the vendor-neutral CISSP certification, while only 18.8% favored the vendor-specific CCNP certification. Participants were also asked how likely they were to promote staff with vendor- neutral or vendor-specific certifications and a slight preference was shown for professionals with vendor-neutral certification. However, this preference for promoting vendor-neutral certified professionals varied depending on the years of experience of the hiring manager. One limitation of this study is the small sample size (N=40). Also, this study did not take into account the differences in the specific skills assessed by these two certifications or the primary technologies employed by these organizations. These two factors could have a bigger impact on employer preferences than the branding of the certification itself. The findings by Gleghorn and Gordon (2012) also contradict the results of the larger study of employer and employee perceptions conducted by Heise (2009). In this study employers were asked about the value of “standardized certifications” versus “individual brand certifications” and the results showed that 55% of employers felt both types were of equal importance while 24% felt that vendor-specific certifications were more valuable (Heise, 2009, p. 45). Although not directly related to certifications, Adams and DeFleur (2006) looked at employer’s perception of graduates from online degree programs versus candidates with degrees from traditional universities and found that the institution issuing the degree was an important factor in employer perceptions. In this study employers were selected randomly across several major metropolitan cities in the United States with a total of 269 employers participating from multiple industries. A quantitative survey showed that employers overwhelmingly favored candidates with a traditional degree (96%) versus one with an online degree (4%). However, an
  26. 26. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 21 important finding from the qualitative component of the survey were “numerous comments that the issue was not whether the degree was earned online or in a traditional sense, but whether the online courses were taken with an institution with a reputation of quality” (Adams & DeFleur, 2006, p. 41). In other words, for these employers the online degree would be more accepted depending on where it came from. One limitation of this study is that employers were not asked to compare degrees from specific institutions, but only the general concept of online or traditional degrees. Also, this study is now 7 years old and online degrees have continued to increase in popularity and are now offered by many well-known schools which may have shifted employer perceptions since that time. The Impact of the Labor Market on Employer Perceptions. A qualitative study of hiring managers in Detroit and Seattle compared employer perceptions of IT workers who held associate degrees to those with bachelor degrees (Van Noy, Jacobs, & Columbia University, 2012). This study found that employer perceptions of these credentials varied between the two cities and may be impacted by the specific characteristics of each labor market. Hiring managers in Seattle expressed more negative perceptions of associate degree holders (46%) than hiring managers in Detroit (26%), indicating that associate degrees signified less academic ability and initiative compared to bachelor degrees. One reason for the difference in employer perceptions offered by the researchers is the higher concentration of candidates with bachelor degrees in Seattle relative to Detroit, although no specific data on the number of degrees was cited (Van Noy et al., 2012). Hiring managers in Seattle tended to view community colleges as more vocational and technical, while hiring managers in Detroit viewed them as more transfer-oriented and a less expensive pathway to higher education. No data was given regarding the number of employers that were interviewed in this study or how those interviews were analyzed. However,
  27. 27. DIGITAL BADGES AS MICRO-CREDENTIALS 22 the results do point to a potential variation in employer perceptions of credentials depending on the specific factors of each local labor market. Also, while employers in both cities felt that the completion of degrees was important and indicated positive attributes, they also indicated that neither degree conveyed all the skills and competencies that they desired in IT staff which could indicate the potential value of an additional form of credentialing. The Impact of Employers’ Personal Experience and Beliefs. A more recent study of employer perceptions of online degrees found that employers’ personal experience with online learning had a direct impact on their perceptions of the value of online degrees (Fogle & Elliott, 2013). In this study 71 employers from multiple industries and regions of the United States completed a quantitative survey on their willingness to hire graduates from online universities. In contrast to the earlier study by Adams and DeFleur (2009), this study showed more acceptance of online degrees among employers. When asked which candidate they would hire, 50% of employers indicated that they would select the candidate with a traditional, on-campus degree. The remaining 50% of employers selected candidates with either a hybrid degree (i.e. 50% online and 50% on-campus) or a completely online degree. However, even more significantly, this study found a strong correlation between the educational background of the respondent and their liklihood of hiring a candidate with an online degree. Employers with a tranditional on-campus degree were twice as likely to hire a candidate with an on-campus degree, while employers with a hybrid or online degree were 14 times as likely to hire candidates with online degrees. The researchers also examined the correlation between other demographic factors such as gender, industry, size of firm, age, and census area. Although there was a slight bias and correlation found against online graduates in smaller firms, none of the other demographic variables yielded a strong correlation with perceptions of online degrees. However,
  28. 28. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 23 one of the limits of this study is the small sample size and the even smaller percentage of respondents who had an online or hybrid degree within that sample. A qualitative study of French employers by Bailly (2008) also supports the findings that an employer’s experience and beliefs play an important role in the perceptions of academic credentials. In this study, over 20 interviews were conducted with individuals involved in the recruiting function in the accounting and retail industries. Bailly found that employers’ views about the origins of employee skills, and whether these skills could be learned or were innate, varied by industry and influenced employers’ perceptions of the overall value of academic credentials. The retail employers believed that the most important characteristic of success was commitment and other innate qualities that could not be learned, while employers in the accounting firm believed that any necessary skills could be learned and therefore valued academic credentials more highly. These findings were correlated with statistics from the Centre d’Études et de Recherches sur les Qualifications (CÉREQ) which showed, at the time of the interviews, that a higher proportion of employees with formal academic qualifications could be found in accounting (32%) than in retail (7%). This study is fairly limited in scope in terms of the sample of employers and the industries that they represent, but it does point to a variation by industry in the overall perception of the value of academic credentials. Conclusion Given the early stages of digital badge implementation, further studies are needed to assess the potential value of these micro-credentials to students and employers. Studies have shown that students are concerned with how to market and differentiate themselves to employers and will pursue credentials they perceive to be valuable to employers. However, digital badges are still very new and students do not yet know how to display or use them effectively and are
  29. 29. DIGITAL BADGES AS MICRO-CREDENTIALS 24 not sure if these credentials will be valued by employers (Cross, Whitelock, & Galley, 2014; Rughiniş & Matei, 2013). For digital badges to be considered valuable by students more education is needed about how to use and display these credentials and more information is needed about the employers who look for or consider these credentials for recruitment. Employer perceptions of the value certifications and degrees varies and could be influenced by a variety of factors including their own familiarity with the credentials, the issuing organization and brand, the specific local labor market characteristics, and even their own personal beliefs. Studies have shown that employers value alternative credentials such as certifications, but at the same time are skeptical about the specific skills that these credentials convey. The Mozilla OBI provides a mechanism to provide transparency about how a digital badge was earned that may solve this concern about certifications. There is an opportunity for digital badges to provide value to employers that is not currently provided by degrees and certification programs if creditability for these micro-credentials can be established. More awareness and education about digital badges is needed so that employers and learners become familiar with them, and more time is needed to see how these credentials evolve. Badge Design & Implementation Four recent case studies were identified that implement digital badges in a learning system. Three of these case studies focus on learning systems designed for adult learners and one case study documents a system designed for both K-12 students and teachers. All four of these case studies relate to computer science education, . which is a limitation of this research. One researcher remarked, “engineering education is especially inclined towards using achievement-type rewards, due to widespread engagement with the gaming culture” (Rughinis, Foley, Restivo, Uhomoibhi, & Helfert, 2013). It may be that computer education courses are
  30. 30. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 25 early adopters of digital badging and which is why more of these examples were found in the literature. Case Study One: IP&T EdTec Course. Instructors at Brigham Young University designed a digital badging system for a 1-credit instructional technology course (course number 286) which teaches secondary education majors technology skills (Randall, Harrison, & West, 2013). Objective. One objective for the digital badge implementation was to motivate students to do better work on each of their assignments. The second objective was to provide a way for students to showcase their skills to employers. Badge Design. Three levels of badging hierarchy were established for this course: the lowest level for small achievements, a second tier for mastering larger technologies (which also corresponded to projects), and a final badge level for course-level mastery. The highest-level badge would be awarded automatically if all project-level badges were completed. In addition, the authors allowed for three additional levels of badges that could be earned after the course was completed as part of a teacher’s ongoing professional development. These additional levels included a strategy level for creating a plan to implement a technology in the classroom, and an applied-level for showing evidence of successfully implementing this technology. After the strategy and applied badges are earned the recipient would automatically receive the highest- level Technology Integration badge.
  31. 31. DIGITAL BADGES AS MICRO-CREDENTIALS 26 Figure 1. IP&T EdTec Badge Hierarchy: Badge levels and types. Recreated from “Giving credit where credit is due: Designing Open Badges for a Technology Integration course” by Daniel L. Randall, J. Buckley Harrison, and Richard E. West, 2013, TechTrends, p. 93. Copyright of TechTrends 2013. Assessment. Projects in this course are evaluated by an instructor or a teaching assistant and graded against a rubric. Automatic assessment and peer assessment were not used for badge awards and assessment. Although this assessment method ensures greater quality control, the authors point out that this assessment method limits scalability of the badging process.
  32. 32. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 27 Technology. The technology used for the creation and issuing of badges for this course was a commercially available solution called the Badge-It-Gadget-Lite plugin for WordPress. Students can display badges from their course on their WordPress sites through WPBadger Display or they can use badgewidgethack.org to display their badges on the web. Unlike traditional credentials, badges can have a set expiration date. Given that the badges in this course relate to specific technology skills and technology can change rapidly, the authors debated how long to make the badges valid for. They were concerned that if the badges expired too rapidly learners would not come back and earn them again so they decided to set the expiration at 10 years. After that time, significant changes to technology would likely have occurred making it necessary to update the badges. Recommendations and Feedback. No data was provided in the case study regarding feedback collected from learners or employers about the perceptions of the badges or badging system or any impact on student motivation or learning. The authors point to the need for further studies regarding the implementation of this system. They also provide a recommendation around the need for a better badging platform that would allow for the bulk issuing of multiple types of badges to many users and then would email each student a link to collect all badges for the course. Case Study Two: Computer Science Student Network (CS2N). Carnegie Mellon University’s Computer Science Student Network (CS2N) is a web-based learning system targeted at K-12 teachers and students for learning about computer science and robotics (Higashi & Shoop, 2012). Students earn badges by completing activities along multiple curriculum pathways as well as for continued participation on the site. Teachers earn badges to become validated instructors of the content and badge issuers.
  33. 33. DIGITAL BADGES AS MICRO-CREDENTIALS 28 Objective. The objectives of the CS2N badge implementation were “providing motivation, articulating curriculum, and serving as lasting indicators of learners’ achievements” (Higashi & Shoop, 2012, p. 423). Badge Design. The CS2N badging hierarchy is designed with four levels of badges, and then aligns to an industry certification at the highest level. At the lowest level, small and medium badges provide motivation as well as document learner progress. At the next level, Knowledge Badges align with major content milestones and assess content proficiency. The fourth level of badges are the Teaching badges and these require human assessment. Once a teacher has completed a Teaching Badge they can then assess other students in the CS2N system. A student who completes this human-verified level of achievement receives a “gold” level badge. The alignment of the badging system to the real-world credentials at the top-most level was done to provide more incentive for completing the pathways and to support, rather than replace, credentials that are already defined and valued by stakeholders. Figure 2. CS2N Badging Hierarchy: Badge types and structure. Reprinted from “The roles of badges in the Computer Science Student Network” by R. Higashi,
  34. 34. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 29 S. Abramovich, R. Shoop, C. Schunn, 2012, GLS 8.0 Conference, Conference Proceedings, p. 424. Copyright of CS2N.org 2012 The graphic design of the badges in the CS2N system provides information on the relative weight of the qualification depending on the size and shape of the badge. The badge name, date of issuance and description appear on the badge. More detail, such as the exact requirements for earning the badge, expiration date and links to the learning pathway, are available by clicking on the badge. Figure 3. CS2N Badge Examples: Example of the CS2N badge design Reprinted from CS2N.org. (n.d.) Badges. Retrieved from: https://www.cs2n.org/teachers/badges Copyright of CS2N.org Another key component of the CS2N badging hierarchy and design are the learning Pathways that are illustrated visually using Badge Maps. The Badge Maps show all of the steps in the learning path, the outcome of the learning path, as well as the learner’s progress within the path. Assessment. The small and medium CS2N badges are awarded based on a computer- administered exam. Teacher and gold level badges involve a human assessment process.
  35. 35. DIGITAL BADGES AS MICRO-CREDENTIALS 30 Technology. No information was provided regarding the underlying technology used to issue and reward badges. The CS2N website indicates that the badges conform to the Mozilla OBI framework and users over the age of 13 can share badges to a Mozilla Backpack or sites such as Facebook and LinkedIn. C22N badges also conform to the Learning Resource Metadata Initiative (LRMI) metadata that allows them to be discovered by search engines and educational databases and identifies the educational standards that the badges align with and represent. Recommendations and Feedback. The authors report that overall learners have enjoyed the badging component of the learning site, but that this positive feedback has not necessarily correlated to an increase in learning performance. No information was provided on feedback from employers or industry stakeholders. A separate qualitative study was conducted with middle school students using the CS2N site to study the motivational effects of digital badges on learners with different levels of prior knowledge (Abramovich, Schunn, & Higashi, 2013). A total of 36 seventh graders and 15 eighth graders participated in the study and students were split into two groups based on the results of a pre-test on proportional reasoning skills. The results of this study found that students who had scored lower on the pre-test, and were less skilled in the content area, were more positively impacted by the participation badges. No correlation was found between the skill badges and motivation in the low-performing group. Students who scored higher on the pre-test did show a correlation between the skill badges and motivation. Students in this group who earned only a few skill badges experienced a drop in expectations for success, whereas students who earned many skill badges experienced an increase in expectations for success. No correlation was found between high-performing students and participation badges. Based on the research conducted so far the authors recommend considering the abilities
  36. 36. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 31 and motivations of learners when choosing the balance between the number of skill and participation badges to design. They also recommend providing as much detail as possible to learners about how to earn badges to counteract any negative impact on motivation (Abramovich et al., 2013). Case Study Three: TRAKLA2 and Daechschen). Digital Badges were integrated into a Data Structures and Algorithms course at Aalto University, in Finland. The first implementation occurred in Spring 2012 and was repeated in 2013 using different technology. A total of 300 students participated in the course between the two implementations (Haaranen, Hakulinen, Ihantola, & Korhonen, 2014). Objective. Digital badges were implemented in this course as a way to encourage and motivate good study habits. Badge Design. A total of eight achievement badges could be earned during the course. Each of the badges related to completing the online course exercises and the badge awards were based either on time management, attention to detail, or good study habits. For example, the time management badges related to submitting exercises before the deadline or completing them the fastest. The attention to detail badges related to the having the least number of submissions to get the correct answers. (Hakulinen, Auvinen, & Korhonen, 2013). There was no explicit badge hierarchy, but earning the most difficult badge within a category would most likely result in earning the other badges in that category. The course designers chose this nested structure for the awards to make it possible for users to earn all of the badges available during the course For the course exercises students were shown a piece of code that they needed to simulate with input data to show the execution of an algorithm. The input data was chosen randomly so
  37. 37. DIGITAL BADGES AS MICRO-CREDENTIALS 32 students could try to solve the exercises multiple times. Each exercise was given a point value and the total points were reduced by 50% if students submitted them after the deadline. Points were not reduced for multiple submissions, but the attention to detail badges were tied to minimizing the number of submissions needed to arrive at the correct answer. Figure 4. TRAKLA2 Badges: Example of badge types and design. Reprinted from “Empirical study on the effect of achievement badges in TRAKLA2 online learning environment” by L. Hakulinen, T. Auvinen, and A. Korhonen, 2013, Learning and Teaching in Computing and Engineering, p. 49. Copyright of IEEE Computer Society 2013.
  38. 38. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 33 Technology. Originally the badging system and course was designed using the TRAKLA2 open source Learning Management System (LMS). In the second iteration of the course the badging system was designed using the A+ LMS which is a Software as a Service (SaaS) platform. This second system was called Daechschen. Daechschen was built using the Django web-framework and integrated with the A+ LMS and API’s. (Haaranen et al., 2014) Recommendations and Feedback. The authors report in a separate qualitative study that feedback collected from students after the course was generally positive, but overall a small number of students actively collected badges (Hakulinen et al., 2013). The recommendation from the authors is to provide some type of external motivation or to tie the badges more closely to the final grade in the course to provide more motivation. Students were able to see the badges they earned, but there was no mechanism to share the badges or make them visible to others. The authors also suggest that having a mechanism to share the badges and make them visible to others would make them more motivating for students. Case Study Four: RL Hit List. Instructors at the University Politehnica of Bucharest used digital badges in a Computer Networks course which is part of the 3rd year curriculum in the Computer Science program (Rughinis et al., 2013). Objective. The goals of implementing the badging system in this course was to publicly recognize top students in a visible public forum, to create a community of students and faculty across multiple years and courses, to motivate students to engage with the course and labs, and to stimulate conversation among students and faculty.
  39. 39. DIGITAL BADGES AS MICRO-CREDENTIALS 34 Badge Design. Currently a total of 26 badges can be earned during the course (https://systems.cs.pub.ro/teaching/courses/rl/hit-list/). Badges are awarded for top scores on the various exams as well as in-class participation and overall involvement in the course. There is no explicit badge hierarchy for the awards. A unique feature of the RL Hit List badges is that they are both digital and physical in nature. In addition to a virtual badge, students also receive a metallic pin badge that is awarded ceremoniously during the course. Figure 5. RL Hit List: Physical Pin Badges Reprinted from “Badge architectures in Engineering Education: Blueprints and Challenges” by Răzvan Rughiniș, 2013, CSEDU. 5th International Conference on Computer Supported Education. Proceedings, p. 5. Copyright of INSTICC Press 2013. From a design standpoint the authors state that the badges and website were designed to be minimalist and professional rather than playful to align with the intended audience of a community of professionals. The course site contains a leaderboard where badges earned are listed by student ID and student name as well as the date when each badge was issued. Assessment. Badges are awarded for top scores on the various exams on the course. Participation badges are more subjective and are awarded by instructors based on in-class participation.
  40. 40. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 35 Technology. No information was provided regarding the technology used to create and issue the badges to participants, but the course site was published using WordPress. Recommendations and Feedback. One challenge cited by the authors has been creating and maintaining the ritual dimension of awarding the material pins during the courses and making sure the award experience is consistent. Another challenge has been the participants’ understanding of the badge architecture and objectives. Initially it was assumed that instructors and students had a clear understanding of the objectives, but feedback has shown that most have understood the badges only as a motivational tool. The recommendation from the author is to provide more detail about the badge architecture and rationale when designing a badge system so that it is well understood by all stakeholders. The last recommendation provided by the authors is to think about badges as a mechanism for driving conversation and to focus on the conversational infrastructure to support students, instructors and the community having active dialogue about these achievements. Conclusion Two of these case studies had an explicit goal of providing learners a way to showcase their skills and achievements and, not surprisingly, both of these systems provided a way for learners to share their badges outside of the learning site (Higashi & Shoop, 2012; Randall et al., 2013). These two case studies also had a more complex badging hierarchy with multiple levels of awards. The lower-level badges in both of these examples were designed for motivation and provided more of a gamification element, while the higher-level badges were skill-based and provided a credential. The other two case studies had a simple badging structure with no hierarchy and the objective of these badge systems was more about influencing behavior or providing recognition
  41. 41. DIGITAL BADGES AS MICRO-CREDENTIALS 36 than providing a way to showcase individual skills (Haaranen et al., 2014; Rughinis et al., 2013). Both of these examples placed more emphasis on the gamification component of badges and no mechanism was provided to share these badges outside of these sites. One of these systems included a leaderboard as well as physical pins that were given as awards which further emphasized the gamification element. One of the challenges of designing digital badges as micro-credentials is the variety of ways digital badges are being used within learning systems and the perception of badges as gamification elements rather than credentials. Some digital badges are not designed to be credentials and recognize much smaller achievement than other badges, which could be confusing for recipients and employers to understand how these various badges compare or have value. The total number of badges that could be earned in each of these systems varied and the number of badges available could also have an impact on their perceived value. More research is needed to understand how the number of badges that can be earned impacts learning understanding and value of these badges. Information on the technology used to create and issue badges was only available for two of the case studies. In one case a commercially available solution was used and in the other case the system was developed internally. More information is needed about the infrastructure requirements to support badging systems. Commercially available badging platforms are making that process easier, especially for small organizations, but more information is needed to compare these solutions and understand their limitations and benefits.
  42. 42. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 37 Chapter 3 Project Development Salesforce.com is a technology company established in 2000 that is the leading provider of cloud-based Customer Relationship Management (CRM) software. In addition to its Software as a Service (SaaS) products, Salesforce.com also provides a platform for building enterprise web and mobile applications. One of the challenges facing the company is how to increase the number of skilled developers that are familiar with the technology to fulfill the demand for Salesforce-skilled talent within the ecosystem of customers and partners. Salesforce.com has an existing certification program managed by the training organization and employers see value in the certifications as a way to qualify candidates. However, the number of people completing these certifications each year has not kept pace with the demand in the marketplace. The Developer Marketing team at Salesforce.com is tasked with growing this developer community and providing learning pathways for developers to increase their skills. The department maintains an extensive library of free online documentation and workbooks and has also produced a MOOC in partnership with Udacity. Although these existing resources have been well received, the department would like to increase the percentage of users who are completing these self-paced training resources and continuing to learn and deepen their knowledge of the platform. The idea for a new learning platform started with the concept of providing assessments or challenges that users could complete and then to have those challenges automatically scored or validated using the Salesforce.com platform API’s. The goal was to provide a way for users to test their knowledge and to better identify what they need to learn as well as to make the learning experience fun and interactive.
  43. 43. DIGITAL BADGES AS MICRO-CREDENTIALS 38 The other goal of this project was to provide better learning paths or guides through the content so that users knew what the next step in their learning should be. Although there is a substantial amount of documentation and training content already available, users have lacked clear direction on how to navigate through that content. Initially digital badges were proposed primarily as a way to motivate users and make the experience fun. I was invited to participate in this project and to contribute to the design of the incentive and award system because of my interest in digital badging. My hope was to encourage to the team to consider these digital badges as potential micro-credentials as well as motivational components of the new system. Background of the Project This project officially kicked off in July 2014 with the goal of rolling out a beta version to customers by mid-October 2014. The project was code-named Trailhead and a small core team was assembled to help prioritize the features and functionality that could be built for this initial release. At Salesforce.com all employees and projects start by creating a document called the V2MOM which stands for vision, values, methods, obstacles and measures (Benioff & Adler, 2009). The first task for the Trailhead core team was to create a V2MOM and come to an agreement about the overall vision for this project. After a few rounds of meetings and discussions the vision was defined as: “Grow the number of skilled Force.com declarative and programmatic developers and increase feature adoption by radically redefining the learning experience; create community-driven, engaging, contextual, scalable, assessment-driven content that motivates learning.” (Salesforce.com Powerpoint presentation, July 2014)
  44. 44. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 39 The goals for the Trailhead project included: • Make learning paths visible, personal, and easily navigable • Break learning down into bite-size modules • Make individual learning visible by recognizing skills and competencies • Motivate learning and make it fun and engaging • Drive awareness and adoption of new features • Provide analytics to measure feature-level adoption • Provide a platform for all learning content that the department produces The metrics of success to be measured included: • number of learning path page views • number of users who completed at least one assessment • total number of assessments completed Project Components A project leader was assigned to lead the core team, Sandeep Bhanot, and the project was divided into multiple work streams with a lead assigned to each of these streams. The work streams for the Trailhead project included: • Design & Implementation (Jacob Lehrbaum): Creating the UI and interface as well as building the features and functionality of the learning platform. • Assessments (Samantha Ready): Creating the assessment questions and mechanism that would test user’s knowledge of each topic. • Learning Content (Kim Shain): Updating existing documentation and learning content to match the Trailhead format as well as leading the content creation of new lessons.
  45. 45. DIGITAL BADGES AS MICRO-CREDENTIALS 40 • Incentives & Learning Paths (Lisa Tenorio): Identifying the components of each learning path, determining how the content would be divided, and designing the gamification and badging system for the platform. • Marketing (Lauren Grau): Defining the nomenclature and branding for the platform, creating the graphic design for the badges and incentives, creating introductory videos and guides, and executing all marketing and promotion of the learning platform. Design & Implementation. One of the initial decisions for the team was whether this learning platform should be built in-house or if we would use commercially available technology to deliver it. Given the short timeframe and the complexity of the integration with the documentation content, the Developer website, and the assessment mechanism, the decision was made to build the Trailhead platform in-house. There are a variety of hosted solutions available for creating, issuing and managing digital badges (see Appendix 1). However, these solutions are limited in terms of the systems that they integrate with. Cred.ly is one of the more popular digital badging platforms and does integrate with the Salesforce CRM, but not with the Force.com developer platform which was a requirement for the assessment mechanism and for displaying the badges within the developer community site. It was also a requirement to integrate this platform with our in-house content management tools so integration with any commercially available solution would have required additional development work and potentially taken longer. One advantage of these commercially available badging solutions is that most of them integrate with the Mozilla Open Badge Infrastructure and enable badges to be shared through the Mozilla Open Backpack. Since we were building Trailhead in-house the team needed to decide if we would integrate the Mozilla OBI for the Trailhead badges. Although not difficult, this
  46. 46. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 41 would require some additional development work. Since it was unclear whether the Trailhead badges would be valuable to users and employers as micro-credentials, it was decided not to implement this for the initial release and to wait and see if this was something we should implement for a future release. Learning Paths. As mentioned earlier, one of the key reasons for developing the new learning platform was to make clear learning paths for users to follow. From a badging design standpoint it was also important to start with the overall structure of the learning paths in order to define the content structure, learning objectives, assessments and rewards. One of the questions about the learning paths was whether those paths should be defined by specific tasks, by user roles or by levels. Another question was whether these paths should be independent or build on each other. As a starting point for the learning paths a mind map drawing of everything a user should know about the Salesforce platform was created with contributions from the technical members of the team. From the mind map, groupings were created to help structure the content by feature area and topic and then that information was put into a spreadsheet and turned into a content outline that could then be mapped against our existing content to see where we were missing content. After a few iterations the content was structured into three levels. One of the biggest debates during the project was what to call each of these content levels and various names were proposed such as lessons, topics, units, paths and modules. Eventually it was decided the three content types would be: trail, module and unit. Each module represented a feature area and would contain 3-5 individual units. Each unit would have an associated challenge or assessment.
  47. 47. DIGITAL BADGES AS MICRO-CREDENTIALS 42 Three trails were identified for the first release and each of these trails each had five to seven content modules. Incentive Framework. Gamification is a popular tactic to motivate user behavior and new learning platforms such as Khan Academy and Treehouse have implemented badges and points to help motivate learners to complete courses. One of the objectives for Trailhead was to motivate more users to complete courses and gamification was seen as a specific tactic to employ to make learning fun and achieve a higher completion rate. Although the initial goal of the Trailhead badges and incentives was primarily gamification, as the project evolved the potential value of badges as a way to make learning visible and to provide clear pathways for learning was also recognized. It was agreed that the gamification aspect of the badges would be a lesser priority to these other goals, although still an important part of the system. One of the early tasks of the projects was to define how many levels the reward system would have and how those levels would work. In addition to the case studies found in the literature, other learning sites which utilize digital badges to teach technology were reviewed including Khan Academy, Code Academy, Code School and Treehouse. Of these sites, Khan Academy was selected as the best model from a badging and incentive standpoint. The elements of the Khan Academy incentive system that seemed most relative to Trailhead included: • An award system that combines points and badges • Participation, community, and skill-based badges • Clear learning paths and rapid feedback via automated assessment • Mastery challenges that enable users to skip over content they may already know One concern for the design of the Trailhead badges was that they would have enough meaning and value for users. An initial question was whether badges should be awarded for
  48. 48. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 43 completing an individual lesson or if they should they be reserved for completing a more substantial learning goal such as a module of learning content (usually 3-5 lessons) or even a broader functional area. At the highest level the learning content for Trailhead was organized into Trails and each Trail included several modules. Another question was if there should be a separate type of award for completing an entire Trail (equivalent to a course) or if there should be any award at all for that level. The initial badge system proposal for Trailhead included three different types of awards: points, badges & patches. Points would be the lowest-level of award provided for motivation and documenting progress, rather than credentialing. The goal of using points at the lowest level rather than badges was to reduce the total number of badges and to reserve badges for more meaningful accomplishments. Points could be earned for completing individual lessons and potentially other types of activities in the future such as community participation or attending events. Badges would be awarded for completing a challenge at the module level, or for completing all of the individual unit challenges in a module. There would also be the opportunity to award special badges in specific feature areas, such as badges for different levels of expertise within a feature area or for community engagement and participation. One concern was making sure that if we did award badges for community-based achievements that they would be clearly distinguishable from the skill-based based badges from a design standpoint. Finally, the initial proposal included a final level of award which was differentiated from the badges by calling it a patch. That final level would be awarded automatically for completing all modules in a particular Trail.
  49. 49. DIGITAL BADGES AS MICRO-CREDENTIALS 44 Figure 6. Trailhead Incentive System: Initial design proposal. Due to the time constraints for the initial release it was not possible to create a challenge or assessment at the module level. So, the decision was made to have badges earned only through the completion of each lesson-level challenge rather than have a separate module-level challenge. This simplified the design, but meant that there was no way for users to skip directly to a master challenge rather than complete all units. There was some resistance from the internal stakeholders to include both points and badges because of the desire to keep this system as simple as possible. However, in the end it was decided to keep points and badges and eliminate the top-level award instead.
  50. 50. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 45 Figure 7. Final Trailhead Incentive System: Final design of the badging and award system for the Trailhead project. One open question that was not addressed in this initial badge design is how the Trailhead badging hierarchy would relate to the existing Salesforce certifications. Salesforce currently offers seven types of certification that are assessed through an online exam proctored by a third party. The content that is currently available in the Trailhead system does not yet cover all of the topics covered by Developer or Administrator certification exams. However, as more content is produced it is possible that users could acquire the skills to prepare for these exams through the Trailhead system. The final challenge in designing the incentive system was determining how many points to award for each activity. Should all lessons be worth a different amount of points depending on the difficulty of the topic? Or, should each lesson be worth the same amount? One factor that needed to be taken into account was that not every lesson had an assessment. Some lessons were planned to be evaluated via a multiple choice assessment rather than through direct product application. However, the multiple choice test capability would not be available for the initial
  51. 51. DIGITAL BADGES AS MICRO-CREDENTIALS 46 release. Therefore, many lessons in the initial release could be marked as complete simply by clicking a button without any type of assessment. These lessons should most likely be worth fewer points then those that required users to implement something directly in the product. Once again other learning platforms were reviewed to evaluate how their point systems were structured and the number of points these systems awarded for a similar achievement. In reviewing these systems there appeared to be two main point structures: those based on denominations of 100 and those based on denominations of 10. Table 1. Points Systems in Learning Platforms. Points given to each lesson in popular online learning platforms. Platform   Points  per  Unit/Lesson   Khan Academy 300 Code School 250 Treehouse 30 Code Academy 7- 8 (1 pt. per exercise) For the Trailhead point system there needed to be enough differentiation between the awards for completing the challenges vs. the multiple choice assessments or other participation awards that may be implemented in the future. It was decided to use a point system based on denominations of 100 to create the most flexibility between different award levels. There was also a feeling that enabling users to gain points more quickly would be more motivating. Table 2. Trailhead Point System. Final proposal for Trailhead point incentive system. Task Points Completing a Unit (No Challenge) 100
  52. 52. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 47 Completing a Unit (With Challenge) 300 Completing all Units in a Module 200 Completing all Units in a Trail 1000 Community Badges (TBD) Points Attending a workshop 100 Attending a webinar 100 Answering a forum question 50 Badge Design. One of the advantages of most digital badging platforms is that they have templates to facilitate the graphic design of badges. However, since the Trailhead badging system was developed in-house we also needed to design the look and feel of the badges. The Creative team within Developer Marketing was responsible for creating the badge design. One requirement for the design was to differentiate the badges that required programming skills from the point-and-click badges with some type of visual treatment. The final design concept was to use circular badges with a simple icon. The icons were chosen to correspond as much as possible to the feature itself. The programmatic badges have a thin border on the perimeter of the circle, while the point and click badges do not.
  53. 53. DIGITAL BADGES AS MICRO-CREDENTIALS 48 Figure 8. Trailhead Profile and Badge Design. The Trailhead badges and user profile page where badges are stored. Some learning platforms use playful names for badges and originally the concept for the Trailhead badges was that they would be more fun and playful with names like “Apex Ninja”. However, because of the interest in seeing how these badges might be used as micro-credentials it was a conscious decision to keep the badge names straightforward. Each badge name represents an area or feature of the platform. The names need to be kept fairly short to fit in the space allotted in the user interface and every badge name must be unique. A total of 11 badges were planned for the initial release, but a longer list, which also contained future badges to be issued, was reviewed and edited to make sure that the names were all consistent, short and unique. Conclusion Although the main objective for Trailhead badges was not the micro-credentialing aspect, the badge design was influenced by the possibility that these badges could serve as credentials in
  54. 54. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 49 the future. The look and feel of the badges was kept simple and professional, rather than playful. The naming of the badges and iconography was chosen to convey as concisely as possible what skills the achievements conveyed. Badges as a reward were also reserved for skill-based achievements of a more significant size and to limit the number of badges overall. Currently badges can be shared within the Salesforce Developer Community site and via social media. The badges do not conform to the Mozilla OBI which would enable greater sharing and transfer of the badges across the web, but it is not clear if that is something that users would value or use. Many employers and job seekers use the Salesforce Developer Community to find opportunities or talent so if the badges are seen as credentials it may be enough to have them displayed within this site.
  55. 55. DIGITAL BADGES AS MICRO-CREDENTIALS 50 Chapter 4 Project Outcome The Trailhead learning platform launched on October 13, 2014 and is accessible via the Salesforce website at: https://developer.salesforce.com/trailhead. The platform launched with a total of 3 Trails, 11 Modules and 41 Units. So far very little marketing of the system has been done, other than some announcements at Dreamforce and some posts via social media. However, many users have already started using system over the last month. Table 3. Trailhead Usage Metrics. Use of Trailhead system as of November 28, 2014. Usage Metrics Total Users who have started at least one unit 1,176 Users who have completed at least one unit 1,116 Users who completed at least one module 635 Users who completed all 11 modules 34 Feedback so far has been overwhelmingly positive. Many users have written blog posts or tweeted about their experience using the platform. Table 4. Trailhead Feedback Metrics. Social media posts since launch. Feedback Metrics Total Tweets 1,832 Facebook posts 80 Blog posts written 17
  56. 56. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 51 Most positive feedback has been about the assessments and about the gamification element of earning points or badges. One of the bloggers who has written about Trailhead commented about the competitive nature of developers in general: “Challenges and automated checking are going to be key factors in the success of this program. The Salesforce developer community are a competitive bunch, and anything involving badges and points gets a lot of interest.” (Bowden, 2014) Another blogger commented on the satisfaction of getting a reward when you get the right answer to a challenge: “First, I love the challenges. They’re not too hard, but they’re not a step-by-step, so I had to really read what they were asking me to do. There is something super satisfying about pushing the button to check your code and getting a reward when you get it right.” (Jameson, 2014) So far, no users have commented on the potential value of the badges as a credential or way to demonstrate skills to potential employers. Users seem motivated to collect the badges without really knowing if they will have any future value other than for bragging rights. Many users who have earned badges have posted screen shots of the badges they’ve earned to social media to share their accomplishments. Issues & Challenges One of the challenges of this project was the short time-frame to plan and execute. Because of the lead time needed to develop the infrastructure and content, there was very little time to research the overall badge design and system before critical decisions needed to be made. Ideally it would have been beneficial to spend more time researching gamification system design specifically to understand best practices for creating gamified learning experiences.
  57. 57. DIGITAL BADGES AS MICRO-CREDENTIALS 52 Some questions have come up from users regarding how Trailhead badges relate to Salesforce certifications. This was a question that was not tackled in the initial release, but will need to be thought through for the future. Having badges map to higher-level certifications may also give the badges more value and credibility as micro-credentials. Future Development Currently the Trailhead team is working on adding additional features and functionality to the system as well as developing new content. One of the main features being worked on is the addition of computer-based multiple choice assessments. Not all units can be or need to be assessed within a Salesforce developer environment. Currently these units have no assessment and points are awarded simply for confirming completion. For each of these units, three to five multiple choice questions will be added depending on the number of learning objectives for the unit. All questions must be answered correctly to complete the unit, but users can attempt the assessment multiple times. Another feature being worked on is to add a leaderboard to the system so information about the top scoring users is visible to all users. The goal is to provide more motivation to complete units and modules by providing this element of competition to the site. Users have indicated that they would like to see this feature with one saying, “a leaderboard would be cool…something that shows the top scorers for the last month, or biggest increase in points in the last week…would give everyone a chance to see their name up in lights” (Bowden, 2014). In addition to the leaderboard, a plan to provide physical rewards such as embroidered patches or t- shirts is also being discussed. A new type of content, called projects, is also being added to the site. Projects are more like tutorials and will take a user through the steps of building something from start to finish, but will
  58. 58. Running Head: DIGITAL BADGES AS MICRO-CREDENTIALS 53 not provide conceptual information or in-depth content on a specific feature area. A proposal for how projects will fit into the overall content map and incentive framework is being reviewed. The current proposal is that projects would reside at the module level and users will receive a badge for completing an entire project. Figure 9. Proposal for integrating projects into incentive framework. The updated Trailhead incentive framework. Because projects give users more of a big picture view of how features work together rather than in-depth knowledge in a specific area, the proposal is to make the badge design for projects visually different from the other skill badges. Another idea under consideration is to make the badge names for the projects more fun and playful since they are not tied to specific features.

×