Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crowdsourcing Expert Performance to Improve Training at Cyber Speed


Published on

Presented at the 2011 I/ITSEC Conference in Orlando, Florida.

  • 80% Win Rate? It's Not a BUG? [Proof Inside] ➤➤
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Crowdsourcing Expert Performance to Improve Training at Cyber Speed

  1. 1. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011 Crowdsourcing Expert Performance to Improve Training at Cyber Speed Janet Cichelli Maureen Baginski Serco North America Serco North America Rockville, Maryland Columbia, Maryland ABSTRACTWith the continued escalation in the volume, sophistication, and complexity of network attacks, the training ofgovernment professionals with the cyber skills needed to operate, maintain and defend computer networks hasbecome increasingly urgent. In fact, maintaining a robust workforce of educated and trained ―cyber-bodyguards‖ is areal and present national security issue. And as threats continue to change and evolve at cyber speed, training mustalso continue to rapidly improve and adapt to meet new and unknown threats.The use of crowdsourcing—a social networking technology that leverages the ―collective experience‖ of thegroup—is rapidly showing promise as a way to capture high-quality experiential knowledge and lessons learned in amore efficient way than previous methods. The human performance domain typically employs cognitive analysisand direct workplace observation to identify what experts know and do and then feed that into improvement oftraining methods and content. But this process can be time consuming, has logistical challenges, and provides a verylimited sampling upon which to make continuous improvements in training. Applying controlled ―internalcrowdsourcing‖ techniques, expert cyber security practitioners and intelligence analysts can be immersed in a game-based environment where their performance patterns are tracked, their insights captured, and new detection patternsdiscerned. These findings can then be fed back—in rapid succession—to improve training scenarios and methods.This paper presents a methodology to help meet cyberspace requirements and address the current talent gap using arapid and agile process for continuous improvement in cyber training. The authors present a framework wherebycrowdsourcing can be employed to capture human performance in a game-based network environment. Theconvergence of crowdsourcing and gaming provides a new and effective way for current training developmentmodels to integrate lessons learned from expert performers and improve personnel proficiency at cyber speed. ABOUT THE AUTHORSMs. Janet Cichelli has over twenty-five years experience in the field of technology management and new conceptdevelopment. She is a recognized solution architect and thought leader in the areas of emerging technologies, humanperformance consulting, and knowledge management systems. She currently serves as Chief Technologist for SercoNorth America, where she provides strategic consulting to Defense and Intelligence organizations in the design andimplementation of technology solutions that enable a high-performing workforce.Dr. Maureen Baginski has a distinguished back-ground with almost three decades of service in the United StatesIntelligence Community and currently serves as Vice President of the Intelligence business and National SecurityAdvisor for Serco North America. Ms. Baginski previously served as the FBI‘s Executive Assistant Director forIntelligence where she was responsible for establishing and managing the FBI‘s first-ever intelligence program. Shealso served at the National Security Agency (NSA), where she held a variety of positions, including SignalsIntelligence (SIGINT) Director, Senior Operations Officer in the National Security Operations Center; ExecutiveAssistant to the Director of NSA/Central Security Service, Chief Officer of the Director; Assistant Deputy Directorof Technology and Systems; and lead analyst for the Soviet Union.2011 Paper No. 11116 Page 1 of 8
  2. 2. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011 Crowdsourcing Expert Performance to Improve Training at Cyber Speed Janet Cichelli Maureen Baginski Serco North America Serco North America Rockville, Maryland Columbia, Maryland INTRODUCTION more vulnerable than we‘d like to be. Attacks on government networks are also ubiquitous. According toWithin days of taking office, the Obama Administration a report by the Center for Strategic and Internationalordered a ―clean slate‖ review of U.S. policies and Studies, NASA and the departments of Defense,structures for addressing the urgent challenges of cyber Homeland Security and Commerce have "all sufferedsecurity. After two months of input from top cyber major intrusions by unknown foreign entities" (Droginexperts, the White House issued a comprehensive report 2010).(Cyber Policy Review, 2009) that states: With this continued escalation in the volume, ―Cyber security risks pose some of the most sophistication, and complexity of network attacks, the serious economic and national security training of government professionals with the cyber challenges of the 21st century… Growing skills needed to operate, maintain and defend computer arrays of state and non-state actors are networks has become increasingly urgent. Brig. Gen. compromising, stealing, changing, or John Davis, director of current operations for the destroying information and could cause critical Defense Departments Cyber Command, said, ―There is disruptions to U.S. systems.‖ a finite amount of human capital associated with the skill set required to do this job. There is a need forSome experts have warned a ―cyber world war‖ is training and education for the skill set needed to do thisalready underway with pervasive espionage and job effectively - we need to grow the workforce‖information extraction activity taking place across the (Beasley, 2011).U.S. cyber infrastructure, and reconnaissance andexploitation of critical infrastructure assets in In fact, maintaining a robust workforce of educated andpreparation for actual attack and disabling of our trained ―cyber-bodyguards‖ is considered by many tocritical systems. Such an attack could begin from a be a real and present national security issue. And asseemingly benign scenario: threats continue to change and evolve at cyber speed, training must also continue to rapidly improve and The crisis began when college basketball fans adapt to meet new and unknown threats. downloaded a free March Madness application to their smart phones. The app hid spyware The current demand for trained and competent cyber that stole passwords, intercepted e-mails and talent far exceeds the availability. So, to ameliorate this created havoc. Soon 60 million cell phones cyber talent gap, we must be able to select and develop were dead. The Internet crashed, finance and people to higher levels of proficiency, and much more commerce collapsed, and most of the nations quickly. There is simply not the time to do it using a electric grid went dark. White House aides traditional, linear military training model. And cyber discussed putting the Army in American cities training must be able to continually improve to outpace (Drogin, 2010). an increasingly assertive and cunning adversary. The human performance domain typically employsScenarios such as these may sound like science fiction, cognitive analysis and direct workplace observation tobut as the world becomes increasingly interconnected identify what experts know and do and then feed thatvia the Internet, our nation‘s ability to safeguard critical into improvement of training methods and content. Butnetworks, such as the electric grid, military assets, the this process can be time consuming, has logisticalfinancial sector and telecommunications, is quickly challenges, and provides a very limited sampling uponbecoming a top national priority. No one knows how which to make the continuous improvements invulnerable we really are because the most costly attacks training. What is needed is a rapid and agile process forhave not been made public. But we are probably a lot continuous training improvement.2011 Paper No. 11116 Page 2 of 8
  3. 3. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011Fortunately, recent developments in Web 2.0 that an individual in an organization could provide?capabilities—specifically crowdsourcing— are rapidly Simply stated, the collective experience, knowledge,showing promise as a way to capture high-quality and insight is richer than any single individual‘s. This isexperiential knowledge and lessons learned that can be broadly acknowledged across government and is aapplied to improve training in a more efficient way. major reason why organizations have increased their efforts to provide knowledge-sharing systems that CROWDSOURCING promote teamwork and collaboration.Fundamentally, crowdsourcing is a social networking Effective internal crowdsourcing in the cybermethod that leverages the ―collective experience‖ of a environment requires a diverse group of participants,group. Wikipedia defines it as, ―the act of outsourcing each representing different specialties and knowledge intasks, traditionally performed by an employee or niche areas. Cyber design, technology, and operationscontractor, to an undefined, large group of people or are far too complex for a group of experts to impart allcommunity (a "crowd"), through an open call‖ of the knowledge and breadth of experience required to( fully understand and analyze the domain and its complex systems. Complexity and diversity in theCrowdsourcing essentially brings problems or cyber domain is comparable to the medical domain,challenges directly to the masses, who serve as 21st where no single practitioner can assess and treat allcentury ―free agents‖ to solve problems and answer diseases and conditions; and therefore must rely on acompelling questions, often in innovative and out-of- large cadre of specialists and researchers. Further, thethe-box ways. Within the context of federal training cyber domain is dynamic and rapidly changing, so thatefforts, crowdsourcing provides the ability to rapidly new knowledge is rarely disseminated in time for allharness the cognitive and creative abilities of large participants to become aware of it and able to put newnumbers of people in a professional community to knowledge to use universally. Through internalcollect orders of magnitude more performance data that crowdsourcing, both explicit and experientialcan then be used to identify training gaps and knowledge can be captured efficiently from a very largeopportunities for improvement. and diverse population.Internal Crowdsourcing A recent position paper (Erickson, 2011) states three main advantages of crowdsourcing, each of which isWhile the results from individual crowdsourcing relevant to the consideration of internal crowdsourcingparticipants can be taken and applied, the power lies in for helping to provide continuous improvements inthe aggregate of participants‘ ideas and actions. training:Crowdsourcing, then, by definition, is most effectivewith very large crowds. However, participation may 1. Speed: produces results more quickly byneed to be limited in a government cyber environment multiplying effort;due to access restrictions to sensitive information and 2. Quality: produces higher quality results byprivileged network operations. But government integrating diverse input;organizations can limit participation and still realize 3. Legitimacy: produces results that are moremany of the benefits by constraining the size and access legitimate by virtue of representing a community ofof the community and practicing internal practice.crowdsourcing. Speed. Some crowdsourcing systems, says Erickson,With internal crowdsourcing, the participants can be add value because they can perform a task more quicklylimited to any subset of the workforce. Examples might than an individual. Classic examples he cites (2011)include: personnel working within a specific include von Ahn‘s ESP Game in which people generateorganization, those in an identified job role, those textual labels for images, or Galaxy Zoo in whichidentified as exceptional performers, or those with people classify images of galaxies as spiral or elliptical.particular job experience and/or meeting training This is usually managed by recruiting large numbers ofprerequisites. people who perform one or two very simple tasks. Their results are then captured and integrated. The true powerAdvantages in this kind of crowdsourcing lies in its ability to create a situation in which many people will perform the taskSo how does internal crowdsourcing provide an even though it may be trivially simple.advantage over the insights and experiential knowledge2011 Paper No. 11116 Page 3 of 8
  4. 4. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011Quality. A second way that crowdsourcing systems add performance. During game play, users will oftenvalue is by producing higher quality results. Wikipedia perform tasks, solve a problem, or make a decision in a( has become a classic example of way they might not otherwise be able to articulate. Butusing crowdsourced content authoring to create articles those insights—or tacit knowledge—can be capturedthat are generally higher in quality than an individual and cycled back into the training improvement processmight produce. Quality typically continues to increase for the update, validation, and enhancement of trainingwith edits, resulting from the diversity of knowledge content (e.g., scenarios) and instructional methods.that its broad base of contributors bring to the writingtask. War gaming is something that military organizations do regularly in order to test their readiness andLegitimacy. The third way that internal crowdsourcing competencies in scenarios that match the real thing asoffers an advantage is that its results are based on real much as possible. In recent years, the U.S. military hasperformance by real people in a highly realistic bought into gaming in a much larger way, through theenvironment. In this way, it is strongly representative use of multi-player, online serious games. The rationaleand could be deemed the best practice approach to a for serious games is to attempt to capture some percenttask or problem. of the large amount of human brainpower expended every day in playing games and harness it toward aCrowdsourcing in Government useful social goal or for training and readiness.Crowdsourcing is gaining momentum within Electronic game play has become widespread for bothgovernment as a means to gather public and serious and recreational use. Our society spends 3organization ideas. The US Department of Homeland billion hours each week playing online gamesSecurity (DHS) has adopted the IdeaFactory (McCann, 2011). According to Luis von Ahn, acrowdsourcing application to encourage brainstorming researcher at Carnegie Melon University, humans spendof new ideas by its employees. DHS is using the nine billion hours alone playing Solitaire every yearplatform to encourage its employees to come forward (Rigby, 2009). Ahn has contrasted this huge number towith new ideas on how to do things. Other employees the number of human-hours it took to accomplish othercan then rate the ideas, comment on them, pick highly significant works in which humanity takes pride:favorites, and forward them to others (Hoover, 2009). the Empire State Building (7 million human-hours) and the Panama Canal (20 million human-hours). TheThe Open Innovation Portal, launched by the U.S. results are both jaw-dropping and very intriguingEducation Department (ED), aims to use crowdsourcing related to harnessing the collective input of the crowdto address educational challenges ranging from high in serious games. If we could take the amount ofschool dropout rates to low reading, math, and science voluntary human effort that was going into playingscores. The initiative is part of a new White House solitaire and channel it to these previously mentionedeffort to encourage innovative collaboration across all projects instead, the Empire State Building could haveindustry sectors. Users also can post ideas and see, been completed in 6.8 hours and the Panama Canalreview, and rate ideas posted by others based on need, could have been built in one day (Barquin, 2011).impact, evidence, innovation, and scalability(Stansbury, 2010). In a thought-provoking TED Talk (2010), Jane McGonigal discussed the tremendous potential thatBut only very recently has crowdsourcing begun to could be realized if the energy and motivation found inemerge as a way to refine and improve training. In fact, today‘s gaming culture could be harnessed and directedthe most tremendous promise is being shown by to develop personnel proficiency in areas of nationalcombining the presence and engagement of online concern. She cited a Carnegie Mellon University study,serious games with the power of crowdsourcing which found that the average young person today in amethods. country like the U.S. with a strong gaming culture will have spent 10,000 hours playing online games by the CONVERGENCE OF CROWDSOURCING & age of 21. This number—10,000 hours—is a very GAMING relevant number for two reasons. First, for children in the United States, 10,080 hours is the exact amount ofSerious games can provide for the constructive time that each will spend in school from fifth grade tochanneling of human insights and experiential high school graduation with perfect attendance.knowledge, thereby providing an exceptionalframework for crowdsourcing expert human2011 Paper No. 11116 Page 4 of 8
  5. 5. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011And the second reason it is relevant is related to crowdsourcing methods to capture soldier actions andMalcom Gladwells ―10,000 hour theory of success‖ as insights. In 2010, the Army partnered with thestated in his book Outliers (2008). This theory is based University of Southern California Institute for Creativeon cognitive science research that states, ―If we can Technology to establish the Mobile Counter-IEDmaster 10,000 hours at effortful study, at anything by Interactive Trainer, or MCIT, at Fort Bragg, N.C. tothe age of 21, we will be virtuosos at it. In fact, we will assist soldiers in recognizing and reacting to improvisedbe as good as the greatest people in the world at that explosive devices. In that time, over 15,000 soldiersparticular thing.‖ nationwide have gone through the modified shipping containers featuring fictional video narratives and aBy combining serious games with controlled ―internal computer game that are now set up at three differentcrowdsourcing‖ techniques, cyber security practitioners bases across the country (Kim, 2010).can be immersed in a game-based environment wheretheir performance patterns are tracked and new insights The Mobile Counter-IED Interactive Trainer (MCIT)and detection patterns discerned. This new information tries to expose soldiers to the real-life environmentis then fed back—in rapid succession—to improve around making and evading IEDs. It was developed ontraining scenarios and methods. Sound far-fetched? In the model of a Hollywood set and has the look and feelfact, several new training projects, combining of the real site (Barquin, 2011). While MCIT in itself iscrowdsourcing and serious games, have just begun to a very effective pre-deployment training tool, theemerge. program is beginning to use crowdsourcing games to capture what returning soldiers have learned about IEDThe United States Navy has begun crowdsourcing detection and avoidance. These soldiers are immersedtactics for fighting Somali pirates and securing the Horn in a game where they are challenged to help predictof Africa, through a new video game project called ambush sites and IED hotspot locations. This ―trainMMOWGLI (Massive Multiplayer Online WarGame first, then go down range, then give feedback‖ trainingLeveraging the Internet). MMOWGLI was developed cycle is providing valuable insights and lessons learnedby the Office of Naval Research (ONR) to help solve that the Army can use to immediately refine training fordifficult strategic problems. The MMOWGLI game, those about to deploy.scheduled to be launched in summer 2011 focusesexclusively on combating Somalian piracy Using serious games for crowdsourcing expert(Ungerleider, 2011). performance has several benefits over other, more common, crowdsourcing methods. Chief among these isMMOWGLI was developed by the ONR with that games can offer participants a strong motivationalassistance from the Institute for the Future and the incentive. When the game is engaging, then playersNaval Postgraduate School. For the Navy, the use of will want to play, and want to continue to play, inmassively multiplayer games to solicit military strategy exchange for the intrinsic enjoyment. This stands inis a no-brainer. Players in the game are not stark contrast to other potential methods ofcompensated for their time and ideas--and the military crowsdsourcing or survey analyses, where thecommand structure has access to informed advice and motivation must either be one of pure altruism or wherestrategy that they otherwise wouldnt. players are compensated to participate.In another promising effort, a crowdsourcing project APPLICABILITY TO THE CYBERcalled World Without Oil (WWO) is self-described as INTELLIGENCE MISSION―a massively collaborative imagining‖ of the first 32weeks of a global oil crisis ( In conjunction with the current efforts to protect and defend networks and systems in the cyber domain, thereWWO is a serious game that invites people from all is a broader need to provide training for conducting all-walks of life to contribute ―collective imagination‖ to source and all-method cyber intelligence operationsconfront the real-world issue of declining oil resources. (and counter-intelligence operations). If we think of theIt is significant in its use of crowdsourcing and games IT and cyber hardware and software infrastructure, withas democratic, collaborative platform for exploring all of its vulnerabilities and points of entry andpossible future scenarios and sparking future-changing exploitation as the means to an end for our adversariesactions. to conduct espionage and attack, we must also consider the motivations, tactics, dependencies, and desiredAnd finally, the U.S. Army has bought into gaming in a consequences for both defending and attacking themuch larger way, recently starting to incorporate cyber domain. The breadth of skills and tools needed to2011 Paper No. 11116 Page 5 of 8
  6. 6. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011execute a comprehensive cyber strategy must go war-gaming exercise with critical infrastructure (food,beyond the technical cyber security expertise used to energy, finance, water, etc.) attack events andmanage and exploit the networks and systems. information infrastructure espionage activities occurring in a world arena. As the game scenarioThe emerging definition of a cyber intelligence analyst unfolds players will discover, report, and respond tois an almost super-human combination of cyber security trends and incidents in order to maintain the integrity,expert and traditional intelligence analyst. The reality trust, and availability of information and resourcesof combining human intelligence, political and cultural within the cyber domain.insight, all-source analysis, and deep cyber technicalknowledge places even greater pressure on training. A Both the CND and COI realms can be merged into angame-based crowdsourcing framework can capture and all-inclusive game scenario where participants can takemanage this very dynamic set of learning requirements, on any of several roles to either game at the high leveland can continue to adapt as many of the rules of exercise, or delve into the low-level cyber defense andanalysis continue to be written. ―hacking‖ activities. Interaction and cooperation between these groups will be highly useful in METHODOLOGY understanding comprehensive cyber strategy in the real world.To be operationally relevant and efficient, a crowdcomputing gaming platform used to solicit cyber Semi-Automated Capture of Cyber Tactics. Using adetection and prevention strategies requires a standard process similar to that employed in MMOWGLIframework and a process to collect, capture, analyze (Ungerleider, 2011), a scenario will trigger a decisionand use performance information. In its most basic point: What do you do now? A text input area recordsform, the crowdsourcing framework must capture and the participant‘s brief answer. Current capabilitiescollect desired INPUT, and provide aggregate and within the cyber intelligence domain allow for semi-actionable OUTPUT. automated collection of both actions and textual input from participants. The process will result in a series ofThe framework also relies on a pre-defined audience. In datasets and hierarchical taxonomies. These datasets arethe cyber training context that is the focus of this paper, built dynamically by large numbers of participants andthat audience is characterized as a large and diverse will be used to collect general, domain‐specific, andsample of accomplished performers. community-biased information. The resulting datasets will have a quantitative confidence assessment appliedThe INPUT collected by the crowdsourcing framework to rank the actions and feedback, as reflected by thewill be in these two categories: crowds activity.1) The activities and actions performed in a given Once player input is collected, fellow players will be situation, and able to comment or rank another player‘s tactic for2) The crowd‘s opinions, decisions and insights validation. Comments can include adding additional collected on the topics and problems presented. information to expand the idea, asking a clarifying question, or challenging the original input. Then, duringThe OUTPUT desired is an aggregate of each round of game play, which can be executed overdata/information that can be analyzed for consensus, an extended period of time, these ideas and tactics willanomalies, or performance themes. continue to mature and be refined.In the Cyber Network Defense (CND) realm, players Automated Game Play Management using Artificialwill be immersed in 3D game play where they monitor Intelligence (AI). While current focus involves thenetwork traffic to take actions that prevent and identify semi-automated capture of performer data, researchsophisticated cyber threats. As the game scenario points to the future ability to include automated gameunfolds, they are required to deal with how their play management for even greater efficiencies. Currentpreventive plans failed or succeeded and must research is being conducted around automated gamedetermine new strategies. play management, using AI analyzers to drive the real- time tuning and adjustment of the task, therebyIn the Cyber Operations and Intelligence (COI) realm, modifying game parameters automatically to meetplayers will interact in a similar 3D game play, but will with a higher level of human, political, social,natural, and cyber activity. This is more of a cyber2011 Paper No. 11116 Page 6 of 8
  7. 7. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011Representing Crowdsourcing Output. At the end of REFERENCESthe planned number of rounds, the game will output thecollected data and information to a repository. From Barquin, Ramon. ―Games and Business Intelligence.‖ B-that, we can display a logical treeing of the information eye-Network blog. 3/15/11. See http://www.b-eye-collected. Using a hyperbolic tree view helps determine relationship, trends, and validity of informationcollected, to inform decisions regarding changes in Beasley, Meg. ―DoD Wants Common Cyber Picture.‖training that are needed. It is important to identify Federal News Radio online. 2/24/11. Seedifferent associations and among entities, and using a view and a hierarchical list view allows the team to visualize these entity relationships as asemantic tree representation of captured ideas, tactics, Cyber Policy Review: Assuring a Trusted and Resilientand information. Information and Communication Infrastructure. 5/29/09. Report issued by the White House. CONCLUSION Drogin, Bob. ―In a Doomsday Cyber Attack Scenario,We now have the means to mobilize the brainpower of Answers Are Unsettling.‖ Los Angeles Times online.our nation‘s finest cyber defenders – using the power of 2/17/10. Seeserious gaming combined with simple crowdsourcing Leveraging this ―collective brain‖ of top cyber-attack17-2010feb17.performers in the federal cyber workforce promises toproduce a higher-quality result in a more efficient way Erickson, Thomas. ―Some Thoughts on a Framework forthan could be produced previously. Crowdsourcing: A Position Paper for the CHI 2011 Workshop on Crowdsourcing and HumanAs envisioned by the ONR, this approach of combining Computation.‖ CHI 2011 Workshop oncrowdsourcing with serious games provides for novel Crowdsourcing and Human Computation. 5/8/11.combinations and complex interactions of ideas that―otherwise might not emerge from more traditional Gladwell, Malcolm. Outliers: The Story of Success. Newwargame approaches‖ (Ungerleider, 2011). Serious York, NY: Little, Brown & Company. can incorporate crowdsourcing, to scale effortsthat might otherwise be difficult for teams to do alone, Hoover, Nicholas J. ―Homeland Security Seeks Newor elicit information or monitor activities that people Ideas.‖ Information Week. 11/4/09. Seemight otherwise be unable to articulate (i.e., making knowledge explicit). nfo-management/221600274.The bottom line is that serious games are becoming Kim, Julia. ―ICT at the Army Science Conference inimportant tools for problem solving that harness crowds Orlando.‖ 12/3/10. ICT blog. Seeof users into contributing their time and knowledge in a way. Through a rapid and agile process, the conference-in-orlando.convergence of crowdsourcing and gaming provides anew and effective way for current training development McCann, Sandi. ―Trends We‘re Watching: Social Gamemodels to integrate lessons learned from expert Marketing.‖ The Heinrich Report blog. 6/14/11. Seeperformers and improve personnel proficiency. game-marketing. ACKNOWLEDGEMENTS McGonigal, Jane. ―Gaming Can Make a Better World.‖The authors would like to acknowledge and thank Mr. TED Talk. Recorded March 2010. SeeJim Schifalacqua, Director of the Cyber Center of at Serco North America, for his an_make_a_better_world.html.contributions and support of this paper. Rigby, Ben. ―Information Age Volunteerism: Open Sourced! Crowdsourced!‖ Tech President blog. 2/6/09. See Paper No. 11116 Page 7 of 8
  8. 8. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2011 entry/information-age-volunteerism-open-sourced- crowdsourced.Stansbury, Meris. ―Feds Turn to ‗Crowdsourcing‘ for Educational Innovation.‖ eSchool News. 5/10/10. See to-crowdsourcing-for-educational-innovation.Ungerleider, Neal. ―Wannabe SEALs Help U.S. Navy Hunt Pirates in Massively Multiplayer Game.‖ FastCompany online. 5/10/11. See massively-multiplayer-pirate-hunting-game.Wikipedia. 2011 Paper No. 11116 Page 8 of 8