This interim report summarizes research on using crowdsourcing as a potential hiring pipeline for federal agencies. A simulated crowdsourcing application was developed and tested with volunteer applicants. Key findings include that crowdsourcing could identify candidates for specialized occupations, but commercial applications require customization. The test was successful, indicating crowdsourcing has potential as a recruitment tool to supplement existing methods. Recommendations include presenting findings to agency leaders and human resource organizations.

1. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Interim Report
by Thomas F. Kaplan
ESHRI/R&D PMO
June 2011
Readers should note that the
enclosed report is only an
example.

2. Project Manager Contact Details
Name: Thomas F. Kaplan
Address: 1900 E Street, NW, Room 3304b, Washington, DC 20415
Telephone: 202-606-4454
Email: thomas.kaplan@opm.gov
Not for general distribution without express permission of the HRS R&D PMO ii

3. Table of Contents
Executive Summary..............................................................................................................1
Introduction.........................................................................................................................2
Objectives............................................................................................................................3
Methodology ....................................................................................................................4
Research Description............................................................................................................5
Results.................................................................................................................................6
Implications..........................................................................................................................7
Recommendations ..............................................................................................................8
Appendices..........................................................................................................................9
Appendix A: Project Request
Appendix B: Potential Uses of Crowdsourcing (extract)
Glossary
References
Not for general distribution without express permission of the HRS R&D PMO iii

4. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Executive Summary
Leveraging crowdsourcing for use as a potential
hiring pipeline may give agency recruiters additional
sources of potential candidate supply (in addition to
applicant inventories). Briefly, this project
attempted to research the uses of crowdsourcing for
use as a potential recruitment tool. Specifically:
In 2009, crowdsourcing applications became widely
available and appeared to have potential for use in as
an adjunct for federal recruiting. A review of
available research at that time indicated there was no
ongoing investigation on the use of crowdsourcing in
federal recruitment.
Since this project involved simulated use of crowdsourcing as a hiring tool, it quickly became
apparent that, in general, commercial “off-the-shelf” crowdsourcing applications were not well
suited for this task. On the other hand, the research indicated that customization of the
crowdsourcing application could assist recruitment efforts in certain circumstances, including
targeted recruitment efforts (e.g., for specialized engineering occupations such as electronics
engineers with specialization in automotive and railroad industries, etc.).
Two existing customers at the Department of Transportation (notably that had difficulty in
surfacing any candidates for electronics engineer positions at the National Highway
Transportation Safety Administration and the Federal Railways Administration) expressed an
interest. This current HRS customer was interested in pursuing a test development application
based on the project’s outcome.
The project, in general, progressed without significant delay in meeting its schedule. The
project scope remained unchanged from its original definition and the budget was adequate to
achieve project’s original intent.
Project lessons learned include:
• Engage in-house and contractor staff early in the R&D process
• Spend more time presenting and discussing research goals and objectives with
potential customers
• Daily, spend more time documenting project actions and outcomes.
Not for general distribution without express permission of the HRS R&D PMO 1
Crowdsourcing
is the act of outsourcing tasks,
traditionally performed by an
employee or contractor, to an
undefined, large group of people
or community (e.g., a crowd),
through an open call.

5. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Introduction
The project objective was to explore potential recruitment pipeline enhancement and adjunct
methods using crowdsourcing opportunities. This project would potentially allow agencies to
develop and access large numbers of candidates differentiated by various recruitment
categories (e.g., veteran, PWTD, etc.). Project outcome could lead to an additional method
for identifying potential job candidates.
The idea for this research project came from an article published in the June 2009 “The
American Journal for Applied Research,” by Ann Benson (Appendix B).
This research is important as it directly supports OPM’s Hiring Reform Initiative. Leveraging
crowdsourcing as a potential hiring pipeline opportunity can give recruiters additional sources
of candidate supply (in addition to applicant inventories). Additionally, crowdsourcing
leverages emerging social media networking toward use as a potential recruitment method.
Utilizing the “power” of social networking may boost recruitment opportunities not presently
available through regular methods.
This project began in October 2010 and completed in September 2011. Project participants
included myself and Samantha Jones, who acted as the crowdsourcing application technician.
Project beneficiaries potentially include HR staffs at two operating administrations at the
Department of Transportation and other defense and civilian agency HR organizations.
Key findings include current crowdsourcing applications must be customized to realize the full
potential of identifying potential candidates for specific occupations. Customization should
yield a useful recruitment application that identifies job candidates not possible through
contemporary methods. Additionally, agency IT operations would need to ensure
maintenance of 508 compliance and IT security.
The remainder of this report describes in detail the research conducted and specific findings.
Not for general distribution without express permission of the HRS R&D PMO 2

6. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Objectives
From the Project Request Form I, Objective (Appendix A): “Use commercially available
crowdsourcing sites (e.g., IdeaScale, etc.) to identify potential candidates for federal
employment.” Additional objectives included 1) determine the usefulness of crowdsourcing as
a recruitment tool, 2) evaluate the degree to which a crowdsourcing tool would need to be
modified, if at all, for recruitment purposes (to meet Title 5 USC requirements), and 3)
establish a brief process description of the information flow between potential applicant, the
crowdsourcing application, and federal recruiters.
Achievement of objectives relied on a non-governmental crowdsourcing application to identify
potential candidates for federal employment. As such, there was no commercial sector
analogue to follow and there was only time enough to develop a limited test application to
gain a proof of concept; hence, application modification was limited, but included a means to
assess candidate interest and academic credentials and basic competencies.
An actual crowdsourcing application, intended to support federal hiring, would likely require
extensive research and planning to be truly effective. For example, researchers completed only
limited flow process charting involving a three-tiered hierarchy; this would likely need to be
expanded to at least a five-tiered hierarchy process description to determine if logical flow
exists between the processes.
While the crowdsourcing application would need to conform to federal IT standards (e.g.,
FIPS-2, etc.) the government would not need to host the application thus allowing greater
leverage of cloud computing opportunities through partnering with the private sector. If
successful, using a crowdsourcing application to identify potential federal job applicants could
reduce agencies’ information technology investment, over time.
The next section describes the project’s methodology.
Not for general distribution without express permission of the HRS R&D PMO 3

7. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Methodology
To achieve the main and additional three objective in the previous section required the
• evaluation of extant crowdsourcing providers,
• developing a Request for Information and its publication on FedBizOps,
• meeting individually with potential commercial crowdsourcing providers to understand
the uses, constraints, and limitations of the crowdsourcing applications,
• contracting with a crowdsourcing provider (IdeaScale) to provide a simulated
environment where potential job applicants would interact with the application, and
• developing interface templates for potential applicants to submit their credentials and
competencies for evaluation and agency follow-up.
Government employee researchers designed and applied several interface templates’ with
specifications including
• notice of user legal and compliance requirements prior to use of the application
• capture of contact information
• statement of academic achievement
• disclosure of certifications and credentials
• self-assessment of general and technical competencies, and other relevant tasks
Fifty pre-selected volunteer test subjects were given access to the crowdsourcing application
and encouraged to enter at least five different fictitious profiles of academic, competency, and
certification achievement for use as a proof of concept. This test generated 237 separate
profiles for use in evaluating the attainment of stated objectives.
The next section presents the research description.
Not for general distribution without express permission of the HRS R&D PMO 4

8. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Research Description
After selecting a crowdsourcing provider, research staff met with the selected crowdsourcing
contractor to evaluate and establish the range of actions needed to complete the project. The
research staff and the contractor developed mock-up web site screens for review by staffs at
the Office of the Chief Information Office and Office of the General Counsel. Research staff
modified web site screens to include statements on all pages stating that the web site was a
simulation for test purposes only.
Research staff provided test crowdsourcing web site information to pre-selected volunteers
who were given basic instructions on the purpose and use of the web site. Input screens were
shown to the volunteers and the research staff answered volunteer’s questions. The fifty
volunteers were provided website access for five days to input data into the crowdsourcing
application. Thirty-six of the fifty volunteers successfully entered all requested data, while the
remaining 24 failed to complete the data entry tasks (due to simulated disruption of website
access by the research staff). Seventy-two percent of the volunteers were able to remain
logged in to the crowdsourcing application over the five days to complete required data entry
tasks.
Thirty-three volunteers completed exit questionnaires describing their experience using the
crowdsourcing application.
The next section presents the research results.
Not for general distribution without express permission of the HRS R&D PMO 5

9. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Results
This test application of a crowdsourcing application as a potential hiring pipeline, in this case
as a precursor to applying for vacant federal positions and applicant inventories for
prospective job candidates, provided the following results:
• General:
Thirty-three of the 36 volunteer applicants completed the on-line crowdsourcing
profile in less than 2 hours.
Twelve of the 36 volunteer applicants indicated they were able to complete the
online crowdsourcing profile in less than one hour.
Five of the 36 volunteer applicants indicated they called researchers at least once
for guidance on using the on-line crowdsourcing application.
Research staff reviewed the 33 completed volunteer profiles and determined that
28 of the profiles achieved 100 percent data completion.
• Technical:
The crowdsourcing application remained operational 24 hours per day during the
five-day test period. Researchers intentionally disrupted volunteer users at random
intervals to simulate network contention often experienced by on-line system users.
All volunteers entered “unscripted” information (i.e., they entered purely individual
personal information)
No volunteer applicants were able to cause the crowdsourcing application to fail to
retain entered data.
• Other:
Research staff was able to retrieve and analyze volunteer applicant data during the
five-day access period.
A review of volunteer applicants exit questionnaires indicated an overall score of 4
out of 5 (5 indicating a high degree of satisfaction) with their experience using the
crowdsourcing application.
The next section presents research implications.
Not for general distribution without express permission of the HRS R&D PMO 6

10. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Implications
Research staff and the project manager deemed the research effort as generally successful
given the limitations and constraints1
of the project. While there may be a need for additional
R&D efforts to fully evaluate crowdsourcing as a potential “hiring” tool, this test indicates
that using a crowdsourcing application to identify potential applicants for federal job vacancies
has merit. Looking forward, it may be possible to replace current agency succession planning
“applicant inventories” with a crowdsourcing application that 1) quickly identifies interested
job applicants, 2) reduces some agency IT costs2
, and 3) gives agencies an additional source
for identifying potential applicants with critically important skills.
The relatively low-cost investment involved in researching the potential use of the
crowdsourcing application (less than $45K) and the use of uncompensated volunteers allowed
researchers to achieve a proof of concept that crowdsourcing is useful tool in identifying
potential job applicants for difficult to fill positions.
The next section presents recommendations.
1
Limitations include the small number of volunteer applicants involved and the short duration of the test
period.
2
Some IT costs are borne by the crowdsourcing provider (e.g., application hosting, help desk services, etc.).
Not for general distribution without express permission of the HRS R&D PMO 7

11. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Recommendations
The following list presents tentative recommendations for evaluation against current business
plans.
• Provide a summary of this report to current customer agency Chief Human Capital
Officers (CHCO) and determine if they would be interested in developing and
implementing a customized solution.
• Present the project findings at the CHCO Training Academy.
• Consider promoting/sharing the results of this research at public venues (e.g.,
Performance Institute, Partnership for Public Service, SHRM, PIHRA, etc.).
Not for general distribution without express permission of the HRS R&D PMO 8

12. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Appendices
Appendix A: Project Request
Appendix B: Potential Uses of Crowdsourcing (extract)
Not for general distribution without express permission of the HRS R&D PMO 9

13. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Appendix A: Project Request
Project Request Part I
1. Project name (without acronyms or abbreviations) and today’s date. Research Hiring
Pipeline Opportunities using Crowdsourcing. August 5, 2010.
2. Requested project description. Explore potential recruitment pipeline
enhancement/adjunct using crowdsourcing opportunities. This project would potentially
allow agencies to develop and access large numbers of candidates differentiated by various
recruitment categories (e.g., veteran, PWTD, etc.). Project outcome could lead to an
additional method for identifying potential job candidates.
3. Practice Area and Sponsor Name. Also identify other Practice Areas that could benefit
from the proposed project. HRS/ESHRINNOV, Thomas F. Kaplan. Unaware of benefit for
other Practice Areas.
4. When did the project begin (state month and year), or is it new? This is a new project.
5. If the request involves a legacy R&D project, what was/is the project name? NA
6. List each key characteristic of the proposed project under the headings:
Feature Function Advantage (technical/economic/social)
Wide availability World Wide Web based Internet based social media provider
3rd
party hosted
Commercially owned and
operated hardware and
software platform
Low cost alternative to equivalent
government owned and operated site
Open
Government
compliant
Facilitates greater
transparency
Helps OPM achieve Presidential mandate
Agency
availability
All Executive Branch agencies
can access a vast hiring
pipeline
Extends OPM leadership capabilities
7. Describe any other special aspect or wider applications of the requested project. None.
8. Give a best estimate of any resources used to date (e.g., 2.0 full-time equivalents, $10,000,
expendable research materials, 100sq ft lab space, 40% use of equipment, etc). 10 hours
or 0.005 FTE of the sponsor’s duty time.
Not for general distribution without express permission of the HRS R&D PMO

14. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
9. Describe the requested R&D efforts in terms of objectives, requirements for human
resources, risk, space, and relate these items to the funds and time required. Use
commercially available crowdsourcing sites (e.g., IdeaScale, etc.) to identify potential
candidates for federal employment. Government staff will maintain the site for an estimate
of 1,000 hours per year. Risk in undetermined. Funds required are approximately $1,000 for
an annual subscription for unlimited users, ideas, and commenters with refund for unused
portion. See table.
Category Funds Risk Space Time
Labor (GS-11/5 DC) w/.284
benefits
$33.92ph x
1.284 = $43.55 x
1000 = $43,550
Moderate 1,000 hours
Non-labor $999.003
Moderate One year
Other NA NA NA NA
10. Explain if a proof of concept and prototype will result from the requested project. If not,
what extra work would need to be done and by whom. Proof of concept and prototype will
likely result from the project.
11. Please add any other information you consider relevant. Leveraging crowdsourcing as a
potential hiring pipeline can give recruiters additional sources of candidate supply in
addition to applicant inventories. Additionally, using reverse auction procurement approach
would likely drive down cost of crowdsourcing subscription. Please refer to attached
commercial crowdsourcing documents for additional information.
3
IdeaScale Subscription for prototyping/proof-of-concept testing.
Not for general distribution without express permission of the HRS R&D PMO

15. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Project Request Part II
1. Project name (without acronyms or abbreviations) matching the project name from
Research and Development Project Request Part I and today’s date. Research Hiring
Pipeline Opportunities using Crowdsourcing, August 5, 2010.
2. Is there a patentable invention/copyrighted product, or could one arise from this R&D
effort? No.
3. Has there been any public disclosure about the project? If yes, please give details. The
project sponsor knows of no known public disclosure.
4. Comment on any R&D or commercialization by other researchers in the field of the
project. Unknown.
5. Please list any companies and organizations that could be interested in the project, details
of any contacts and reasons for the likely interest. IdeaScale, Innocentive, Ninesigma, etc.
No contacts as of yet. Its likely these companies would show interest in the R&D effort since
it would (further) demonstrate their relevance in the federal marketplace.
6. Please name any similar services or products that already exist. Unknown.
7. Please compare the new service or product with any existing alternative that provides
similar outcomes. NA.
8. Please list the types of customers and end-users that are likely to purchase or benefit from
the new product or service. Customers include federal agencies and State/local agencies
and others interested in identifying potential public sector job candidates.
9. Please provide an estimate of market locations and annual market size of the new service
or product. Market locations are worldwide since host platform is virtual (via World-Wide
Web). Annual market size is undetermined at this time; however, the market potential is
vast.
10. Is there a need for Government approval for the release of the service or product? Yes.
11. Please complete a Project Cost Estimate Worksheet and attach it to this project request.
Not for general distribution without express permission of the HRS R&D PMO

16. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Project Cost Estimate Worksheet
Project Name: Research Hiring Pipeline Opportunities using Crowdsourcing
Today’s Date: August 5, 2010
Planned Project Start Date: January 1, 2011
Planned Project End Date: December 31, 2011
One-year subscription for IdeaScale with unlimited use is $999.
Not for general distribution without express permission of the HRS R&D PMO
Cost Category Year 1 Year 2 Total
Salaries $42,269
Benefits 0.00
Awards 0.00
Total Salaries and Benefits $42,269
Travel
Transportation of Things
Communications/Utilities/Misc
Printing
Training
Contracts $999
Other Services
Supplies
Equipment
Total non-Salaries and Benefits $999
Total $43,268
Federal Employee Hours 1,000

17. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Appendix B: Potential Uses of Crowdsourcing (extract)
The following extract came from the June 2009 edition of “The American Journal for Applied
Research,” by Ann Benson.
“Identifying potential government employees with critically needed skills can be a laborious
effort. Searching online resume banks, conducting job fairs, communicating with college and
university placement offices may not yield candidates in areas such as electronics engineering
and IT architecture and design.
Recruiters need a new approach that leverages the ‘power’ of crowdsourcing. Consider the
following scenario, which presents a possible adaptation of crowdsourcing to create a “job
interest pipeline.”
Agency A’s succession plan identifies a future skill shortage of electronics engineers. The
agency has attempted to identify potential future candidates through traditional means
(including USA Jobs recruiting for future needs) with limited success.
At the same time, Bill Murphy is in his senior year at the DeForest University pursuing a
degree in general engineering with an emphasis in electronics. Lately, Bill Murphy is
wondering about his future employment opportunities when he learns about the “Job
Opportunity Bank” (The Bank) at his university’s placement office.
Bill learns more about The Bank. The Bank allows college level students to place their
personal and academic profiles online for review by government agencies. Bank students
invite professors and other students to comment on the veracity of the profiled student and to
add their comments. Bank students control which comments are allowed to be viewed by
recruiters and hiring managers.”
Not for general distribution without express permission of the HRS R&D PMO

18. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Glossary
Note: A glossary is optional.
Term Definition
Crowdsourcing The act of outsourcing tasks, traditionally performed by an
employee or contractor, to an undefined, large group of people
or community (e.g., a crowd), through an open call.
Job Interest Pipeline An online database of candidates interested in certain
occupations.
Not for general distribution without express permission of the HRS R&D PMO

19. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
References
Identify footnotes, references, and cross-references.
Note: References is optional.
Not for general distribution without express permission of the HRS R&D PMO

20. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
Plain English Summary
This summary format is here as a reminder and it should not be included in the TOC of the
final report. All final (not interim) projects must have a plain English summary.
Project Title: Research of Hiring Pipeline Opportunities using Crowdsourcing
Researcher: Thomas F. Kaplan
Organization: U.S. Office of Personnel Management, Human Resources Solutions,
center for Emerging Solutions and HR Innovations, Research and
Development Program Management Office
Telephone: (202) 606-4454
Fax: (202) 606-1982
Email: thomas.kaplan@opm.gov
Objectives Research the use of commercially available crowdsourcing sites (e.g.,
IdeaScale, etc.) to identify potential candidates for federal
employment. Additionally, 1) determine the usefulness of
crowdsourcing as a recruitment tool, 2) evaluate the degree to which
a crowdsourcing tool would need to be modified, if at all, for
recruitment purposes (to meet Title 5 USC requirements), and 3)
establish a brief process description of the information flow between
potential applicant, the crowdsourcing application, and federal
recruiters.
Methodology To achieve the main and additional three objective in the previous
section required the evaluation of extant crowdsourcing providers,
developing a Request for Information and its publication on
FedBizOps, meeting individually with potential commercial
crowdsourcing providers, contracting with a crowdsourcing provider,
and developing interface templates for potential applicants to submit
their credentials and competencies for evaluation and agency follow-
up.
Research
Description
Research staff met with the selected crowdsourcing contractor to
evaluate and establish the range of actions needed to complete the
project. The research staff and the contractor developed mock-up
web site screens for review by staffs at the Office of the Chief
Information Office and Office of the General Counsel. Research staff
provided test crowdsourcing web site information to pre-selected
volunteers who were given basic instructions on the purpose and use
Not for general distribution without express permission of the HRS R&D PMO

21. FY 2011 R&D Project Report
Research of Hiring Pipeline
Opportunities using Crowdsourcing
of the web site. Input screens were shown to the volunteers and the
research staff answered volunteer’s questions. The fifty volunteers
were provided website access for five days to input data into the
crowdsourcing application. Thirty-six of the fifty volunteers
successfully entered all requested data.
Results This test application of a crowdsourcing application as a potential
hiring pipeline, in this case as a precursor to applying for vacant
federal positions and applicant inventories for prospective job
candidates, indicated an overall score of 4 out of 5 (5 indicating a
high degree of satisfaction) with their experience using the
crowdsourcing application.
Implications Looking forward, it may be possible to replace current agency
succession planning “applicant inventories” with a crowdsourcing
application that 1) quickly identifies interested job applicants, 2)
reduces some agency IT costs, and 3) gives agencies an additional
source for identifying potential applicants with critically important
skills. The relatively low-cost investment involved in researching the
potential use of the crowdsourcing application (less than $45K) and
the use of uncompensated volunteers allowed researchers to achieve a
proof of concept that crowdsourcing is useful tool in identifying
potential job applicants for difficult to fill positions.
Recommendations Provide a summary of this report to current customer agency Chief
Human Capital Officers (CHCO) and determine if they would be
interested in developing and implementing a customized solution.
Additionally, present the project findings at the CHCO Training
Academy, and consider promoting/sharing the results of this research
at public venues (e.g., Performance Institute, Partnership for Public
Service, SHRM, PIHRA, etc.).
Not for general distribution without express permission of the HRS R&D PMO