Your SlideShare is downloading. ×
0
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Needs Assessment Webinar to AFC of ISPI
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Needs Assessment Webinar to AFC of ISPI

437

Published on

A webinar on needs assessment presented to the Armed Forces Chapter of ISPI

A webinar on needs assessment presented to the Armed Forces Chapter of ISPI

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
437
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Introduce Team: BIOS:
  • On 11 September 2008 a new Project Management Process was implemented within the client organization following a one day training event. The Project Officers were advised that each project assigned to them would consist of a project charter, a project plan, bi-weekly status reports, and a final project report. From September 2008 to January 2009, only two of the seven Project Officers completed all the status reports as required. In October the Project Officers requested that the bi-weekly reporting requirement be dropped to monthly. It was, however, reporting continued to drop off. In January 2009, the project status reporting requirement was put on hold pending the outcome of this needs assessment.
  • DO POLL HERE When you conduct a needs assessment do you include data gathering?
  • There are all kinds of ways to collect DATA What is DATA? DATA is Wikipedia: Data are often viewed as a lowest level of abstraction from which information and knowledge are derived. Dictionary.com: The plural of datum. Individual facts, statistics, or items of information: Merriam-Webster: factual information (as measurements or statistics) used as a basis for reasoning, discussion, or calculation Seems everyone agrees that data is a snippet that we can look at with other snippets and turn it into information based on how we – the analysts – perceive it! Knowing what data is, is fine… now we need to know how to get it!
  • Definition “ Qualitative or quantitative data collected for governmental, research, education, or service purposes and available to researchers in usable raw data forms and formats” (Schensul, Schensul, and LeCompte, 1999, p. 201). Purpose: Archival Data is information that may or may not relate to the need being assessed. For example, data may pertain to a different organization, a time period, or else it may have been collected by people other than the practitioner who will use it. Archival Data is different from other data sources such as interviews, focus groups, surveys, and observations. While these sources can be influenced by the HPT practitioner, archival data cannot. In other words, the practitioner can only work with the archival data already available.
  • Ways You Can Use Archival Data As a general rule, first-hand data is preferable to archival data. However, there are several situations in which the use of archival data can be beneficial to a Needs Assessment. Should Use Archival Data If: data exists that is relevant to the group being assessed primary data is not available obtaining primary data would be overly costly or time-consuming the practitioner conducting the needs assessment has insufficient resources to obtain accurate primary data the practitioner seeks to compare an existing situation to those outside of the organization the practitioner seeks to identify trends among other groups (Explain this point – trend across several groups taking the same course for example) Should Not Use Archival Data If: available archival data requires significant manipulation to be relevant to the needs assessment being conducted the practitioner is not capable of analyzing the data properly (e.g., lacks knowledge in statistical analysis) public data is not readily available obtaining Archival Data is cost-prohibitive the methodology to obtain the Archival Data is flawed the accuracy of the Archival Data cannot be verified How We Used Archival Data: To determine actual reporting frequency by project officer and by project To determine exemplar vs average vs poor performers To assist in creation of BOTG
  • Pros Cost effective Limited time commitment Can help validate observations or other findings (UNICEF, 2005) Can help to identify data trends May provide a larger context for the data (UNICEF, 2005) Society-wide, larger amounts of information are being stored electronically (IBM, 2007), increasing the quantity of archival data available Cons Data is often not a direct match to current situation Data may require difficult manipulation to make it relevant Some secondary data may only be available at a cost (Schensul, Schensul, and LeCompte, 1999) Data may not be current
  • DO Poll here I would use archival data to... Correct answer is Find trends among other groups (Y)
  • DEFINITION: “A data-collection method used with individuals that can be conducted in person, by phone, or by computer technology (such as online cameras, or videoconferencing).” Gupta et al. (2007) p.309 Individuals rather than groups, as more than 1 person may be considered a “focus group” we all know group dynamics can come into play with focus groups may shut down some interviewees, arguments, and the “Boss is in the room” our means of interviewing via technology are increasing all the time PURPOSE: (your general purpose might be…)
  • WAYS TO USE/APPROACH interviews: (you might use them to…) use to get by-in: if done on client, mgrs., employees > (helps with ownership of findings and willingness to support change effort) use to help gage change mgt. progress and acceptance use as a tool in evaluation of the initiative’s results as part of a Performance Analysis (especially is doing one apart from the NA as part of your Cause Analysis as part of your Change Mgt. process to see how the initiative is being received as part of your Evaluation to deepen or expand your knowledge in a certain area or field POLLING QUESTION : OVERALL FORMAT/ QUESTION FORMS : Structured (questions laid out, limit possible answers, very focused) Semi-structured (some questions laid out, maybe probes written down to remind interviewee to expand on their answers, open to giving interviewee some latitude in their responses while still keeping the interview focused) Unstructured (much more of a relaxed approach, may still have a general focus, but very open to where the interviewee takes you) QUESTION FORMS: Open ended (gives the person lots of room to respond as they see fit, may give you things you would not have thought of) Closed ended (somehow limits the person’s response, maybe used especially when you are trying to make sure the data will be easily quantifiable) Probing (maybe written down under your main open ended q’s to remind you to make sure to get more detailed information on something incase the interviewee does not hit on it during his open response) HOW WE USED THE INTERVIEWS : some questions based on BEM factors as a means to juxtapose supervisor perception with employee perception to add qualitative data in addition to our qualitative data
  • PROS: (electronic) gives reach to hard-to-get-at people (outpost in Alaska) (electronic) convenience for interviewee, maybe interviewer also (electronic) give control / flexibility to interviewee cut / past features (electronic) good for limited budget programs (travel, time…) (electronic) may be set up for anonymity / confidentiality (in-person) good interviewer can build trust and rapport CONS: (in-person) no chance for anonymity, person may be afraid to tell it as it is (in-person) nervousness on part of interviewee, may change answers due to same
  • DO POLL HERE The most common method of collecting data is... Correct answer is INTERVIEWS
  • Take a side step here to talk about the tool that was important for data gathering and led to smooth analysis, which is the cause analysis worksheet. This tool, which was developed by Roger Chevalier, has it’s origins in Gilbert’s Behavior Engineering Model and Kurt Lewin’s force field analysis tools. If you’re not familiar with the Force Field Analysis, it is a method for listing, discussing, and evaluating the various forces for and against a proposed change – which in this case would be a gap. The cause analysis worksheet overlays these two models, which we’ll see in a minute. There is quite a bit of background information on this model, and we would encourage you to read the reference noted in the job aid, which is titled “the evolution of a performance analysis job aid” – because it has evolved quite interestingly over the past few years.
  • DEFINITION “ A data-collection method typically used to gather data from many people at one time; can be mailed, e-mailed, or offered online” (Gupta, 2007, p. 313). PURPOSE To collect qualitative and quantitative data of things that cannot be observed, such as opinions, knowledge, and feelings. WAYS YOU CAN USE IT
  • In our Needs Assessment, we used surveys to gather data regarding expectations, feedback, knowledge and skill levels, consequences, incentives, motives – all of which fall into the Six Factors of Gilbert’s BEM, which is what we used to develop and structure our questions (Information, Resources, Incentives, Knowledge, Capacity, and Motives). We revised and refined our questions several times and decided to conduct a pilot survey. After the pilot survey we again revised our questions and then sent out the final survey. We used 4 point Likert-scale matrix questions, grouped together in sets, with each set followed by an open-ended question. We chose to conduct the survey electronically, via Opinio web-based software. We had an 86% participant rate, which we fill gave us sufficient data. PILOT Six sets of matrix questions, 27 total Four-point Likert scale (Strongly Disagree, Disagree, Agree, Strongly Agree) Open-ended textbox after each set Five questions regarding survey construction Time requirement, unclear questions, terminology, etc. Two participants FINAL Questions revised based on Pilot survey Six sets of matrix questions, 29 total Four-point Likert scale (Strongly Disagree, Disagree, Agree, Strongly Agree) Open-ended textbox after each set 86% completion rate
  • There are several pros & cons outlined in the job aid we’ve provided, and here are the ones we found most relevant to our needs assessment project. PROS Electronic surveys don’t cost anything to send out unless you have to pay for the survey tool. But there are some tools out there, like survey monkey, which are free. Questions are the same for everyone so there’s no variability between surveys like with interviews. Whenever possible, it’s good to send surveys to a larger group than you actually need to ensure you get sufficient data. (e.g., if you want 50 participants, maybe send it to 100 people) CONS It’s easy to ignore or forget about a survey, so reminder notes are a good idea. Participants may misread or misinterpret questions, which is another reason pilot surveys are beneficial. PROS Quantitative data that is gathered is easy to compile and analyze; Time Commitment – much quicker to take a 5-10 minute survey than 30-60 minute interview Participant size – In turn, you can have a larger participant size because data is more manageable Question preparation – no variability between participants No time constraints – they don’t have to be completed within 10 minutes CONS Paper surveys can be expensive (paper, ink, postage, etc.) Electronic surveys – require computer, internet Additional Info – limited to the questions in the survey
  • DO POLL Fill in the blank: I_____complete surveys I receive at work
  • DO 6X WEBINAR EVAL POLLING QUESTIONS HERE
  • Transcript

    • 1. Needs Assessment Methods and Tools Brett Christensen Kayleen Grage Mike Hajba Barb Spice
    • 2. Outline <ul><li>Background of the project </li></ul><ul><li>Methods employed </li></ul><ul><ul><ul><li>Archival Data </li></ul></ul></ul><ul><ul><ul><li>Interviews </li></ul></ul></ul><ul><ul><ul><li>Cause Analysis </li></ul></ul></ul><ul><ul><ul><li>Surveys </li></ul></ul></ul><ul><li>Lessons Learned </li></ul><ul><li>Q & A </li></ul><ul><li>Your evaluation of this webinar </li></ul>
    • 3. Dilbert on Data Collection
    • 4. Background <ul><li>Sep 2008: Client implements Project Management Process </li></ul><ul><li>Project status reports inconsistent </li></ul><ul><li>Frequency of reports is reduced </li></ul><ul><ul><ul><li>Project reporting still drops </li></ul></ul></ul><ul><li>Project reporting suspended </li></ul><ul><li>Needs Assessment requested </li></ul>
    • 5. Background <ul><li>Client requests assistance from BSU IPT 529 (Needs Assessment) class </li></ul><ul><li>Project Plan drafted, reviewed, and approved by client and professor </li></ul>
    • 6. Data Collection Methods
    • 7. Archival Method <ul><li>Definition </li></ul><ul><ul><ul><li>“ Qualitative or quantitative data collected for governmental, research, education, or service purposes and available to researchers in usable raw data forms and formats” (Schensul et al., 1999) </li></ul></ul></ul><ul><li>Purpose </li></ul><ul><ul><ul><li>To find already existing information that relates to the need being assessed </li></ul></ul></ul>
    • 8. Using Archival Data <ul><li>Ways you can use archival data </li></ul><ul><ul><ul><li>If primary data is not available/too expensive </li></ul></ul></ul><ul><ul><ul><li>To identify trends among other groups </li></ul></ul></ul><ul><li>How we used archival data </li></ul><ul><ul><ul><li>To determine actual reporting frequency by project officer and by project </li></ul></ul></ul><ul><ul><ul><li>To determine exemplar vs. average vs. poor performers </li></ul></ul></ul><ul><ul><ul><li>To assist in creation of Behaviour Over Time Graph (BOTG) </li></ul></ul></ul>
    • 9. Archival Data Pros & Cons <ul><li>Pros </li></ul><ul><ul><ul><li>Cost effective </li></ul></ul></ul><ul><ul><ul><li>Good for trend analysis and identification </li></ul></ul></ul><ul><ul><ul><li>More and more data is available electronically </li></ul></ul></ul><ul><li>Cons </li></ul><ul><ul><ul><li>Data requires manipulation to make it relevant </li></ul></ul></ul><ul><ul><ul><li>Currency of data can be an issue </li></ul></ul></ul>
    • 10. Dogbert on Data Mining
    • 11. Dogbert on Qualitative Data
    • 12. Interview Method <ul><li>Definition </li></ul><ul><ul><ul><li>“ A data-collection method used with individuals that can be conducted in person, by phone, or by computer technology (such as online cameras, or videoconferencing)” (Gupta et al., 2007) </li></ul></ul></ul><ul><li>Purpose </li></ul><ul><ul><ul><li>To get information and opinions about some issue in the current needs assessment </li></ul></ul></ul>
    • 13. Using Interviews <ul><li>Ways you can use interviews </li></ul><ul><ul><ul><li>Collect data for a needs assessment, study, paper, etc. </li></ul></ul></ul><ul><ul><ul><li>Source of more in-depth data (follow-up) </li></ul></ul></ul><ul><ul><ul><li>Source to triangulate other data </li></ul></ul></ul><ul><li>How we used interviews </li></ul><ul><ul><ul><li>Source to triangulate survey and archived data </li></ul></ul></ul><ul><ul><ul><li>Via phone and web conferencing software </li></ul></ul></ul><ul><ul><ul><li>Semi-structured approach, with probes </li></ul></ul></ul>
    • 14. Interview Pros and Cons <ul><li>Pros </li></ul><ul><ul><ul><li>Allows for a good build of rapport </li></ul></ul></ul><ul><ul><ul><li>Can use/see body language if in-person </li></ul></ul></ul><ul><ul><ul><li>Easy, wide reach with technology </li></ul></ul></ul><ul><li>Cons </li></ul><ul><ul><ul><li>Resource intensive (people, travel, time, etc.) </li></ul></ul></ul><ul><ul><ul><li>Strong interviewee/weak interviewer can derail an interview </li></ul></ul></ul><ul><ul><ul><li>Differences in responses due to different interviewers </li></ul></ul></ul>
    • 15. Dogbert on Interviews
    • 16. Cause Analysis Worksheet <ul><li>ORIGINS </li></ul><ul><li>Behavior Engineering Model (BEM) (Gilbert, 2007) </li></ul><ul><li>Force Field Analysis (Lewin, 1947) </li></ul><ul><li>Cause Analysis Worksheet (Chevalier, 2007) </li></ul>WHAT IT IS HOW TO USE IT BENEFITS
    • 17. Cause Analysis Worksheet <ul><li>ORIGINS </li></ul><ul><li>Behavior Engineering Model (BEM) (Gilbert, 2007) </li></ul><ul><li>Force Field Analysis (Lewin, 1947) </li></ul><ul><li>Cause Analysis Worksheet (Chevalier, 2007) </li></ul><ul><li>WHAT IT IS </li></ul><ul><li>Identifies current and desired performance with a defined reasonable goal </li></ul><ul><li>Structured tool to guide the development of information for data gathering and subsequent analysis using BEM factors and sub factors </li></ul><ul><li>Incorporates weighting of data </li></ul>HOW TO USE IT BENEFITS
    • 18. Cause Analysis Worksheet Example BEM Factor Force Field Weighting BEM Sub Factors
    • 19. Cause Analysis Worksheet Adaptation for Project BEM Factor BEM Sub Factors FACTORS DRIVING FORCES RESTRAINING FORCES INFORMATION Clear Expectations 1a. Expectations for the project status reporting process are clear. Relevant Feedback 1b. Feedback given to me on my project status reports is timely. 1c. I receive feedback on my project status reports for good performance. 1d. I receive feedback on my project status reports for not so good performance. 1e. Feedback I have received on my performance on my project status reports is relevant. Relevant guides/job aids 1f. The handout (containing job aids, templates, and examples) for project status reporting is useful. Performance coaching 1g. I receive coaching regarding the quality of my performance on project status reports. Purpose/use of the reports 1h. I understand how my project status reports will be used by others.
    • 20. Cause Analysis Worksheet <ul><li>ORIGINS </li></ul><ul><li>Behavior Engineering Model (BEM) (Gilbert, 2007) </li></ul><ul><li>Force Field Analysis (Lewin, 1947) </li></ul><ul><li>Cause Analysis Worksheet (Chevalier, 2007) </li></ul><ul><li>WHAT IT IS </li></ul><ul><li>Identifies current and desired performance with a defined reasonable goal </li></ul><ul><li>Structured tool to guide the development of information for data gathering and subsequent analysis using BEM factors and sub factors </li></ul><ul><li>Incorporates weighting of data </li></ul><ul><li>HOW TO USE IT </li></ul><ul><li>Identify gap </li></ul><ul><li>Using BEM factors, revise sub factors relative to problem </li></ul><ul><li>Place results of data on weighted scale for analysis </li></ul>BENEFITS
    • 21. Cause Analysis Worksheet <ul><li>ORIGINS </li></ul><ul><li>Behavior Engineering Model (BEM) (Gilbert, 2007) </li></ul><ul><li>Force Field Analysis (Lewin, 1947) </li></ul><ul><li>Cause Analysis Worksheet (Chevalier, 2007) </li></ul><ul><li>WHAT IT IS </li></ul><ul><li>Identifies current and desired performance with a defined reasonable goal </li></ul><ul><li>Structured tool to guide the development of information for data gathering and subsequent analysis using BEM factors and sub factors </li></ul><ul><li>Incorporates weighting of data </li></ul><ul><li>HOW TO USE IT </li></ul><ul><li>Identify gap </li></ul><ul><li>Using BEM factors, revise sub factors relative to problem </li></ul><ul><li>Place results of data on weighted scale for analysis </li></ul><ul><li>BENEFITS </li></ul><ul><li>Systematic approach </li></ul><ul><li>Links data gathering directly with analysis </li></ul><ul><li>Data is logically grouped by factor </li></ul>
    • 22. Dogbert on Cause Analysis
    • 23. Surveys <ul><li>Can be used before, after, or with other data collection methods </li></ul><ul><li>Collect quantitative and qualitative data </li></ul><ul><li>Collect opinions, knowledge, feelings </li></ul><ul><li>Written, face-to-face, phone, electronic </li></ul>
    • 24. Using Surveys <ul><li>Collaboratively written with Interview data </li></ul><ul><li>Structured around Gilbert’s BEM and Chevalier’s Cause Analysis worksheet </li></ul><ul><li>Reviewed and revised several times </li></ul><ul><li>Pilot survey, revisions, final survey </li></ul><ul><li>Electronic via Opinio </li></ul><ul><li>Emailed invitation with link </li></ul>
    • 25. Surveys: Pros & Cons <ul><li>PROS </li></ul><ul><ul><ul><li>Inexpensive </li></ul></ul></ul><ul><ul><ul><li>Consistency in questions and presentation </li></ul></ul></ul><ul><ul><ul><li>Participants can fill out surveys on their own time </li></ul></ul></ul><ul><ul><ul><li>Can send to a larger participant pool than needed </li></ul></ul></ul><ul><ul><ul><li>Can be anonymous </li></ul></ul></ul><ul><li>CONS </li></ul><ul><ul><ul><li>Completion rate is generally low </li></ul></ul></ul><ul><ul><ul><li>Questions may be misinterpreted </li></ul></ul></ul><ul><ul><ul><li>Participants can’t ask for clarification </li></ul></ul></ul><ul><ul><ul><li>Participants’ motivation/mood can impact </li></ul></ul></ul><ul><ul><ul><li>Unable to probe for additional information </li></ul></ul></ul>
    • 26. Dogbert on Surveys
    • 27. Lessons Learned Key Points Things to Consider Planning <ul><li>Factor in time constraints </li></ul><ul><li>Be thorough </li></ul><ul><li>Develop clear understanding of data to be collected </li></ul><ul><li>Data gathering tools feed data analysis tools </li></ul>Instrument Design <ul><li>Keep it simple </li></ul><ul><li>Design for user friendly administration </li></ul><ul><li>Pilot test instruments </li></ul>
    • 28. Lessons Learned Key Points Things to Consider Interviews <ul><li>Keep interview focused </li></ul><ul><li>Determine if recording will enhance or inhibit data </li></ul><ul><li>Provide feedback for in-process changes as additional interviews are conducted </li></ul>Surveys <ul><li>Simple is better </li></ul><ul><li>Comments provide good information </li></ul><ul><li>Know your purpose </li></ul><ul><li>Allow extra time to write and rewrite questions </li></ul><ul><li>Plan to pilot your survey </li></ul><ul><li>Keep the survey as short and concise as possible </li></ul>
    • 29. Questions
    • 30. How Did We Do? <ul><li>Six quick questions… </li></ul>
    • 31. References <ul><li>BFMartin.ca. The Dilbert search works. Retrieved June 18, 2009, from http://bgmartin.ca/finder </li></ul><ul><li>Chevalier, R. (2007). A manager’s guide to improving workplace performance. New York: American Management Association. </li></ul><ul><li>Gilbert, T. (2007). Human competence: Engineering worthy performance (Tribute ed.). San Francisco, CA: Pfeiffer. </li></ul><ul><li>Gupta, K., Sleezer, C., & Russ-Eft, D. (2007). A practical guide to needs assessment (2nd ed.). San Francisco, CA: Pfeiffer/ASTD. </li></ul><ul><li>Lewin, K. (1947). Frontiers in group dynamics: Concept, method, and reality in social science, equilibria and change. Human Relations 1 (1), 5-41. </li></ul><ul><li>Schensul, S., Schensul, J., & LeCompte, M. (1999). Essential ethnographic methods: Observations, interviews and questionnaires (Vol. 2). Walnut Creek, CA: Alta Mira Press. </li></ul>

    ×