Integrated Performance Assessment: Linking Individual Development to Industry Standards

2,639 views

Published on

This research presentation was presented at the 2007 AECT Conference in Anaheim, CA.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,639
On SlideShare
0
From Embeds
0
Number of Embeds
73
Actions
Shares
0
Downloads
43
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Integrated Performance Assessment: Linking Individual Development to Industry Standards

  1. 1. Integrated Performance Assessment (IPA) Linking Individual Development to Industry Standards Tammé McCowin Doctoral Candidate University of Phoenix 1 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  2. 2. Agenda Topic Time (Mins.) Literature Review 10 ISD Performance Inventory 5 Scale Development Process 10 Peer Activity: Using the ISD Performance 40 Inventory Research Participation 5 Questions and Answers 5 2 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  3. 3. Literature Review Topics • Symptoms and Issues • Problem Statement • Research Purpose • Research Question • Literature Search • Research Findings • IPA Research Methodology 3 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  4. 4. Symptoms and Issues • Instructional technology (IT) professionals must continually update and develop their skills in an ever- changing world market due to global competition, new methods and tools, fluctuations in the market, business acumen, strategy, positioning, and technological innovations (Lui, Gibby, Quiros, & Demp, 2002; Katz, 2005) • There is a demand for IT professionals to continually adapt to agile work environments while acquiring new competencies in a rapidly changing & Hempstreet, 1998; Katz, 2005) industry (Curtis, Hefley, & Miller, 2002; Lui et al., 2002; Lui, Jones, 4 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  5. 5. Symptoms and Issues • IT professionals “have a responsibility to keep their skills current” (Rothwell & Kazanas, 2004, p. 386) • It is an important and essential competency to update and improve one’s knowledge, skills, and abilities (Richey, Fields, Foxon, Roberts, Spannaus, & Spector, 2005) • Skill utilization, extension, and versatility are critical components for sustaining professional success and technical professionals must continually extend their knowledge and skill capabilities to grow, innovate, and lead (Katz, 2005) 5 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  6. 6. Problem Statement • Effective job performance can be mitigated when no means is available to accurately and objectively assess the performance of IT professionals on industry defined competency standards Instructional Technology (IT) 6 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  7. 7. Research Purpose • Career Planning and Professional Development o To provide a valid and reliable measurement methodology to guide career planning and professional development activities • Leadership Development o To develop technical leadership by focusing on professional competencies, maturation, experience, and context • Selection and Guidance Counseling o To provide an effective and objective measurement methodology to aid employers and educators with making better decisions about job and student applicants (e.g. classification, placement, educational programming, recruitment, and selection) 7 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  8. 8. Research Question • What are the valid and reliable competencies, methods, and tools for assessing the preparation and performance of IT professionals? Why ask this question? What someone thinks he or she is capable of doing, how others know that he or she is capable, and how that capability may be demonstrated is not always in alignment. Instructional Technology (IT) 8 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  9. 9. Literature Search Number of References 9 9 8 7 26 reference sources found 6 6 5 5 Total 4 4 3 2 2 1 0 0 Q ue s t ia P ro / P s y ER IC D is s e rt a t io ns ETR &D P IQ Literary Sources 9 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  10. 10. Literature Search Number of Research Studies 11 12 10 8 Total 6 18 research studies found 4 4 1 2 1 1 0 IS D P ra c t ic e CD MD TD Ot he r Study Type Competency Development (CD), Model Development, (MD), Tool Development (TD) 10 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  11. 11. Study Findings • ISD Practice Studies o Software tools for empowering instructional developers (Gayeski, 1991) o What do instructional designers actually do? An initial investigation of expert practice (Rowland, 1992) o Designer thinking: How novice and experts think about instructional design (Perez & Emery, 1995) o Context sensitive instructional design models: A response to design theory, practice, and criticism (Wedman & Tessmer, 1995) o The present and future of ID practice (Winer & Vazquez-Abad, 1995) o Interactive multimedia design and production processes (Lui & Hempstreet, 1998) o A history of instructional design and technology: Part I: A history of instructional media 2001a) (Reiser, 11 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  12. 12. Study Findings • ISD Practice Studies o A history of instructional design and technology: Part II: A history of instructional design (Reiser, 2001b) o The challenge of being an instructional designer for new media development: A view from the practitioners (Lui, Gibby, Quiros, & Demps, 2002) o Paradigms in the theory and practice of education and training design (Visscher-Voerman & Gustafson, 2004) o Instructional design practice: Career environments, job roles, and a climate of change (Larson & Lockee, 2004) 12 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  13. 13. Study Findings • CD Studies o Roles and competencies of instructional design as identified by expert instructional designers: a qualitative study* (Atchison, 1996) o An examination of the instructional design competencies written by the International Board of Standards for Training, Performance, and Instruction* (Song, 1998) o Instructional design competencies: The standards (3rd ed.) (Richey, Fields. Foxon, Roberts, Spannaus, & Spector, 2001) o Building a foundation for the tomorrow: Skill standards for information technology 2003 edition (NWCET, 2003) Competency Development (CD) * Denotes dissertation study 13 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  14. 14. Study Findings • MD Studies o ID model construction and validation: A multiple intelligences case (Tracey & Richey, 2007) o The construction and validation of an instructional systems design model incorporating multiple intelligences* (Tracey, 2001) • TD Studies o The validity of a multiple-choice, paper and pencil instrument in discriminating between masters and nonmasters of instructional design* (Stepp, 1990) • Other o The comparative study of information competencies-using Bloom’s taxonomy (Ven & Chuang, 2005) Model Development, (MD), Tool Development (TD) * Denotes dissertation study 14 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  15. 15. IPA Research Methodology assessed Perceived demonstrated 1 2 3 4 ISD Performance Multiple Intelligences Integrated Skills Integrated Inventory ISD Construct Assessment Performance Study Map Study Study Assessment Study RQ RQ1 H1 RQ What are the valid and What is the relationship IT professionals who are What is the relationship reliable competencies between MI Constructs able to correctly between perceived for assessing the and ISD competencies? respond to a large assessed and preparation and number of competency- demonstrated RQ2 performance of IT intelligence items will performance? What combination of MI professionals? exhibit skill integration. Constructs and ISD competencies influence H2 an IT professional’s skill IT professionals who are integration? not able to correctly respond to a large RQ3 number of competency- What combination of MI intelligence items will Constructs and ISD exhibit skill imbalance. competencies influence an IT professional’s skill imbalance? Research Question (RQ), Hypotheses (H), Instructional Systems Design (ISD) 15 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  16. 16. ISD Performance Inventory Topics • Scale Composition • Rating Scale • Sample Professional Foundation Scale 16 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  17. 17. Scale Composition Scale C PS SI Professional Foundations 5 26 63 Planning and Analysis 7 30 59 Design and Development 6 32 58 Implementation and Management 5 34 56 Digital Media 5 35 63 Technical Writing 5 30 52 Web Development and Administration 8 55 92 Total 41 242 443 *Scale items were written to align to industry standards. *International Board of Standards for Training Performance and Instruction (IBSTPI) C - Competencies, PS - Performance Statements, SI - Scale Items *National Workforce Center for Emerging Technologies (NWCET) 17 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  18. 18. Rating Scale Main Scale Equivalent Scale 1. Absolutely need 1. Novice 2. Strongly need 2. Advanced Beginner 3. Definitely need 3. Competent 4. Somewhat need 4. Proficient 5. No need 5. Expert Rating oneself as an expert on an individual item does not imply that you cannot or do not need to ever make improvements. Rather it is an indication of where you believe you are to date regarding where you are in your career and the current state of the industry and based on the standards that are relevant right now. 18 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  19. 19. Sample Professional Foundations Scale C1: Communicate Effectively in visual and oral form Performance Scale Item Rating Scale Statement Create messages that When designing instruction, write learner centric content accommodate learner that is factual, clear, unambiguous, and tied to relevant 1 2 3 4 5 needs and supporting visual aids to improve learning and characteristics, content, and comprehension of subject matter. objectives When designing instruction, write instructional objectives that clearly describe a measurable performance. 1 2 3 4 5 When designing instruction, write learner centric content that maps directly to documented instructional 1 2 3 4 5 objectives. Write and edit text to When designing instruction, write learner centric content produce messages that is clear, concise, and grammatically correct. 1 2 3 4 5 that are clear, When designing instruction, edit learner centric content concise, and grammatically that is clear, concise, and grammatically correct. 1 2 3 4 5 correct. 1 – Absolutely need, 2 – Strongly need, 3 – Definitely need, 4 –Somewhat need, 5 – No need 19 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  20. 20. Scale Development Process Topics • Four Step Scale Development Process • Construct Delineation • Item Generation • Pilot Studies • Validation Studies • Scale Sample Sizes 20 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  21. 21. Four Step Scale Development Process 1 2 3 4 Construct Item Pilot Test Validation Study Delineation Generation Create higher order Develop item pool Selects small study Select random sample factors group samples Create response format Administer instrument Create factors Administer instrument to samples and collect Conduct expert reviews to samples and collect data Create variables data Conduct factor and Create control variables Conduct factor and reliability analysis reliability analysis Conduct concurrent Refine instrument validity analysis 21 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  22. 22. Factor-Item Structure Relationships 1 2 3 Scales Professional Foundations Planning and Analysis Knowledge Higher Ordered = = Design and Development Domains Factors Implementation and Management (7) (7) Digital Media Technical Writing Web Development and Administration Competency Factors ISDC = = Statements (41) (41) (41) Performance Variables = = Statements Scale Items (242) (242) (443) Performance Statements (PS), Instructional Systems Design Competencies (ISDC) 22 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  23. 23. Item Generation Nine-Step Review Process Qualitative Review Step 1: Determine statement’s behavioral characteristics Step 2: Determine statement’s unitary value Step 3: Determine statement’s clarity Step 4: Determine statement’s duality focus Step 5: Determine statement’s functional purpose Step 6: Determine statement’s face validity Quantitative Review Step 7: Determine inter-rater reliability of initial item pool Step 8: Determine statement’s measurement type Step 9: Determine statement’s skill level 23 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  24. 24. Pilot Studies • Two sample groups will be Group Samples pulled to complete pilot Phase One Instrument Development studies • Study 1: 10-20 participants • Study 2: 10-20 participants • Group 1 will be used to test the skill inventory for reliability and factor structure relationships • Group 2 will be used to refine the skill inventory for construct validity and to further confirm factor structure relationships 24 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  25. 25. Validation Studies • There are two sample Group Samples study groups needed to Phase Two Instrument Validation • Study 3: 276-460 participants complete validation • Study 4: 75 participants studies • Study 3 is to finalize the All participants who register to skill inventory and confirm participate will be randomly placed in one or more study groups. Email test-retest reliability communication will be sent to you at the start of study 1 to explain the • Study 4 is to establish details and list which group or groups you have been randomly concurrent and predictive placed in. validity 25 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  26. 26. Scale Sample Sizes Scale C PS SI Sample Professional Foundations 5 26 63 189-315 Planning and Analysis 7 30 59 177-295 Design and Development 6 32 58 174-290 Implementation and Management 5 34 56 168-280 Digital Media 5 35 63 189-315 Technical Writing 5 30 52 156-260 Web Development and Administration 8 55 92 276-460 Total 41 242 443 276-460 The sample size of a scale is generally five times the size of a scale (DeVellis, 2003; Netemeyer, Bearden, & Sharma, 2003) *International Board of Standards for Training Performance and Instruction (IBSTPI) C - Competencies, PS - Performance Statements, SI - Scale Items *National Workforce Center for Emerging Technologies (NWCET) 26 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  27. 27. Peer Activity: Using the ISD Performance Inventory Topics • Reviewer Types • Instructions to Self Raters • Instructions to Third-party Raters • Peer Activity: How to Use the ISD Performance Inventory 27 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  28. 28. Reviewer Types • Self Rater – An individual who voluntarily uses the ISD Performance Inventory to rate themselves to obtain performance feedback • Third-party Rater – An individual who voluntarily uses the ISD Performance Inventory to rate another person to provide performance feedback (e.g. peer, teacher, manager, team member, etc.) 28 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  29. 29. Instructions to Self-Raters • You will be presented with items for each scale • Rate the level of professional development you require on each item using the rating scale • Use the ISD Skill Level Response Format Descriptions to judge your professional development on each item relative to the rating scale 29 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  30. 30. Instructions to Third-party Raters • Use the interview rubric to interview your partner • Rate the level of professional development your partner requires on each item using the rating scale • Use the ISD Skill Level Response Format Descriptions to judge his or her professional development on each item relative to the rating scale 30 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  31. 31. Peer Activity • Select a partner and How to Use the ISD determine your role (5-mins) Performance Inventory • Third-party rater interviews C1: Communicate Effectively in visual and oral form his or her partner (15-mins) Performance Scale Item Rating Scale Statement Create messages that When designing instruction, write learner centric content accommodate learner that is factual, clear, unambiguous, and tied to relevant 1 2 3 4 5 needs and • Self rater and third-party supporting visual aids to improve learning and characteristics, content, and comprehension of subject matter. objectives When designing instruction, write instructional objectives that clearly describe a measurable 1 2 3 4 5 performance. raters complete the When designing instruction, write learner centric content that maps directly to documented instructional 1 2 3 4 5 objectives. Write and edit text When designing instruction, write learner centric content to produce messages that is clear, concise, and grammatically correct. 1 2 3 4 5 that are clear, inventory (10-mins) When designing instruction, edit learner centric content concise, and grammatically that is clear, concise, and grammatically correct. 1 2 3 4 5 correct. • Compare and share findings (5-mins) • Reporting and analytics (5- mins) 40-mins 31 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  32. 32. Research Participation Topics • Eligibility Requirements • Participation Benefits • How to Participate 32 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  33. 33. Eligibility Requirements • Anyone who falls into one of the instructional design roles or both and has performed the functions of any of these roles Instructional Designer Role Instructional Developer Role analyst, course developer, content editor, content developer, eLearning developer, eLearning specialist, electronic publisher, instructional developer, information evaluator, communications, designer, architect, information developer, creative writer, document specialist, programmer, project manager, media instructional designer, interaction designer, designer/specialist, multimedia online publisher, project manager, technical author/specialist, multimedia developer, communicator, technical writer, trainer, training developer, user interface developer, training designer/specialist, usability web developer, web media developer, designer, user interface designer, web technologist, teachers, students designer, teacher, students 33 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  34. 34. Participation Benefits • When you register and fully participate in the research study you will – Receive your choice of one online training course to promote professional development toward linking your skills to industry standards – Contribute to the research and validation of a online for performance assessment tool professional development – Earn a chance to win either one 30GB video iPod or one of two 4GB iPod Nanos 34 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  35. 35. How to Participate • To participate you must: o Register at the research website, provide your name, email address, contact number, and some general demographic information o Acknowledge and give your formal consent to participate by selecting the terms and conditions check box on the registration form o Finalize your registration by clicking the link in the confirmation email that will be sent to you http://www.iparesearch.org http://iparesearch.blogspot.com Let’s transform professional development together! Share the link with your colleagues and friends! 35 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development
  36. 36. Questions and Comments? 36 2007 International AECT Conference, 10/27/2007 © 2007 IPA Research Foundation Transforming Professional Development

×