Your SlideShare is downloading. ×
0
Welcome!
Please …
1. Take a seat with your grant team members (or
anywhere if you are solo).
2. Grab some coffee.
3. Compl...
Accessing the Moodle Website
Before we get started, we want to make sure everyone has
successfully accessed the private Go...
Golden LEAF Essential Skills Initiative
Summer Evaluation Institute
June 27, 2013
The Friday Institute – Raleigh, NC
Golden LEAF Essential Skills Initiative
Evaluation Team
The Friday Institute at NCSU
• Jeni Corn
• Latricia Townsend
• Rob...
Agenda
• Welcome and Introduction
• The Evaluation Process & Logic Models
• Team Logic Modeling
• LUNCH
• Asking Good Ques...
The Golden LEAF Essential Skills Initiative
“Supporting collaborative programs aimed at increasing
the talent pool of a hi...
The Initiative Evaluation
• Grant artifacts
• CC registrar data
• Student surveys
• Instructor surveys
• Employer surveys
...
Who Is in the Room?
One person from each grant team
stand up and please share:
• What Community Colleges your
team is repr...
The Evaluation Process
Plan, Do, Study, Act (PDSA)
Loyola University Stritch
School of Medicine, 2011,
http://www.stritch.luc.edu/lu
men/MedEd/so...
Five Evaluation Steps
1. Logic Model: Identify the critical elements of the
workforce development program.
2. Evaluation Q...
Communicate & Collaborate
• Include key people in the
evaluation planning process:
• Especially the primary users of
the r...
Keep it Simple and Focused
• An evaluation doesn’t
have to be big! Match
the number of
evaluation questions in
your plan t...
STEP 1: Logic Model
Identify the critical elements of the workforce
development program
Formative Evaluation Steps
1.Logic Model: Identify the critical
elements of the workforce
development program.
2. Evaluati...
ACTIVITY
Write answers to questions 1-3, answering them in that order, on post-
its. You will likely have multiple answers...
What is a logic model?
A logic model is a
graphic representation
of the relationships
among the key
elements of a project:...
What is a logic model?
What is a logic model?
www.ScienceCartoonsPlus.com
What is a logic model?
Inputs Activities Outputs
Short-term
Outcomes
Medium-
term
Outcomes
Long-term
Outcomes
What is
inve...
What is a logic model?
Terms aren’t consistently
used … meaning is what
matters.
What is a logic model?
Whenever possible, it is helpful to try to use
the “SMART Goals” strategy for creating your target
...
Strategies Short Term Goals Long Term Goals
Regional career work-based
experiences supported by
business community
Partner...
Team Logic Modeling 11:00am – 12:15pm
1. In your grant teams, on a computer, draft a logic
model for your grant. (Use
PPT,...
Lunch!
12:00 – 12:30pm
Pack up your tables a little, grab
lunch, and find a new table to sit at
with a conversation topic ...
Asking Good Evaluation
Questions
Formative Evaluation Steps
1. Logic Model: Identify the critical elements of the
workforce development program.
2. Evaluat...
ACTIVITY
What do you think of when you think of
the term “evaluation question”?
Look at your logic model. Jot down 1 or
2 ...
Developing Evaluation Questions
• Evaluation questions provide the direction and
foundation for the entire evaluation.
• T...
Developing Evaluation Questions
Why do we need to ask good questions?
• To determine what is really important, and to whom...
Developing Evaluation Questions
Every evaluation question can’t be answered - finding out the
answers costs time, money an...
Keep it Simple and Focused
• An evaluation doesn’t
have to be big! Match
the number of
evaluation questions in
your plan t...
Developing Evaluation Questions
The main types of evaluation questions are:
1. Questions about STRATEGIES: these questions...
Developing Evaluation Questions
Quick tips for writing good questions:
1. Try to avoid simple “yes or no” questions
2. Con...
Developing Evaluation Questions
IMPLEMENTATION/STRATEGY
questions:
- How many hours of sleep am I getting
each week? (quan...
ACTIVITY
Brainstorm a list of evaluation questions, and write them down.
1. Brainstorm implementation questions about the ...
Balanced Scorecard
What is the Balanced Scorecard?
• A 4-quadrant framework that provides a comprehensive
picture of performance
• Originally...
Development of the Balanced Scorecard
Combined
Evaluation
Strategy
Workforce
Education
Internal ProcessPartnerships
Learni...
Combined
Evaluation
Strategy
Workforce Education
Outcomes associated with changes
in educational access and
achievement in...
Gallery Walk – What are the data sources?
Combined
Evaluation
Strategy
Workforce Education
Increase access to programs in ...
Break!
For 10 minutes.
Hope Opportunity Jobs
Dr. Kristen Corbell, Coordinator of Research Projects
North Carolina Community College System Office...
Hope Opportunity Jobs
Data Initiative Overview
The Data Initiative includes the
review, assessment and revision of our
cur...
Hope Opportunity Jobs
Data Initiative Motivation
• Concerns over data quality at the college and state level
• Increasing ...
Hope Opportunity Jobs
Data Initiative Objectives
• Expand the individual and collective inquisitiveness of stakeholders
wi...
Hope Opportunity Jobs
Data Initiative Objectives
• Develop focused topic-based data marts that will serve a wide
variety o...
Hope Opportunity Jobs
Continuous
Improvement
Cycle
Hope Opportunity Jobs
Culture of Curiosity / Strategic Initiatives
49
Culture of Curiosity
Represents individual and colle...
Hope Opportunity Jobs
How the Essential Skills Grant fits in with the
Culture of Curiosity
• Within this grant you are ask...
Hope Opportunity Jobs
Data Quality, Consistency, and Accountability
51
Data Quality and Consistency
Ensures information is...
Hope Opportunity Jobs
How the Essential Skills Grant fits in with Data
Quality and Consistency
• Using the data definition...
Hope Opportunity Jobs
Data Collection and Warehousing
53
Data Extraction, Transformation, and Loading
Procedural routines ...
Hope Opportunity Jobs
Culture of Evidence / Policy Implications
54
Culture of Evidence / Policy Implications
•Determinatio...
Wrap-Up
Calendar of Evaluation Activities – Year One
ACTIVITY DATE
Summer Institute 2013 June 2013
Grant Coordinator Phone Intervi...
Please Take 5 Minutes to Complete Survey
Go to GLF ES Moodle site.
Go to the Summer Institute 2013 Section
Click on the Es...
Thank You
Questions? Comments? Concerns?
Have a good trip home!
Michael Maslin Item
#: 8476186
Upcoming SlideShare
Loading in...5
×

Glfes summer institute2013_raleigh_final

61

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
61
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • The Data Initiative is one of our system initiatives. We have approximately 200 people across the state involved in the initiative. This Data initiative will review, assess, and revise current data collections and reporting processes. The end result will be a data system that has accurate and accessible information for all 58 colleges that will foster the culture of data driven decision making to address questions and inform policy.
  • Concerned about data qualityDemands for dataNeed to have decision makers having access to dataImprove reporting toolsHave sound data to base policies on
  • Your focus today will be directly related to the Culture of Curiosity and strategic initiatives. However, this will effect your data entry for the data quality, consistency, and accountability. Data that are already collected in Colleague and sent up in the flat file will be extracted and provided to the FI for quantitative analyses. Kristen will be working with the evaluation team throughout the grant regarding data being provided.
  • Transcript of "Glfes summer institute2013_raleigh_final"

    1. 1. Welcome! Please … 1. Take a seat with your grant team members (or anywhere if you are solo). 2. Grab some coffee. 3. Complete the GLF Evaluation Capacity Assessment. 4. Get connected to the wireless network: NCSU Guest. Instructions are in your folder. 5. Get access to the GLF Essential Skills Evaluation “Moodle” website (where we will organize project information). Instructions are in your folder.
    2. 2. Accessing the Moodle Website Before we get started, we want to make sure everyone has successfully accessed the private Golden LEAF Essential Skills Evaluation Moodle website. • Instructions are in your folder. • Raise your hand if you are having difficulty and an evaluation team member will assist you.
    3. 3. Golden LEAF Essential Skills Initiative Summer Evaluation Institute June 27, 2013 The Friday Institute – Raleigh, NC
    4. 4. Golden LEAF Essential Skills Initiative Evaluation Team The Friday Institute at NCSU • Jeni Corn • Latricia Townsend • Rob Maser • Malinda Faber • Laurie Brummit NCSU College of Education • Diane Chapman • Julia Storberg-Walker NC Community College System • Kristen Corbell
    5. 5. Agenda • Welcome and Introduction • The Evaluation Process & Logic Models • Team Logic Modeling • LUNCH • Asking Good Questions • Balanced Scorecard • BREAK • Culture of Curiosity & Data Definitions • Closing and Q&A
    6. 6. The Golden LEAF Essential Skills Initiative “Supporting collaborative programs aimed at increasing the talent pool of a highly skilled technical workforce aligned with identified employment opportunities in tobacco-dependent, economically distressed and/or rural communities of North Carolina.” 14 Grants 22 Community Colleges 14 School Districts
    7. 7. The Initiative Evaluation • Grant artifacts • CC registrar data • Student surveys • Instructor surveys • Employer surveys • Focus groups • Interviews • NC Education Research Data Center administrative data • NC Community College System administrative data • Employment Securities Commission workforce data Evaluation Questions Data Sources TBD
    8. 8. Who Is in the Room? One person from each grant team stand up and please share: • What Community Colleges your team is representing • One sentence description of your grant’s main focus
    9. 9. The Evaluation Process
    10. 10. Plan, Do, Study, Act (PDSA) Loyola University Stritch School of Medicine, 2011, http://www.stritch.luc.edu/lu men/MedEd/softchalkhdht/C MEFacDevWebPage/CMEFacD evWebPage10.html CYCLE OF CONTINUOUS IMPROVEMENT
    11. 11. Five Evaluation Steps 1. Logic Model: Identify the critical elements of the workforce development program. 2. Evaluation Questions: Ask important questions about your workforce development program. 3. Data Sources: Identify what data are available to help answer your questions and determine the additional data needed. 4. Data Analysis: Collect, analyze, and interpret data to answer your questions. 5. Decisions: What changes to your workforce development program should you make based on your result. Repeat steps 1-5 11
    12. 12. Communicate & Collaborate • Include key people in the evaluation planning process: • Especially the primary users of the results. • This is a great opportunity to build a shared understanding and buy-in for your project. • Communicate regularly that this is on-going process of improvement – “This is not a gotcha evaluation.”
    13. 13. Keep it Simple and Focused • An evaluation doesn’t have to be big! Match the number of evaluation questions in your plan to your resources. • Focus on efficient, effective data collection strategies. A basic chart is a great way to organize your own thinking and to share easily the plan with others. Evaluation Questions Data Sources
    14. 14. STEP 1: Logic Model Identify the critical elements of the workforce development program
    15. 15. Formative Evaluation Steps 1.Logic Model: Identify the critical elements of the workforce development program. 2. Evaluation Questions: Ask important questions about your workforce development program. 3. Data Sources: Identify what data are available to help answer your questions and determine the additional data needed. 4. Data Analysis: Collect, analyze, and interpret data to answer your questions. 5. Decisions: What changes to your workforce development program should you make based on your result. Repeat steps 1-5 15
    16. 16. ACTIVITY Write answers to questions 1-3, answering them in that order, on post- its. You will likely have multiple answers - write one idea per post-it. 1. Ultimately, what good is your program going to do (for students, employers, etc.)? 2. If you are successful implementing grant activities, what will your program accomplish this year? 3. What are the major activities of your program? (How are you going to spend your time, money, and other resources?) Arrange your post-its on a piece(s) of paper from left to right, from major activities (3), to immediate results (2), to the ultimate good the program is going to do (1). 3  2  1 .
    17. 17. What is a logic model? A logic model is a graphic representation of the relationships among the key elements of a project: • inputs, • strategies, • objectives, • long-term goals. A logic model … • Helps to articulate the key elements of the project. • Enables evaluation efficiency and effectiveness. • Promotes stakeholder buy-in by helping clarify how the project works. • Provides a great opportunity to involve stakeholders in planning. They are living documents – they can change over time!
    18. 18. What is a logic model?
    19. 19. What is a logic model? www.ScienceCartoonsPlus.com
    20. 20. What is a logic model? Inputs Activities Outputs Short-term Outcomes Medium- term Outcomes Long-term Outcomes What is invested in the process? Time, money, human resources, partners, equipment, etc. What are the major activities the program will entail? What will be done with the time, money, etc.? How many participants are involved? How many teachers and/or students are reached? What is produced? Within life of grant: What changes in knowledge, skills, attitudes, motivations, or awareness resulted? Within 1-3 Years: What changes in behaviors, practices, policies, or procedures resulted? After 3 or more Years: What are the ultimate goals and did changes results? Student achievement, teacher retention, school culture, etc.
    21. 21. What is a logic model? Terms aren’t consistently used … meaning is what matters.
    22. 22. What is a logic model? Whenever possible, it is helpful to try to use the “SMART Goals” strategy for creating your target outcomes: Specific Measurable Attainable Results-oriented Time-bound E.g.: Increase number of students taking advanced science courses by 25% by fall 2015.
    23. 23. Strategies Short Term Goals Long Term Goals Regional career work-based experiences supported by business community Partnerships to provide enrichment activities for students and differentiated PD for teachers Curriculum alignment for vertical and cross curriculum through PLCs Increased access to content specific technology for science and math Increase enrollment in Algebra 1 for middle school students and upper level math and science Increase percentage proficiency in Math and Science Increase use of inquiry-based learning in lessons Raise STEM interest and awareness Increase student engagement and achievement in the STEM courses 23 Development of Leadership for supporting STEM instruction PD in Inquiry-Based learning Increase student interest and teacher awareness in STEM careers Promote STEM educator effectiveness through professional development Early diagnosis and intervention for college readiness assessment
    24. 24. Team Logic Modeling 11:00am – 12:15pm 1. In your grant teams, on a computer, draft a logic model for your grant. (Use PPT, Word, Inspiration, …) 2. After drafting your logic model, reflect on your PMPs - are there changes you want to make? 3. (You will get another 20 minutes to reflect on your PMPs in the afternoon.)
    25. 25. Lunch! 12:00 – 12:30pm Pack up your tables a little, grab lunch, and find a new table to sit at with a conversation topic you’re interested in (look for table tents). THIS AFTERNOON: Asking Good Questions; Balanced Scorecard; Community College System Data Initiative.
    26. 26. Asking Good Evaluation Questions
    27. 27. Formative Evaluation Steps 1. Logic Model: Identify the critical elements of the workforce development program. 2. Evaluation Questions: Ask important questions about your workforce development program. 3. Data Sources: Identify what data are available to help answer your questions and determine the additional data needed. 4. Data Analysis: Collect, analyze, and interpret data to answer your questions. 5. Decisions: What changes to your workforce development program should you make based on your result. Repeat steps 1-5 27
    28. 28. ACTIVITY What do you think of when you think of the term “evaluation question”? Look at your logic model. Jot down 1 or 2 examples of what you think could be evaluation questions for your program.
    29. 29. Developing Evaluation Questions • Evaluation questions provide the direction and foundation for the entire evaluation. • The process for identifying the questions to be answered by the evaluation is critical. EVALUATION QUESTIONS Data Collection Data Analysis Results
    30. 30. Developing Evaluation Questions Why do we need to ask good questions? • To determine what is really important, and to whom • Considering “to whom” will help you when you need to decide what kind of data you want to collect to answer the particular question. • To focus data collection efforts • What do we need to find out? • How will we collect that information?
    31. 31. Developing Evaluation Questions Every evaluation question can’t be answered - finding out the answers costs time, money and people. Pick the most important questions that provide the most valuable information to users.
    32. 32. Keep it Simple and Focused • An evaluation doesn’t have to be big! Match the number of evaluation questions in your plan to your resources. • This will help you focus on efficient, effective data collection strategies. A basic chart is a great way to organize your own thinking and to share easily the plan with others. Evaluation Questions Data Sources
    33. 33. Developing Evaluation Questions The main types of evaluation questions are: 1. Questions about STRATEGIES: these questions ask about how well the strategies were implemented. 2. Questions about OBJECTIVES: these questions ask about impacts. Logic models are great guides for developing evaluation questions.
    34. 34. Developing Evaluation Questions Quick tips for writing good questions: 1. Try to avoid simple “yes or no” questions 2. Consider QUANTITY questions, e.g: – “How many” – “How much” – “How often” 3. Consider QUALITY questions, e.g.: – “How well” – “How effectively” – “In what ways” 4. Be able to be tuned-in to unexpected results.
    35. 35. Developing Evaluation Questions IMPLEMENTATION/STRATEGY questions: - How many hours of sleep am I getting each week? (quantity) - How soundly am I sleeping? (quality) IMPACT/OBJECTIVE questions: - How much weight have I lost? (quantity) - How has my stress level changed? (quality)
    36. 36. ACTIVITY Brainstorm a list of evaluation questions, and write them down. 1. Brainstorm implementation questions about the program’ strategies For example: THE STRATEGY IS: Teachers attend 4 quarterly sessions on science kits. A QUANTITY QUESTION: How many faculty/staff attended the sessions? A QUALITY QUESTION: How did the participants rate the quality of the sessions? 2. Brainstorm impact questions to evaluate how well the outcomes are being met. Crosswalk your list of evaluation questions with your PMP. Do the activities and outcome measures in your PMP capture all of your evaluation questions and the data you would use to answer them? Fill in any gaps.
    37. 37. Balanced Scorecard
    38. 38. What is the Balanced Scorecard? • A 4-quadrant framework that provides a comprehensive picture of performance • Originally designed for business and industry, now regularly adapted to different sectors – Original four quadrants are financial, customer, internal processes, learning and growth • Adapted to the GLF Essential Skills project, the quadrants are partnerships, workforce education, internal processes, learning and growth • GLF evaluation objective: Aggregate reporting of four quadrants
    39. 39. Development of the Balanced Scorecard Combined Evaluation Strategy Workforce Education Internal ProcessPartnerships Learning & Growth
    40. 40. Combined Evaluation Strategy Workforce Education Outcomes associated with changes in educational access and achievement including dual degree and 3rd party certifications Internal Process Outcomes associated with changes in enrollment and capacity Partnerships Outcomes associated with partners (community, industry, education) including partnership strength, activities and impact Learning & Growth Outcomes associated with changes in grantee knowledge, skills, attitudes, and capabilities Essential Skills Evaluation Balanced Scorecard
    41. 41. Gallery Walk – What are the data sources? Combined Evaluation Strategy Workforce Education Increase access to programs in targeted areas Increase program completion in targeted areas attributed to grant proposal and collaborative partnerships Increased dual degree opportunities attributed to grant proposal and collaborative partnerships Increase in 3rd party credentials attributed to grant proposal and collaborative partnerships Internal Process Increase enrollments in targeted courses and programs Increase capacity attributed grant proposal Increase capacity attributed collaborative relationships Partnerships Increase students served Increase student employment in targeted industries Strengthen and increase targeted industry partnerships Increase activity with partners Learning & Growth Increase learning attributed to collaborative partnerships (community, students, industry, colleges)
    42. 42. Break! For 10 minutes.
    43. 43. Hope Opportunity Jobs Dr. Kristen Corbell, Coordinator of Research Projects North Carolina Community College System Office Basic Skills Focus Area Team Chair, Data Initiative Coordinating Team member, Data Access and Advisory Team member, External Sources Focus Area Team Member Data Initiative Overview 4/15/2014
    44. 44. Hope Opportunity Jobs Data Initiative Overview The Data Initiative includes the review, assessment and revision of our current data collection and reporting processes that results in a robust data system which provides accurate and accessible information fostering a culture of data-driven decision making which addresses research questions and informs policies. 44
    45. 45. Hope Opportunity Jobs Data Initiative Motivation • Concerns over data quality at the college and state level • Increasing demands for data from federal, state, and grant-providing agencies/organizations • Expectations of collaboration between state agencies to share data • The disconnect between data access and decision makers at the college and state level • Increasing demands on colleges from accreditation bodies to practice and exhibit informed decision-making • Current reporting tools do not adequately supporting the increasing research and reporting demands • The need to ensure policies are based on sound data 45
    46. 46. Hope Opportunity Jobs Data Initiative Objectives • Expand the individual and collective inquisitiveness of stakeholders within the NC Community College System to ask research questions that inform decisions and policies. • Develop data definitions based on research needs and reporting requirements. • Ensure data quality and consistency through appropriate and uniform entering and collecting of information across the System so data extracted are valid and reliable. • Incorporate a data review process that ensures accountability through the validation of submitted data. 46
    47. 47. Hope Opportunity Jobs Data Initiative Objectives • Develop focused topic-based data marts that will serve a wide variety of analytical research needs including, but not limited to, strategic initiatives, grants, and business and student centric needs. • Expand information accessibility by making all data users aware of the availability of predefined reports, web-based dashboards, statistical software, and training. • Enhance advanced analytical capabilities to empower researchers focusing on specific topics and initiatives. • Utilize research findings and reports to educate policy makers on the extent to which various factors impact outcomes, including student enrollment, student learning, student completion, faculty and staff development, budget allocation, etc. 47
    48. 48. Hope Opportunity Jobs Continuous Improvement Cycle
    49. 49. Hope Opportunity Jobs Culture of Curiosity / Strategic Initiatives 49 Culture of Curiosity Represents individual and collective inquisitiveness to ask research questions that inform decisions and policies Data Needs •Strategic Initiative Variables •Reporting Mandates •Research Needs Data Definitions Developed based on research needs and reporting requirements
    50. 50. Hope Opportunity Jobs How the Essential Skills Grant fits in with the Culture of Curiosity • Within this grant you are asking questions about how the innovation at your college(s) is impacting student success and employment outcomes. These questions are building your culture of curiosity. • From your questions, you are identifying the data that will be collected to answer the questions. • For analysis throughout and at the end of the grant, all data must be collected with clear definitions from the beginning and carried throughout. 10/17/2012
    51. 51. Hope Opportunity Jobs Data Quality, Consistency, and Accountability 51 Data Quality and Consistency Ensures information is collected and entered appropriately and uniformly across the system so data extracted for reporting purposes is valid and reliable Data Definitions Direct what, where, when, and how information is entered into the data collection systems Data Entry Proper understanding of data definitions will sustain the quality of the data collection and reporting processes Data Review Process which ensures accountability through the validation of data submitted for reporting purposes
    52. 52. Hope Opportunity Jobs How the Essential Skills Grant fits in with Data Quality and Consistency • Using the data definitions, your college should ensure that the data are collected and entered uniformly. • We would suggest reviewing the data entered prior to submission to ensure the data make sense. For instance, make sure dates entered are logical, program codes are correct, and student grades are recorded on time for submission with the semester files. 10/17/2012
    53. 53. Hope Opportunity Jobs Data Collection and Warehousing 53 Data Extraction, Transformation, and Loading Procedural routines of extracting data files from local systems to a centralized warehouse Data Marts The creation of specific data sets dedicated to strategic initiatives and research focus areas Data Access The ability to query variables within data marts for basic informational purposes and detailed research needs Data Collection and Warehousing Centralized and technical processes that impact the way in which data is transferred and transformed between data systems and stored for reporting purposes
    54. 54. Hope Opportunity Jobs Culture of Evidence / Policy Implications 54 Culture of Evidence / Policy Implications •Determination of optimal decisions and policies based on the analysis and reporting of collected data •Likely to result in development of additional research and evaluation needs General Reporting Automated informational reports updated routinely Advanced Analytics Research and analysis focused on specific topics and initiatives to inform current and future practices. Research Findings & Reporting Educate policy makers on the extent of impact of various factors on outcomes, including student learning and progress
    55. 55. Wrap-Up
    56. 56. Calendar of Evaluation Activities – Year One ACTIVITY DATE Summer Institute 2013 June 2013 Grant Coordinator Phone Interview Fall 2013 Site Visit Fall-Spring 2013 Surveys Winter 2014
    57. 57. Please Take 5 Minutes to Complete Survey Go to GLF ES Moodle site. Go to the Summer Institute 2013 Section Click on the Essential Skills Professional Development Summer 2013 Survey link. Thank you.
    58. 58. Thank You Questions? Comments? Concerns? Have a good trip home! Michael Maslin Item #: 8476186
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×