Data-Driven Decision Making

1,109 views

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,109
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • For those viewing this presentation after downloading it from the Naviance Network: Please view in presentation mode as some animations obscure the full content of the presentation.
  • It goes deeper than this. Not only are educators drowning in data, it’s being used against them.
  • Refocusing on outcomes lets you shape the debate.
  • Data used for this example considers staff activities and student usage 7/15/12 – 4/30/13.
  • In the context of this district, training serves to increase the percentage of students logged in. Consulting and NSI serve to deepen the interaction by increasing the number of logins per student.
  • What did we learn? – Don’t make assumptions. Don’t force your opinion on the results.
  • Data-Driven Decision Making

    1. 1. Data-Driven Decision Making Amy Reitz Manager, Consulting Services Hobsons K-12
    2. 2. Agenda Overview of Data Driven Decision Making Building a Data Culture • Staff Workshops • Student Workshops
    3. 3. Overview of Data-Driven Decision Making
    4. 4. Data-Driven Decision Making is… DDDM D3M The collection and analysis of data to make decisions that improve student success. Continual evaluation accompanied by incremental changes. Translation of data into knowledge and actionable strategies. Collaboration and communication throughout the school, district and community.
    5. 5. Theme 1: Use data to make decisions Data Decisions
    6. 6. What we want to happen Helpful Data I need to…
    7. 7. What happens in reality Teacher Evaluations Partner Assessments SAT GPA ACT Attendance Activities Standardized Test Scores Dem ographics Classroom Activities PSAT AP IBIEP CoursePla SchoolHistory 1600 4.0 32 1.7 365 ???
    8. 8. Theme 2: Focus on outcomes
    9. 9. How do you measure success? Staff and students have completed all of their assigned tasks. Students are career and college ready. Productivity Outcome
    10. 10. Theme 3: Dig deeper Illustrated via an example: If staff members from a school attend NSI, is the Naviance student usage at that school positively affected? Following slides are from a case study for a large urban district with staff members attending NSI 2012.
    11. 11. Impact of NSI on Student Usage *Active Student User = Student that has logged in at least once.
    12. 12. Dig deeper What is wrong with this analysis? • It doesn’t consider the past. • Schools with NSI attendees could have already had higher usage. How can it be improved? • Compare growth rates.
    13. 13. Impact of NSI on Student Usage *Active Student User = Student that has logged in at least once.
    14. 14. Dig deeper What is wrong with this analysis? • It doesn’t consider other support. • Schools with NSI attendees could have had additional support from Naviance staff. How can it be improved? • Include another variable: interaction with a consultant.
    15. 15. Impact of NSI on Student Usage Average Logins per Student *Average logins per student include students with 0 logins. Schools without NSI Attendees Schools with NSI Attendees Schools without Consultant Interaction Schools with Consultant Interaction 1.5 2.9 4.7 5.8
    16. 16. Dig deeper What is wrong with this analysis? • It doesn’t consider more than 2 variables. • Schools with NSI attendees could have had additional training or other student variables could influence usage. How can it be improved? • Analyze multiple variables to build a predictive model.
    17. 17. Multivariable Regression y = 10x + 2w – 5z outcome variables coefficients
    18. 18. Impact of NSI on Student Usage Output of Multi-Variable Regression for Unique Logins: Positive Coefficients
    19. 19. Impact of NSI on Student Usage Output of Multi-Variable Regression for Total Logins: *Total logins for students with 1+ login. Positive Coefficients Statistically Insignificant (p > 0.05)
    20. 20. Impact of NSI on Student Usage Regression Coefficients Unique Logins Total Logins NSI 0.012 0.96 Consulting 0.122 2.78 Training 0.321 N/A
    21. 21. Dig deeper What is wrong with this analysis? • It doesn’t consider individual student variables. - Did an analysis with gender and class year, but they were statistically insignificant. Still many other variables that could be included. How can it be improved? • Additional variables. • Sensitivity analysis and other statistical models. • Larger sample size across multiple schools, districts, and regions.
    22. 22. Building a Data Culture
    23. 23. DDDM is a culture To be truly effective, DDDM needs input from everyone. Everyone needs to see value and be invested in collecting and analyzing data. Staff need to openly collaborate and take action based on data. In some cases, this requires a huge attitudinal shift. Be prepared to facilitate.
    24. 24. Building a Data Culture: Staff Workshops
    25. 25. Staff Workshops Involve multiple staff members from various roles in the development of data processes. Collaborate to make the best possible decisions. Use data for decisions and information, not just compliance.
    26. 26. Staff Workshop: Report Review Purpose: Review the reports in Naviance and identify needs. Activities: • Review reports in Naviance. • Identify helpful reports. • For each report, determine: - Audience: Who should receive this report? - Parameters: Which students/tasks/variables should be included? - Frequency: When and how often should this report be run? Next Steps: • Determine data needed to populate report. - Ensure data is collected during activities throughout the year. • Customize and schedule reports in Naviance.
    27. 27. Staff Workshop: KPIs & Outcomes Purpose: Define the key performance indicators and outcomes that are important. Activities: • Brainstorm student outcomes. What does it mean for students to be successful? • For each outcome, determine associated KPIs. - Appendix: Key Performance Indicators Next Steps: • Document and communicate KPIs and outcomes. • Map KPIs and outcomes to Naviance activities and reports.
    28. 28. Staff Workshop: Identify Variables Purpose: Identify variables that should be tracked to link to outcomes and KPIs. Activities: • Review identified outcomes and KPIs. • Brainstorm variables that could impact outcomes. • Determine how variables are tracked and stored. - SIS - Naviance Activities - Naviance Surveys - Other Next Steps: • Incorporate into Naviance activities and data collection. - Appendix: Data Collection in Naviance • Develop maintenance plan.
    29. 29. Staff Workshop: Survey Development Purpose: Create surveys to collect data and inform decisions. Activities: • Review previously identified needs. - Direct data collection. - Indirect collection through reflection and and feedback. • Brainstorm and organize questions. Next Steps: • Setup surveys in Naviance. • Incorporate survey(s) into activities throughout the year.
    30. 30. Staff Workshop: Scope & Sequence Purpose: Define a plan for the activities that need to occur throughout the year. Activities: • Review available activities in Naviance. • Review previously identified data needs. • Review suggested activities in Naviance Implementation Guide and Naviance Network. • Develop a plan for the activities to be completed by students and staff throughout the year. Next Steps: • Document and communicate scope and sequence. • Map to tasks in Success Planner and assign to students.
    31. 31. Staff Workshop: Data Validation Purpose: Review imported and input data to verify accuracy and completeness. Activities: • Review data import history in Naviance. • Review student profiles. Note inaccuracies or incomplete entries. - For imported data, correct the source. - For input data, correct in Naviance. • Review regular reports to verify accuracy. Next Steps: • Determine any patterns to identify processes to be corrected.
    32. 32. Staff Workshop: Progress Reviews Purpose: Regularly review progress against KPIs and scope and sequence. Activities: • Review regularly scheduled reports. • Identify successes and discuss lessons learned. • Identify lagging indicators. Brainstorm causes and discuss solutions. Next Steps: • Make changes to address challenges.
    33. 33. Staff Workshops What else have you done at your school or district? What else have you done at your school or district?
    34. 34. Building a Data Culture: Student Workshops
    35. 35. Student Workshops Get relevant input from students. Help students understand data driven decision making. Bolster college going culture. Supplement college and career planning activities.
    36. 36. Student Workshop: Data Validation Purpose: Verify student demographics, contact information, and other profile information. Activities: • Provide background on importance of profile information. • Have students review their Naviance profiles for incorrect information. - Add email addresses. - Note inaccuracies to be corrected in source by staff (SIS).
    37. 37. Student Workshop: College & Career Survey Purpose: Introduce students to basic principles of data collection in the context of post-secondary planning and readiness. Activities: • Provide background on survey basics: question development, tools, distribution. • Students develop and distribute college and career surveys. • Provide background on basic survey analysis. • Students analyze and present their results.
    38. 38. Student Workshop: Identify Variables Purpose: Identify additional variables and challenge students to consider the challenges and steps leading to their post-secondary plan. Activities: • Provide students with an outcome to consider. • Provide background information about the concept of variables and some possible variables affecting the identified outcome. • Students brainstorm and present the variables they think are important.
    39. 39. Student Workshops What else have you done at your school or district? What else have you done at your school or district?
    40. 40. Questions & Discussion
    41. 41. Appendices Key Performance Indicators Data Collection in Naviance Additional Resources
    42. 42. Your Feedback Matters! Thank you for attending the Naviance Summer Institute 2013! We greatly appreciate your feedback, please complete a brief evaluation for this session at: http://go.naviance.com/evaluations
    43. 43. Appendix: Key Performance Indicators
    44. 44. Focusing your analysis Outcomes are the ultimate goal. Variables are the many data points for each student. They include everything that affects a student’s outcomes. Key performance indicators are measurements to determine if you are on track to attain a particular outcome. This appendix includes suggested KPIs using data collected in Naviance. Note: The source for the following slides in this appendix is the 2012 NSI presentation by Todd Bloom: KPIs for College and Career Readiness.
    45. 45. Student Growth & Proficiency Grade Point Average Test score averages • PLAN • PSAT • SAT • ACT • State assessment(s) International Baccalaureate scores International Baccalaureate scores by course % of students who used PrepMe at least once % of students who complete the learning style assessment % of students who complete Do What You Are assessment % of students who complete Career Key assessment % of students who complete a Course Plan Course Plan Rigor distribution
    46. 46. College Planning College Power Score distribution Alignment of Course Demand Forecast with college readiness curriculum determined by school/district Student interest in specific courses that school/district indicate align with college readiness goals Number of applications for individual colleges Number of applications for individual colleges % of students who submit one or more college applications % of students admitted to one or more colleges % of students who intend to attend college after graduation Meaningful and up-to-date scholarship database available for student use
    47. 47. Career Planning % of students who identify careers and career clusters of interest % of students interested in professional careers % of students interested in technical careers % of students interested in careers with specific characteristics, such as STEM, that are determined by the school/district
    48. 48. Student Engagement % of students who report they understand the knowledge and skills necessary for success in their careers of interest % of students who set goals % of students who met goal % of students who completed tasks that align with college and career readiness as determined by the school/district(e.g. FAFSA completion, internship/ mentorship requirement) % of students who report understanding their learning styles % of students who report they have explored colleges and careers based on learning style assessment % of students who report they understand the links between careers, preparation needed, college major and projected income
    49. 49. Alumni Performance % of students who enrolled in college % of students who completed college degrees % of students who completed college degrees within a specified timeframe % of students with positive perceptions of college and career readiness % of students satisfied with teaching or other specified aspects of their K-12 experience % of students who are satisfied with their post high school plans % of students who enrolled in remedial college mathematics, English or other courses % of students who completed remedial college math, English or other courses
    50. 50. Appendix: Data Collection in Naviance
    51. 51. Collecting Data in Naviance Data comes into Naviance from various sources in multiple ways. This means that data quality can vary. Poor data quality means less accurate analysis. This appendix will cover some of the ways you can improve the quality of data you are collecting. Data is input into Naviance via various methods. Some methods include: • Student Information (data import) • College Planning • Career Planning • Course Planning • Success Planning • Surveys
    52. 52. Collect Data: Student Info Keep student data up to date. • Automate data import • Define a process for importing test scores regularly • Import all data that would be helpful for analysis. • Review data import and data import templates to determine what is missing and update. - Data import file: setup > data import > data import history > view - Templates located in the Help Library.
    53. 53. Collect Data: College Planning • Engage students before senior year to add colleges to their prospective list. • Use Senior Survey to improve accuracy of outcome reports. • Turn off the option to allow students to delete active applications. • connections > family connection > select and update optional features > delete active applications • Import college application data from previous years.
    54. 54. Collect Data: Career Planning • Create a scope & sequence for career planning activities that students will complete in grades 6- 12. • Use CCR Curriculum to improve rollout and add context for students. • Leverage class or advisory time to work through activities with students to ensure completion. • Setup computers at career fairs for students to add careers to their list.
    55. 55. Collect Data: Success Planning • Use Success Planning to assign tasks that improve career and college data. • Link tasks to activities where possible. • For example, instead of manually marking that a student completed a workshop, create a post-workshop survey and assign the survey as a task. • Utilize the built-in tasks. • Schedule planner reports to regularly assess progress.
    56. 56. Collect Data: Course Planner • Configure the rigor levels for the College Power Score. • courses > configuration > total potential course rigor • Configure the rigor level for courses in the course catalog. • Can be included in the course catalog import. • Can be set manually in Naviance - courses > course catalog > edit > instructional level
    57. 57. Collect Data: Surveys • Send out frequent, short surveys instead of long, annual surveys. • Opt for question types that make aggregate analysis easier. • Send surveys as a direct link via email (doesn’t require students to login). • connections > e-mail > send email to a group of students and parents > select options and attach survey > preview and send > “check this box if you want to include a direct link to the survey”
    58. 58. • Appendix: Additional Resources
    59. 59. Naviance Resources • Naviance Network Community Forums: http://community.naviance.com/t5/Community- Forums/ct-p/succeed • Naviance Network Help Library, Reporting Section: http://community.naviance.com/t5/Reporting/tkb- p/Reporting%40tkb
    60. 60. Workshop Resources • ATLAS – Looking at Data: http://www.nsrfharmony.org/protocol/doc/atlas_l ooking_data.pdf * • Data.gov in the Classroom, Education Materials: http://www.data.gov/education/page/datagov- classroom * Thanks to a participant in the NSI 2012 DDDM session for this suggestion.
    61. 61. MS Office Resources • Office Support: http://office.microsoft.com/en- us/support/ • VLOOKUP (joining data in Excel): http://office.microsoft.com/en-us/excel- help/vlookup-HP005209335.aspx • Excel Review, Duke University: https://faculty.fuqua.duke.edu/~pecklund/ExcelR eview/ExcelReview.htm
    62. 62. Misc Stats and Analysis Resources • Data Mining: The Tool of the Information Age Revolution, Rajan Patel, Stanford (recorded webinar): http://myvideos.stanford.edu/player/slplayer.aspx? coll=2e431434-84e4-4de0-81c9- 76035c36a18f&co=12138da9-eab8-405b-a06f- cc11f12e5871&w=true • Introduction to Statistics and Data Analysis, University of Michigan (open course materials): http://open.umich.edu/education/lsa/statistics250/sp ring2013

    ×