Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A Pipeline for Modeling Automated
Scoring Using Python, R and
Jupyter Notebooks
Copyright © 2015 by Educational Testing Se...
Machine Learning &
Educational Assessment
A Pythonic Love Story
Nitin Madnani, Anastassia Loukina & Lei Chen
Copyright © 2...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Ed...
Upcoming SlideShare
Loading in …5
×

Pipeline for Modeling Automated Scoring (PyData NYC 2015)

814 views

Published on

Slides from my PyData NYC 2015 talk.

Published in: Engineering
  • DOWNLOAD FULL MOVIE, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. MOVIE 4K,FHD,HD,480P here { https://tinyurl.com/yybdfxwh }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Pipeline for Modeling Automated Scoring (PyData NYC 2015)

  1. 1. A Pipeline for Modeling Automated Scoring Using Python, R and Jupyter Notebooks Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Nitin Madnani, Anastassia Loukina & Lei Chen
  2. 2. Machine Learning & Educational Assessment A Pythonic Love Story Nitin Madnani, Anastassia Loukina & Lei Chen Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141
  3. 3. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Testing Service • A non-profit educational organization founded in 1947, headquartered in Princeton, New Jersey (N≊3500). • Designs and administers global as well as domestic educational assessments (GRE®, TOEFL®, PRAXIS® etc.) • Conducts and publishes extensive research on psychometrics, statistics, cognitive science, and computer science.[1] • Mission: To advance quality and equity in education by providing fair and valid assessments, research and related services. 3 [1] http://search.ets.org/researcher/
  4. 4. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Two Parts • Part 1: What makes educational assessment a challenging application for machine learning?
 
 • Part 2: How does Python help us address some of these challenges at ETS? 4
  5. 5. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessment Machine Learning Part 1
  6. 6. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6
  7. 7. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz
  8. 8. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz Homework Assignment
  9. 9. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz MOOC Assignments Homework Assignment
  10. 10. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz MOOC Assignments K-12 Standardized Tests Homework Assignment
  11. 11. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz MOOC Assignments Teacher Certification K-12 Standardized Tests Homework Assignment
  12. 12. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz MOOC Assignments Teacher Certification K-12 Standardized Tests Homework Assignment Practice Tests
  13. 13. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz MOOC Assignments TOEFL/IELTS Teacher Certification K-12 Standardized Tests Homework Assignment Practice Tests
  14. 14. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz GRE MOOC Assignments TOEFL/IELTS Teacher Certification K-12 Standardized Tests Homework Assignment Practice Tests
  15. 15. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz GRE MOOC Assignments TOEFL/IELTS Teacher Certification GMAT K-12 Standardized Tests Homework Assignment Practice Tests
  16. 16. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 6 Classroom Quiz GRE MOOC Assignments TOEFL/IELTS GED Teacher Certification GMAT K-12 Standardized Tests Homework Assignment Practice Tests
  17. 17. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 7 Classroom Quiz GRE MOOC Assignments TOEFL/IELTS GED Teacher Certification GMAT K-12 Standardized Tests Homework Assignment Practice Tests
  18. 18. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 8 Classroom Quiz GRE MOOC Assignments TOEFL/IELTS GED Teacher Certification GMAT K-12 Standardized Tests Homework Assignment High Stakes Practice Tests
  19. 19. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 “High Stakes” • A test with results that have important, direct consequences for the test-takers.
 
 • A test-taker would want to understand what their score means and how it maps to what they did on the test. 9
  20. 20. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 10 Classroom Quiz GRE MOOC Assignments TOEFL/IELTS GED Teacher Certification GMAT K-12 Standardized Tests Homework Assignment High Stakes Practice Tests
  21. 21. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessments 11 Classroom Quiz GRE MOOC Assignments TOEFL/IELTS GED Teacher Certification GMAT K-12 Standardized Tests Homework Assignment High Stakes Practice Tests
  22. 22. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 The GRE • Graduate Record Examination, designed and administered by ETS. • Used by at least 3000 colleges and universities across the world for graduate school applications to MS, MBA & PhD programs.[1] • ~575,000 test-takers from ~200 countries between July 2013 and June 2014 (50% women, 45% men). [2] • Three sections: • Verbal Reasoning • Quantitative Reasoning • Analytical Writing 12 [2] http://www.ets.org/s/gre/pdf/snapshot_test_taker_data_2014.pdf[1] https://www.ets.org/s/gre/pdf/gre_aidi_fellowships.pdf
  23. 23. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 The GRE • Graduate Record Examination, designed and administered by ETS. • Used by at least 3000 colleges and universities across the world for graduate school applications to MS, MBA & PhD programs.[1] • ~575,000 test-takers from ~200 countries between July 2013 and June 2014 (50% women, 45% men). [2] • Three sections: • Verbal Reasoning • Quantitative Reasoning • Analytical Writing 13 [2] http://www.ets.org/s/gre/pdf/snapshot_test_taker_data_2014.pdf[1] https://www.ets.org/s/gre/pdf/gre_aidi_fellowships.pdf
  24. 24. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 GRE Analytical Writing 14 “As people rely more and more on technology to solve problems, the ability of humans to think for themselves will surely deteriorate.” 
 Directions: Write a response in which you discuss the extent to which you agree or disagree with the statement and explain your reasoning for the position you take. https://www.ets.org/gre/revised_general/prepare/analytical_writing/issue/scoring_guide Score 6. Outstanding articulates a clear and insightful position develops the position fully well-focused, well-organized analysis conveys ideas fluently and precisely demonstrates superior facility with English Score 1. Fundamentally Deficient provides little/no evidence of understanding disorganized or extremely brief severe problems with sentence structure pervasive errors in grammar incoherent and meaning not clear …
  25. 25. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Scoring essays Given the stakes, our scoring methodology must maximize: • Accuracy: how accurately does the assigned score measure the analytical skills of the test-taker? • Interpretability: how easily can test-takers understand why they was assigned a particular score and what that score means? 15
  26. 26. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Scoring essays Given the stakes, our scoring methodology must maximize: • Accuracy: how accurately does the assigned score measure the analytical skills of the test-taker? • Interpretability: how easily can test-takers understand why they was assigned a particular score and what that score means? 15 It would also be nice to minimize: • Cost: how efficiently can we score each test (how much money can we save the test-taker in fees)?
  27. 27. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Scoring essays 16
  28. 28. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Scoring essays 17 Essay Scoring Guide Trained Human Readers High Accuracy Medium Interpretability High Cost Option 1
  29. 29. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Scoring essays 18 Essay Scoring Guide Features Automated Scoring System (Machine Learning) Medium Accuracy (Choice of) High Interpretability Low Cost Option 2
  30. 30. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Scoring essays 19 Essay Scoring Guide One Trained Human Reader Scoring Guide Features Automated Scoring System (Machine Learning) Final score Human Score System Score As good as using two human readers[1]. [1] http://www.ets.org/Media/Research/pdf/RD_Connections2.pdf
  31. 31. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 E-rater 20 Scoring Guide Features Automated Scoring System (Machine Learning)
  32. 32. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 E-rater 20 “Essay Rater”
 Linear regression trained on older essays written to the same topic and scored by human readers.
 Features errors in grammar (e.g., subject-verb agreement) usage errors (incorrect prepositions/articles) mechanics errors (capitalization, spelling) errors in style (repetitious word use) discourse structure (presence of a thesis statement, main points) vocabulary sophistication essay organization Automated Essay Scoring With e-rater® V.2, The Journal of Technology, Learning, and Assessment, Volume 4(3), 2006 Scoring Guide Features Automated Scoring System (Machine Learning)
  33. 33. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 E-rater & Research 21 E-rater still an active area of research at ETS
 Design new features; examine their effect on performance, and whether they overlap with existing features. Try more sophisticated machine learning models (higher accuracy worth lower interpretability?) Last year, 10 new e-rater features proposed just for GRE! GRE one of a dozen assessments, e-rater one of many automated scoring engines Research untenable for a large group (>15 scientists) without a standardized pipeline. Scoring Guide Features Automated Scoring System (Machine Learning)
  34. 34. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Need an end-to-end machine learning pipeline that can: • Work on (almost) all platforms, • Read features in any tabular format and clean it up, • Efficiently apply filtering, scaling and transformations, • Train any specified model with those features, and • Generate a standardized, detailed report of performance on unseen essays. 22 Ideal research pipeline
  35. 35. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessment Machine Learning
  36. 36. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Educational Assessment Machine Learning Python Part 2
  37. 37. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 25 Python Pipeline Input Preprocess Model Evaluate Report Input final self-contained report
  38. 38. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 26
  39. 39. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 26 Model Name (str) Training Features (csv/tsv/xls) Unseen Test Features (csv/tsv/xls) Feature Definitions (json)
  40. 40. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 26 Model Name (str) Training Features (csv/tsv/xls) Unseen Test Features (csv/tsv/xls) Feature Definitions (json) 1.Input • Read files into data frames • Check for missing feature columns, exclude others • Filter out non-numeric and blank values • Standardize essay ID and essay score column names pandas
  41. 41. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 26 Model Name (str) Training Features (csv/tsv/xls) Unseen Test Features (csv/tsv/xls) Feature Definitions (json) Training Data Frame (Raw) Test Data Frame (Raw) 1.Input • Read files into data frames • Check for missing feature columns, exclude others • Filter out non-numeric and blank values • Standardize essay ID and essay score column names pandas
  42. 42. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 27 Model Name (str) Feature Definitions (json) Training Data Frame (Raw) Test Data Frame (Raw)
  43. 43. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 27 Model Name (str) Feature Definitions (json) Training Data Frame (Raw) Test Data Frame (Raw) 2.Preprocess • Filter out user-flagged rows, if so specified • Remove feature outliers & “intelligently” apply feature transformations (log, inv, sqrt, etc.), if available • Standardize all features (center and scale) numpy + pandas
  44. 44. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 27 Model Name (str) Feature Definitions (json) Training Data Frame (Raw) Test Data Frame (Raw) Training Data Frame (Processed) Test Data Frame (Processed) 2.Preprocess • Filter out user-flagged rows, if so specified • Remove feature outliers & “intelligently” apply feature transformations (log, inv, sqrt, etc.), if available • Standardize all features (center and scale) numpy + pandas
  45. 45. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 28 Model Name (str) Feature Definitions (json) Training Data Frame (Processed) Test Data Frame (Processed)
  46. 46. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 28 Model Name (str) Feature Definitions (json) Training Data Frame (Processed) Test Data Frame (Processed) 3.Model • Train regression/classification model via SKLL API or R • Grid-search using a task-appropriate objective • Serializes model to disk (using joblib) R + skll
  47. 47. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 28 Model Name (str) Feature Definitions (json) Training Data Frame (Processed) Test Data Frame (Processed) 3.Model • Train regression/classification model via SKLL API or R • Grid-search using a task-appropriate objective • Serializes model to disk (using joblib) R + skll SKLL (pronounced “skull”) provides an API and command-line utilities to make it much simpler to run common scikit-learn experiments with pre-generated features. (Presented by @dsblanch at PyData 2013 & 2014) https://github.com/EducationalTestingService/skll
  48. 48. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 28 Model Name (str) Feature Definitions (json) Training Data Frame (Processed) Test Data Frame (Processed) 3.Model • Train regression/classification model via SKLL API or R • Grid-search using a task-appropriate objective • Serializes model to disk (using joblib) R + skll
  49. 49. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 28 Model Name (str) Feature Definitions (json) Training Data Frame (Processed) Test Data Frame (Processed) 3.Model • Train regression/classification model via SKLL API or R • Grid-search using a task-appropriate objective • Serializes model to disk (using joblib) R + skll Serialized model
  50. 50. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 29 Training Data Frame (Processed) Test Data Frame (Processed) Serialized model
  51. 51. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 29 Training Data Frame (Processed) Test Data Frame (Processed) Serialized model 4.Evaluate • Use serialized model to compute test set predictions • Trim and re-scale predictions to match training data • Compute a set of standard evaluation metrics by comparing predictions to test set human scores skll + pandas
  52. 52. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 29 Training Data Frame (Processed) Test Data Frame (Processed) Serialized model Test Data Predictions Evaluation Statistics 4.Evaluate • Use serialized model to compute test set predictions • Trim and re-scale predictions to match training data • Compute a set of standard evaluation metrics by comparing predictions to test set human scores skll + pandas
  53. 53. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 30 Test Data Predictions Evaluation Statistics Training Data Frame (Processed) Test Data Frame (Processed) Serialized model
  54. 54. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 30 Test Data Predictions Evaluation Statistics Training Data Frame (Processed) Test Data Frame (Processed) Serialized model 5.Report • Determine what report sections should be included • Merge pre-existing section templates (.ipynb files) • Dynamically Run final .ipynb file (via ExecutePreprocessor and environment variables) • Convert report to HTML using HTMLExporter jupyter + seaborn + pandas
  55. 55. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 30 Test Data Predictions Evaluation Statistics Training Data Frame (Processed) Test Data Frame (Processed) Serialized model Final Report (.ipynb) Final Report (html) 5.Report • Determine what report sections should be included • Merge pre-existing section templates (.ipynb files) • Dynamically Run final .ipynb file (via ExecutePreprocessor and environment variables) • Convert report to HTML using HTMLExporter jupyter + seaborn + pandas
  56. 56. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Demo 31
  57. 57. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Summary • Machine learning in high-stakes educational assessment requires additional number crunching to verify accuracy and interpretability. • Need a pipeline to compare a large number of research experiments using a standardized, easy-to-read report. • The scientific Python stack makes it super easy to implement all stages of the pipeline! • In progress • Release under open-source license (2016 release) • A CherryPy/JS web-app to allow wider reach 32
  58. 58. Copyright © 2015 by Educational Testing Service. All rights reserved. ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). MEASURING THE POWER OF LEARNING is a trademark of ETS. 30141 Questions? 33 https://github.com/EducationalTestingService https://github.com/desilinguist @haikuman

×