Downloaded 11 times
Uploaded byPhilip Piety
Learning Analytics and Knowledge (LAK) 14 Education Data Sciences
The document discusses the emerging field of Education Data Sciences (EDS). It outlines four main ideas: 1) A sociotechnical paradigm shift in how data is conceived, from external to internal/contextual. 2) The notion of EDS, which includes academic analytics, educational data mining/learning analytics, learner analytics/personalization, and systemic instructional improvement. 3) Common features across these communities, including rapid change, boundary issues, disruption to evidence practices, and ethics/privacy. 4) A proposed framework for EDS that recognizes different social/temporal levels, digital fluidity across contexts, and values in design. The field of EDS analyzes learning, organizations and systems through an interdisciplinary lens.
More Related Content
![Action Research Proposal Presentation [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/actionresearchproposalpresentationautosaved-220404044459-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Learning Analytics and Knowledge (LAK) 14 Education Data Sciences
Learning Analytics and Knowledge (LAK) 14 Education Data Sciences
- 1. Education Data Sciences FramingEmergent Practices for Analytics of Learning, Organizations, and Systems Philip J. Piety Ed Info Connections ppiety@edinfoconnections.com Daniel T. Hickey Learning Sciences, School of Education Indiana University dthickey@indiana.edu MJ Bishop Center for Innovation and Excellence in Learning & Teaching University System of Maryland mjbishop@usmd.edu 1
- 2.
- 3. Four Big Ideas 1.Sociotechnical paradigm shift 2. Notion of Education Data Sciences (EDS) – Academic/Institutional Analysis – Learning Analytics/Educational Data Mining – Learning Analytics/Personalization – Systemic Instructional Improvement 3. Common features across these communities 4. Framework for EDS 3
- 4. SOCIOTECHNICAL PARADIGM SHIFT INCONCEPTION OF DATA From External/Distant/Artificial to Internal/Current/Contextual 4
- 5.
- 6. The Educational DataMovement Understanding how the organizational model of education is similar to/different from other fields is key to understanding the educational data movement. 1980 – 1990 - 2000 - 2010 Finance Manufacturing Retail Health Care Education 6
- 7. The Educational DataMovement Understanding how the organizational model of education is similar to/different from other fields is key to understanding the educational data movement. 1980 – 1990 - 2000 - 2010 Finance Manufacturing Retail Health Care Education 7
- 8. The Educational DataMovement 8
- 9. Early Childhood K-12 Post Secondary Continuing/ Career Individuals Cohorts OrganizationsSystems Scale of Educational Context EducationalLevel(Age) The EDS Landscape 9
- 10. Early Childhood K-12 Post Secondary Continuing/ Career Individuals Cohorts OrganizationsSystems Scale of Educational Context EducationalLevel(Age) Academic/ Institutional Analytics Academic/Institutional Analytics 10
- 11.
- 12. Systemic/ Instructional Improvement Early Childhood K-12 Post Secondary Continuing/ Career Individuals Cohorts OrganizationsSystems Scale of Educational Context EducationalLevel(Age) Academic/ Institutional Analytics Systemic/Instructional Improvement 12
- 13. Systemic/Instructional Improvement “In manyways, the practice of data use is out ahead of research. Policy and interventions to promote data use far outstrip research studying the process, context, and consequences of these efforts. But the fact that there is so much energy promoting data use and so many districts and schools that are embarking on data use initiatives means that conditions are ripe for systematic, empirical study.” Coburn, Cynthia E., and Erica O. Turner. "Research on data use: A framework and analysis." Measurement: Interdisciplinary Research & Perspective 9.4 (2011): 173-206. 13
- 14. Systemic/ Instructional Improvement Early Childhood K-12 Post Secondary Continuing/ Career Individuals Cohorts OrganizationsSystems Scale of Educational Context EducationalLevel(Age) Academic/ Institutional Analytics EDM/Learning Analytics 14
- 15.
- 16. Systemic/ Instructional Improvement Early Childhood K-12 Post Secondary Continuing/ Career Individuals Cohorts OrganizationsSystems LearnerAnalytics/ Personalization Scale of Educational Context EducationalLevel(Age) Academic/ Institutional Analytics LearnER Analytics/Personalization 16
- 17.
- 18. Systemic/ Instructional Improvement Early Childhood K-12 Post Secondary Continuing/ Career Individuals Cohorts OrganizationsSystems LearnerAnalytics/ Personalization Scale of Educational Context EducationalLevel(Age) Academic/ Institutional Analytics D. Flipped Classrooms C. Early Warning Systems A. School to College Analyses B. Teacher Preparation Efficacy Evaluation Boundary Conditions 18
- 19. COMMON FEATURES &FACTORS IN EDUCATIONAL DATA SCIENCES A unified perspective for Educational Data Science 19
- 20. Five Common Featuresin EDS 1. Rapidly changing - Indicative of sociotechnical movement 2. Boundary issues - All communities touch on other communities 3. Disruption in evidentiary practices - Big data is disrupting all the sectors 4. Visualization, interpretation, and culture - Dashboards, representations, APIs, open data 5. Ethics, privacy and governance - FERPA & COPPA 20
- 21. Four Factors thatMake All Educational Data Unique • Human/social creation –Most requires human manipulation • Measurement imprecision –Reliability issues are huge • Comparability challenges –Validity creates “wicked problems” • Fragmentation –Systems can’t talk to each other 21
- 22. SOME COMMON PRINCIPLES Aunified perspective for Educational Data Science 22
- 23.
- 24. Recognize Social/Temporal Levels Timescale Context Targeted Educational Content Time Frame Formatof Educational Evidence Appropriate Formative Function for Students Ideal Formative Functions for Others Immediate Curricular Activity (lesson) Minutes Event-oriented observations (Informal observations of the enactment of the activity) Discourse during the enactment of a particular activity. Teacher: Refining discourse during the enactment of a particular activity. Close Curricular Routines (chapet/unit) Days Activity-oriented quizzes (semi-formal classroom assessments) Discourse following the enactment of chapter, quiz. Teacher: Refining the specific curricular routines and providing informal remediation to students. Proximal Entire Curricula Weeks Curriculum-oriented exams (Formal classroom assessments) Understanding of primary concepts targeted in curriculum. Teacher/curriculum developer: providing formal remediation and formally refining curricula. Distal Regional/Na tional Content Standards Months Criterion-referenced tests (external tests aligned to content standards) Administrators: Selection of curricula that have the largest impact on achievement in broad content domains. Remote National Achieve- ment Years Norm-referenced external tests standardized across years (ex: ITBS, NAEP) Policy makers: Long-term impact of policies on broad achievement targets. 24
- 25. Digital Fluidity State Longitudinal DataSystems District Data Warehouses and Teacher Evaluation Systems Learning Tools-Driven Analytics School Teams School Leaders District Curriculum District Leaders Teacher Planning Individual Students State Analysis 25
- 26. Values in Design Infrastructureand Tools Context Organizational and Political Context •routines •access to data •leadership •time •norms •power relations Processes of data use •noticing •interpreting •constructing implications •data components •linkages •time span covered •Infrastructure boundaries •data quality •technology features 26
- 27.
- 28. Four Big Ideas 1.Sociotechnical paradigm shift 2. Notion of Education Data Sciences (EDS) – Academic/Institutional Analysis – Learning Analytics/Educational Data Mining – Learning Analytics/Personalization – Systemic Instructional Improvement 3. Common features across these communities 4. Framework for EDS 28
- 29. Education Data Sciences FramingEmergent Practices for Analytics of Learning, Organizations, and Systems Philip J. Piety Ed Info Connections ppiety@edinfoconnections.com Daniel T. Hickey Learning Sciences, School of Education Indiana University dthickey@indiana.edu MJ Bishop Center for Innovation and Excellence in Learning & Teaching University System of Maryland mjbishop@usmd.edu 29
Editor's Notes
- #2 Phil is with ed info connections in DCI am with IUMJ Bishop is the director of CIELT for the university system of Maryland
- #4 Let me summarize the four main points in our paper and in this talk.Paradigm shift in conception of data. It is a sociotechnological one.I will then argue that this movement involves four distinct fields and that together these fields might be most productive if merged into one broader notion of EDSWe then discuss five common features that unite the currently disparate EDS communities WE then
- #5 The scientific community describe by Kuhn are giving way to highly contextual and current paradigms and educational data is advancing more quickly than we know what do with it.This has created fragmentation that limits knowledge sharing and advance with different communities, different names, and different practices.It is not just that technology is driving the society but the society is driving the technology
- #6 We Hold These Truths to be Self Evident:Digital tools create vast quantities and categories of data: This datastorm is like nothing we have ever seen beforeQualitative Shift from institutional control of data, think personalized learning, people controlling their own learning. Bring your data with you like in health care.There is just so much more knowledge Th
- #7 The points here are twofold:There are parallels with other fields Even though we argue that education is different and has special properties, the movements in broad terms are similar and (very importantly) there are those who expect that education's transformation will parallel other fields. We are not arguing this, but there are parallels and this causes us to re-examine what about education is fundamentally different, unique.Education is both coming to this transformative stage late and with expectations to quickly catch up and so the time other fields have had to develop mature infrastructures was greater than education has leading to stresses in the process.Late to the game but that means that people expect EDS to be like finance and that is not goingThis is big and irrevesable.: OK, now we are in an age of big data and analytics and there is a lot of promise, a lot of potential, and a lot of need. At the same time, the field needs to develop a professional corps to deal with the new opportunity and challenges. It is not the
- #9 Now, this plugs my book. In doing so we can say that this book provides a broad argument for some of what to consider in this new paradigm, although there is more of a focus on K-12 than there could be. Specifically, the book:Compares education to other fields Explains using education data is necessary/difficultSynthesizes strands of education/organizational researchUsing design science and learning sciences framework
- #11 Traditionally called Institutional Research. Began as extension of analytic work and reporting many IHEs have been doing for years. FOCUSED ON INSTITUTIONAL NEEDSEarly warning systemsRecently, moving forward with new analyses including:Who gets accessHow they are admittedWhere they progress through the systemWhat is occurring in finance, fundraising, administration, grants management, etc.Moverment toward open standards (OAAI
- #12 Traditionally called Institutional Research. Began as extension of analytic work and reporting many IHEs have been doing for years. FOCUSED ON INSTITUTIONAL NEEDSEarly warning systemsRecently, moving forward with new analyses including:Who gets accessHow they are admittedWhere they progress through the systemWhat is occurring in finance, fundraising, administration, grants management, etc.Moverment toward open standards (OAAI
- #13 Largely from US with direct support No Child Left Behind (NCLB)Test focusedLots of talk about DDDMLimited by its data—lets just call it test driven decision making or even single test driven decision making.Ten years in an millions of dollars later we don’t have very much—But the notion of State Longitudinal Data Systems is big and includes more than test scoresNo conferences, no journals, some special issues but it is quite fragmentedlots of jobs and income for testing companiesSpecial issue of journalNo evidence of systemic validity, lots of evidence of negative consequenceIs getting a lot of attention because they will be using test scores to evaluated teacher performance
- #14 Largely from US with direct support No Child Left Behind (NCLB)Test focusedAccumulating research literature across many education communitiesLots of talk about DDDMLimited by its data—lets just call it test driven decision making or even single test driven decision making.Ten years in an millions of dollars later we don’t have very much—But the notion of State Longitudinal Data Systems is big and includes more than test scoresNo conferences, no journals, lots of jobsSpecial issue of journalNo evidence of systemic validity, lots of evidence of negative consequenceIs getting a lot of attention because they will be using test scores to evaluated teacher performance
- #15 Both Learning Analytics & Educational Data Mining (EDM) emerged around same time (2005)Have similar roots in digital learning environments. Now share many aspects. Learning Analytics was more rooted in Includes use of data from LMSData Mining is cognitive tutors and videotamesBig focus at NSF right now (Big Data in Education)Example of broad convergence
- #16 How many of you think that these should merge?Lot of parallels with Learning Sciences and CSCL
- #17 Macro or “learner” levelNow focus on “adaptive” mode considering:Time to revisit ATI, the availability of data may change things. Information-driven personalization
- #18 Historically learners differences was cognitive (Hence the early Aptitude Treatment Interaction Work
- #19 These four areas are examples of the kinds of emerging data-intensive activities that work across the different parts of the data sciences
- #21 1. Rapid changing indicative of sociotechnical movementLike the steam engine and the printing press, lots of innovation across multiple sectors2. Boundary IssuesFor example, flipped classrooms stretch across systemic improvement and personalization, early warning systems stretch across EDM and Institutional Analytics.3. Disruption in evidentiary practicesAcross all four of these communities, there have also been questions about how to use different kinds of information that were previously not available; how to make high quality inferences using the different kinds of evidence in ways appropriate to the context4. Visualization, interpretation, and cultureAcross these four areas we also see the emergence of issues around visualization and interpretation of information. Visualization is usually the lead element in these discussions as different representational schemes. These include “dashboards” that rank and sort individuals and other similar tools used to make sense of the vast amounts of diverse information available in these four communities. 5. Ethics and privacyAcross all four of these areas are issues of ethics and privacy; how the collection and use of the information about learners and teachers can be done responsibly while also safeguarding the privacy of those whose information is captured.
- #22 Appreciate the Distinctive Character of Education DataIn important ways this field is like the other areas where data are used in other domains such as health care, finance, and industry. In some other ways, educational data has unique properties. These unique properties include:Human/social creation. Unlike most other fields that use data, much of educational data requires human manipulation, which increases the possibility of error and manipulation. Some have focused on cheating and gaming of the system in the area of tests, but this property is actually much more pervasive and affects areas like special education planning and school improvement plans as well as assessments. Measurement imprecision. Educational data can be rife with issues of precision, especially when assessments of student learning or systemic capabilities are used. Compared to blood pressure readings or financial transactions, educational assessments are noisy. They can be sensitive to student background, instructional techniques, circumstances of testing, and the likeComparability challenges. Comparisons across different areas of educational data can be sometimes impacted by contact variation. For example, different schools are often compared for many different kinds of analyses. However, programmatic variation often occurs from school to school and those programmatic differences may not always be apparent in the data streams. Fragmentation. The world of educational data is fragmented. Many different organizations hold parts of educational information and there are still incomplete and partially adopted technical standards which impacts the ability to link some data without specific extra work. There are a number of efforts to create interoperable data standards. While progress has been made in these areas the road forward will be difficult as the governance of educational data is highly decentralized owing to the US Constitution’s delegation of authority for education to states and across the states there are many different approaches to state-district interactions and almost 20,000 district and charter providers.
- #24 The message here is that these six fields (not counting computer science) are all influential in the EDS, but computer science is more so because it is generative with the development of different innovations that will end up rippling through other fields and then to EDS. Roy Pea was behind this thinking.
- #26 The concept here is that the same artifacts can serve multiple purposes. This ties back nicely to the paradigm shift because historically data were developed for single purposes and often externally. Now, data is generated from many places and being used for many purposes.
- #27 This is really the sociotechnical thesis that argues that the artifacts – any artifacts – are not neutral. They are both adopted and adapted. They shape contexts and so the implications of them are far beyond their singular instrumental use. For example, think of NCLB testing. These are more than simply tests that measured student achievement. They cast a huge shadow over following years and were exploited by conservatives to help break the public’s confidence in public schools
- #28 The Gates Data Quality Campaign likes to use the phrase data should be like a flashlight not a hammer. I go further to say it is like lenses, imperfect, but generally better than nothing. Educational data is rarely truth, but often an indication of something to pay attention to.
- #29 Let me summarize the four main points in our paper and in this talk.Paradigm shift in conception of data. It is a sociotechnological one.I will then argue that this movement involves four distinct fields and that together these fields might be most productive if merged into one broader notion of EDSWe then discuss five common features that unite the currently disparate EDS communities WE then
- #30 Phil is with ed info connections in DCI am with IUMJ Bishop is the director of CIELT for the university system of Maryland