The document discusses different types of experiments that can be conducted in educational research and practice, particularly in online and blended contexts. It describes laboratory, design, embedded, and randomized controlled experiments. It provides examples of using internet tools and online platforms like Khan Academy to conduct experiments. The document advocates the REPEAT framework for iterative experimental research integrated with practice. It emphasizes early collaboration with practitioners and extending interventions over time.
What is Learning Analytics? Slides from a talk at a pre-conference seminar on learning analytics at the EMINENT conference, European Schoolnet, Pädagogishe Hochschule Zürich, 12 November 2014.
The OERu from the inside out and the outside inwitthaus
Presentation given at SAIDE (South African Institution of Distance Education) in Johannesburg, 15 June 2012. The audience included friends of mine from the University of the Witwatersrand and consultants in the fields of adult basic education and training, and so I included an overview of the whole field of OERs before sharing what I had learnt about the OER university through my TOUCANS research.
Presentation overview:
Part 1: OERs... the story so far
Part 2: The OERu from the inside out (views of people working within OERu network institutions)
Part 3: The OERu from the outside in (UK HEI views on the OERu concept)
Please note that this is work in progress and findings are indicative.
This is an updated version of an invited talk I presented at the European Research Council-Brussels (Scientific Seminar): "Love for Science or 'academic prostitution'".
It has been updated to be presented at my home institution (Instituto de Astrofísica de Andalucía - CSIC) in a scientific seminar (14 June 2013).
I have included some new slides and revised others.
I present a personal revision (sometimes my own vision) of some issues that I consider key for doing Science. It was focused on the expected audience, mainly Scientific Officers with background in different fields of science and scholarship, but also Agency staff.
Abstract: In a recent Special issue of Nature concerning Science Metrics it was claimed that " Research reverts to a kind of 'academic prostitution' in which work is done to please editors and referees rather than to further knowledge."If this is true, funding agencies should try to avoid falling into the trap of their own system. By perpetuating this 'prostitution' they risk not funding the best research but funding the best sold research.
Given the current epoch of economical crisis, where in a quest for funds researchers are forced into competitive game of pandering to panelists, its seems a good time for deep reflection about the entire scientific system.
With this talk I aim to provoke extra critical thinking among the committees who select evaluators, and among the evaluators, who in turn require critical thinking to the candidates when selecting excellent science.
I will present some initiatives (e.g. new tracers of impact for the Web era- 'altmetrics'), and on-going projects (e.g. how to move from publishing advertising to publishing knowledge), that might enable us to favor Science over marketing.
What is Learning Analytics? Slides from a talk at a pre-conference seminar on learning analytics at the EMINENT conference, European Schoolnet, Pädagogishe Hochschule Zürich, 12 November 2014.
The OERu from the inside out and the outside inwitthaus
Presentation given at SAIDE (South African Institution of Distance Education) in Johannesburg, 15 June 2012. The audience included friends of mine from the University of the Witwatersrand and consultants in the fields of adult basic education and training, and so I included an overview of the whole field of OERs before sharing what I had learnt about the OER university through my TOUCANS research.
Presentation overview:
Part 1: OERs... the story so far
Part 2: The OERu from the inside out (views of people working within OERu network institutions)
Part 3: The OERu from the outside in (UK HEI views on the OERu concept)
Please note that this is work in progress and findings are indicative.
This is an updated version of an invited talk I presented at the European Research Council-Brussels (Scientific Seminar): "Love for Science or 'academic prostitution'".
It has been updated to be presented at my home institution (Instituto de Astrofísica de Andalucía - CSIC) in a scientific seminar (14 June 2013).
I have included some new slides and revised others.
I present a personal revision (sometimes my own vision) of some issues that I consider key for doing Science. It was focused on the expected audience, mainly Scientific Officers with background in different fields of science and scholarship, but also Agency staff.
Abstract: In a recent Special issue of Nature concerning Science Metrics it was claimed that " Research reverts to a kind of 'academic prostitution' in which work is done to please editors and referees rather than to further knowledge."If this is true, funding agencies should try to avoid falling into the trap of their own system. By perpetuating this 'prostitution' they risk not funding the best research but funding the best sold research.
Given the current epoch of economical crisis, where in a quest for funds researchers are forced into competitive game of pandering to panelists, its seems a good time for deep reflection about the entire scientific system.
With this talk I aim to provoke extra critical thinking among the committees who select evaluators, and among the evaluators, who in turn require critical thinking to the candidates when selecting excellent science.
I will present some initiatives (e.g. new tracers of impact for the Web era- 'altmetrics'), and on-going projects (e.g. how to move from publishing advertising to publishing knowledge), that might enable us to favor Science over marketing.
Presentation given at NUI, Galway 2019-04-11 for Open Science Week.
An overview of Early Career Researchers, their innovation and contribution towards Open Infrastructure
Keynote at 4th Annual KnowEscape Conference, Sofia, Bulgaria (Feb 24, 2017). http://knowescape.org/knowescape2017/
Yes, we’re open: Open science & altmetrics
Abstract: Open Science is en vogue – especially after Carlos Moedas, EU-Commissioner for Research, Science and Innovation, has outlined his vision for Europe along the lines of ‘open innovation, open science, open to the world’. Open science describes the transition of ‘publishing as fast as possible’ towards ‘sharing knowledge as fast as possible’. Several reasons explain the move towards openness, it is expected, for example, that open science will increase the efficiency of science. Of course, digital media and web-based environments are keys to this development, but it also requires a systemic change to transform open science from a nice-to-have-feature into the default way of performing research. Altmetrics, i.e. social media-based metrics, are often considered drivers of open science and essential tools for changing the reward system in science. When looking closer, though, severe tensions between features as well as expectations of open science and altmetrics become apparent. The talk will argue that open science only can enfold its potential if ‘openness’ is fully embraced and supported by open metrics.
We were trying to figure out the issue on why most of our students refuse to think and unable to think creative and critically. Here's the ideas of the problems.
These slides are about Quality Education, this presentation will help you to find the factors, dimensions and approaches of quality education, and will make you aware of issues and problems which are affecting the quality of education
Presentation given at NUI, Galway 2019-04-11 for Open Science Week.
An overview of Early Career Researchers, their innovation and contribution towards Open Infrastructure
Keynote at 4th Annual KnowEscape Conference, Sofia, Bulgaria (Feb 24, 2017). http://knowescape.org/knowescape2017/
Yes, we’re open: Open science & altmetrics
Abstract: Open Science is en vogue – especially after Carlos Moedas, EU-Commissioner for Research, Science and Innovation, has outlined his vision for Europe along the lines of ‘open innovation, open science, open to the world’. Open science describes the transition of ‘publishing as fast as possible’ towards ‘sharing knowledge as fast as possible’. Several reasons explain the move towards openness, it is expected, for example, that open science will increase the efficiency of science. Of course, digital media and web-based environments are keys to this development, but it also requires a systemic change to transform open science from a nice-to-have-feature into the default way of performing research. Altmetrics, i.e. social media-based metrics, are often considered drivers of open science and essential tools for changing the reward system in science. When looking closer, though, severe tensions between features as well as expectations of open science and altmetrics become apparent. The talk will argue that open science only can enfold its potential if ‘openness’ is fully embraced and supported by open metrics.
We were trying to figure out the issue on why most of our students refuse to think and unable to think creative and critically. Here's the ideas of the problems.
These slides are about Quality Education, this presentation will help you to find the factors, dimensions and approaches of quality education, and will make you aware of issues and problems which are affecting the quality of education
As India aspires for economic growth, it will need to invest in Education. Here's an examination of what is ailing the system, and recommendations for amending these.
Using Experiments and Cognitive Science Research to Improve the Design of Onl...Joseph Jay Williams
The recent explosion of online educational resources has the potential to reorganize how we learn – from K-12 and university to the workplace and the informal learning we do every day. It also raises new questions and opportunities for research that crosses the many disciplines relevant to designing computer programs that help people learn. For example, HCI and cognitive science can provide complementary perspectives in investigating how to design the content and instructional features of an online course, such that a person processes and stores that information in a way that successfully guides their future behavior. Online educational environments provide new optimism in tackling challenges like these because they can be instrumented to collect an unprecedented scale and diversity of data, and allow iterative sequences of experiments to be embedded in authentic educational contexts with real students.
This talk presents one approach to this kind of research, using experimental comparisons to test the effects of modifying online mathematics exercises to include motivational messages and question prompts for people to explain, the design of which is guided by the psychological literature on motivation and learning. A combination of laboratory experiments and experiments embedded in real-world online education platforms (like www.KhanAcademy.org) reveal that prompting people to explain “why?” facts are true drives them beyond memorization to uncover underlying principles and patterns, and that teaching such self-questioning strategies may accelerate student learning. Motivational messages appear to have limited benefits if they are simply encouraging or aimed at raising confidence, but do increase how much effort students invest if the messages emphasize that aptitude is malleable and can be improved through persistence. Several planned experiments are presented which also use this paradigm of adding minimal but effective textual changes to online exercises to achieve practical impact and explore basic cognitive science questions about learning.
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partn...Simon Bates
Keynote lecture at 2016 NTU Learning and Teaching Seminar - Students as Partners in Learning and Teaching. In this keynote, I will consider the role of students as partners in learning with reference to what current research can tell us about how people learn, what students have to say about what supports their learning, and where technology can help.
Research Webinar: OERS and Cognitive ScienceiNACOL
This webinar provides practical information on how to use published research findings and make contact with cognitive scientists in order to improve K-12 and university students’ learning from digital online resources, like Khan Academy videos or interactive mathematics exercises. The webinar focuses on how students’ motivation and grades have been increased by helping them believe they can take charge of their learning and become smarter, and how students can be supported in reflective thinking and seeking deep understanding, when questions and prompts for students to explain are inserted in videos and interactive exercises
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformationalthe Hartsook Letter
Keynote Speaker: Vijay Kumar
Dr. Vijay Kumar is Senior Associate Dean of Undergraduate Education and Director of the Office of Educational Innovation and Technology, DUE at MIT. In this capacity he provides leadership for sustainable technology-enabled educational innovation at MIT. In his prior roles at MIT as Assistant Provost and Director of Academic Computing, as well at other institutions, Vijay provided leadership for units engaged in delivering infrastructure and services for the effective integration of information technology and media services in education. Vijay was the Principal Investigator of O.K.I (Open Knowledge Initiative), a MIT-led collaborative project to develop an open architecture for enterprise educational applications.
Vijay is a member of the Advisory Committee of MIT OpenCourseWare (OCW) and a member of the steering committee for I-Campus, the MIT-Microsoft Alliance initiative for educational technology. He is the Executive officer for MIT's Council on Educational Technology. Vijay also served on the Applications Strategy Council for Internet2, as a Trustee of the Corporation for Research and Education Networking (CREN), Chair of the Boards of the Seminars on Academic Computing (SAC) Snowmass, CO and NERCOMP.
Vijay is an active champion of open education efforts: As an honorary Advisor to India's National Knowledge Commission he has been engaged in advancing Open and technology enabled initiatives for educational access and quality in India; He is actively involved in efforts, such as those supported by the Hewlett Foundation, and Curriki to advance the use of Open Educational Resources for improving educational access and quality. He is also an advisor to the Open University of Catalonia.
Vijay has recently co-edited "Opening Up Education: The Collective Advancement of Education through Open Technology, Open Content and Open Knowledge " (MIT Press, August 2008), a book sponsored by the Carnegie Foundation for the Advancement of Teaching.
Doha College Mobile Learning Conference 2014: Learning from ResearchKevin Burden
Dr. Kevin Burden (The University of Hull) argues that like many educational technologies in the past, whilst we know fairly well WHAT works when students have access to a mobile device, we have virtually no idea WHY it works. Design Based Research (DBR) offers an opportunity to unlock this mystery and in so doing help to replicate and extend the use of mobile technologies in ways which have not even been imagined yet
Learning Analytics: Seeking new insights from educational dataAndrew Deacon
CPUT Fundani TWT - 22 May 2014
Analytics is a buzzword that encompasses the analysis and visualisation of big data. Current interest results from the growing access to data and the many software tools now available to analyse this data in Higher Education, through platforms such as Learning Management Systems. This seminar provides an overview of current applications and uses of learning analytics and how it can help institutions of learning better support their learners. The illustrative examples look at institutional and social media data that together provide rich insights into institutional, teaching and learning issues. A few simple ways to perform such analytics in a context of Higher Education will be introduced.
Personalized Learning: Expanding the Social Impact of AIPeter Brusilovsky
Slide of my keynote talk at SIAIA '23 workshop held at AAAI 2023:
The use of AI in Education could be traced to the early days of AI. While the publicity associated with the most recent wave of AI applications rarely mentions education, it is through the improvement in education AI could achieve an impressive social impact. In particular, the AI ability to personalize the learning process could make a large difference in a context where learners' knowledge could be radically different from learner to learner. Modern computer and internet technologies can now bring the power of learning in the forms of MOOCs, online textbooks, and zoom courses truly worldwide. Yet, without personalization, the potential of these technologies is not fully leveraged. In this talk, I will review several generations of research on personalized learning and discuss tools, technologies, and infrastructures for personalized learning that we are currently exploring.
Data-Driven Education: Using Big Educational Data to Improve Teaching and Learning. Keynote slides for 15th International Conference on Web-Based Learning, ICWL 2016, Rome, Italy, October 26–29.
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...Joseph Jay Williams
Paper at CHI 2019, PDF at tiny.cc/icepdf.
Digital educational resources could enable the use of randomized
experiments to answer pedagogical questions that
instructors care about, taking academic research out of the
laboratory and into the classroom. We take an instructorcentered
approach to designing tools for experimentation that
lower the barriers for instructors to conduct experiments. We
explore this approach through DynamicProblem, a proof-ofconcept
system for experimentation on components of digital
problems, which provides interfaces for authoring of experiments
on explanations, hints, feedback messages, and learning
tips. To rapidly turn data from experiments into practical improvements,
the system uses an interpretable machine learning
algorithm to analyze students’ ratings of which conditions are
helpful, and present conditions to future students in proportion
to the evidence they are higher rated. We evaluated the system
by collaboratively deploying experiments in the courses
of three mathematics instructors. They reported benefits in
reflecting on their pedagogy, and having a new method for
improving online problems for future students.
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Joseph Jay Williams
Abstract: How can online learning platforms provide useful information about pedagogy to instructors teaching online, while ensuring that course teams are not constrained in leveraging their teaching expertise to personalize their MOOC? The scientific literature on learning and education provides hundreds of detailed studies, which can be synthesized to identify effective instructional strategies, and mined for examples of how an instructional strategy can be implemented in a specific environment, set of educational materials, or student population. This talk illustrates this approach, by presenting a worksheet guide that supports MOOC designers in using two instructional strategies: increasing student motivation to think through challenges by designing exercises which encourage students to see their intelligence as malleable, and enhancing deep understanding with questions and prompts for students to explain. The talk explains how these two instructional strategies are motivated by both existing literature and recently conducted experimental studies. It also presents the specific details of how the guide is targeted at MOOC instructors and provides them with multiple actionable strategies they can use in their courses.
How can Cognitive Science improve Online Learning & Education?Joseph Jay Williams
Slides from Google Tech Talk by Joseph Jay Williams. The video presentation is on Youtube: http://youtu.be/VKW5lZqBWgI
Title: How can Cognitive Science improve Online Learning at Google and Google in Education?
Abstract: Knowledge and technology that maximizes human learning has financial value for Google in customer education and internal training, as well as social value for the public initiatives of Google in Education. Recent research in Cognitive Science provides complementary insights to those gained from practical experience and the research in Computer Science, Education and other Learning Sciences. This talk considers how learning can be improved by: (1) Asking questions and requesting explanations; (2) Presenting specific examples to illustrate abstract principles; (3) Using tests as pedagogical rather than assessment tools. Moreover, online education provides the unique opportunity of hybrid research that is simultaneously applied and academic. Online environments satisfy the scientific requirements of randomized experiments and precise control, as well as the practical need for ecological validity, fidelity, and scalable dissemination. The Cognitive Science focus on identifying both similarities and differences across learning contexts positions it well for doing research that simultaneously advances public education and a corporate mission. In addition to presenting ongoing research at Khan Academy and MOOCs like EdX, I discuss how analogous principles can be explored in teaching end-users Google Power Search, internal training, and customer education.
Bio: Joseph Jay Williams (For resources on Cognitive Science, Online Education, Ed-Tech see: www.josephjaywilliams.com/education, sites.cognitivescience.co/learn, or www.learningresearch.net) does Cognitive Science research on how generating explanations promotes learning, and Online Education work on improving learning from mathematics exercises (Khan Academy), increasing motivation to learn by changing people's beliefs about intelligence (Project for Education Research that Scales: www.perts.net), teaching metacognitive & learning strategies in Massive Open Online Courses (EdX), and using technology to change educational and health habits. He is finishing his PhD in Psychology at UC Berkeley, and also has interests in consulting for corporate e-learning and training, web development for online education, using journalism to disseminate research to practitioners, and education in online search and problem-solving for students and entrepreneurs.
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Jay Williams
Talk on how Cognitive Science can be applied to corporate e-learning and instructional design. 30 minutes, at Bloombsburg Corporate Advisory Council Meeting. Covers problem-based learning, explanation, analogy, comparison, retrieval practice, testing effects, mixing effects, teaching Google search.
How to Split Bills in the Odoo 17 POS ModuleCeline George
Bills have a main role in point of sale procedure. It will help to track sales, handling payments and giving receipts to customers. Bill splitting also has an important role in POS. For example, If some friends come together for dinner and if they want to divide the bill then it is possible by POS bill splitting. This slide will show how to split bills in odoo 17 POS.
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
Students, digital devices and success - Andreas Schleicher - 27 May 2024..pptxEduSkills OECD
Andreas Schleicher presents at the OECD webinar ‘Digital devices in schools: detrimental distraction or secret to success?’ on 27 May 2024. The presentation was based on findings from PISA 2022 results and the webinar helped launch the PISA in Focus ‘Managing screen time: How to protect and equip students against distraction’ https://www.oecd-ilibrary.org/education/managing-screen-time_7c225af4-en and the OECD Education Policy Perspective ‘Students, digital devices and success’ can be found here - https://oe.cd/il/5yV
The Indian economy is classified into different sectors to simplify the analysis and understanding of economic activities. For Class 10, it's essential to grasp the sectors of the Indian economy, understand their characteristics, and recognize their importance. This guide will provide detailed notes on the Sectors of the Indian Economy Class 10, using specific long-tail keywords to enhance comprehension.
For more information, visit-www.vavaclasses.com
We all have good and bad thoughts from time to time and situation to situation. We are bombarded daily with spiraling thoughts(both negative and positive) creating all-consuming feel , making us difficult to manage with associated suffering. Good thoughts are like our Mob Signal (Positive thought) amidst noise(negative thought) in the atmosphere. Negative thoughts like noise outweigh positive thoughts. These thoughts often create unwanted confusion, trouble, stress and frustration in our mind as well as chaos in our physical world. Negative thoughts are also known as “distorted thinking”.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdfTechSoup
In this webinar you will learn how your organization can access TechSoup's wide variety of product discount and donation programs. From hardware to software, we'll give you a tour of the tools available to help your nonprofit with productivity, collaboration, financial management, donor tracking, security, and more.
Operation “Blue Star” is the only event in the history of Independent India where the state went into war with its own people. Even after about 40 years it is not clear if it was culmination of states anger over people of the region, a political game of power or start of dictatorial chapter in the democratic setup.
The people of Punjab felt alienated from main stream due to denial of their just demands during a long democratic struggle since independence. As it happen all over the word, it led to militant struggle with great loss of lives of military, police and civilian personnel. Killing of Indira Gandhi and massacre of innocent Sikhs in Delhi and other India cities was also associated with this movement.
Thesis Statement for students diagnonsed withADHD.ppt
Experiments in Educational Research and Practice
1. Experiments in Educational Research
& Practice
Joseph Jay Williams
josephjaywilliams@gmail.com
www.josephjaywilliams.com
Lytics Lab, Office of the Vice Provost of Online
Education, Graduate School of Education, Stanford
2. Experiments in Educational Research & Practice
•
Kinds of Experiments
–
–
–
–
•
Laboratory
Design experiment (Brown, 1992)
In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)
Randomized Controlled Trial/Intervention
Experiments & Internet, Online Education
–
Internet-facilitated Laboratory studies
•
•
•
–
–
–
–
•
•
Mechanical Turk (tiny.cc/mechanicalturk )
Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )
“A/B testing” more widespread
Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)
In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)
Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)
Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
REPEAT Approach to Experimental Research on Online/Blended Resources
Identifying a Practical Online Context for your Research Question
–
–
–
–
Partner with platform *early* in research
“Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)
Examples of Blended Learning Resources & MOOCs
Examples of resources that support Randomized Trials
–
–
Interventions extended over time
Bridging studies
•
•
Engaging with Practitioners
–
–
How to communicate value of experiments
Consulting
•
•
•
•
–
–
•
Educational companies & publishers (marcy.baughman@pearson.com )
Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer)
Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange)
Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)
Virtual collaboration on practical resources (www.learnnetwork.net )
Experiments & …
–
–
–
•
Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,
Computer Supported Collaborative Learning (Rose)
Cognitive Tutors (Aleven, Ritter, Koedinger)
EDM & AIED (Stamper, Gordon, Pardos)
Conducting Experiments (tiny.cc/conductingexperiments)
7. Integrate Research & Practice
• Randomized assignment
• Experimental Control
• Rich data
• Real-world environment
• Authentic activities
• Practical Challenges
• Generalizable theories
• “in vivo” experiments
• Diverse populations
• Practical improvements
• Disseminate research
• Generate Funding
8. Experiments & Internet, Online Education
– Internet-facilitated Laboratory studies
• Mechanical Turk (tiny.cc/mechanicalturk )
• Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )
• “A/B testing” more widespread
– Iterative Design & Improvement (Amy Collier, Stanford Office of Online
Learning)
– In vivo motivation experiment on Khan Academy (Williams et
al, 2013, MOOCshop)
– Conducting Randomized Trials over the Internet
(Paunesku, Greene, Williams)
– Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
9. In-vivo motivation experiment on Khan Academy
Jascha Paunesku,
• Williams,Sohl-Dickstein Haley, & SohlDickstein, MOOCshop, AIED
• Embed messages in online Khan Academy exercises
• Effect of Growth Mindset beyond encouragement?
10. Implicit beliefs about Intelligence
• On a scale from 1 to 10, how much do you agree
that?
– Your intelligence is something very basic about you that you can’t
change very much.
– No matter how much intelligence you have, you can always change it
quite a bit.
• Growth vs. Fixed Mindset (Dweck, 2007)
11. Experimentally manipulate added messages
Practice-as-usual Message
Growth Mindset Message
Positive
Some of these problems are hard. Do your best!
Remember, the more you practice the smarter you become!
12. Design
Practice-as-usual
• Growth Mindset Message
• Positive Message
•
•
•
"Remember, the more you practice the
smarter you become.”,
"Mistakes help you learn. Think hard to
learn from them.”
•
"Some of these problems are hard. Just
do your best."
"This might be a tough problem, but we
know you can do it.”
• 50 000+ students per condition
• Dependent measures:
– Number of problems completed
– Number correct on first attempt
– Accuracy
13. R.E.P.E.A.T. Framework for Online Education Research
•
•
•
•
•
•
Realistic
Experimental
Product
Evaluated
Accessible
Theoretically motivated
• REPEAT – iteratively improve through revision &
collaboration
13
14. Research & Practice in Online Education
• APS symposium: Williams, Saxberg, Means, Mitros.
(2013). Online Learning and Psychological Science:
Opportunities to integrate research and practice.
• Williams, J. J., Renkl, A., Koedinger, K., Stamper, J.
(2013). Online Education: A Unique Opportunity for
Cognitive Scientists to Integrate Research and
Practice. In M. Knauff, M. Pauen, N. Sebanz, & I.
Wachsmuth (Eds.), Proceedings of the 35th Annual
Conference of the Cognitive Science Society.
15. Experiments in Educational Research & Practice
•
Kinds of Experiments
–
–
–
–
•
Laboratory
Design experiment (Brown, 1992)
In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)
Randomized Controlled Trial/Intervention
Experiments & Internet, Online Education
–
Internet-facilitated Laboratory studies
•
•
•
–
–
–
–
•
•
Mechanical Turk (tiny.cc/mechanicalturk )
Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )
“A/B testing” more widespread
Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)
In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)
Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)
Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
REPEAT Approach to Experimental Research on Online/Blended Resources
Identifying a Practical Online Context for your Research Question
–
–
–
–
Partner with platform *early* in research
“Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)
Examples of Blended Learning Resources & MOOCs
Examples of resources that support Randomized Trials
–
–
Interventions extended over time
Bridging studies
•
•
Engaging with Practitioners
–
–
How to communicate value of experiments
Consulting
•
•
•
•
–
–
•
Educational companies & publishers (marcy.baughman@pearson.com )
Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer)
Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange)
Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)
Virtual collaboration on practical resources (www.learnnetwork.net )
Experiments & …
–
–
–
•
Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,
Computer Supported Collaborative Learning (Rose)
Cognitive Tutors (Aleven, Ritter, Koedinger)
EDM & AIED (Stamper, Gordon, Pardos)
Conducting Experiments (tiny.cc/conductingexperiments)
16. Identifying a Practical Online Context for your Research Question
– Partner with platform *early* in research
– “Customer Development”: Engaging with
Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)
– Examples of Blended Learning Resources & MOOCs
– Examples of resources that support Randomized Trials
• Go to the source: Empirically test value of instructional prescriptions & professional
development (University of Virginia,
– Interventions extended over time
– Bridging studies
17. Engaging with Practitioners
– How to communicate value of experiments
– Consulting
• Educational companies & publishers (marcy.baughman@pearson.com )
• Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will
Thalheimer)
• Medical Education & Health Behavior (Medbiquitous, International Society for
Internet Interventions, cognitivescience.co/behaviorchange)
• Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
– Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)
– Virtual collaboration on practical resources (www.learnnetwork.net )
19. Experiments in Educational Research & Practice
•
Kinds of Experiments
–
–
–
–
•
Laboratory
Design experiment (Brown, 1992)
In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)
Randomized Controlled Trial/Intervention
Experiments & Internet, Online Education
–
Internet-facilitated Laboratory studies
•
•
•
–
–
–
–
•
•
Mechanical Turk (tiny.cc/mechanicalturk )
Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )
“A/B testing” more widespread
Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)
In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)
Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)
Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
REPEAT Approach to Experimental Research on Online/Blended Resources
Identifying a Practical Online Context for your Research Question
–
–
–
–
Partner with platform *early* in research
“Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)
Examples of Blended Learning Resources & MOOCs
Examples of resources that support Randomized Trials
–
–
Interventions extended over time
Bridging studies
•
•
Engaging with Practitioners
–
–
How to communicate value of experiments
Consulting
•
•
•
•
–
–
•
Educational companies & publishers (marcy.baughman@pearson.com )
Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer)
Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange)
Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)
Virtual collaboration on practical resources (www.learnnetwork.net )
Experiments & …
–
–
–
•
Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,
Computer Supported Collaborative Learning (Rose)
Cognitive Tutors (Aleven, Ritter, Koedinger)
EDM & AIED (Stamper, Gordon, Pardos)
Conducting Experiments (tiny.cc/conductingexperiments)
Editor's Notes
Reframing Education to teachers – it’s about INVESTIGATION. Evidence-based practice.*Teacher buy-in, teacher CONTRIBUTION. Teacher modification.Unnatural things researchers are doing…Vs. Teachers doing things anyway, trying it out. Researchers are being involved.“Experiment”. It’s a comparison of instruction… (less the theory, more about its effectiveness).Personalization.Bridging studies.It’s on a case-by-case basis.What’s similar, what’s different.Still pretty diverse.Population demographics.Motivational issues – being paid.Are your assumptions correct? Just having to IMPLEMENT it concretely is valuable, as a DESIGN exercise.Cons:I AM CHOOSING just the right contexts. Most people start with others.> Can you adapt your research question to make it easier to run in a Bridging Study?Interventions:Short & Fat.Long & Skinny.
Features. Heuristics. Guiding principles.Context of how this fits in the larger context.
Reframing Education to teachers – it’s about INVESTIGATION. Evidence-based practice.*Teacher buy-in, teacher CONTRIBUTION. Teacher modification.Unnatural things researchers are doing…Vs. Teachers doing things anyway, trying it out. Researchers are being involved.“Experiment”. It’s a comparison of instruction… (less the theory, more about its effectiveness).Personalization.Bridging studies.It’s on a case-by-case basis.What’s similar, what’s different.Still pretty diverse.Population demographics.Motivational issues – being paid.Are your assumptions correct? Just having to IMPLEMENT it concretely is valuable, as a DESIGN exercise.Cons:I AM CHOOSING just the right contexts. Most people start with others.> Can you adapt your research question to make it easier to run in a Bridging Study?Short & Fat.Long & Skinny.
Reframing Education to teachers – it’s about INVESTIGATION. Evidence-based practice.*Teacher buy-in, teacher CONTRIBUTION. Teacher modification.Unnatural things researchers are doing…Vs. Teachers doing things anyway, trying it out. Researchers are being involved.“Experiment”. It’s a comparison of instruction… (less the theory, more about its effectiveness).Personalization.Bridging studies.It’s on a case-by-case basis.What’s similar, what’s different.Still pretty diverse.Population demographics.Motivational issues – being paid.Are your assumptions correct? Just having to IMPLEMENT it concretely is valuable, as a DESIGN exercise.Cons:I AM CHOOSING just the right contexts. Most people start with others.> Can you adapt your research question to make it easier to run in a Bridging Study?Short & Fat.Long & Skinny.