Measuring and Evaluating Educational Research Literacy in Higher Education: A Synthesis of Mechanisms and Discoveries
•Download as PPTX, PDF•
0 likes•3 views
Conference presentation slides used in ACE2022 in Tokyo.
Recommended
Recommended
More Related Content
Similar to Measuring and Evaluating Educational Research Literacy in Higher Education: A Synthesis of Mechanisms and Discoveries
Similar to Measuring and Evaluating Educational Research Literacy in Higher Education: A Synthesis of Mechanisms and Discoveries (20)
More from Jingjing Lin
More from Jingjing Lin (20)
Recently uploaded
Recently uploaded (20)
Measuring and Evaluating Educational Research Literacy in Higher Education: A Synthesis of Mechanisms and Discoveries
- 1. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Measuring and Evaluating Educational Research Literacy in Higher Education: A Synthesis of Mechanisms and Discoveries Dr. Jingjing Lin Developer of ResearchIC.com Assistant Professor, Center for IT-based Education, Toyohashi University of Technology 1
- 2. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Introduction The 14th Asian Conference on Education (2022), Tokyo 2 Input Output Training Competency Career success Research Literacy Scientific literature Scientific literature
- 3. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Introduction (continued) Research literacy Domains Education sciences Medicine, nursing, health sciences Other domains The 14th Asian Conference on Education (2022), Tokyo 3
- 4. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Introduction (continued) Practitioners nurses psychiatry residents social workers chaplains schoolteachers Pre-Service Teachers or PST Researchers The 14th Asian Conference on Education (2022), Tokyo 4 RL training
- 5. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Research question How is ERL trained, measured, and evaluated in the population of researchers and research students in higher education? The 14th Asian Conference on Education (2022), Tokyo 5
- 6. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Methodology • Systematic literature review (SLR) • Peer-review journal articles written in English, reported RL intervention(s) or measurement at the higher education level in the domain of education science, collected empirical data from actual participants, and were available in full text. • PRISMA flow chart The 14th Asian Conference on Education (2022), Tokyo 6
- 7. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Results Overview of 11 studies 7 The 14th Asian Conference on Education (2022), Tokyo Modes: Publication year: 2017 Country: Germany Research type: Quantitative Research design: Cross-sectional survey, or pre-test/post-test Participants: PST Instrument: LeScEd booklet
- 8. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Measure instruments of ERL • LeScEd booklet • two test sets (avoid testing effect) • to measure: information literacy, statistical literacy, ethical reasoning • forced-choice tasks • 40-minute test time (30% tasks were left out on average; it is fine) • Tested and validated using • Pretest–posttest control group or cross-sectional/longitudinal surveys • large samples of PST students (e.g., 2,113 in one study and 1,655 in another study) The 14th Asian Conference on Education (2022), Tokyo 8 (Schladitz et al., 2013)
- 9. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Measure instruments of ERL • Psychological Research Inventory of Concepts (PRIC) • 20-item vignette-based multiple choice measure • to measure: research methodology and statistics knowledge • either yes/no or mostly open-ended responses to the questions • 30 minutes per test • Tested and validated using • cross-sectional and longitudinal surveys • 971 participants of both psychology and non-psychology backgrounds The 14th Asian Conference on Education (2022), Tokyo 9 Veilleux and Chapman (2017b)
- 10. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Measure instruments of ERL • Research Literacy Inventory • 18 self-reported statements, based on the work by Shank & Brown (2013) • five-point Likert scale • Tested and validated using • Pre-test and post-test with control group • 62 PST students • Research literacy survey • 15-item self-report survey • to measure: source/literature search (SLS) and research integration methods (RIM) • Tested and validated using • Pre-test and post-test with control group • 48 PST students 10 The 14th Asian Conference on Education (2022), Tokyo Gutman and Genser (2017) Han and Schuurmans-Stekhoven (2017)
- 11. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) The 14th Asian Conference on Education (2022), Tokyo 11 ERL training interventions
- 12. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Summary of findings • Long-term evidence suggests that self-regulated asynchronous learning benefits students’ retention of RL skills more than synchronous learning (Gutman & Genser, 2017). • “Short-term intensive training may have provided theoretical knowledge of what information to process and how to process it, but it is insufficient for students to fully comprehend its application in their own practice” (Han & Schuurmans- Stekhoven, 2017, p. 38). • No distinct advantage was found between multiple-choice questions and open- ended questions; therefore, both formats can be used in ERL tests. (Schladitz et al., 2017) • The most validated instruments is LeScEd booklet. The 14th Asian Conference on Education (2022), Tokyo 12
- 13. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Summary of findings • The most common format of intervention was course. • Web-based technology (e.g., CMS, twitter) was used to support delivery of intervention. • The interventions were mostly for PST students in education science. • The interventions were often general research education, introducing research process and methods as a general experience. • Half interventions had high replicability and the other half had low replicability. • Online journal club is not a common training intervention format in ERL training interventions; How to read scientific literature was not a focus for any of the examined eight interventions. The 14th Asian Conference on Education (2022), Tokyo 13
- 14. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) 14 The 14th Asian Conference on Education (2022), Tokyo
- 15. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) 100 Online Journal Club Events They have joined: • 12 educational researchers • 11 universities • 7 countries/regions The 14th Asian Conference on Education (2022), Tokyo 15
- 16. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Join as a host • [2023 & 2024] Open call for 100 Online Journal Club or OJC Events in education sciences • ResearchIC.com – funded by JSPS KAKENHI Grant Number 22K13755 • https://bit.ly/3zYhsNj 16 The 14th Asian Conference on Education (2022), Tokyo
- 17. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) Measuring and Evaluating Educational Research Literacy in Higher Education: A Synthesis of Mechanisms and Discoveries Email: lin@cite.tut.ac.jp Websites: https://researchic.com https://jingjing-lin.com 17
- 18. ResearchIC.com Hosting and Attending 100 Online Journal Club Events in Education Sciences (2023-2024) References • Beaudry, J. S., & Miller, L. (2016). Research literacy: A primer for understanding and using research. Guilford Publications. The 14th Asian Conference on Education (2022), Tokyo 18
Editor's Notes
- Good afternoon. My name is Jingjing Lin. I am an assistant professor from Toyohashi University of Technology in Japan. I manage the course management system at my university, I also use my skills of Moodle to develop ResearchIC.com, which is a platform that allow researchers and research students in education science to host and attend online journal club events for free. Today, I would like to share with you some associated research on the ResearchIC project.
- Reading and writing scientific literature are like breathing in and breathing out for a researcher. We read others’ publications as input to merge with our own thoughts and produce scientific publications of our own as output. This work process needs years of training to develop a good level of research literacy that can support our career success.
- Research literacy is the ability to locate, understand, discuss, and evaluate different types of research; to communicate accurately about them; and to use findings for academic and professional purposes. (Beaudry & Miller, 2016) It is an ability often segmented by research domains. When we talk about educational research literacy, it is likely different from medical research literacy. This is because different domains’ scholars approach phenomena with different epistemological views and analysis tools. In the current literature on research literacy, the health-related subject area is the most active, whereas education science is much less active.
- I also found that research literacy training is quite often targeting frontline practitioners in different domains. The primary common purpose is to advocate research-informed practices. However, researchers as frontline practitioners in research work are rarely studied in the research literacy publications. How to advocate research-informed research practice by researchers and research students is highly omitted by scholars who publish on the topic of research literacy.
- Therefore, I was driven to understand better educational research literacy for researchers and research students in the domain of education science. How is it trained, measured, and evaluated.
- To answer this question, I conducted a systematic literature review using two databases: Scopus, and Web of science. I searched using the keyword “research literacy” and you can see that there are not many articles returned as results. I limited the results to include only peer-reviewed journal articles written in English, which reported research literacy intervention or measurement in the field of education science, was empirical studies, and I could access to its full text file. After data cleaning and screening, a total of 11 articles were included for further analysis.
- Let’s look at the overview of 11 studies. The first study was published in 2002, and 2017 is the year with the most publications. Germany is the most productive on this topic. 54% of studies are quantitative. The prevalent research design approach is to use cross-sectional survey. The pre-service teacher students are the most used sample. A frequent appearance of a measure instrument called LeScEd booklet was found. When it comes to the concept of research literacy per se, a few studies mixed it with information literacy, data literacy, or treat it as one whole without breaking it into sub-concepts. A group of researchers in Germany continuously published from the “learning the science of education” project since 2013, and their four publications are included in this review. They treats ERL as a concept that is a combination of three sub-concepts, they are information literacy, statistical literacy, and ethical reasoning.
- Now let’s have a look at four measure instruments. The first one is the most reported instrument. It breaks into two test sets. Participants can take one of them in 40 minutes. It is not required to finish the test. All tasks are forced-choice answer type. This instrument has been tested with rigorous procedure and with large sample sizes.
- The second one is not in education science but in psychology. However, psychology highly influences the education theories and development. Therefore, I kept it in the analysis. This instrument is a 20-item vignette-based multiple choice measure. Participants answer most open ended questions, and the test takes 30 minutes. Same as the first instrument, this one also went through rigorous validation procedure and was tested with large sample size.
- The third instrument and the fourth instrument used smaller sample size and were both self-report survey type.
- Eight studies introduced training interventions. Most used format is course that last for one semester. Only one university placed the intervention as a two-year long research practice for students. Half used web-based technologies to assist the training delivery. I evaluated the replicability of each intervention and found half of eight studies described the intervention in sufficient details that allow its replication to other contexts.
- I would like to use some sentences to summarize the findings of this review. First, when the training is designed as self-regulated learning and asynchronous learning, it benefits students’ RL retention more. Short-term intensive training is not enough for training research literacy. Both multiple-choice questions and open-ended questions can be used to test ERL. The most validated instrument to measure ERL is LeScEd booklet.
- Most ERL training interventions were in the format of a semester course, with PST students as the participants. The training content was mainly general research education. None introduces how to read scientific literature as a research skill, and none used online journal club as a long-term ERL training format.
- All these findings provide theoretical and practical evidence to support the initiative of organizing a global scale online journal club events in the domain of education science, and evaluate its influence on participants’ ERL changes before and after these events. ResearchIC is such a research and development initiative. I developed the platform ResearchIC.com using Moodle and it is now inviting global educational researchers as event hosts to contribute two 30-minute videos in these events.
- Starting from two weeks ago, the open call attracted 12 educational researchers from 11 universities in 7 countries and regions to join the 100 OJC Events as hosts.
- If you are interested to join and host one event, please visit ResearchIC.com, or scan the QR code. Existing accounts such as LinkedIn, Google, and Facebook can be used to directly login to the system.
- Thank you very much for your attention. I look forward to your participating.