Seeking Empirical Validity in an Assurance of Learning SystemRochell McWhorter
Ā
Business schools have established measurement tools to support their AoL systems and assess student achievement of learning objectives. However, business schools have not required their tools be empirically validated thus ensuring that they measure what they are intended to measure. We propose confirmatory factor analysis (CFA) be utilized by business schools to evaluate AoL measurement systems. We illustrate a CFA model used to evaluate the measurement tools at our College. Our approach is in initial steps currently evaluating individual measurement tools, but working towards developing a system that can evaluate the entire AoL measurement systems.
Authors: Sherry Avery, Rochell McWhorter, Roger Lirely and H. Harold Doty, The University of Texas at Tyler Contact author: rmcwhorter@uttyler.edu
Note: This is the last authorsā copy of this work. The final edited, definitive, and distributed copy is published in the Journal of Education for Business, 2014; 89: 156-164,
doi: 10.1080/08832323.2013.800467
Seeking Empirical Validity in an Assurance of Learning SystemRochell McWhorter
Ā
Business schools have established measurement tools to support their AoL systems and assess student achievement of learning objectives. However, business schools have not required their tools be empirically validated thus ensuring that they measure what they are intended to measure. We propose confirmatory factor analysis (CFA) be utilized by business schools to evaluate AoL measurement systems. We illustrate a CFA model used to evaluate the measurement tools at our College. Our approach is in initial steps currently evaluating individual measurement tools, but working towards developing a system that can evaluate the entire AoL measurement systems.
Authors: Sherry Avery, Rochell McWhorter, Roger Lirely and H. Harold Doty, The University of Texas at Tyler Contact author: rmcwhorter@uttyler.edu
Note: This is the last authorsā copy of this work. The final edited, definitive, and distributed copy is published in the Journal of Education for Business, 2014; 89: 156-164,
doi: 10.1080/08832323.2013.800467
A seminar drawn from two projects that explored a range of assessment practices, and examined how they are implemented by establishing and comparing attitudes to assessment amongst tutors and students within three ODL environments: University of London International Programmes, Kingās College London (ODL programmes) and the Open University.
Governing Quality Of Online Content Through Threshold Standards: Facilitating...Charles Darwin University
Ā
A presentation outlining different approaches to ensuring quality of technology enhanced learning and teaching in higher education. Please cite: Sankey. M. (2017). Governing Quality Of Online Content Through Threshold Standards: Facilitating A Consistent Learning Experience. Online e-Learning Summit 2017. Sydney, 20-21 June.
What philosophical assumptions drive the teacher/teaching standards movement ...Ferry Tanoto
Ā
What philosophical assumptions drive the teacher/teaching standards movement today? Are standards dangerous?
Week 4 - Reading highlights
Falk, B., 2002 and Tuinamuana, K., 2011
A seminar drawn from two projects that explored a range of assessment practices, and examined how they are implemented by establishing and comparing attitudes to assessment amongst tutors and students within three ODL environments: University of London International Programmes, Kingās College London (ODL programmes) and the Open University.
Governing Quality Of Online Content Through Threshold Standards: Facilitating...Charles Darwin University
Ā
A presentation outlining different approaches to ensuring quality of technology enhanced learning and teaching in higher education. Please cite: Sankey. M. (2017). Governing Quality Of Online Content Through Threshold Standards: Facilitating A Consistent Learning Experience. Online e-Learning Summit 2017. Sydney, 20-21 June.
What philosophical assumptions drive the teacher/teaching standards movement ...Ferry Tanoto
Ā
What philosophical assumptions drive the teacher/teaching standards movement today? Are standards dangerous?
Week 4 - Reading highlights
Falk, B., 2002 and Tuinamuana, K., 2011
The Roman Empire A Historical Colossus.pdfkaushalkr1407
Ā
The Roman Empire, a vast and enduring power, stands as one of history's most remarkable civilizations, leaving an indelible imprint on the world. It emerged from the Roman Republic, transitioning into an imperial powerhouse under the leadership of Augustus Caesar in 27 BCE. This transformation marked the beginning of an era defined by unprecedented territorial expansion, architectural marvels, and profound cultural influence.
The empire's roots lie in the city of Rome, founded, according to legend, by Romulus in 753 BCE. Over centuries, Rome evolved from a small settlement to a formidable republic, characterized by a complex political system with elected officials and checks on power. However, internal strife, class conflicts, and military ambitions paved the way for the end of the Republic. Julius Caesarās dictatorship and subsequent assassination in 44 BCE created a power vacuum, leading to a civil war. Octavian, later Augustus, emerged victorious, heralding the Roman Empireās birth.
Under Augustus, the empire experienced the Pax Romana, a 200-year period of relative peace and stability. Augustus reformed the military, established efficient administrative systems, and initiated grand construction projects. The empire's borders expanded, encompassing territories from Britain to Egypt and from Spain to the Euphrates. Roman legions, renowned for their discipline and engineering prowess, secured and maintained these vast territories, building roads, fortifications, and cities that facilitated control and integration.
The Roman Empireās society was hierarchical, with a rigid class system. At the top were the patricians, wealthy elites who held significant political power. Below them were the plebeians, free citizens with limited political influence, and the vast numbers of slaves who formed the backbone of the economy. The family unit was central, governed by the paterfamilias, the male head who held absolute authority.
Culturally, the Romans were eclectic, absorbing and adapting elements from the civilizations they encountered, particularly the Greeks. Roman art, literature, and philosophy reflected this synthesis, creating a rich cultural tapestry. Latin, the Roman language, became the lingua franca of the Western world, influencing numerous modern languages.
Roman architecture and engineering achievements were monumental. They perfected the arch, vault, and dome, constructing enduring structures like the Colosseum, Pantheon, and aqueducts. These engineering marvels not only showcased Roman ingenuity but also served practical purposes, from public entertainment to water supply.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Ā
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Ā
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
ā¢ The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
ā¢ The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate āany matterā at āany timeā under House Rule X.
ā¢ The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
Model Attribute Check Company Auto PropertyCeline George
Ā
In Odoo, the multi-company feature allows you to manage multiple companies within a single Odoo database instance. Each company can have its own configurations while still sharing common resources such as products, customers, and suppliers.
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
Ā
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
A Strategic Approach: GenAI in EducationPeter Windle
Ā
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Francesca Gottschalk - How can education support child empowerment.pptxEduSkills OECD
Ā
Francesca Gottschalk from the OECDās Centre for Educational Research and Innovation presents at the Ask an Expert Webinar: How can education support child empowerment?
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdfTechSoup
Ā
In this webinar you will learn how your organization can access TechSoup's wide variety of product discount and donation programs. From hardware to software, we'll give you a tour of the tools available to help your nonprofit with productivity, collaboration, financial management, donor tracking, security, and more.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Ā
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
2024.06.01 Introducing a competency framework for languag learning materials ...Sandy Millin
Ā
http://sandymillin.wordpress.com/iateflwebinar2024
Published classroom materials form the basis of syllabuses, drive teacher professional development, and have a potentially huge influence on learners, teachers and education systems. All teachers also create their own materials, whether a few sentences on a blackboard, a highly-structured fully-realised online course, or anything in between. Despite this, the knowledge and skills needed to create effective language learning materials are rarely part of teacher training, and are mostly learnt by trial and error.
Knowledge and skills frameworks, generally called competency frameworks, for ELT teachers, trainers and managers have existed for a few years now. However, until I created one for my MA dissertation, there wasnāt one drawing together what we need to know and do to be able to effectively produce language learning materials.
This webinar will introduce you to my framework, highlighting the key competencies I identified from my research. It will also show how anybody involved in language teaching (any language, not just English!), teacher training, managing schools or developing language learning materials can benefit from using the framework.
Acetabularia Information For Class 9 .docxvaibhavrinwa19
Ā
Acetabularia acetabulum is a single-celled green alga that in its vegetative state is morphologically differentiated into a basal rhizoid and an axially elongated stalk, which bears whorls of branching hairs. The single diploid nucleus resides in the rhizoid.
A Proposed Theoretical Model For Evaluating E-Learning
1. A PROPOSED THEORETICAL MODEL FOR
EVALUATING E-LEARNING
Brenda Mallinson, Norman Nyawo
Rhodes University
ABSTRACT
The deployment of e-learning offers an opportunity to build the skills required for the 21st century knowledge-based
economy. It is important to be able to evaluate various e-learning systems and analyse their efficacy. The focus of this
paper is to investigate the area of e-learning evaluation in order to discover or formulate a framework or model that
would assist the successful evaluation of e-learning in Higher Education Institutions (HEIs). The manner in which
organisations currently implement e-learning evaluation is investigated. This paper critically assesses four current models
and determines how applicable they are to HEIs. Finally, the various perspectives are synthesised and inform the creation
of a new theoretical model for the implementation of successful e-learning evaluation. The proposed model attempts to
address the identified shortcomings, and is suggested for use as a guideline for evaluating e-learning in HEIs.
KEYWORDS
Evaluation, E-Learning, Higher Education Institutions
1. INTRODUCTION
Rosenberg (2006) redefines e-learning as āthe use of Internet technologies to create and deliver a rich
learning environment that includes a broad array of instruction and information resources and solutions, the
goal of which is to enhance individual and organizational performanceā. Active learning strategies placing
the student at the heart of the education process can now be supported by a range of media deployed by an
HEI. Mostert and Hodgkinson-Williams (2006) report that the high level of hardware/software availability,
together with pervasive Internet access, are reflected in the growing prevalence of e-learning in HEIs.
An important goal of e-learning is that it should be equivalent to or better than learning provided by
conventional methods such as classroom-based instruction (Leung, 2003) and, as such, justify the return on
investment (ROI). Although there has been a significant increase in the use of e-learning in mainstream
education, very little research has been conducted to justify its use (Aivazidis et al., 2006), and the evaluation
of e-learning solutions is only partially resolved (Voigt and Swatman, 2004). HEIs considering the use of e-
learning are increasingly aware of the need for quality in both the development and implementation of their
online solutions, and evaluation of these systems will promote quality maintenance.
This study investigates how e-learning is or should be evaluated in HEIs in order to ascertain whether
their various e-learning technologies are providing them with a positive ROI. Current research on e-learning
evaluation, the purpose of evaluation, the motivation for evaluating e-learning systems, and the reasons why
some institutions may not want to evaluate their systems are investigated. Existing evaluation models are
examined, and an approach to e-learning evaluation that is designed to deal with all stages of the e-learning
cycle is shown. Finally, a new theoretical model is proposed to promote the effective evaluation of e-
learning. It is suggested that e-learning takes place in a social context and therefore any evaluation methods
and their impact on outcomes should take the surrounding constraints into consideration.
IADIS International Conference e-Learning 2008
411
3. ā¢ Evaluation is expensive and difficult: some organisations may lack the proper budget, skills and time
to evaluate their e-learning systems effectively (Horton, 2001). For smaller organisations or
institutions, evaluation may result in budget and time over-runs.
ā¢ Evaluation is political: the notion of evaluation often results in personnel feeling some discomfort
and even organisational paranoia. Instructors that use the traditional methods of teaching may feel
threatened if evaluation compares their methods to e-learning systems (Horton, 2001).
ā¢ Credibility of e-learning: the launch of questionable e-learning courseware combined with some less
successful e-learning implementations has bruised the image of e-learning and critics use this to
discredit the necessity of evaluation (Van Dam, 2004).
It was found that there is no single model that can be used when it comes to the evaluation of e-learning
in Higher Education Institutions. The following models have been adapted by various authors in an attempt
to formulate a suitable e-learning evaluation model. The two main schools of thought are one that follows the
traditional Kirkpatrick inspired views and another that follows a systematic approach to e-learning.
3. MODIFIED KIRKPATRICK MODELS
Many professionals turn to Kirkpatrickās model comprising four ordered structured levels because it has
become an industry standard for evaluation. Most evaluations take a layered approach using the basic model
of: Level 1 ā Response (Reaction); Level 2 ā Learning; Level 3 ā Performance (Behaviour); and Level 4 -
Results (Horton, 2001). The first model examined is Van Damās (2004) expanded Kirkpatrick model, with
two new levels inserted: Level 0 (Participation) was added as e-learning participation has evolved and
become an important factor in evaluation. Participation can be measured by counting the number of hits on
the website, downloads, live plays, orders, unique users, live e-learning attendance and overall usage; and the
additional Level 3 (Job application), which is related to Level 0 (Participation).
Table 1. Van Damās modified Kirkpatrick evaluation model (Van Dam, 2004)
Level Name Description
Level 0 Participation This focuses on the level of participation and interaction with the application.
Level 1 Response
(Reaction)
Was the course liked by students? Was it completed? This level gauges the learnersā satisfaction
with the training program.
Level 2 Learning Did the students gain any knowledge or skills? This level verifies improvement in skill,
acquisition of knowledge, or positive change in attitude.
Level 3 Job Application Did they use it? This level ascertains whether the acquired skills were later used.
Level 4 Performance
(Behaviour)
Did the course improve student performance? This level determines the impact of training on
behaviour, on-the-job performance and application of learned skill.
Level 5 Results Was there a good ROI for the institution? This level ascertains whether the training program
achieved or impacted desired end-results.
The second model is the result of Beal (2007) proposing that the ADDIE model (Analysis, Design,
Development, Implementation, and Evaluation) be used in conjunction with Kirkpatrickās model. The most
widely used methodology for developing new education and training programs is called Instructional
Systems Design (ISD). This approach provides a step-by-step system for the evaluation of students' needs,
the design and development of training materials, and the evaluation of the effectiveness of the training
intervention. Almost all ISD models are based on the generic ADDIE model (Beal 2007). Each step has an
outcome that feeds the subsequent step and the five phases represent a dynamic, flexible guideline for
building effective training and performance support tools. Usually evaluation design for e-learning only takes
place at the end of the development process when ideally it should take place at the beginning. The
Evaluatorās Project Report Summary (Table 2), which integrates the ADDIE model with Kirkpatrickās
model, is a projection of how useful variant evaluation can be and that it should not be left to the āStepsā
suggested by Kirkpatrick. The most important link phase is the Evaluation phase that focuses on how well
participants have mastered the learning content and the effectiveness of the training programme or
application.
IADIS International Conference e-Learning 2008
413
5. The fourth model is Voigt and Swatmanās (2004) suggested use of Frickeās model, which includes nine
evaluation forms that consider a variety of prescriptive and descriptive research questions. Frickeās model is
designed to deal with both the stages of the e-learning system life cycle and a variety of learning
environments (Voigt and Swatman, 2004). This model also emphasises the importance of context when
evaluating e-learning systems. E-learning in a social context is an open system, in that the system influences
the environment and vice versa, making it vulnerable to a number of external/internal contextual forces.
It is clear that any context-situated learning research must first define what should be evaluated and where
context comes into play. Frickeās model established a popular framework for the design and evaluation of
multimedia-based instruction. Fricke identified the five evaluation categories: Instructional conditions;
Instructional outcomes; Instructional methods; Assumptions; and General considerations. The two last
categories, in particular, help to integrate contextual information into evaluation design: āAssumptionsā help
to clarify norms and values underlying the evaluation design and 'General conditions' describe the non-
scientific nature of evaluations (Voigt and Swatman, 2004). The model suggests that evaluation be seen as an
ongoing process in the quest for transparency and better decision quality (Table 4).
Table 4. Frickeās Evaluation Criteria: Contextual Variables (Voigt and Swatman, 2004)
C1 Learner's previous knowledge, attitudes & experiences C6 Implicit learning and instructional theories
C2 Content to be learned C7 Explicit learning and instructional theories
C3 Instructional outcomes C8 Priorities of learning outcomes
C4 Instructional methods C9 Financial resources and skills available
C5 Instructional settings C10 Political guidelines
5. CRITICAL ANALYSIS OF THE CURRENT MODELS
Van Damās (2004) adapted Kirkpatrick model is a good summary of the important steps that should be
included in any evaluation process of a training programme or application. The two extra levels added by
Van Dam (2004) make the model more applicable to various contexts. However, the model is caught in the
trap of adhering to Kirkpatrickās generic model, which neglects the idea that systematic approaches are used
to design and develop these e-learning systems; thus it is important to take into consideration the systems
design model (Reeves and Hedberg, 2003). Jochems et al. (2004) highlight that Kirkpatrickās model is partial
and has to be revised conceptually to be applicable, particularly in e-learning environments.
Bealās (2007) integrated model is a good evaluation framework for e-learning systems as it takes into
account the systematic approach to evaluation. Reeves and Hedberg (2003) highlight how important the ISD
approach is for developing, and evaluating education and training programmes. As the model implies some
sort of iteration, it allows for a more thorough evaluation process that can guide evaluators to give a more
detailed, systems approach. This more schematic approach is aligned with current best practice and
educational standards. A disadvantage of this model is that it fails to ask the crucial questions regarding the
experience that the users gained from the training application, or how well the training helped them perform
on the job. These questions could be addressed by the use of Kirkpatrickās guidelines (Jochems et al., 2004).
The most important part of any evaluation model is to query the effectiveness of the evaluation process. This
is not expressed clearly in the ADDIE model, which has been criticised by some as being too systematic: too
linear, too inflexible, too constraining, and even too time-consuming to implement. As an alternative, there
are a variety of systemic design models that emphasise a more holistic, iterative approach to development.
Rather than developing the instruction in phases, the entire development team works together from the start
to rapidly build modules that can be tested with the student audience, and then revised based on their
feedback. Although this approach to development has many advantages when it comes to the creation of e-
learning, there are practical challenges in the management of resources. Frequently, training programmes
must be developed under a fixed and often limited budget and schedule. While it is easy to allocate people
and time to each step in the ADDIE model, it is harder to plan deliverables when there are no distinct steps.
The eLSE model focuses on user testing and obtaining direct user feedback to ascertain whether the
training application is effective. Evaluation patterns are created that can be used repeatedly, standardising the
whole procedure. The breaking down of the evaluation into a systematic inspection and a user-based
IADIS International Conference e-Learning 2008
415
7. Figure 1. Proposed Model for E-Learning Evaluation
In order to understand the new model (Figure 1), it is important to re-examine some misconceptions that
have been made by the current models and comment on how the proposed model directly improves on them.
Misconception 1: Level 4 of Kirkpatrickās model is superior. Kirkpatrickās Levels 1 to 4 measure
different aspects but level 4 is often described as a āhigherā level of evaluation. There is the view that level 4
is the pinnacle of the model as it is concerned with ROI and results.
Misconception 2: Level 3 is difficult to measure. Many measures are not appropriate or not sensitive
enough to detect changes in learnersā behaviours. It is difficult to ask the correct questions and obtain
accurate, truthful responses from people. Human behaviour is generally difficult to measure, thus the
measurement methods are not 100% reliable.
Misconception 3: Evaluation equals effectiveness. This is not necessarily true; the evaluation should focus
on the learning aspect (Level 2) of the subject, while effectiveness focuses on whether training has produced
intended results (Level 3 and 4). Evaluation and effectiveness are linked but they should not necessarily
arranged in continuum as they are in Kirkpatrickās model.
Misconception 4: The waterfall approach is the most suitable method. This approach has its own
disadvantages such as no fair division of phases in the life cycle. Not all the problems related to a phase are
resolved during the same phase; instead all those problems related to one phase are carried on to the next
phase and need to be resolved there. This takes up much of the time of the next phase. The proposed model
thus uses a spiral approach in its systematic evaluation so that it can avoid the problems of carrying over
issues into the next phase by solving the problems in further iterations.
Misconception 5: The external variables to the evaluation process are not necessary. Most of the
traditional evaluation models overlook the impact of external variables on the evaluation process. The
IADIS International Conference e-Learning 2008
417