SlideShare a Scribd company logo
Running head: SEARCHING FOR EMPIRICAL VALIDITY
Seeking Empirical Validity in an Assurance of Learning System
Sherry Avery, Rochell McWhorter, Roger Lirely, H. Harold Doty
College of Business and Technology
The University of Texas at Tyler
Contact Email: rmcwhorter@uttyler.edu
Note: This is the last authors’ copy of this work. The final edited, definitive, and distributed copy is
published in the Journal of Education for Business, 2014; 89: 156-164,
doi: 10.1080/08832323.2013.800467
Abstract
Business schools have established measurement tools to support their AoL systems and assess
student achievement of learning objectives. However, business schools have not required their
tools be empirically validated thus ensuring that they measure what they are intended to measure.
We propose confirmatory factor analysis (CFA) be utilized by business schools to evaluate AoL
measurement systems. We illustrate a CFA model used to evaluate the measurement tools at our
College. Our approach is in initial steps currently evaluating individual measurement tools, but
working towards developing a system that can evaluate the entire AoL measurement systems.
Keywords: AACSB, Assurances of Learning, Assessment, Confirmatory Factor Analysis
A decade ago, the Association to Advance Collegiate Schools of Business (AACSB)
International ratified new accreditation requirements including the addition of assurance of
learning (AoL) standards for continuous improvement (Martell, 2007). As part of this addition,
schools seeking to earn or maintain AACSB accreditation must develop a set of defined learning
goals and subsequently collect relevant assessment data to determine direct educational
achievement (LeClair, 2012; Sampson & Betters-Reed, 2008). The establishment of the mission-
driven assessment process requires “well-documented systematic processes to develop, monitor,
SEARCHING FOR EMPIRICAL VALIDITY 2
evaluate, and revise the substance and deliver of the curricula on learning” (Romero, 2008, p.
253).
With establishment of the 2003 AACSB standards, all schools “must develop assessment
tools that measure the effectiveness of their curriculum” (Pesta & Scherer, 2011, p. 164). As a
response to this outcomes assessment mandate, a number of schools created models to depict and
track their assessment functions (Betters-Reed, Nitkin & Sampson, 2008; Gardiner, Corbitt, &
Adams 2010; Zocco, 2011). However, the question arises as to the validity of system models for
measuring learning outcomes—does the model measure what it purports to measure—do the
learning experiences accomplish the learning goals outlined in the systems model? This question
is a very important one because once validity of a measurement system is established, then it
provides confidence in a program and quality of assurance in achieving the school’s mission
(Baker, Ni & Van Wart, 2012).
The purpose of this article is to illustrate development of an empirically-based AoL
System which may be used by other business schools seeking accreditation. Relevant literature
on this topic will be examined next.
Review of Literature
A search of the literature for an empirically-validated AoL system yielded results for
research covering either the validation of AoL tools or validation of an Aol model. Each is
discussed in the following section.
Measures of Validity in AoL Assessment Tools
Measures of validity associated with AoL learning outcomes were located in business
literature by reviewing articles that described locally-developed assessment tools and externally
validated instruments. For instance, researchers developed an assessment tool to explore
SEARCHING FOR EMPIRICAL VALIDITY 3
students’ self-efficacy toward service and civic participation. They utilized traditional scale
development and confirmatory factor analysis (CFA) and simultaneous factor analysis in several
populations (SIFASP) for insuring the validity and reliability of their instrument to measure AoL
criteria for ethics and social responsibility (Weber, Weber, Sleeper & Schneider, 2004). Another
tool offered was a content valid assessment exam created to measure business management
knowledge (Pesta & Scherer, 2011).
Also, a matrix presented by Harper and Harder (2009) depicts demonstrated abilities
intersected with competency clusters; the clusters were developed from literature describing
valid research into “the kinds of knowledge and skills that are known to be necessary for success
as a practitioner in the MIS field” (p. 492). However, no statistical measures of validity were
provided. Additionally, instances of use of externally validated instruments such as the revised
version of the Defining Issues Test (DIT-2) to assess ethical reasoning instruction in
undergraduate cost accounting (Wilhelm & Czyzewski, 2012) and use of the CAPSIM computer
simulation to assess business strategy integration (Garrett, Marques & Dhiman, 2012), were
found.
Measures of Validity in AoL Assessment Models
Various models have been offered for outcomes measurement as part of a processed-
based approach for meeting AoL standards (i.e. Beard, Schwieger & Surendran, 2008; Betters-
Reed, Nitkin & Sampson, 2008; Hess & Siciliano, 2007) but without statistical evidence or
discussion of validity measures. However, the search of literature found an article by Zocco
(2011) that presented a recursive model to address and document continuous and systematic
improvement and discussed validity issues surrounding the application of recursion to a process
such as AoL. Although helpful for looking at school improvement, the model does not measure
SEARCHING FOR EMPIRICAL VALIDITY 4
validity of the model itself. Therefore, the review of literature offered several tools and a model
with validity calculations, however, no example of an empirically-validated system was found.
Case Study: Assurances of Learning at [blinded]
During the past five years, the College of Business [removed for blinding purposes,
hereafter referred to as “College”] has conducted a complete redesign of its AACSB Assurance
of Learning system. To understand the rationale for this design change it is important to explore
several key drivers of this decision, especially in light of the fact that our prior AoL system was
cited as one or our “best practices” during our last Maintenance of Accreditation visit. At last
visit, the College operated three different and largely unrelated assessment systems: one for the
AACSB, [names of 2 other accreditation bodies removed for blinding purposes]. In some ways,
these independent assessment systems simplified accreditation reporting: each system was
tailored to the specifics of a single accrediting body and the data associated with one system
were not considered in collaboration with data collected for a different accrediting body. For
example, AACSB and the College’s assessment procedures were treated as completely
independent. This approach simplified reporting, but hindered integrating different assessment
data in the larger curriculum management process.
A second major contextual factor relevant to our AoL process was feedback from our last
AACSB visit that recommended revisions to the vision and mission statements for the College
and AACSB AoL processes. As part of that revision process, the College clarified its mission
and identified five core values.
Based largely on these contextual factors, faculty determined we were at an ideal point to
design a new single integrated assessment model to meet the needs of each of our accrediting
bodies. Further, we determined that the new system should be linked to the new mission by
SEARCHING FOR EMPIRICAL VALIDITY 5
incorporating the core values as learning outcomes, and that we should attempt to assess the
validity of the system in terms of both the theoretical model used to design the system and the
measurement model used to organize the data collection. These additional steps would allow
more confidence in the evidence-based changes we were making in program structure and course
curriculum. The full scale implementation began in the 2010-11 school year; our model is more
fully described next.
Faculty-Driven Process
AoL in the College is a faculty-driven process. Oversight of this process is charged to
the AoL Committee, a committee comprised of a faculty chair, the undergraduate program
director, the graduate programs coordinator and four at-large faculty members. The composition
of the Committee provides cross-sectional representation of all disciplines and programs in the
College.
The Committee works closely with our faculty to ensure that each learning objective is measured
periodically, at least twice during each five-year period but generally more often. The faculty
employ a variety of measurement strategies, including major field tests, embedded test questions,
case analyses, observation of student presentations, activity logs, simulations and other class
assignments and/or projects. Analyses of results guide the Committee in its work with the
faculty to develop and implement appropriate actions to ensure curricula and pedagogy are
managed in a manner enhancing student learning and development. Figure 1 illustrates how the
SEARCHING FOR EMPIRICAL VALIDITY 6
AoL assessment process operates in a continuous improvement mode.
FIGURE 1
AoL Curriculum Management Process at [Blinded]
Conceptual Framework
The AoL system in the College is based on a set of shared core values: professional
proficiency, technological competence, global awareness, social responsibility and ethical
courage as seen in Figure 2. These mission-based core values form the framework for our
comprehensive, empirically-validated AoL models for both the Bachelor in Business
Administration program and the Master of Business Administration Program, as well as other
College programs that are outside the scope of AACSB accreditation. AoL in the College has
evolved to the point where our current system is second-generation, that is, it is the culmination
SEARCHING FOR EMPIRICAL VALIDITY 7
of an assessment of the AoL system itself. Many of the best features of the prior system were
retained, including assessment of discipline-based knowledge, communication skills, and the use
of quantitative tools and business technology. The result of this process is a value-based
conceptual framework whose efficacy can be tested empirically using confirmatory factor
analysis. To our knowledge, our College is the first AACSB-accredited program to design an
empirically-validated AoL system. Figure 2 depicts the conceptual framework of our AoL
system for the BBA program.
Figure 2
Method
Data Collection. The faculty developed ten learning objectives to support the five
learning goals of the College. A measurement tool was designed for each objective, such as the
Major Field Test or rubrics. Assessment was conducted within required core business courses
that included students across all College of Business majors. Students were generally either
junior or seniors in one of the business majors. Results were then collected and compiled
centrally in an administrative function within the College.
College of Business & Technology
Creating Leaders for a Better Tomorrow
Learning Goal:
Graduates have the
knowledge and
communication skills to
succeed in the business
profession.
Ethical
Courage
Social
Responsibility
Global
Awareness
Technological
Competence
Professional
Proficiency
Learning Goal:
Graduates use the
information sources and
tools associated with
their chosen
profession.
Learning Goal:
Graduates incorporate
global considerations in
business activities.
Learning Goal:
Graduates evaluate the
social consequences of
alternative outcomes
when making business
decisions.
Learning Goal:
Graduates understand
ethical considerations
and their impacts.
Cognitive:
GA1 - Students
demonstrate
awareness of global
issues and
perspectives.
Behavioral:
GA2 - Students are
knowledgeable of
global issues and
perspectives that
may impact
business activities.
Behavioral:
SR1 - Students
exhibit an
understanding of
the social
consequences of
business activities.
Cognitive:
EC1 - Students
understand legal
and ethical
concepts.
Behavioral:
EC2 - Students
make ethical
decisions.
Behavioral:
TC2 - Students are
able to use
business software,
data sources, and
tools.
Cognitive:
TC1 - Students
understand
information
systems and their
role in business
enterprises.
Behavioral:
PP3 - Students are
able to deliver a
presentation that is
focused, well-
organized, and
includes the
appropriate verbal
and nonverbal
behaviors.
Cognitive:
PP1 - Students
demonstrate that
they are
knowledgeable
about current
business theory,
concepts,
methodology,
terminology and
practices.
PP2 - Students can
prepare a business
document that is
focused, well-
organized, and
mechanically
correct.
Bachelor of Business Administration Updated: 7/11/2012
SEARCHING FOR EMPIRICAL VALIDITY 8
Analysis Approach. Several of the measurement tools included a number of items that
collectively assessed the specific learning objective. Confirmatory factor analysis (CFA) was
conducted to assess empirical validity of the items measures and learning objectives. CFA was
chosen for assessment because it tests how well the measured variables represent the constructs
(Hair, Black, Babin, Anderson & Tatham, 2005).
The items that comprise each construct are identified prior to running the CFA analysis.
We then confirm or reject that the items properly reflect the construct, in this case the learning
objective. CFA was conducted on six of the learning objectives. We were unable to run CFA
for some learning objectives because they were either measured by a single item, the sample size
was too small, or data was binary, therefore negating the applicability of using CFA. Table 1
details the learning objectives, measurement tools, and when the CFA was conducted. In the
following section, we discuss the general approach used in the CFA analysis. Then, we follow up
with two examples of the CFA analysis; the first example is empirically valid, and the second
example was not empirical valid. We used a combination of the software tools SPSS and AMOS
SEARCHING FOR EMPIRICAL VALIDITY 9
to support the analysis.
For each of the CFA analyses, we followed a 3-step approach documented in many of the
leading academic journals: (1) review of the raw data, (2) assessment of model fit, and (3)
assessment of construct validity. Prior to the CFA analyses, we reviewed the data for sample
SEARCHING FOR EMPIRICAL VALIDITY 10
size, outliers, missing data, and normality. We determined if the sample size was adequate for
the model based on suggested requirements that range from 5 to 20 observations per variable.
(Hair et al., 2005). The existence of outliers along with their potential impact on normality and
the final results were assessed at both the univariate and multivariate level by reviewing the
Mahalanobis distance (D2) calculation for each case. Next, we identified missing data and
assessed the potential impact on analysis. A case that has a substantially different value from
other D2 calculations is a potential outlier. Then, we identified the amount of missing data and
assessed the potential impact on the analysis. Finally, we assessed normality by reviewing both
skewness and kurtosis at the univariate and multivariate level. Values of 0 represent a normal
distribution. For skewness, less than 3 is acceptable (Chou & Bentler, 1995; Kline, 2005). For
kurtosis, Kline (2005) stated that less than 8 is reasonable with greater than 10 indicating a
problem and over 20 an extreme problem.
We evaluated how well the data fit the measurement model using Analysis of Moment
Structures (AMOS) software (see http://www.utexas.edu/its/help/spss/526). We used maximum-
likelihood estimations which is a widely used approach and is fairly robust to violations of
normality and produces reliable results under many circumstances (Hair et al., 2005; Marsh,
Balla & McDonald, 1998). First, we evaluated the Chi-Square (χ2) statistic which measures the
difference between the observed and estimated covariance matrices. It is the only statistic that
has a direct statistical significance and is used as the basis for many other fit indices (Hair et al.,
2005). Statistical significance in this case indicates that an error of approximation or estimation
exists. Many researchers question the validity of the chi-square statistics (Bentler, 1990), so if it
is significant additional indices should be used to assess overall evaluate model fit. The root
mean square error of approximation (RMSEA) is a standardized measure of the lack of fit of the
SEARCHING FOR EMPIRICAL VALIDITY 11
data to the model (Steiger, 1990). It is fairly robust in terms of small sample size; i.e. 250 or
less. Thresholds of .05 to .08 have been suggested with Hu and Bentler (1999) recommending a
cutoff close to .06. The Bentler-Bonnet (1980) Non-Normed Fit Index (NNFI) was used because
it also works well with small sample sizes (Bedeian, 2007). Generally .90 or better is considered
adequate fit with Hu and Bentler (1999) suggesting a threshold of .95 or better for good fit. The
non-normed fit index (NNFI) and comparative fit index (CFI) are incremental fit indices in that
they assess model fit by referring a baseline model (Bentler 1990; Bentler & Bonett, 1980; Hu &
Bentler, 1999). The NNFI and CFI generate values between 0 and 1 with .90 or greater
representing adequate fit (Hu & Bentler, 1999; Bedeian 2007).
The final step was to assess construct validity which is the extent to which a set of
measured items accurately reflect the theoretical latent construct the items were designed to
measure (Hair et al., 2005). The standardized factor loadings should be statistically significant
and .5 or higher (Hair et al., 2005). Convergence validity was assessed by calculating the
average variance extracted (AVE) and construct reliability (CR). The average percent of
variance extracted among a set of construct items is a summary indicator of convergence. AVE
of .5 or higher suggests adequate convergence. Less than 5 indicates that on average more error
remains in the items than variance explained by the latent construct (Fornell & Larcker, 1981).
High CR reliability indicates that the measures consistently represent the same latent construct.
Values of .6 or 7 are acceptable, with .7 or higher indicating good reliability (Nunnally, 1978).
Results. The purpose of this article is to illustrate the method we used to assess empirical
validity of our learning objectives to aid other business schools in their AoL journey. It is not
our goal to suggest that our measurement tools or learning objectives should be universally
adopted. Therefore for illustration of our process, we will limit our discussion to the overall fit
SEARCHING FOR EMPIRICAL VALIDITY 12
of our constructs and then discuss in detail two examples of how we used CFA; an example of a
valid measure (business knowledge) and an example of a measure that requires some
modifications (oral communication).
Table 2 documents the model fit indices for the CFA analyses performed for six of the
learning objectives. The learning objectives for business knowledge, written communication,
global awareness context, region, and perspectives are valid for construct reliability and model
fit. Therefore we are reasonably confident that these objectives adequately measure the learning
goals established by the College. The model fit for Oral Communication was below suggested
thresholds.
Table 2
Fit Indices
Composite
reliability
Variance
Extracted
χ2
RMSEA RMSR NNFI CFI
Learning Objective
Business Knowledge (n=218)
.853 .409 61.35** .077 11.66 .932 .949
Written Communication (n=147) .798 .498 6.163** .119 .011 .926 .975
Oral Communication (n=161) .62 .246 120.24** .148 .033 .693 .770
Global Awareness Context (n=151) .85 .397 19.65** .089 .336 .958 .975
Global Awareness Region (n=151) .89 .562 8.183 .000 .192 1.003 1.000
Global Awareness Perspectives (n=90) .96 .73 96.643** .208 .031 .867 .905
**p<.05
Business knowledge. Students take the ETS Major Field Test for the Bachelor’s Degree
in Business (MFT) in a capstone class in their senior year. The MFT is a widely used
standardized exam for business students (www.ets.org/mft). The capstone class is required for all
business majors and the MFT is administered in all sections of the capstone class thus ensuring
that all business majors participate in the assessment prior to graduation. To ensure that students
take the exam seriously and give their best effort, their results are reflected in their course grade.
Two hundred and eighteen responses were obtained from the exams administered in 2010 –
2011. Nine composite scores from the exam are used to assess the overall business knowledge
SEARCHING FOR EMPIRICAL VALIDITY 13
of the student. (See Table 3 for a listing of these nine items.) A review of the data found no
missing data or significant outliers. The sample size divided by the number of responses to the
number of variables (218/9 = 24) was well above the recommended threshold range of 5 to 20.
The kurtosis and skewness statistics were less than the recommended thresholds of 8 and 3,
which indicates only a slight departure from normality. Therefore we were reasonably confident
in proceeding to the next phase of the analysis, evaluating model fit by running a confirmatory
factor analysis on the item measures.
The χ2
was significant, however the RMSEA was .077, below the recommended threshold
of .08. The RMSEA is parsimonious in that it considers the impact of the number of variables in
their calculations, so it is a better indicator of model fit than χ2. The NNFI and CFI was .932
and .949, well above the recommended threshold of .90. Overall, the model fit is acceptable.
SEARCHING FOR EMPIRICAL VALIDITY 14
In assessing construct validity of the items, we noted that all items were statistically
significant, however one item measure, quantitative analysis, fell below the recommended
loading of .50. The composite reliability of .853 was well above the recommended threshold of
.6 and the variance extracted was slightly below the recommended threshold at .50. Overall
there is evidence that the item measures adequately reflect the latent construct of business
knowledge. However, further analysis is needed to determine the cause of the low factor loading
of quantitative analysis. We provided Figure 3 as a visual representation of the CFA model for
this construct.
Oral communication. The students’ oral communication skills were measured by a rubric
assessed oral presentation assignment administered in the business communication course, which
is part of the required core curriculum. The business communication professor for all sections of
the class completed the assessment. There is only one business communication professor, thus
ensuring consistency of the measurement process. The rubric is comprised of nine item measures
(see Table 3 for a list of these items). Data was obtained from the 2010-2012 assessments which
resulted in 161 observations for the CFA of the oral communication construct. A review of the
data found two missing observations for the item measure conclusion and one missing
SEARCHING FOR EMPIRICAL VALIDITY 15
observation for eye contact. Because the impact of missing data was small, we used the mean
imputation method for the missing observations. We also identified one potential outlier; we
deleted the case on a trail run and found that it did not have a significant impact on normality or
the results. The sample size ratio (161/9 = 17.9) is in the recommended threshold range of 5 to
20. The multivariate kurtosis statistic of 23.393 was well above the recommended thresholds of
8, which provides evidence of a departure from normality. The univariate skewness statistics
were below the threshold of 3.
Because of the non-normal distribution, we attempted to run an asymptotically
distribution-free estimation method to contact the CFA. Unfortunately this resulted in an
inadmissible solution because of the existence of a negative error variance. Therefore we used
the Maximum-Likelihood estimation technique which often provides reasonable results with
departures from normality. The χ2
was significant, however the adjusted chi-square ratio (χ2
/df)
was 4.453, below the recommended threshold of 5. The RMSR was .033 well below the
recommended threshold of .10 while the RMSEA was .145, above the recommended threshold of
.08. The NNFI and CFI was .693 and .770, below the recommended threshold of .90. Our
overall assessment is the model fit is poor. The RMSEA, NNFI, and CFI are impacted by the
model complexity which could be an indication that the number of variables in the model
affected model fit.
Two of the items were not statistically significant: conclusion and fillers. Only two of
the nine items were greater than the recommended threshold of .50; projection and pace. The
composite reliability of .62 met the recommended threshold of .6. The variance extracted of .246
was well below the recommended threshold of .50. Our conclusion is that the item measures do
not accurately reflect the latent construct oral communications.
SEARCHING FOR EMPIRICAL VALIDITY 16
Discussion
The purpose of this article is to highlight an AACSB-accredited program as a case study
of its design of an empirically-validated AoL system and to demonstrate how empirical validity
improved our AoL system. When appropriate, we used confirmatory factor analysis to validate
the measurement instruments used to assess student achievement of the learning objectives
established by the faculty and stakeholders of the College. We provided a description of the CFA
process used to assess the empirical validity of the learning objectives. Finally we illustrated the
process by discussing the results of the validation process on two learning objectives.
For the business knowledge learning objective, we found that both the model fit and
construct reliability are valid. However, we noted that factor loading for quantitative analysis
was much lower than the other item measures. In reviewing the raw scores, not surprisingly, we
found that our students do not perform as well in the quantitative analysis topic when compared
to the other topics covered by the MFT indicating that even though we had a valid measure of
business knowledge, our students need improvement in their quantitative skills. These result
prompted us to examine the curriculum of the class where much of the quantitative methods are
taught.
SEARCHING FOR EMPIRICAL VALIDITY 17
For the oral communication learning objective, we found that the model fit was poor,
construct reliability was low, and many of the item measures from the rubric did not load. These
results prompted us to examine the measurement tool used to assess achievement oral
communication competency. Corrective action includes a review of the rubric and also the
process used to collect the data.
Conclusion and Limitations. We found value in, and therefore will continue to
empirically validate the AoL learning objectives using confirmatory factor analysis. The
validation process has increased the support of the AoL system by faculty that understand and
are trained in the research process. We have received positive feedback from the AACSB and
higher education associations on our validation process. Most importantly, it provides
confidence in the tools we are using to measure student achievement of the learning objectives.
Now that the process and the supporting models are in place, it will be relatively simple to
continue the validation process in order to continually improve. Because of the method we used
to capture the assessments, we are able to use the same validation process for both AACSB and
SACS accreditations.
We are continually striving to improve our validation process. One limitation is that we
are unable to simultaneously assess the empirical validity of the entire model of the learning
objectives. Assessment is conducted by class, rather than individual student across all classes.
Therefore we do not have data for one student for all learning objectives. To address this
limitation, we are evaluating both in-house and commercially developed databases to track
student data across classes and semesters. For example, the University of Central Florida (UCF)
developed an in-house database for tracking of student data (Moskal, Ellis, & Keon, 2008).
Taskstream is a commercially available database for tracking data (https://www.taskstream.com).
SEARCHING FOR EMPIRICAL VALIDITY 18
References
Baker, D. L., Ni, A. Y., & Van Wart, M. (2012). AACSB assurance of learning: Lessons learned
in ethics module development. Business Education Innovation Journal, 4(1), 19-27.
Beard, D., Schwieger , D., & Surendran, K. (2008). Integrating soft skills assessment through
university, college, and programmatic efforts at an AACSB accredited institution.
Journal of Information Systems, 19(2), 229-240.
Bedeian, A. G. (2007). Even if the tower is 'Ivory,' It Isn't 'White:' Understanding the
consequences of faculty cynicism. Academy Of Management Learning & Education, 1, 9.
doi:10.2307/40214514
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107,
238-246.
Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of
covariance structures. Psychological Bulletin, 88, 588-606.
Betters-Reed, B. L., Nitkin, M. R., Sampson, S. D. (2008). An assurance of learning success
model: Toward closing the feedback loop. Organization Management Journal, 5, 224-
240.
Chou, C. P., & Bentler, P. M. (1995). Estimates and tests in structural equation modeling. In R.
Hoyle (Ed.), Structural Equation Modeling (pp. 37-59). Thousand Oaks, CA: Sage.
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable
variables and measurement eror. Journal of Marketing Research, 18, 39-50.
Gardiner, L. R., Corbitt, G., & Adams, S. J. (2010). Program assessment: Getting to a practical
how-to model. Journal of Education for Business, 85(3), 139-144.
doi:10.1080/08832320903258576
SEARCHING FOR EMPIRICAL VALIDITY 19
Garrett, N., Marques, J., & Dhiman, S. (2012). Assessment of business programs: A review of
two models. Business Education & Accreditation, 4(2), 17-25.
Hair, J. F., Black, B., Babin, B., Anderson, R. E., & Tatham, R. L. (2005). Multivariate Data
Analysis, 6th
edition. Saddle River, NJ: Prentice-Hall.
Harper, J. S., & Harder, J. T. (2009). Assurance of Learning in the MIS Program. Decision
Sciences Journal of Innovative Education, 7(2), 489-504. doi:10.1111/j.1540-
4609.2009.00229.x
Hess, P. W., & Siciliano, J. (2007). A research-based approach to continuous improvement in
business education. Organization Management Journal, 4(2), 135-147.
Hu, L., & Bentler, P. (1999). Cutoff criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives. Structural Equation Modeling-A
Multidisciplinary Journal, 6(1), 1-55.
Kline, R. B. (2005). Principles and Practice of Structural Equation Modeling (2nd
ed.). New
York: Guilford.
LeClair, D. (2012). Broadening our view of Assurance of Learning. Retrieved from
http://aacsbblogs.typepad.com/dataandresearch/2012/02/broadening-our-view-of-
assurance-of-learning.html
Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indices in confirmatory
factor analysis: Effects of sample size. Psychological Bulletin, 103, 391-411.
Martell, K., (2007). Assurance of learning (AoL) methods just have to be good enough. Journal
of Education for Business,(82) 4, 241-243.
SEARCHING FOR EMPIRICAL VALIDITY 20
Moskal, P., Ellis, T., & Keon, T. (2008). Assessment in Higher Education and the management
of student-learning data. Academy of Management Learning & Education, (2), 269.
doi:10.2307/40214542
Nunnally, J. C. (1978) Psychometric theory (2nd
ed.). New York: McGraw-Hill.
Pesta, B., & Scherer, R. (2011). The Assurance of Learning Tool as Predictor and Criterion in
Business School Admissions Decisions: New Use for an Old Standard? Journal Of
Education For Business, 86(3), 163. doi:10.1080/08832323.2010.492051
Romero, E. J. (2008). AACSB accreditation: Addressing faculty concerns. Academy of
Management Learning & Education, 7(2), 245-255.
Sampson, S. D., & Betters-Reed, B. L. (2008). Assurance of Learning and outcomes assessment:
A case study of assessment of a marketing curriculum. Marketing Education Review,
18(3), 25-36.
Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation
approach. Multivariate Behavioral Research 25(2), 173-180.
Weber, P., Weber, J. E., Sleeper, B. J., & Schneider, K. C. (2004). Self-Efficacy toward service,
civic participation and the business student: Scale development and validation. Journal of
Business Ethics, 49(4), 359-369.
Wilhelm, W. J., & Czyzewski, A. B. (2012). Ethical reasoning instruction in undergraduate cost
accounting: A non-intrusive approach. Academy of Educational Leadership Journal,
16(2), 131-142.
Zocco, D. (2011). A recursive process model for AACSB assurance of learning. Academy of
Educational Leadership Journal. 15(4), 67-91.

More Related Content

What's hot

EDU 675 Entire Course NEW
EDU 675 Entire Course NEWEDU 675 Entire Course NEW
EDU 675 Entire Course NEW
shyamuopnine
 
9.john egbeazien oshodi -69-72
9.john egbeazien oshodi -69-729.john egbeazien oshodi -69-72
9.john egbeazien oshodi -69-72
Alexander Decker
 
Reviewing the Research and PEAC Recommendations around Principal Evaluation
Reviewing the Research and PEAC Recommendations around Principal EvaluationReviewing the Research and PEAC Recommendations around Principal Evaluation
Reviewing the Research and PEAC Recommendations around Principal Evaluation
Richard Voltz
 
Predictors of Success: Student Achievement in Schools
Predictors of Success: Student Achievement in SchoolsPredictors of Success: Student Achievement in Schools
Predictors of Success: Student Achievement in Schools
School Improvement Network
 
Ajman University
Ajman University Ajman University
Ajman University
Lee Schlenker
 
Heliyon d-21-02382 r2 (1)
Heliyon d-21-02382 r2 (1)Heliyon d-21-02382 r2 (1)
Heliyon d-21-02382 r2 (1)
Pedro Prestes
 
A1070109
A1070109A1070109
A1070109
IJERD Editor
 
If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...
If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...
If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...
Ulf-Daniel Ehlers
 
Dr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle Iwundu
Dr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle IwunduDr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle Iwundu
Dr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle Iwundu
guestfa49ec
 
2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...
2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...
2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...Thomas McCabe, PMP, CSSMBB, CSM, PMI-ACP
 
Dubrowski assessment and evaluation
Dubrowski assessment and evaluationDubrowski assessment and evaluation
Dubrowski assessment and evaluation
Adam Dubrowski
 
Week1 Assessment Overview
Week1 Assessment OverviewWeek1 Assessment Overview
Week1 Assessment OverviewIPT652
 
Assessment in higher education final draft 05-11-16 (1) final
Assessment in higher education   final draft 05-11-16 (1) finalAssessment in higher education   final draft 05-11-16 (1) final
Assessment in higher education final draft 05-11-16 (1) final
midhat12
 
Framework for teaching evaluation instrument. 2013 edition
Framework for teaching evaluation instrument. 2013 editionFramework for teaching evaluation instrument. 2013 edition
Framework for teaching evaluation instrument. 2013 edition
Rafael Mireles
 

What's hot (16)

EDU 675 Entire Course NEW
EDU 675 Entire Course NEWEDU 675 Entire Course NEW
EDU 675 Entire Course NEW
 
Cdst12 ijtel
Cdst12 ijtelCdst12 ijtel
Cdst12 ijtel
 
9.john egbeazien oshodi -69-72
9.john egbeazien oshodi -69-729.john egbeazien oshodi -69-72
9.john egbeazien oshodi -69-72
 
Reviewing the Research and PEAC Recommendations around Principal Evaluation
Reviewing the Research and PEAC Recommendations around Principal EvaluationReviewing the Research and PEAC Recommendations around Principal Evaluation
Reviewing the Research and PEAC Recommendations around Principal Evaluation
 
Predictors of Success: Student Achievement in Schools
Predictors of Success: Student Achievement in SchoolsPredictors of Success: Student Achievement in Schools
Predictors of Success: Student Achievement in Schools
 
Ajman University
Ajman University Ajman University
Ajman University
 
Heliyon d-21-02382 r2 (1)
Heliyon d-21-02382 r2 (1)Heliyon d-21-02382 r2 (1)
Heliyon d-21-02382 r2 (1)
 
A1070109
A1070109A1070109
A1070109
 
If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...
If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...
If we open it will they come? Towards a new OER Logic Model (by Ulf-Daniel Eh...
 
Dr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle Iwundu
Dr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle IwunduDr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle Iwundu
Dr. W.A. Kritsonis, Dissertation Committee for La'Shonte Nechelle Iwundu
 
2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...
2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...
2011 Spring - IDEA Diagnostic - Fairfield U - Student Feedack - Prof McCabe, ...
 
Dubrowski assessment and evaluation
Dubrowski assessment and evaluationDubrowski assessment and evaluation
Dubrowski assessment and evaluation
 
Standards libraries higher education
Standards libraries higher educationStandards libraries higher education
Standards libraries higher education
 
Week1 Assessment Overview
Week1 Assessment OverviewWeek1 Assessment Overview
Week1 Assessment Overview
 
Assessment in higher education final draft 05-11-16 (1) final
Assessment in higher education   final draft 05-11-16 (1) finalAssessment in higher education   final draft 05-11-16 (1) final
Assessment in higher education final draft 05-11-16 (1) final
 
Framework for teaching evaluation instrument. 2013 edition
Framework for teaching evaluation instrument. 2013 editionFramework for teaching evaluation instrument. 2013 edition
Framework for teaching evaluation instrument. 2013 edition
 

Viewers also liked

ESL My personal info presentation, name,job, nationality.
ESL My personal info presentation, name,job, nationality.ESL My personal info presentation, name,job, nationality.
ESL My personal info presentation, name,job, nationality.
TeacherEls
 
Leveraging Green Computing for Increased Viability and Sustainability
Leveraging Green Computing for Increased Viability and SustainabilityLeveraging Green Computing for Increased Viability and Sustainability
Leveraging Green Computing for Increased Viability and Sustainability
Rochell McWhorter
 
Cindy k greco presentation1
Cindy k greco   presentation1Cindy k greco   presentation1
Cindy k greco presentation1
CindyKGreco
 
Virtual HRD: Utilizing Pinterest as Informal Learning Tool
Virtual HRD: Utilizing Pinterest as Informal Learning ToolVirtual HRD: Utilizing Pinterest as Informal Learning Tool
Virtual HRD: Utilizing Pinterest as Informal Learning ToolRochell McWhorter
 
Instructing Through Second Life
Instructing Through Second LifeInstructing Through Second Life
Instructing Through Second Life
Rochell McWhorter
 
An Initial Conceptualization of Virtual Scenario Planning
An Initial Conceptualization of Virtual Scenario PlanningAn Initial Conceptualization of Virtual Scenario Planning
An Initial Conceptualization of Virtual Scenario Planning
Rochell McWhorter
 
Creating Virtual Communities of Practice with the Visual Social Media Platfor...
Creating Virtual Communities of Practice with the Visual Social Media Platfor...Creating Virtual Communities of Practice with the Visual Social Media Platfor...
Creating Virtual Communities of Practice with the Visual Social Media Platfor...
Rochell McWhorter
 
Technology Development: Learning from Project Failures
Technology Development: Learning from Project FailuresTechnology Development: Learning from Project Failures
Technology Development: Learning from Project Failures
Rochell McWhorter
 

Viewers also liked (8)

ESL My personal info presentation, name,job, nationality.
ESL My personal info presentation, name,job, nationality.ESL My personal info presentation, name,job, nationality.
ESL My personal info presentation, name,job, nationality.
 
Leveraging Green Computing for Increased Viability and Sustainability
Leveraging Green Computing for Increased Viability and SustainabilityLeveraging Green Computing for Increased Viability and Sustainability
Leveraging Green Computing for Increased Viability and Sustainability
 
Cindy k greco presentation1
Cindy k greco   presentation1Cindy k greco   presentation1
Cindy k greco presentation1
 
Virtual HRD: Utilizing Pinterest as Informal Learning Tool
Virtual HRD: Utilizing Pinterest as Informal Learning ToolVirtual HRD: Utilizing Pinterest as Informal Learning Tool
Virtual HRD: Utilizing Pinterest as Informal Learning Tool
 
Instructing Through Second Life
Instructing Through Second LifeInstructing Through Second Life
Instructing Through Second Life
 
An Initial Conceptualization of Virtual Scenario Planning
An Initial Conceptualization of Virtual Scenario PlanningAn Initial Conceptualization of Virtual Scenario Planning
An Initial Conceptualization of Virtual Scenario Planning
 
Creating Virtual Communities of Practice with the Visual Social Media Platfor...
Creating Virtual Communities of Practice with the Visual Social Media Platfor...Creating Virtual Communities of Practice with the Visual Social Media Platfor...
Creating Virtual Communities of Practice with the Visual Social Media Platfor...
 
Technology Development: Learning from Project Failures
Technology Development: Learning from Project FailuresTechnology Development: Learning from Project Failures
Technology Development: Learning from Project Failures
 

Similar to Seeking Empirical Validity in an Assurance of Learning System

A Proposed Theoretical Model For Evaluating E-Learning
A Proposed Theoretical Model For Evaluating E-LearningA Proposed Theoretical Model For Evaluating E-Learning
A Proposed Theoretical Model For Evaluating E-Learning
Christina Bauer
 
Journal raganit santos
Journal raganit santosJournal raganit santos
Journal raganit santos
essayumiyuri
 
PROF ED 7 PPT.pptx
PROF ED 7 PPT.pptxPROF ED 7 PPT.pptx
PROF ED 7 PPT.pptx
DessAlla
 
Assessment Of Business Programs A Review Of Two Models
Assessment Of Business Programs  A Review Of Two ModelsAssessment Of Business Programs  A Review Of Two Models
Assessment Of Business Programs A Review Of Two Models
Ashley Carter
 
This week, you examine cellular processes that are subject to alte
This week, you examine cellular processes that are subject to alteThis week, you examine cellular processes that are subject to alte
This week, you examine cellular processes that are subject to alte
TakishaPeck109
 
CURRICULUM EVALUATIONTOPICSTODUSCUSS.pptx
CURRICULUM EVALUATIONTOPICSTODUSCUSS.pptxCURRICULUM EVALUATIONTOPICSTODUSCUSS.pptx
CURRICULUM EVALUATIONTOPICSTODUSCUSS.pptx
TrisciaJoyNocillado
 
School based assesment
School based assesmentSchool based assesment
School based assesmentHajar Hamdan
 
Assessing Service-Learning
Assessing Service-LearningAssessing Service-Learning
Assessing Service-Learning
Kristen Flores
 
Ikin McClenaghan formatted JAN 5_15
Ikin McClenaghan formatted JAN 5_15Ikin McClenaghan formatted JAN 5_15
Ikin McClenaghan formatted JAN 5_15Peter McClenaghan
 
The following is a discussion on some key factors for a school to consider wh...
The following is a discussion on some key factors for a school to consider wh...The following is a discussion on some key factors for a school to consider wh...
The following is a discussion on some key factors for a school to consider wh...malacia
 
Department IE Assessment Process
Department IE Assessment ProcessDepartment IE Assessment Process
Department IE Assessment Process
Kelly McMichael
 
Effect of performance appraisal system on staff performance in
Effect of performance appraisal system on staff performance inEffect of performance appraisal system on staff performance in
Effect of performance appraisal system on staff performance in
Alexander Decker
 
CURRICULUM DEVELOPMENT CYCLE.pdf
CURRICULUM DEVELOPMENT CYCLE.pdfCURRICULUM DEVELOPMENT CYCLE.pdf
CURRICULUM DEVELOPMENT CYCLE.pdf
Victor Rosales
 
Data Driven Decision Making Presentation
Data Driven Decision Making PresentationData Driven Decision Making Presentation
Data Driven Decision Making Presentation
Russell Kunz
 
Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)
Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)
Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)Research in Action, Inc.
 
A Novel Expert Evaluation Methodology Based On Fuzzy Logic
A Novel Expert Evaluation Methodology Based On Fuzzy LogicA Novel Expert Evaluation Methodology Based On Fuzzy Logic
A Novel Expert Evaluation Methodology Based On Fuzzy Logic
Leslie Schulte
 
2022_eat_framework_-aug_.pdf
2022_eat_framework_-aug_.pdf2022_eat_framework_-aug_.pdf
2022_eat_framework_-aug_.pdf
Rebecca665273
 
Using a standards alignment model as a framework for doctoral candidate asses...
Using a standards alignment model as a framework for doctoral candidate asses...Using a standards alignment model as a framework for doctoral candidate asses...
Using a standards alignment model as a framework for doctoral candidate asses...CPEDInitiative
 
White_Paper-The_Art_of_Evaluation
White_Paper-The_Art_of_EvaluationWhite_Paper-The_Art_of_Evaluation
White_Paper-The_Art_of_EvaluationSandy Barker
 
Chapter 7 050213 124524
Chapter 7 050213 124524Chapter 7 050213 124524
Chapter 7 050213 124524
Brenda Rachel Marie
 

Similar to Seeking Empirical Validity in an Assurance of Learning System (20)

A Proposed Theoretical Model For Evaluating E-Learning
A Proposed Theoretical Model For Evaluating E-LearningA Proposed Theoretical Model For Evaluating E-Learning
A Proposed Theoretical Model For Evaluating E-Learning
 
Journal raganit santos
Journal raganit santosJournal raganit santos
Journal raganit santos
 
PROF ED 7 PPT.pptx
PROF ED 7 PPT.pptxPROF ED 7 PPT.pptx
PROF ED 7 PPT.pptx
 
Assessment Of Business Programs A Review Of Two Models
Assessment Of Business Programs  A Review Of Two ModelsAssessment Of Business Programs  A Review Of Two Models
Assessment Of Business Programs A Review Of Two Models
 
This week, you examine cellular processes that are subject to alte
This week, you examine cellular processes that are subject to alteThis week, you examine cellular processes that are subject to alte
This week, you examine cellular processes that are subject to alte
 
CURRICULUM EVALUATIONTOPICSTODUSCUSS.pptx
CURRICULUM EVALUATIONTOPICSTODUSCUSS.pptxCURRICULUM EVALUATIONTOPICSTODUSCUSS.pptx
CURRICULUM EVALUATIONTOPICSTODUSCUSS.pptx
 
School based assesment
School based assesmentSchool based assesment
School based assesment
 
Assessing Service-Learning
Assessing Service-LearningAssessing Service-Learning
Assessing Service-Learning
 
Ikin McClenaghan formatted JAN 5_15
Ikin McClenaghan formatted JAN 5_15Ikin McClenaghan formatted JAN 5_15
Ikin McClenaghan formatted JAN 5_15
 
The following is a discussion on some key factors for a school to consider wh...
The following is a discussion on some key factors for a school to consider wh...The following is a discussion on some key factors for a school to consider wh...
The following is a discussion on some key factors for a school to consider wh...
 
Department IE Assessment Process
Department IE Assessment ProcessDepartment IE Assessment Process
Department IE Assessment Process
 
Effect of performance appraisal system on staff performance in
Effect of performance appraisal system on staff performance inEffect of performance appraisal system on staff performance in
Effect of performance appraisal system on staff performance in
 
CURRICULUM DEVELOPMENT CYCLE.pdf
CURRICULUM DEVELOPMENT CYCLE.pdfCURRICULUM DEVELOPMENT CYCLE.pdf
CURRICULUM DEVELOPMENT CYCLE.pdf
 
Data Driven Decision Making Presentation
Data Driven Decision Making PresentationData Driven Decision Making Presentation
Data Driven Decision Making Presentation
 
Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)
Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)
Assessment Selection Paper-Herman_Heritage_Goldschmidt (2011)
 
A Novel Expert Evaluation Methodology Based On Fuzzy Logic
A Novel Expert Evaluation Methodology Based On Fuzzy LogicA Novel Expert Evaluation Methodology Based On Fuzzy Logic
A Novel Expert Evaluation Methodology Based On Fuzzy Logic
 
2022_eat_framework_-aug_.pdf
2022_eat_framework_-aug_.pdf2022_eat_framework_-aug_.pdf
2022_eat_framework_-aug_.pdf
 
Using a standards alignment model as a framework for doctoral candidate asses...
Using a standards alignment model as a framework for doctoral candidate asses...Using a standards alignment model as a framework for doctoral candidate asses...
Using a standards alignment model as a framework for doctoral candidate asses...
 
White_Paper-The_Art_of_Evaluation
White_Paper-The_Art_of_EvaluationWhite_Paper-The_Art_of_Evaluation
White_Paper-The_Art_of_Evaluation
 
Chapter 7 050213 124524
Chapter 7 050213 124524Chapter 7 050213 124524
Chapter 7 050213 124524
 

More from Rochell McWhorter

Giving Back: Exploring Service-Learning in an Online Environment
Giving Back: Exploring Service-Learning in an Online EnvironmentGiving Back: Exploring Service-Learning in an Online Environment
Giving Back: Exploring Service-Learning in an Online Environment
Rochell McWhorter
 
18th Annual Family History Fair
18th Annual Family History Fair18th Annual Family History Fair
18th Annual Family History Fair
Rochell McWhorter
 
Beyond the Walls of the Classroom: Technology Trends for 2012
Beyond the Walls of the Classroom: Technology Trends for 2012Beyond the Walls of the Classroom: Technology Trends for 2012
Beyond the Walls of the Classroom: Technology Trends for 2012
Rochell McWhorter
 
Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...
Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...
Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...Rochell McWhorter
 
Exploring the emergence of virtual human resource development
Exploring the emergence of virtual human resource developmentExploring the emergence of virtual human resource development
Exploring the emergence of virtual human resource development
Rochell McWhorter
 
Exploring the Emergence of Virtual Human Resource Development
Exploring the Emergence of Virtual Human Resource Development Exploring the Emergence of Virtual Human Resource Development
Exploring the Emergence of Virtual Human Resource Development
Rochell McWhorter
 
Virtual HRD, Technology & Distance Learning
Virtual HRD, Technology & Distance LearningVirtual HRD, Technology & Distance Learning
Virtual HRD, Technology & Distance Learning
Rochell McWhorter
 
Using Virtual World Technologies Across the Disciplines
Using Virtual World Technologies Across the DisciplinesUsing Virtual World Technologies Across the Disciplines
Using Virtual World Technologies Across the Disciplines
Rochell McWhorter
 
Virtual HRD Presentation
Virtual HRD PresentationVirtual HRD Presentation
Virtual HRD Presentation
Rochell McWhorter
 
The Emergence of Virtual Human Resource Development
The Emergence of Virtual Human Resource DevelopmentThe Emergence of Virtual Human Resource Development
The Emergence of Virtual Human Resource Development
Rochell McWhorter
 
Adult Learning In A Virtual Environment, AHRD 2008
Adult Learning In A Virtual Environment, AHRD 2008Adult Learning In A Virtual Environment, AHRD 2008
Adult Learning In A Virtual Environment, AHRD 2008
Rochell McWhorter
 
Scenario Planning as the Development of Leadership
Scenario Planning as the Development of LeadershipScenario Planning as the Development of Leadership
Scenario Planning as the Development of Leadership
Rochell McWhorter
 

More from Rochell McWhorter (12)

Giving Back: Exploring Service-Learning in an Online Environment
Giving Back: Exploring Service-Learning in an Online EnvironmentGiving Back: Exploring Service-Learning in an Online Environment
Giving Back: Exploring Service-Learning in an Online Environment
 
18th Annual Family History Fair
18th Annual Family History Fair18th Annual Family History Fair
18th Annual Family History Fair
 
Beyond the Walls of the Classroom: Technology Trends for 2012
Beyond the Walls of the Classroom: Technology Trends for 2012Beyond the Walls of the Classroom: Technology Trends for 2012
Beyond the Walls of the Classroom: Technology Trends for 2012
 
Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...
Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...
Teaching Social Skills to Students with Autism Spectrum Disorders in Virtual ...
 
Exploring the emergence of virtual human resource development
Exploring the emergence of virtual human resource developmentExploring the emergence of virtual human resource development
Exploring the emergence of virtual human resource development
 
Exploring the Emergence of Virtual Human Resource Development
Exploring the Emergence of Virtual Human Resource Development Exploring the Emergence of Virtual Human Resource Development
Exploring the Emergence of Virtual Human Resource Development
 
Virtual HRD, Technology & Distance Learning
Virtual HRD, Technology & Distance LearningVirtual HRD, Technology & Distance Learning
Virtual HRD, Technology & Distance Learning
 
Using Virtual World Technologies Across the Disciplines
Using Virtual World Technologies Across the DisciplinesUsing Virtual World Technologies Across the Disciplines
Using Virtual World Technologies Across the Disciplines
 
Virtual HRD Presentation
Virtual HRD PresentationVirtual HRD Presentation
Virtual HRD Presentation
 
The Emergence of Virtual Human Resource Development
The Emergence of Virtual Human Resource DevelopmentThe Emergence of Virtual Human Resource Development
The Emergence of Virtual Human Resource Development
 
Adult Learning In A Virtual Environment, AHRD 2008
Adult Learning In A Virtual Environment, AHRD 2008Adult Learning In A Virtual Environment, AHRD 2008
Adult Learning In A Virtual Environment, AHRD 2008
 
Scenario Planning as the Development of Leadership
Scenario Planning as the Development of LeadershipScenario Planning as the Development of Leadership
Scenario Planning as the Development of Leadership
 

Recently uploaded

Brand Analysis for an artist named Struan
Brand Analysis for an artist named StruanBrand Analysis for an artist named Struan
Brand Analysis for an artist named Struan
sarahvanessa51503
 
In the Adani-Hindenburg case, what is SEBI investigating.pptx
In the Adani-Hindenburg case, what is SEBI investigating.pptxIn the Adani-Hindenburg case, what is SEBI investigating.pptx
In the Adani-Hindenburg case, what is SEBI investigating.pptx
Adani case
 
Mastering B2B Payments Webinar from BlueSnap
Mastering B2B Payments Webinar from BlueSnapMastering B2B Payments Webinar from BlueSnap
Mastering B2B Payments Webinar from BlueSnap
Norma Mushkat Gaffin
 
ikea_woodgreen_petscharity_dog-alogue_digital.pdf
ikea_woodgreen_petscharity_dog-alogue_digital.pdfikea_woodgreen_petscharity_dog-alogue_digital.pdf
ikea_woodgreen_petscharity_dog-alogue_digital.pdf
agatadrynko
 
ikea_woodgreen_petscharity_cat-alogue_digital.pdf
ikea_woodgreen_petscharity_cat-alogue_digital.pdfikea_woodgreen_petscharity_cat-alogue_digital.pdf
ikea_woodgreen_petscharity_cat-alogue_digital.pdf
agatadrynko
 
Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...
Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...
Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...
SOFTTECHHUB
 
Agency Managed Advisory Board As a Solution To Career Path Defining Business ...
Agency Managed Advisory Board As a Solution To Career Path Defining Business ...Agency Managed Advisory Board As a Solution To Career Path Defining Business ...
Agency Managed Advisory Board As a Solution To Career Path Defining Business ...
Boris Ziegler
 
Auditing study material for b.com final year students
Auditing study material for b.com final year  studentsAuditing study material for b.com final year  students
Auditing study material for b.com final year students
narasimhamurthyh4
 
Training my puppy and implementation in this story
Training my puppy and implementation in this storyTraining my puppy and implementation in this story
Training my puppy and implementation in this story
WilliamRodrigues148
 
Meas_Dylan_DMBS_PB1_2024-05XX_Revised.pdf
Meas_Dylan_DMBS_PB1_2024-05XX_Revised.pdfMeas_Dylan_DMBS_PB1_2024-05XX_Revised.pdf
Meas_Dylan_DMBS_PB1_2024-05XX_Revised.pdf
dylandmeas
 
Bài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.doc
Bài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.docBài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.doc
Bài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.doc
daothibichhang1
 
Discover the innovative and creative projects that highlight my journey throu...
Discover the innovative and creative projects that highlight my journey throu...Discover the innovative and creative projects that highlight my journey throu...
Discover the innovative and creative projects that highlight my journey throu...
dylandmeas
 
Observation Lab PowerPoint Assignment for TEM 431
Observation Lab PowerPoint Assignment for TEM 431Observation Lab PowerPoint Assignment for TEM 431
Observation Lab PowerPoint Assignment for TEM 431
ecamare2
 
Search Disrupted Google’s Leaked Documents Rock the SEO World.pdf
Search Disrupted Google’s Leaked Documents Rock the SEO World.pdfSearch Disrupted Google’s Leaked Documents Rock the SEO World.pdf
Search Disrupted Google’s Leaked Documents Rock the SEO World.pdf
Arihant Webtech Pvt. Ltd
 
Sustainability: Balancing the Environment, Equity & Economy
Sustainability: Balancing the Environment, Equity & EconomySustainability: Balancing the Environment, Equity & Economy
Sustainability: Balancing the Environment, Equity & Economy
Operational Excellence Consulting
 
Top mailing list providers in the USA.pptx
Top mailing list providers in the USA.pptxTop mailing list providers in the USA.pptx
Top mailing list providers in the USA.pptx
JeremyPeirce1
 
An introduction to the cryptocurrency investment platform Binance Savings.
An introduction to the cryptocurrency investment platform Binance Savings.An introduction to the cryptocurrency investment platform Binance Savings.
An introduction to the cryptocurrency investment platform Binance Savings.
Any kyc Account
 
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
bosssp10
 
The effects of customers service quality and online reviews on customer loyal...
The effects of customers service quality and online reviews on customer loyal...The effects of customers service quality and online reviews on customer loyal...
The effects of customers service quality and online reviews on customer loyal...
balatucanapplelovely
 
Premium MEAN Stack Development Solutions for Modern Businesses
Premium MEAN Stack Development Solutions for Modern BusinessesPremium MEAN Stack Development Solutions for Modern Businesses
Premium MEAN Stack Development Solutions for Modern Businesses
SynapseIndia
 

Recently uploaded (20)

Brand Analysis for an artist named Struan
Brand Analysis for an artist named StruanBrand Analysis for an artist named Struan
Brand Analysis for an artist named Struan
 
In the Adani-Hindenburg case, what is SEBI investigating.pptx
In the Adani-Hindenburg case, what is SEBI investigating.pptxIn the Adani-Hindenburg case, what is SEBI investigating.pptx
In the Adani-Hindenburg case, what is SEBI investigating.pptx
 
Mastering B2B Payments Webinar from BlueSnap
Mastering B2B Payments Webinar from BlueSnapMastering B2B Payments Webinar from BlueSnap
Mastering B2B Payments Webinar from BlueSnap
 
ikea_woodgreen_petscharity_dog-alogue_digital.pdf
ikea_woodgreen_petscharity_dog-alogue_digital.pdfikea_woodgreen_petscharity_dog-alogue_digital.pdf
ikea_woodgreen_petscharity_dog-alogue_digital.pdf
 
ikea_woodgreen_petscharity_cat-alogue_digital.pdf
ikea_woodgreen_petscharity_cat-alogue_digital.pdfikea_woodgreen_petscharity_cat-alogue_digital.pdf
ikea_woodgreen_petscharity_cat-alogue_digital.pdf
 
Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...
Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...
Hamster Kombat' Telegram Game Surpasses 100 Million Players—Token Release Sch...
 
Agency Managed Advisory Board As a Solution To Career Path Defining Business ...
Agency Managed Advisory Board As a Solution To Career Path Defining Business ...Agency Managed Advisory Board As a Solution To Career Path Defining Business ...
Agency Managed Advisory Board As a Solution To Career Path Defining Business ...
 
Auditing study material for b.com final year students
Auditing study material for b.com final year  studentsAuditing study material for b.com final year  students
Auditing study material for b.com final year students
 
Training my puppy and implementation in this story
Training my puppy and implementation in this storyTraining my puppy and implementation in this story
Training my puppy and implementation in this story
 
Meas_Dylan_DMBS_PB1_2024-05XX_Revised.pdf
Meas_Dylan_DMBS_PB1_2024-05XX_Revised.pdfMeas_Dylan_DMBS_PB1_2024-05XX_Revised.pdf
Meas_Dylan_DMBS_PB1_2024-05XX_Revised.pdf
 
Bài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.doc
Bài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.docBài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.doc
Bài tập - Tiếng anh 11 Global Success UNIT 1 - Bản HS.doc
 
Discover the innovative and creative projects that highlight my journey throu...
Discover the innovative and creative projects that highlight my journey throu...Discover the innovative and creative projects that highlight my journey throu...
Discover the innovative and creative projects that highlight my journey throu...
 
Observation Lab PowerPoint Assignment for TEM 431
Observation Lab PowerPoint Assignment for TEM 431Observation Lab PowerPoint Assignment for TEM 431
Observation Lab PowerPoint Assignment for TEM 431
 
Search Disrupted Google’s Leaked Documents Rock the SEO World.pdf
Search Disrupted Google’s Leaked Documents Rock the SEO World.pdfSearch Disrupted Google’s Leaked Documents Rock the SEO World.pdf
Search Disrupted Google’s Leaked Documents Rock the SEO World.pdf
 
Sustainability: Balancing the Environment, Equity & Economy
Sustainability: Balancing the Environment, Equity & EconomySustainability: Balancing the Environment, Equity & Economy
Sustainability: Balancing the Environment, Equity & Economy
 
Top mailing list providers in the USA.pptx
Top mailing list providers in the USA.pptxTop mailing list providers in the USA.pptx
Top mailing list providers in the USA.pptx
 
An introduction to the cryptocurrency investment platform Binance Savings.
An introduction to the cryptocurrency investment platform Binance Savings.An introduction to the cryptocurrency investment platform Binance Savings.
An introduction to the cryptocurrency investment platform Binance Savings.
 
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
Call 8867766396 Satta Matka Dpboss Matka Guessing Satta batta Matka 420 Satta...
 
The effects of customers service quality and online reviews on customer loyal...
The effects of customers service quality and online reviews on customer loyal...The effects of customers service quality and online reviews on customer loyal...
The effects of customers service quality and online reviews on customer loyal...
 
Premium MEAN Stack Development Solutions for Modern Businesses
Premium MEAN Stack Development Solutions for Modern BusinessesPremium MEAN Stack Development Solutions for Modern Businesses
Premium MEAN Stack Development Solutions for Modern Businesses
 

Seeking Empirical Validity in an Assurance of Learning System

  • 1. Running head: SEARCHING FOR EMPIRICAL VALIDITY Seeking Empirical Validity in an Assurance of Learning System Sherry Avery, Rochell McWhorter, Roger Lirely, H. Harold Doty College of Business and Technology The University of Texas at Tyler Contact Email: rmcwhorter@uttyler.edu Note: This is the last authors’ copy of this work. The final edited, definitive, and distributed copy is published in the Journal of Education for Business, 2014; 89: 156-164, doi: 10.1080/08832323.2013.800467 Abstract Business schools have established measurement tools to support their AoL systems and assess student achievement of learning objectives. However, business schools have not required their tools be empirically validated thus ensuring that they measure what they are intended to measure. We propose confirmatory factor analysis (CFA) be utilized by business schools to evaluate AoL measurement systems. We illustrate a CFA model used to evaluate the measurement tools at our College. Our approach is in initial steps currently evaluating individual measurement tools, but working towards developing a system that can evaluate the entire AoL measurement systems. Keywords: AACSB, Assurances of Learning, Assessment, Confirmatory Factor Analysis A decade ago, the Association to Advance Collegiate Schools of Business (AACSB) International ratified new accreditation requirements including the addition of assurance of learning (AoL) standards for continuous improvement (Martell, 2007). As part of this addition, schools seeking to earn or maintain AACSB accreditation must develop a set of defined learning goals and subsequently collect relevant assessment data to determine direct educational achievement (LeClair, 2012; Sampson & Betters-Reed, 2008). The establishment of the mission- driven assessment process requires “well-documented systematic processes to develop, monitor,
  • 2. SEARCHING FOR EMPIRICAL VALIDITY 2 evaluate, and revise the substance and deliver of the curricula on learning” (Romero, 2008, p. 253). With establishment of the 2003 AACSB standards, all schools “must develop assessment tools that measure the effectiveness of their curriculum” (Pesta & Scherer, 2011, p. 164). As a response to this outcomes assessment mandate, a number of schools created models to depict and track their assessment functions (Betters-Reed, Nitkin & Sampson, 2008; Gardiner, Corbitt, & Adams 2010; Zocco, 2011). However, the question arises as to the validity of system models for measuring learning outcomes—does the model measure what it purports to measure—do the learning experiences accomplish the learning goals outlined in the systems model? This question is a very important one because once validity of a measurement system is established, then it provides confidence in a program and quality of assurance in achieving the school’s mission (Baker, Ni & Van Wart, 2012). The purpose of this article is to illustrate development of an empirically-based AoL System which may be used by other business schools seeking accreditation. Relevant literature on this topic will be examined next. Review of Literature A search of the literature for an empirically-validated AoL system yielded results for research covering either the validation of AoL tools or validation of an Aol model. Each is discussed in the following section. Measures of Validity in AoL Assessment Tools Measures of validity associated with AoL learning outcomes were located in business literature by reviewing articles that described locally-developed assessment tools and externally validated instruments. For instance, researchers developed an assessment tool to explore
  • 3. SEARCHING FOR EMPIRICAL VALIDITY 3 students’ self-efficacy toward service and civic participation. They utilized traditional scale development and confirmatory factor analysis (CFA) and simultaneous factor analysis in several populations (SIFASP) for insuring the validity and reliability of their instrument to measure AoL criteria for ethics and social responsibility (Weber, Weber, Sleeper & Schneider, 2004). Another tool offered was a content valid assessment exam created to measure business management knowledge (Pesta & Scherer, 2011). Also, a matrix presented by Harper and Harder (2009) depicts demonstrated abilities intersected with competency clusters; the clusters were developed from literature describing valid research into “the kinds of knowledge and skills that are known to be necessary for success as a practitioner in the MIS field” (p. 492). However, no statistical measures of validity were provided. Additionally, instances of use of externally validated instruments such as the revised version of the Defining Issues Test (DIT-2) to assess ethical reasoning instruction in undergraduate cost accounting (Wilhelm & Czyzewski, 2012) and use of the CAPSIM computer simulation to assess business strategy integration (Garrett, Marques & Dhiman, 2012), were found. Measures of Validity in AoL Assessment Models Various models have been offered for outcomes measurement as part of a processed- based approach for meeting AoL standards (i.e. Beard, Schwieger & Surendran, 2008; Betters- Reed, Nitkin & Sampson, 2008; Hess & Siciliano, 2007) but without statistical evidence or discussion of validity measures. However, the search of literature found an article by Zocco (2011) that presented a recursive model to address and document continuous and systematic improvement and discussed validity issues surrounding the application of recursion to a process such as AoL. Although helpful for looking at school improvement, the model does not measure
  • 4. SEARCHING FOR EMPIRICAL VALIDITY 4 validity of the model itself. Therefore, the review of literature offered several tools and a model with validity calculations, however, no example of an empirically-validated system was found. Case Study: Assurances of Learning at [blinded] During the past five years, the College of Business [removed for blinding purposes, hereafter referred to as “College”] has conducted a complete redesign of its AACSB Assurance of Learning system. To understand the rationale for this design change it is important to explore several key drivers of this decision, especially in light of the fact that our prior AoL system was cited as one or our “best practices” during our last Maintenance of Accreditation visit. At last visit, the College operated three different and largely unrelated assessment systems: one for the AACSB, [names of 2 other accreditation bodies removed for blinding purposes]. In some ways, these independent assessment systems simplified accreditation reporting: each system was tailored to the specifics of a single accrediting body and the data associated with one system were not considered in collaboration with data collected for a different accrediting body. For example, AACSB and the College’s assessment procedures were treated as completely independent. This approach simplified reporting, but hindered integrating different assessment data in the larger curriculum management process. A second major contextual factor relevant to our AoL process was feedback from our last AACSB visit that recommended revisions to the vision and mission statements for the College and AACSB AoL processes. As part of that revision process, the College clarified its mission and identified five core values. Based largely on these contextual factors, faculty determined we were at an ideal point to design a new single integrated assessment model to meet the needs of each of our accrediting bodies. Further, we determined that the new system should be linked to the new mission by
  • 5. SEARCHING FOR EMPIRICAL VALIDITY 5 incorporating the core values as learning outcomes, and that we should attempt to assess the validity of the system in terms of both the theoretical model used to design the system and the measurement model used to organize the data collection. These additional steps would allow more confidence in the evidence-based changes we were making in program structure and course curriculum. The full scale implementation began in the 2010-11 school year; our model is more fully described next. Faculty-Driven Process AoL in the College is a faculty-driven process. Oversight of this process is charged to the AoL Committee, a committee comprised of a faculty chair, the undergraduate program director, the graduate programs coordinator and four at-large faculty members. The composition of the Committee provides cross-sectional representation of all disciplines and programs in the College. The Committee works closely with our faculty to ensure that each learning objective is measured periodically, at least twice during each five-year period but generally more often. The faculty employ a variety of measurement strategies, including major field tests, embedded test questions, case analyses, observation of student presentations, activity logs, simulations and other class assignments and/or projects. Analyses of results guide the Committee in its work with the faculty to develop and implement appropriate actions to ensure curricula and pedagogy are managed in a manner enhancing student learning and development. Figure 1 illustrates how the
  • 6. SEARCHING FOR EMPIRICAL VALIDITY 6 AoL assessment process operates in a continuous improvement mode. FIGURE 1 AoL Curriculum Management Process at [Blinded] Conceptual Framework The AoL system in the College is based on a set of shared core values: professional proficiency, technological competence, global awareness, social responsibility and ethical courage as seen in Figure 2. These mission-based core values form the framework for our comprehensive, empirically-validated AoL models for both the Bachelor in Business Administration program and the Master of Business Administration Program, as well as other College programs that are outside the scope of AACSB accreditation. AoL in the College has evolved to the point where our current system is second-generation, that is, it is the culmination
  • 7. SEARCHING FOR EMPIRICAL VALIDITY 7 of an assessment of the AoL system itself. Many of the best features of the prior system were retained, including assessment of discipline-based knowledge, communication skills, and the use of quantitative tools and business technology. The result of this process is a value-based conceptual framework whose efficacy can be tested empirically using confirmatory factor analysis. To our knowledge, our College is the first AACSB-accredited program to design an empirically-validated AoL system. Figure 2 depicts the conceptual framework of our AoL system for the BBA program. Figure 2 Method Data Collection. The faculty developed ten learning objectives to support the five learning goals of the College. A measurement tool was designed for each objective, such as the Major Field Test or rubrics. Assessment was conducted within required core business courses that included students across all College of Business majors. Students were generally either junior or seniors in one of the business majors. Results were then collected and compiled centrally in an administrative function within the College. College of Business & Technology Creating Leaders for a Better Tomorrow Learning Goal: Graduates have the knowledge and communication skills to succeed in the business profession. Ethical Courage Social Responsibility Global Awareness Technological Competence Professional Proficiency Learning Goal: Graduates use the information sources and tools associated with their chosen profession. Learning Goal: Graduates incorporate global considerations in business activities. Learning Goal: Graduates evaluate the social consequences of alternative outcomes when making business decisions. Learning Goal: Graduates understand ethical considerations and their impacts. Cognitive: GA1 - Students demonstrate awareness of global issues and perspectives. Behavioral: GA2 - Students are knowledgeable of global issues and perspectives that may impact business activities. Behavioral: SR1 - Students exhibit an understanding of the social consequences of business activities. Cognitive: EC1 - Students understand legal and ethical concepts. Behavioral: EC2 - Students make ethical decisions. Behavioral: TC2 - Students are able to use business software, data sources, and tools. Cognitive: TC1 - Students understand information systems and their role in business enterprises. Behavioral: PP3 - Students are able to deliver a presentation that is focused, well- organized, and includes the appropriate verbal and nonverbal behaviors. Cognitive: PP1 - Students demonstrate that they are knowledgeable about current business theory, concepts, methodology, terminology and practices. PP2 - Students can prepare a business document that is focused, well- organized, and mechanically correct. Bachelor of Business Administration Updated: 7/11/2012
  • 8. SEARCHING FOR EMPIRICAL VALIDITY 8 Analysis Approach. Several of the measurement tools included a number of items that collectively assessed the specific learning objective. Confirmatory factor analysis (CFA) was conducted to assess empirical validity of the items measures and learning objectives. CFA was chosen for assessment because it tests how well the measured variables represent the constructs (Hair, Black, Babin, Anderson & Tatham, 2005). The items that comprise each construct are identified prior to running the CFA analysis. We then confirm or reject that the items properly reflect the construct, in this case the learning objective. CFA was conducted on six of the learning objectives. We were unable to run CFA for some learning objectives because they were either measured by a single item, the sample size was too small, or data was binary, therefore negating the applicability of using CFA. Table 1 details the learning objectives, measurement tools, and when the CFA was conducted. In the following section, we discuss the general approach used in the CFA analysis. Then, we follow up with two examples of the CFA analysis; the first example is empirically valid, and the second example was not empirical valid. We used a combination of the software tools SPSS and AMOS
  • 9. SEARCHING FOR EMPIRICAL VALIDITY 9 to support the analysis. For each of the CFA analyses, we followed a 3-step approach documented in many of the leading academic journals: (1) review of the raw data, (2) assessment of model fit, and (3) assessment of construct validity. Prior to the CFA analyses, we reviewed the data for sample
  • 10. SEARCHING FOR EMPIRICAL VALIDITY 10 size, outliers, missing data, and normality. We determined if the sample size was adequate for the model based on suggested requirements that range from 5 to 20 observations per variable. (Hair et al., 2005). The existence of outliers along with their potential impact on normality and the final results were assessed at both the univariate and multivariate level by reviewing the Mahalanobis distance (D2) calculation for each case. Next, we identified missing data and assessed the potential impact on analysis. A case that has a substantially different value from other D2 calculations is a potential outlier. Then, we identified the amount of missing data and assessed the potential impact on the analysis. Finally, we assessed normality by reviewing both skewness and kurtosis at the univariate and multivariate level. Values of 0 represent a normal distribution. For skewness, less than 3 is acceptable (Chou & Bentler, 1995; Kline, 2005). For kurtosis, Kline (2005) stated that less than 8 is reasonable with greater than 10 indicating a problem and over 20 an extreme problem. We evaluated how well the data fit the measurement model using Analysis of Moment Structures (AMOS) software (see http://www.utexas.edu/its/help/spss/526). We used maximum- likelihood estimations which is a widely used approach and is fairly robust to violations of normality and produces reliable results under many circumstances (Hair et al., 2005; Marsh, Balla & McDonald, 1998). First, we evaluated the Chi-Square (χ2) statistic which measures the difference between the observed and estimated covariance matrices. It is the only statistic that has a direct statistical significance and is used as the basis for many other fit indices (Hair et al., 2005). Statistical significance in this case indicates that an error of approximation or estimation exists. Many researchers question the validity of the chi-square statistics (Bentler, 1990), so if it is significant additional indices should be used to assess overall evaluate model fit. The root mean square error of approximation (RMSEA) is a standardized measure of the lack of fit of the
  • 11. SEARCHING FOR EMPIRICAL VALIDITY 11 data to the model (Steiger, 1990). It is fairly robust in terms of small sample size; i.e. 250 or less. Thresholds of .05 to .08 have been suggested with Hu and Bentler (1999) recommending a cutoff close to .06. The Bentler-Bonnet (1980) Non-Normed Fit Index (NNFI) was used because it also works well with small sample sizes (Bedeian, 2007). Generally .90 or better is considered adequate fit with Hu and Bentler (1999) suggesting a threshold of .95 or better for good fit. The non-normed fit index (NNFI) and comparative fit index (CFI) are incremental fit indices in that they assess model fit by referring a baseline model (Bentler 1990; Bentler & Bonett, 1980; Hu & Bentler, 1999). The NNFI and CFI generate values between 0 and 1 with .90 or greater representing adequate fit (Hu & Bentler, 1999; Bedeian 2007). The final step was to assess construct validity which is the extent to which a set of measured items accurately reflect the theoretical latent construct the items were designed to measure (Hair et al., 2005). The standardized factor loadings should be statistically significant and .5 or higher (Hair et al., 2005). Convergence validity was assessed by calculating the average variance extracted (AVE) and construct reliability (CR). The average percent of variance extracted among a set of construct items is a summary indicator of convergence. AVE of .5 or higher suggests adequate convergence. Less than 5 indicates that on average more error remains in the items than variance explained by the latent construct (Fornell & Larcker, 1981). High CR reliability indicates that the measures consistently represent the same latent construct. Values of .6 or 7 are acceptable, with .7 or higher indicating good reliability (Nunnally, 1978). Results. The purpose of this article is to illustrate the method we used to assess empirical validity of our learning objectives to aid other business schools in their AoL journey. It is not our goal to suggest that our measurement tools or learning objectives should be universally adopted. Therefore for illustration of our process, we will limit our discussion to the overall fit
  • 12. SEARCHING FOR EMPIRICAL VALIDITY 12 of our constructs and then discuss in detail two examples of how we used CFA; an example of a valid measure (business knowledge) and an example of a measure that requires some modifications (oral communication). Table 2 documents the model fit indices for the CFA analyses performed for six of the learning objectives. The learning objectives for business knowledge, written communication, global awareness context, region, and perspectives are valid for construct reliability and model fit. Therefore we are reasonably confident that these objectives adequately measure the learning goals established by the College. The model fit for Oral Communication was below suggested thresholds. Table 2 Fit Indices Composite reliability Variance Extracted χ2 RMSEA RMSR NNFI CFI Learning Objective Business Knowledge (n=218) .853 .409 61.35** .077 11.66 .932 .949 Written Communication (n=147) .798 .498 6.163** .119 .011 .926 .975 Oral Communication (n=161) .62 .246 120.24** .148 .033 .693 .770 Global Awareness Context (n=151) .85 .397 19.65** .089 .336 .958 .975 Global Awareness Region (n=151) .89 .562 8.183 .000 .192 1.003 1.000 Global Awareness Perspectives (n=90) .96 .73 96.643** .208 .031 .867 .905 **p<.05 Business knowledge. Students take the ETS Major Field Test for the Bachelor’s Degree in Business (MFT) in a capstone class in their senior year. The MFT is a widely used standardized exam for business students (www.ets.org/mft). The capstone class is required for all business majors and the MFT is administered in all sections of the capstone class thus ensuring that all business majors participate in the assessment prior to graduation. To ensure that students take the exam seriously and give their best effort, their results are reflected in their course grade. Two hundred and eighteen responses were obtained from the exams administered in 2010 – 2011. Nine composite scores from the exam are used to assess the overall business knowledge
  • 13. SEARCHING FOR EMPIRICAL VALIDITY 13 of the student. (See Table 3 for a listing of these nine items.) A review of the data found no missing data or significant outliers. The sample size divided by the number of responses to the number of variables (218/9 = 24) was well above the recommended threshold range of 5 to 20. The kurtosis and skewness statistics were less than the recommended thresholds of 8 and 3, which indicates only a slight departure from normality. Therefore we were reasonably confident in proceeding to the next phase of the analysis, evaluating model fit by running a confirmatory factor analysis on the item measures. The χ2 was significant, however the RMSEA was .077, below the recommended threshold of .08. The RMSEA is parsimonious in that it considers the impact of the number of variables in their calculations, so it is a better indicator of model fit than χ2. The NNFI and CFI was .932 and .949, well above the recommended threshold of .90. Overall, the model fit is acceptable.
  • 14. SEARCHING FOR EMPIRICAL VALIDITY 14 In assessing construct validity of the items, we noted that all items were statistically significant, however one item measure, quantitative analysis, fell below the recommended loading of .50. The composite reliability of .853 was well above the recommended threshold of .6 and the variance extracted was slightly below the recommended threshold at .50. Overall there is evidence that the item measures adequately reflect the latent construct of business knowledge. However, further analysis is needed to determine the cause of the low factor loading of quantitative analysis. We provided Figure 3 as a visual representation of the CFA model for this construct. Oral communication. The students’ oral communication skills were measured by a rubric assessed oral presentation assignment administered in the business communication course, which is part of the required core curriculum. The business communication professor for all sections of the class completed the assessment. There is only one business communication professor, thus ensuring consistency of the measurement process. The rubric is comprised of nine item measures (see Table 3 for a list of these items). Data was obtained from the 2010-2012 assessments which resulted in 161 observations for the CFA of the oral communication construct. A review of the data found two missing observations for the item measure conclusion and one missing
  • 15. SEARCHING FOR EMPIRICAL VALIDITY 15 observation for eye contact. Because the impact of missing data was small, we used the mean imputation method for the missing observations. We also identified one potential outlier; we deleted the case on a trail run and found that it did not have a significant impact on normality or the results. The sample size ratio (161/9 = 17.9) is in the recommended threshold range of 5 to 20. The multivariate kurtosis statistic of 23.393 was well above the recommended thresholds of 8, which provides evidence of a departure from normality. The univariate skewness statistics were below the threshold of 3. Because of the non-normal distribution, we attempted to run an asymptotically distribution-free estimation method to contact the CFA. Unfortunately this resulted in an inadmissible solution because of the existence of a negative error variance. Therefore we used the Maximum-Likelihood estimation technique which often provides reasonable results with departures from normality. The χ2 was significant, however the adjusted chi-square ratio (χ2 /df) was 4.453, below the recommended threshold of 5. The RMSR was .033 well below the recommended threshold of .10 while the RMSEA was .145, above the recommended threshold of .08. The NNFI and CFI was .693 and .770, below the recommended threshold of .90. Our overall assessment is the model fit is poor. The RMSEA, NNFI, and CFI are impacted by the model complexity which could be an indication that the number of variables in the model affected model fit. Two of the items were not statistically significant: conclusion and fillers. Only two of the nine items were greater than the recommended threshold of .50; projection and pace. The composite reliability of .62 met the recommended threshold of .6. The variance extracted of .246 was well below the recommended threshold of .50. Our conclusion is that the item measures do not accurately reflect the latent construct oral communications.
  • 16. SEARCHING FOR EMPIRICAL VALIDITY 16 Discussion The purpose of this article is to highlight an AACSB-accredited program as a case study of its design of an empirically-validated AoL system and to demonstrate how empirical validity improved our AoL system. When appropriate, we used confirmatory factor analysis to validate the measurement instruments used to assess student achievement of the learning objectives established by the faculty and stakeholders of the College. We provided a description of the CFA process used to assess the empirical validity of the learning objectives. Finally we illustrated the process by discussing the results of the validation process on two learning objectives. For the business knowledge learning objective, we found that both the model fit and construct reliability are valid. However, we noted that factor loading for quantitative analysis was much lower than the other item measures. In reviewing the raw scores, not surprisingly, we found that our students do not perform as well in the quantitative analysis topic when compared to the other topics covered by the MFT indicating that even though we had a valid measure of business knowledge, our students need improvement in their quantitative skills. These result prompted us to examine the curriculum of the class where much of the quantitative methods are taught.
  • 17. SEARCHING FOR EMPIRICAL VALIDITY 17 For the oral communication learning objective, we found that the model fit was poor, construct reliability was low, and many of the item measures from the rubric did not load. These results prompted us to examine the measurement tool used to assess achievement oral communication competency. Corrective action includes a review of the rubric and also the process used to collect the data. Conclusion and Limitations. We found value in, and therefore will continue to empirically validate the AoL learning objectives using confirmatory factor analysis. The validation process has increased the support of the AoL system by faculty that understand and are trained in the research process. We have received positive feedback from the AACSB and higher education associations on our validation process. Most importantly, it provides confidence in the tools we are using to measure student achievement of the learning objectives. Now that the process and the supporting models are in place, it will be relatively simple to continue the validation process in order to continually improve. Because of the method we used to capture the assessments, we are able to use the same validation process for both AACSB and SACS accreditations. We are continually striving to improve our validation process. One limitation is that we are unable to simultaneously assess the empirical validity of the entire model of the learning objectives. Assessment is conducted by class, rather than individual student across all classes. Therefore we do not have data for one student for all learning objectives. To address this limitation, we are evaluating both in-house and commercially developed databases to track student data across classes and semesters. For example, the University of Central Florida (UCF) developed an in-house database for tracking of student data (Moskal, Ellis, & Keon, 2008). Taskstream is a commercially available database for tracking data (https://www.taskstream.com).
  • 18. SEARCHING FOR EMPIRICAL VALIDITY 18 References Baker, D. L., Ni, A. Y., & Van Wart, M. (2012). AACSB assurance of learning: Lessons learned in ethics module development. Business Education Innovation Journal, 4(1), 19-27. Beard, D., Schwieger , D., & Surendran, K. (2008). Integrating soft skills assessment through university, college, and programmatic efforts at an AACSB accredited institution. Journal of Information Systems, 19(2), 229-240. Bedeian, A. G. (2007). Even if the tower is 'Ivory,' It Isn't 'White:' Understanding the consequences of faculty cynicism. Academy Of Management Learning & Education, 1, 9. doi:10.2307/40214514 Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606. Betters-Reed, B. L., Nitkin, M. R., Sampson, S. D. (2008). An assurance of learning success model: Toward closing the feedback loop. Organization Management Journal, 5, 224- 240. Chou, C. P., & Bentler, P. M. (1995). Estimates and tests in structural equation modeling. In R. Hoyle (Ed.), Structural Equation Modeling (pp. 37-59). Thousand Oaks, CA: Sage. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement eror. Journal of Marketing Research, 18, 39-50. Gardiner, L. R., Corbitt, G., & Adams, S. J. (2010). Program assessment: Getting to a practical how-to model. Journal of Education for Business, 85(3), 139-144. doi:10.1080/08832320903258576
  • 19. SEARCHING FOR EMPIRICAL VALIDITY 19 Garrett, N., Marques, J., & Dhiman, S. (2012). Assessment of business programs: A review of two models. Business Education & Accreditation, 4(2), 17-25. Hair, J. F., Black, B., Babin, B., Anderson, R. E., & Tatham, R. L. (2005). Multivariate Data Analysis, 6th edition. Saddle River, NJ: Prentice-Hall. Harper, J. S., & Harder, J. T. (2009). Assurance of Learning in the MIS Program. Decision Sciences Journal of Innovative Education, 7(2), 489-504. doi:10.1111/j.1540- 4609.2009.00229.x Hess, P. W., & Siciliano, J. (2007). A research-based approach to continuous improvement in business education. Organization Management Journal, 4(2), 135-147. Hu, L., & Bentler, P. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling-A Multidisciplinary Journal, 6(1), 1-55. Kline, R. B. (2005). Principles and Practice of Structural Equation Modeling (2nd ed.). New York: Guilford. LeClair, D. (2012). Broadening our view of Assurance of Learning. Retrieved from http://aacsbblogs.typepad.com/dataandresearch/2012/02/broadening-our-view-of- assurance-of-learning.html Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indices in confirmatory factor analysis: Effects of sample size. Psychological Bulletin, 103, 391-411. Martell, K., (2007). Assurance of learning (AoL) methods just have to be good enough. Journal of Education for Business,(82) 4, 241-243.
  • 20. SEARCHING FOR EMPIRICAL VALIDITY 20 Moskal, P., Ellis, T., & Keon, T. (2008). Assessment in Higher Education and the management of student-learning data. Academy of Management Learning & Education, (2), 269. doi:10.2307/40214542 Nunnally, J. C. (1978) Psychometric theory (2nd ed.). New York: McGraw-Hill. Pesta, B., & Scherer, R. (2011). The Assurance of Learning Tool as Predictor and Criterion in Business School Admissions Decisions: New Use for an Old Standard? Journal Of Education For Business, 86(3), 163. doi:10.1080/08832323.2010.492051 Romero, E. J. (2008). AACSB accreditation: Addressing faculty concerns. Academy of Management Learning & Education, 7(2), 245-255. Sampson, S. D., & Betters-Reed, B. L. (2008). Assurance of Learning and outcomes assessment: A case study of assessment of a marketing curriculum. Marketing Education Review, 18(3), 25-36. Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research 25(2), 173-180. Weber, P., Weber, J. E., Sleeper, B. J., & Schneider, K. C. (2004). Self-Efficacy toward service, civic participation and the business student: Scale development and validation. Journal of Business Ethics, 49(4), 359-369. Wilhelm, W. J., & Czyzewski, A. B. (2012). Ethical reasoning instruction in undergraduate cost accounting: A non-intrusive approach. Academy of Educational Leadership Journal, 16(2), 131-142. Zocco, D. (2011). A recursive process model for AACSB assurance of learning. Academy of Educational Leadership Journal. 15(4), 67-91.