• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
An Inventory of Conventional and Technology-Enabled Direct and Indirect Assessment Methods
 

An Inventory of Conventional and Technology-Enabled Direct and Indirect Assessment Methods

on

  • 340 views

This document contains brief descriptions and examples of direct and indirect assessment methods and techniques.

This document contains brief descriptions and examples of direct and indirect assessment methods and techniques.

Statistics

Views

Total Views
340
Views on SlideShare
340
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft Word

Usage Rights

CC Attribution-NonCommercial LicenseCC Attribution-NonCommercial License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    An Inventory of Conventional and Technology-Enabled Direct and Indirect Assessment Methods An Inventory of Conventional and Technology-Enabled Direct and Indirect Assessment Methods Document Transcript

    • An Inventory of Conventional and Technology-Enabled Direct and Indirect Assessment MethodsDirect MethodsConventional or Technology-Enabled Standardized Instruments: Tests and InventoriesDescription: Historically, standardized instruments, such as objective tests, have servedas the primary direct method to assess student learning. Content or disciplinary expertsidentify the standard content, knowledge and tasks that students should know and be ableto perform. In addition, they determine what constitutes levels of achievement based onthe construction, design, and sequencing of questions or prompts. Students’ achievementis referenced against other student groups’ achievement on the same instrument, referredto as norm-referencing. Primarily designed to make decisions about students,standardized instruments perform a gate keeping role. They certify competence in aprofession or field, such as in the case of licensure examinations for nursing students, orprogram-level knowledge or skills, such as in the case of general education tests. Resultsare also used to place student in appropriate courses or to identify level of achievement atpoints in students’ studies, such as in the case of rising junior examinations.What They Provide: - Content and tasks developed by external experts within fields or 1
    • programs of study -Psychometric approach to assessment that values quantitative methods of interpreting student achievement -Evidence of what students know or can do within the universe and framework of questions, prompts, and tasks of an instrument - Evidence to make gate-keeping decisions such as professional certification or end-of-study achievement or to meet state mandates that valuemeasurement as proof of learning -Evidence to track student learning if instrument can be used formatively and if results have utility for programs and the institution -Quick and easy adoption and efficient objective scoring -History of validity and reliability studies -One possible source of evidence within an institutional commitment to assessing student learning through multiple lensesWhat Standardized Instruments Do Not Provide: 2
    • -Evidence of the strategies, processes, and ways of knowing, understanding, and behaving that students draw upon or apply to represent learning -Evidence of the complex and diverse ways in which humans construct and generate meaning -Alignment with institution- and program-level learning outcome statements and students’ learning histories -Realistic timeframes or contexts that reflect how humans solve problems, seek additional information or resources, correct mistakes, or reposition their thinking. Students respond within a timeframe that might wellaffect their decisions or actions, such as deciding to make a last-minute guessamong options in a question. -Highly useful results that directly relate to pedagogy and educational practices. Results relate to the construct of the instrument itself and what it is designed to measure. Patterns of student performance reported in scales or scores identify discrete areas of performance, such as a skill level, that identify strengths and weaknesses in curricular or co-curricular attention.However, these patterns do not assist in learning about why students responded in 3
    • the ways they did. Or did they learn successful strategies for selecting ormaking “good guesses?”Some Examples: Instruments that test general education knowledge and abilities include the following: * Academic Profile: http://www.ets.org/hea/acpro * Collegiate Assessment of Academic Proficiency (CAAP): http://www.act.org/caap/index.html * Collegiate Learning Assessment (CLA) * MAAP Instruments dedicated to measuring specific skills include: * ACCUPLACER : (http://www.collegeboard.com/highered/apr/accu/accu.html * Watson-Glazer Critical Thinking Appraisal * California Critical Thinking Skills Test * Tasks in Critical Thinking * Reflective Judgment Inventory * Measure of Intellectual Development * e-Write, a component of ACT’s COMPASS/ESL systemExamples of achievement tests in a particular field of study or profession include: * Graduate Record Examinations Subject Tests: 4
    • http://www.gre.org/pbstest.html#testreg * The PRAXIS Series: Professional Assessment for Beginning Teachers: http://www.ets.org/praxis/index.html * Area Concentration Achievement Tests: www.collegeoutcomes.com * Graduate Management Admission Test: http://www.mba.com/mba Examples of inventories that assess students’ knowledge include: include: * Force Concept Inventory (FCI)--designed to assess the initial knowledge state of students before they begin undergraduate physics courses. Studentsrespond to a series of force-related problems or statements that reveal their knowledge (Halloun and Hestenes,1985). Knowledge surveys Locally Designed Tests and InventoriesIn a collective and shared commitment to assessing student learning, core working groupsmay well determine that no appropriate standardized instruments exists that aligns withinstitution- and program-level outcomes. That decision generates the design of local testsor inventories or use of existing instruments that have an institutional history of providinguseful and reliable results. Technological advancements, such as WebCT 5
    • (http://www.webct.com/transform) and Blackboard(http://www.blackboard.com/products/ls/index.htm) offer an online environment forconstructing some locally developed instruments.What Local Tests or Inventories Provide: -Strong alignment of content and format with learning outcome statements and course-based assessment methods students’ have experienced along their learning histories -Useful results that can be interpreted within the local contexts of teaching and learning and then used to improve student learning -Opportunity to establish local instrument criteria that reflect what an institution and its programs value in educational practices. -Opportunity for faculty, staff, administrators, students, teaching assistants, tutors, intern advisors, advisory board members, and institutionalresearchers, for example, to contribute their perspectives on what should beassessed and how it should be assessed.What Local Tests or Inventories Do Not Provide -Immediate reliability and validity results that verify content, construct, format and consistency in scoring, unless instruments have been pilot tested and evaluated over several semesters. Thus, time to pilot test aninstrument for an institution’s representative student populations is anecessary component of an institutional commitment to assessing for learning. 6
    • Conventional or Technology-enabled Authentic Performance-based MethodsAuthentic performance-based methods prompt students to represent their learning inresponse to assignments and projects that are embedded into their educationalexperiences. These methods value divergent thinking and responding, as opposed toconvergent thinking, most typically represented in standardized tests. Focusing on howstudents think, problem solve, react, interpret, or express themselves becomes the focusof these kinds of direct methods. Further, these methods can easily be embedded intostudents’ continuum of learning, providing evidence of their growth over time oftendemonstrated in self-reflective writing and responses to feedback from those whocontribute to their education, such as peers, internship advisors, external reviewers orevaluators, faculty and staff. What They Do Provide: -representation of integrated learning -direct alignment with students’ learning experiences -opportunities for students to reflect on and receive formative feedback about their learning and development -student-generated opportunities to demonstrate learning, as opposed to test- generated occasions _opportunities for students to create as in multi-media contextsWhat They Do Not Provide: -Easily quantifiable evidence given the complexity they capture -Efficient scoring opportunities 7
    • Range of Authentic Assessments: • Portfolio or e-portfolio—collection of multiple kinds of student-generated texts stored electronically or in paper form. Developments in technology now make it possible for students to create digital portfolios. This method of storage and collection provides a longitudinal representation of learning, demonstratinghow students make meaning within their contexts for learning through assignments, projects, narrative self-analyses and self-reflection. They provide an opportunity for students to demonstrate how they integrate learning over time from multiple learning experiences and opportunities within the curriculum,co-curriculum, and learning experiences that extend beyond their formal educational experiences. They provide a valuable source of evidence for institution- and program-level learning by providing a range of texts that represent student learning. Webfolios, digital learning records, or electronic learning records are other current terms for this method. The development of 2.0 electronic portfolios offers students a context to demonstrate their learning processes as well as learning products. Two major national resources that focus on technological developments in electronic portfolios are the following: 1. the newly founded global academic organization, Association for Authentic, Experiential and Evidence-based Learning (AAEEBL: http://www.aaeebl.org/) and 2. the community of practice, Electronic Portfolio Action and Communication (EPAC: http://epac.pbworks.com). Cambridge, Cambridge, and Yancey’s 2009 8
    • publication, 2.0 Electronic Portfolios: Emergent Reseearchon Implementation and Impact (Stylus Publishing,LLC), provides a taxonomy of electronic portfolios and describes the potential of 2.0 electronic portfolios For information about Alverno Colleges Diagnostic Digital Portfolio, go to: http://www.ddp.alverno.edu/. • Learning Record Online A running record of students’ learning is the Learning Record Online (LRO). Originally developed for K-12 and now being developed for higher education, the LRO provides formative and summative evidence of students’ learning within their curricula and against agreed upon criteria and standards of judgment. Like a portfolio, the LRO provides ongoing evidence of students’ emerging learning within contexts for learning, including evidence of their reflective process, their metacognition. Facilitated by online technology, educators and others interested in learning about students’ development are able to aggregate or disaggregate groups to draw inferences and make decisions about students’ progress and eventual levels of achievement: . http://www.cwrl.utexas.edu/~syverson/olr/* Capstone project —a culminating independent research, collaborative, or professionalproject at the end of students’ careers that provide evidence of how students solverepresentative higher order disciplinary, professional or interdisciplinary problems oftenrepresented in more than one kind of text, that is in writing as well as in speaking or invisual texts, such as poster presentations. These projects provide evidence of how well 9
    • students integrate, synthesize, and transfer learning; in addition they also can provideevidence of how well students integrate institution-level outcomes. This method can alsobe integrated as a formative means of assessment. The beginning of a second year ofgraduate study or the beginning of a third year of students’ undergraduate study mightindex a time for students to demonstrate accumulated learning. Senior theses or seniorresearch projects are also examples of capstone projects that provide opportunity forobservers to assess students’ masterly level within a field of study, discipline, orprofession.* Performances, products, creations—required over time as well as at the end ofstudents’ studies, their work represents how they interpret, express, and constructmeaning. Traditionally, the arts’ faculty have formatively assessed students’performances to provide them with immediate feedback that shapes their futureperformances. This approach provides not only students with immediate feedback butalso faculty and others who contribute to students’ learning.* Visual Representation--Representing learning visually through charting, graphing, mapping, for example, provides students with alternative ways to represent their learning, often a practice within disciplines such as mathematics and the sciences. Mathematicians often represent developments in their thinking in mind maps:http://jonathan.mueller.faculty.noctrl.edu/toolbox/examples/seaver/mindmappingtask.htm 10
    • Visual representation offers a way to assess how well students make connections orunderstand a concept. In addition, visual representation extends students’ repertoire ofmaking meaning, developing a versatility in forms of representation that respond to thedifferent needs of audiences, contexts, and the purposes of communication. The BiologyTeaching Homepage provides examples of different kinds of conceptual maps, well asprocedures for incorporating them into students learning.http://www.fed.cuhk.edu.hk/~johnson/misconceptions/concept_map/cmapguid.html . TheInnovative Learning Group presents different forms of thinking maps.file://A:ILG%20%20Thinking%20Maps%20Information.htm* Case studies-- used over time, as well as at the end of students’ studies, case studies, often used in business programs, provide opportunity to assess students’ problem- solving abilities within a major program of study or along students’ continuum to determine how well they are integrating the learning expressed in institutional learning outcomes—knowledge, perspectives, abilities, values, attitudes. Parallel case studies used over time provide evidence of students’ abilities to solve representative disciplinary, professional, or more generalized problems. In addition, they provide evidence of student’s writing. Adding to the dimensionality of traditional paper- or document-based case studies are online case studies that incorporate multi-media into the case study and hyperlinks for users to read or see related resources. Harvard’s* Professional or disciplinary practices—engaging students in practices that preparethem for the kinds of problems, activities, or situations individuals address not only in 11
    • their fields of study but also as contributors to society and local communities. The HarrellProfessional Development Center in the University of Florida’s College of Medicine hascreated an environment that permits observation of medical students interacting withpatients. Audio visual equipment captures, records, and displays these interactions.Replaying these interactions provides opportunities for faculty and students to assessstudents’ knowledge, understanding, behaviors and dispositions. file://A:Harrell%20Professional%20Development%20and%20Assessment%20Center.ht*Team-based or Collaborative Projects--With rare exception, humans work with otherhumans during most of their lives in a range of workplace, social, and communityenvironments. Team-based or collaborative projects are direct methods that enableassessment of individuals’ knowledge, understanding, behaviors and attitudes, as well astheir ability to work with others to achieve a final product or solve a problem. Oftenvideotaped, groups of faculty, staff, and students themselves have access to immediateresults that inform students as well as educators. Alverno College’s institutional exampleon page x illustrates assessment that focuses on individual students as well as theircollective achievement. Community service projects provide another source of evidenceof how well students apply or transfer or integrate learning to solve a problem as well asan opportunity for students to self-reflect on how they contributed and what and how theylearned from the experience.Wikis and podcasts represent two technologically-based assessment opportunities:student teams create new work and contributions are date and time stamped to document 12
    • who contributed material to a project. For example, at the end of a program of studyteams of students might collaborate to create a culminating assignment on a wiki site,designed to assess how well majors integrate knowledge, abilites, habits of mind, waysof thinking and knowing.Educational online gaming-Internships or Practica—How students actually apply or transfer their cumulativelearning can be assessed in internship or practica. That is, authentic experiences becomea direct method of assessment, providing opportunity for students to demonstrate thedimensions of their learning within the context of a real environment. A step beyondsimulations, these direct methods assesses how well students translate their cumulativelearning into actual practice.Oral examinations—often used at the end of a professional program or a graduateprogram, as in an oral doctoral defense (University of Arizona, Graduate College:http://grad.admin.arizona.edu/degreecert/ppfoedc.htm), oral examinations provideopportunities for students to represent how well they integrate learning and apply it tosolving a case study or problem, responding to guided questions, or presenting a product.Measurement Research Associates provides some guidelines for conducting effective oralexaminations: http://www.measurementresearch.com/media/standardizedoral.pdf. 13
    • Simulations—are now possible through virtual reality programs such as Second Lifeperhaps the most used virtual reality Virtual labs, virtual patients Data miningmerlot or Ariadne offer studnets that opportunity  Conversion--converts one form of representation into another form of representation, such as into numerical or statistical, scientific, humanistic, sociological, psychological, or artistic forms of representationThose that provide opportunity for students to construct their own interpretaions, oridentity new or unepected problems  Data mining--collects, records, analyzes, and interprets data (Merlot)  Problem identification/solution—identifies a new or unique problem or poses a new or unique way to view or solve a problem (Here’s a problem and the solution; are there other solutions?)  “Conceptual collisions” (Federation of American Scientists)—brings “previously disconnected knowledge sources into contact for the first time.”  Projection: identifies consequences, results, resulting conditions, challenges, actions, responsesSecondLife sites such as Virtual Ability Island, the Theorist Project, and Sci-LandsConventional and Technologically-enabled Indirect MethodsHelpful in deepening interpretations of student learning are indirect methods, methodsthat focus on perceptions of student learning by asking students or others to respond to aset or series of questions. Indirect methods function to complement direct methods ratherthan to substitute for them. 14
    • What Indirect Methods Provide: -Evidence of students attitudes, perceptions, experiences -Evidence that may help to explain student performance levelsWhat Indirect Methods Cannot Provide: -Work that represents evidence of student learning unless an instrument asks students to produce a text as evidenceSome Representative Examples: -Focus groups with representative students to probe a specific issue that may have been identified in a survey or identified in patterns of student performance as a result of formative or summative assessments. Van Aken and collaborators describe the successful design of a focus group to obtain perceptions of historically under-represented students enrolled in an engineering program (1999). -Interviews with groups of students representing the institutional population. Tracking a cohort of students is one way of assessing learning forformative and summative purposes. -Student, alumni, employer and faculty-staff surveys and questionnaires that provide information about students’ or others’ perceptions of students’ educational experiences and the institutions’ impact on their learning. 15
    • Alumni questionnaires and surveys provide a retrospective view ofgraduates’ educational experience and create an opportunity forthem to recommend improvements in education based on what is relevant totheir current employment, profession, or graduate education. Faculty-staff surveys or questionnaires provide perceptions of student learning—that is what students are able and not able to demonstrate based on classroom-based assessments or observations of student behavior. -ACTs surveys for adult learners, alumni, entering students, withdrawing students, and for institutional services: www.act.org -College Student Experiences Questionnaire (CSEQ): Community College Survey of Student Engagement (CCSSE) http://www.ccsse.org/aboutccsse/aboutccsse.html -Community College Student Experiences Questionnaire (CCSEQ): http://www.people.memphis.edu/~coe_cshe/CCSEQ_main.htm - The National Center for Higher Education Management Systems Comprehensive Alumni Assessment Survey: www.nchems.org -National Survey of Student Engagement (NSSE): http://www.iub.edu/~nsse/ 16
    • -Noel-Levitz Student Satisfaction Inventories: http://www.noellevitz.com/library/research/satisfaction.asp#ssi--Incoming Freshman Assessments (such as CIRP):--First-year expericne surveys such as YFCY:-Small Group Instructional Diagnosis (SGID)—a facilitated small group interview process conducted by a trained interviewer who asks students to identify what and how they have learned and what kinds of learningobstacles they are facing or have faced. Used initially at mid-point in acourse to provide feedback to a faculty member about how students areor are not learning, this method can also provide useful formative assessment data at the program-, department-, or institution-levels ,providing students’ first-hand explanations for dominant patterns ofweakness in their work: http://www.miracosta.edu/home/gfloren/sgid.htm--Student Assessment of Their Learning Gains (SALG)—a customizable online end-of-course survey that asks students to rate the components of a course, such as lab work or discussion groups, and the degree to which these components contributed to their learning. This survey could well beused for program-, department-, or institution-level assessment if a sampleof students were asked to rate how well and why the components of 17
    • learning they have been explosed to have contributed to their learning of stated learning outcomes in their program of study . Given that the survey is customizable, institutions can add additional kinds of questions that probe students’ perceptions of their learning. http://www.miracosta.edu/home/gfloren/sgid.htmInventories that track how well students have changed or shifted attitudes, values,sensitivities, perspectices based on self-perception or responses to situations or casestudies.* Some Representative Examples include: --Global Perspective Inventory (GPI)—46-item on-line inventory designed to assess the effect of cultural experiences on learners’ global perspectives through three dimensions: cognitive, intrapersonal and interpersonal (https://gpi.central.edu/) --Intercultural Development Inventory (IDI)-- 44-item inventory based paper and pencil instrument designed to assess the extent of an individual’s intercultural sensitivity along a continuum that ranges from ethnocentrism to ethnorelativism, identifying a person’s ability to shift from denial of difference to integration of difference (http://www.intercultural.org). * A list of current inventories frequently used in student services is available at: 18
    • The Office of Institutional Research and Planning, The University of Texas at Arlington. Student Affairs-Related Outcomes Instruments: Summary Information. Inventories that gauge students’ , such as the Intercultural Development Inventory(IDI) and the GPI that guage the extent to which studnets have shifted or changedattitudes, values or even perspectives representa yet another kind of indirect methodofAssessment that may provide evidence of how much or to what degree students changetheir views based on their learning either in the curriculum or co=curriculum. Often usedas pre- and post-tests in a program or service, these kinds of assessments provideevidence of how much or to what degree students chage their views, attitudes or beliefsExamples of Graphic Organizershttp://www.eed.state.ak.us/tls/Frameworks/mathsci/ms5_2as1.htm#graphicorganizers 19