32 Yeas of Assessment at UMass Boston | February 9, 201232 Years of Assessment atUMass Boston:History, Philosophy and StructuresNeal Bruss & Mark Pawlak, General Education Committee
32 Yeas of Assessment at UMass Boston | February 9, 2012General Education at UMass BostonIn 2002, a new general education program was put in place with thefollowing principles as its foundation▸ Critical analysis and logical thought▸ Verbal and quantitative reasoning▸ Human diversity▸ Principal approaches to knowledgeCore elements include▸ Writing--English Composition I & II▸ First Year & Intermediate Seminar▸ Mathematics/Quantitative Reasoning▸ Distribution courses in several areas▸ Writing Proficiency Requirement▸ Capstone in major
32 Yeas of Assessment at UMass Boston | February 9, 2012Assessing Student Learning in QuantitativeReasoning:A Dynamic, Evolving ProcessMark PawlakDirector, Academic Support and Quantitative Reasoning ProgramsChair, Quantitative Reasoning Assessment CommitteeUniversity of Massachusetts Boston
32 Yeas of Assessment at UMass Boston | February 9, 2012The Mathematics/Quantitative Reasoning Requirement▸ All students must “demonstrate competence inmathematics/quantitative reasoning.”▸ B.S. students must take Calculus I (a traditional Calculus course)▸ B.A. students have several choices:– Test into PreCalculus or Calculus– Take Statistics (taught by Math, Psychology, Sociology,Economics, etc.)– Take College Algebra– Take a Quantitative Reasoning course▸ Each semester, approximately 200 - 250 students choose a QR course.
32 Yeas of Assessment at UMass Boston | February 9, 2012Quantitative Reasoning CoursesMath 114Q▸ Mathematics offers the primary QRcourse▸ This is the lowest-level course Mathoffers▸ Prerequisite: (outdated) placement test▸ All sections are taught in a computer lab▸ QR faculty are primarily adjuncts (90%)▸ Class size is small (20 - 23 students)▸ Many students come out ofdevelopmental math courses and are mathphobic; have weak skills; hate math▸ All topics are motivated using real worlddata, course follows an investigationspedagogy▸ Technology is used as a tool and as wayof seeing patterns.QR CourseGuidelines▸ Required topics are:descriptive statisticslinear modelsexponential models or probabilityuse of technology (graphingcalculators or computers)▸ Students learning outcomes:engage in critical reading andanalysisspeak, listen and write effectivelyuse technology to further learningwork independently andcollaborativelyreason logically and quantitatively
32 Yeas of Assessment at UMass Boston | February 9, 2012Quantitative Reasoning Assessment Committee(QuAC)▸ A sub-committee of the Faculty Council General EducationCommittee▸ Cross-disciplinary membership▸ Reviews and recommends new QR courses▸ Ensures that intended capabilities are addressed in current QR courses▸ Assesses teaching and student learning in current QR courses▸ Monitors student compliance with QR requirement▸ Reviews compliance with QR requirement across colleges (since2004)
32 Yeas of Assessment at UMass Boston | February 9, 2012Three Phases of QR Assessment at UMassBoston▸ I. Pilot Phase (1998-2000): Curriculum development &course modification via student questionnaire & facultydebriefing▸ II. Implementation: (2000-2003): Evaluation of faculty &teaching using student portfolios.▸ III. Current:(2003-ongoing): Assessment of student learningusing common final exam and student questionnaire.
32 Yeas of Assessment at UMass Boston | February 9, 2012Student self-assessmentSelf-reflection:▸ Students complete an “automathography” at the beginning ofthe semester and a reflection at the end of the semester – thisis viewed only by the instructorCourse assessment and evaluation:▸ Student questionnaire given to all students▸ Demographic data▸ Self-assessment of technical skills and QR skills▸ Attempts to measure attitudinal change▸ Administered online with support from the mathematicsdepartment▸ Faculty log in to view their students’ responses and course-wide responses.
32 Yeas of Assessment at UMass Boston | February 9, 2012Sample Questions – Student Questionnaire
32 Yeas of Assessment at UMass Boston | February 9, 2012Faculty self-assessment▸ QR faculty meet each semester to “de-brief”▸ Discussions include▸ What worked▸ What didn’t work▸ Issues of support and training▸ Student learning outcomes and course objectives▸ Assessment information▸ Resources and information
32 Yeas of Assessment at UMass Boston | February 9, 2012Portfolio reading▸ Each semester, new faculty and selected veteran facultysubmit a sample of portfolios of student work.▸ Portfolios include student “automathography” and end ofsemester reflection; evidence of data analysis; written work;final exam▸ The QR Assessment Committee reads the portfolios and givesfeedback to the individual faculty.▸ Results are used to enhance faculty training and development.(Note: In recent years portfolio reading has been abandoned for lack ofresources to support it)
32 Yeas of Assessment at UMass Boston | February 9, 2012Holistic grading of common final exam problems▸ QR faculty agree on a set of problems that will appear on allfinals across sections▸ Generally 5-6 problems, covering main student learningoutcomes▸ After semester grades have been submitted a sample of finalsfrom each section is graded holistically▸ Initially, holistic grading was done by QuAC members▸ Inspired by a PKAL conference on assessment, QR faculty nowparticipate in the holistic grading▸ End of semester QR faculty debriefing and holistic examgrading are now combined.
32 Yeas of Assessment at UMass Boston | February 9, 2012Common Final Exam sample problem1. An article about the US Postal Service in the New York Times on December 4,2011 stated thatFirst-class mail — items like bills and letters that require a 44-cent stamp —fell 6.6 percent in 2010 alone, continuing a five-year-long plunge. Last year …there were 9.3 billion pounds of ‘standard mail’ — the low-cost postagecategory available to mass advertisers — but only 3.7 billion of first-class mail.www.nytimes.com/2011/12/04/sunday-review/the-junking-of-the-postal-service.html)a. A graphic next to the article said that standard mail in 2010 amounted to 30.3pounds for every adult and child in the US. Verify this 30.3 pounds per personfigure.b. How many pounds of first class mail did the Post Office deliver in 2009?c. First class mail mostly consists of bills, credit card statements, personal lettersand greeting cards. First class postage is 44 cents for the first ounce and 20 centsfor each additional ounce. Estimate the total cost of the postage on first class mailin 2010.
32 Yeas of Assessment at UMass Boston | February 9, 2012Scoring Rubric – sample question2 = full understanding; 1 = partial understanding;0 = little or no understandingmean medianDemonstrated ability to identify and extract relevant datafrom complex verbal text and apply it to problems.1.4 2.0Able to verify textual claims through an appropriate set ofcalculations.1.0 1.0Accurately performed backward percentage calculation todetermine a prior year value.0.9 1.0Performed a complex calculation involving estimation andmultiple unit conversions.1.0 1.0
32 Yeas of Assessment at UMass Boston | February 9, 2012Phase III Outcomes:What we have learned▸ Students appreciate how math (algebra) can be meaningful in the realworld▸ Students gain proficiency in using Excel and value it for use in futurecoursework & employment▸ Student writing about data is more descriptive than analytic▸ Faculty uniformly address QR topics and competencies with variedemphasis▸ Some mathematics topics remain challenges for teaching and learning
32 Yeas of Assessment at UMass Boston | February 9, 2012Assessing QR AssessmentLessons▸ Importance of continualevolution and refinement ofinstruments and methods▸ Adequate resources for facultyrelease time & administrativesupport are needed to conductassessment; collect and analyzedata▸ Importance of faculty involvedin modifying course/programbased on results of assessmentChallenges▸ Timely compilation of commonfinal exam▸ Tardy or negligentadministration of questionnaire▸ Compiling questionnaire data;timely feedback to faculty▸ Conducting timely holisticgrading; assessing results anddiscussing with faculty▸ Faculty buy-in to course andprogram modifications based onassessment
32 Yeas of Assessment at UMass Boston | February 9, 2012Resources▸ The National Numeracy Network: http://serc.carleton.edu/nnn/index.html▸ SIGMAA on Quantitative Literacy: http://sigmaa.maa.org/ql/▸ Common Sense Mathematics (UMass Boston)http://quantitativereasoning.net/See also:▸ “Quantitative Reasoning at the University of Massachusetts Boston,” M. Mastand M. Pawlak, in Current Practices in Quantitative Literacy, edited by R.Gillman, MAA publications, 2006Contacts:▸ email@example.com▸ firstname.lastname@example.org