The 52nd Annual Conference_June 24-26, 2016 @ Osaka University

1. Compare and Contrast of Student Course
Evaluation (SCE) between Waseda and UW
YAMAGISHI, Naoji
Center for Higher Education Studies (CHES)
Waseda University, Tokyo, Japan
Japan Comparative Education Society
The 52nd Annual Conference
June 24-26, 2016 @ Osaka University

2. 1
Outline
1. Accountability and the Backgrounds in Japanese H.Ed.
• Grand Transformation in Japanese Higher Education
• Japan’s Policy Issues
• H.Ed.’s Responses
2. Compare & Contrast between Waseda and UW
• Who is in charge?
• SCE and its features
• Compare & Contrast
3. Issues：What makes the difference？
Yamagishi

3. １. Accountability: Grand Transformation in Japanese H.Ed.
2
Global competitions & Knowledge-based economy
• Ask me my three main priorities for government…education, education and
education (Blair, Oct. 1, 1996)
 H.Ed. has become the key for economic success for nations, organizations, &
individuals.
Quality of H.Ed. has become EVERYONE’s stakes.
Japan’s demographic concern: declining in the # of 18 years old and increasing
the # of institutions falling below their quota
• 18-year-old population was about 2million in 1992, but declined to about
1.2million in 2011.
• the # of private institutions was 372 in 1990 → 605 in 2012
• among private 4-year-colleges, about 45% falls below their quota
• seating capacity (# of enrolled/ # of applicants) = 92%
• about 46% of applicants were admitted without admission tests
Knock, and the (college) door will be opened to you (Matthew 7:7-8 ).
Larger investment on H.Ed., but ambiguous quality. Are college graduates
sufficiently educated?
Yamagishi

4. １. Accountability: Grand Transformation in Japanese H.Ed.
3Yamagishi

5. １. Accountability: Grand Transformation in Japanese H.Ed.
4
Yamagishi
The # of applicants
(thousand)
100% & beyond
(left axis)
The # of private
institutions
• Applicant-to-seat ratio and the number of applicants (private inst.)
The # of applicants
(right axis)
Above 50% below 100%
(left axis)
Below 50%
(left axis)
Source:https://www.jri.co.jp/MediaLibrary/file/seminar/110602_359/handout_1
10602_359_01.pdf

6. １. Accountability: Japan’s policy stance
5
Deregulation of University Act (1991) :
• allowed H.Ed. Inst. to have considerable discretion over curriculum design,
degree-grating criteria, etc.
Quality Assurance through Accountability on Student Learning
• Self-assessment & evaluation was stipulated as task for each institution.
(University Act, 1991)
• Third party evaluation became compulsory for all H.Ed. Inst. (2004)
 Institutional as well as program recognition
• Central Council for Education’s report to the government
 Focus on student learning (2008)
 Establish institution-wide academic management and PDCA cycle (2012)
 Annually open results of assessment of student learning to the public (2012)
• Enforcement Regulations for the School Education Law (2011)
 Mandatorily disclose 9 indicators of institution-wide information
• Public financial support (by MEXT): total budget ￥20 billion for 300 institutions
for improving education quality that includes establishing institution-wide
academic management and utilizing student course evaluation)
Yamagishi

7. １. Accountability: H.Ed. Responses
6
National Survey 1 (MEXT, 2014) (N=766 institutions <response rate 100%>)
• carried out SCE: 80% institutions（2008）→95％（2012)
• opened the aggregated results to the public: 18%
• opened the aggregated results to students and faculty only: 39%
• not open at all (1%)
National Survey 2 (The U of Tokyo, 2014) (n=555 institutions <response rate 71%>)
• Who can access SCE data: university executives (20%), IRer (14%), each
school/department (93%)
 Widely spread, but what is in need? (Central Council for Education, 2012)
• The most commonly used tool for grasping student learning is SCE & portfolio :
36.5% for presidents (n=684), 40.1% for deans (n=1,929)
• Important thing for improving education program is to establish a supporting
system: 91.6% presidents (most important + important) presidents (n=684) said
 How to manage SCE and how to use SCE results becomes a significant issue.
Yamagishi

8. 2. Compare & Contrast: Waseda & UW
7
The University of Washington as a pioneer of SCE
• UW opened its door in 1861.
• UW is publicly run, and is one of “Public Ivys.”
• 7 undergraduate schools and 16 graduate schools
• # of enrolled students: about 45,000
• SCE began in the U.S. during about 1920’s (Fujita & kawashima, 2014).
• UW was one of the three pioneers of SCE (Fujita & kawashima, 2014).
Waseda University
• Waseda established in 1882.
• Waaseda is a private institution.
• 13 ungraduated schools and 21 graduate schools
• # of enrolled students: about 53,000
• SCE at Waseda began in 2001.
• Waseda revised its SCE in 2015, by referring to UW’s
Yamagishi

9. 2．Compare & Contrast: Waseda & UW
8Yamagishi
Who is in charge of SCE: Waseda
Who is in charge of conducting SCE Center for Higher Education Studies (CHES)
Who develops/revises of SCE CTLT Faculty WG within CHES
Who analyzes the data an outside contractor
On organizational chart CTLT is a committee within CHES
The director of CHES is the Senior Executive Vice President for Academic Affairs &
Provost
Expected roles
・executes course evaluation
・reports the results (both school and course level) to each relevant
insturctor
・reports the results (both school and course level) to the evalution
committee of each relevant school
・reports to the aggregated results (insitutional level) to the committee of the
heads of academic affairs of each school
・opens the aggregated results (insitutional level) to the public on the
university HP
Staff
・8 administrative staff, 2 of them (collateral office with Academic Affairs Division) in
charge of SCE
・will be replaced by others thourgh job rotation

10. 2．Compare & Contrast: Waseda & UW
9Yamagishi
Who is in charge of SCE: UW
Who is in charge of conducting SCE Office of Educational Assessment (OEA)
Who develops/revises SCE OEA
Who analyzes the data OEA (IASystem)
On organizational chart is part of Undergraduate Academic Affairs
reports to the Vice Provost & Dean for Undergraduate Academic Affairs
Expected roles provides course evaluation service at UW & to other insts.
provides placement testing & classroom test scoring at UW Seatle Campus
provides different types of assessment at all levels at UW
Staff 16 staff members
Some are resarch specialits in quatitative & qualitative methods
(e.x.) the director with Ph.D. in Education (Measurement & Evaluation)
(e.x.) research staff with Ph.D. or graduate degrees in Psychometrics
Service rates for departments at UW
・free of charge for online
・relative to cost of service for paper ($.26/form)
Evaluation forms $.10/form
Optical scannning $.16/from
Billed to departments
Service rates for outside UW (online) Based on the # of enrollments
(e.x.) Enrollment category 1 (total FTE <2,000)
・$1.25 (annual licensing fee/ enrolled student)
・$3,500 (initial setup fee)
(e.x.) Enrollment category 6 (total FTE >20,000)
・$1.00 (annual licensing fee/ enrolled student)
・$3,500 (initial setup fee)
(e.x.) 40,000 FTE $1.00*40,000+$3,500=$43,500

11. 2．Compare & Contrast: Waseda & UW
10Yamagishi
SCE and its features: Waseda
History began in 2001
online evaluation was introduced in 2004
currently both paper and online types are used
# of courses evaluated about 12,000 courses annually
Functions assists faculty in instrucational development
helps schools to know and improve its quality of education
provides schools with reference information for teaching award
Types of forms
4 forms
・Lecture, Seminar, Skill Acquisition & Lab, Distance Learning
Form structure 4 sets of items on a given evaluation form
1set. course registration requirement Compulsory/Required, Compulsory Electives, Electives
2set. students' self-evaluation
(e.x.) Indicate the time you spent on preparation and review for each course
outside of the class.
3set. students' reviw on course
(e.x.) The instructor effectively facilitated your participation in the form of
questions and constructive comments.
4set. open ended questions
Please state below any ideas you can think of in order for this course to be
more beneficial for future students.
Access to the results Instructors
School Evaluation Committee at each school

12. 2．Compare & Contrast: Waseda & UW
11
Yamagishi
SCE and its features: UW
History began in 1920's
Instructional Assessment System (IASystem) was created in the early 1970's
online evaluation was introduced in autum 2013
currently both paper and online types are used
# of courses evaluated about 12,000 courses annually
more than 60 other post-secondary institutions use IASystem
Functions assists faculty in instrucational development
informs administrative decision-making in such as merit, promositon, and tenure
helps students to select courses
Types of forms 14 forms (e.x.) Small Lecture, Seminar, Lab, Distance Learning, Project, and such
Form structure 4 sets of items on a given evaluation form
1set. Summative items the first four items, common in all forms
(e.x.) The course as a whole was
2set. Formative items about 18 items related to particular instructional forms
(e.x.) Quality of questions or problems raised by instructor was
Instructors can add their own questions
3set. Relative to other courses the last nine items, common in all forms
student academic challenge and engagement
(e.x.) The amount of effort you put into this course was
4set. open ended questions (e.x.) What aspencts of this class detracted from your learnig?
Access to the results: Everyone with UW Net-ID
(e.x.)
Students, Instructors, School deans and department chairs,
Faculty Curriculum Committee, Tenure/Promostion Committee, Executives (Provost/President)

13. 2．Compare & Contrast: Waseda & UW
12Yamagishi
Waseda UW
Ability to evaluate courses either online or on
paper
〇 〇
Summative questions on all forms to allow
cross-course and cross-instuructor
〇
（1st set)
Formative questons to provide feedback on
specific aspects of each course
〇 〇
（2nd set)
Index of student academic challenge and
engagement
〇
（3rd set): Challenge & Engagement Index (CEI)
Ability to add instructor generated questions 〇 〇
Integrated data archiving and reporting for
online and paper evaluations
〇
Multi-level reproting by course, department,
and school
〇
Institutional norms for comparison 〇
Online interface for couse evaluation
management
〇
（Waseda net Course＠Navi）
〇
（faculty portal @ UW NetID)
Responses to be connected to an individual
student
〇
（Waseda net Course＠Navi）
〇
（IASystem）
Reponders' anonimity for instructors 〇 〇
Research application ○
(without IRB: Institutional Review Board )
Specialized research staff 〇
Charge × for online,
〇 for paper
Finance ・university budget
・partially state-funded through university budget
・partially self-sustaining

14. 3. Issues：What makes the difference?
13
 Finance
• OEA is partially state-funded and partially self-sustaining.
• OEA can extend their business by selling more SCE service to other institutions.
 Human Resource
• OEA’s HR includes those with strong graduate training in appropriate fields.
 Research
• SCE data can be studied, and the research results can be published without any
commission review.
 Right to Know
• Everyone with UW Net-ID can view on-line Course Evaluation Catalogue
(CEC) ; CEC displays SCE results of each course with its instructor(s)’
name on.
• SCE results are considered as institutional public goods; the members
can share the information.
• The display of evaluation results was originally required by ASUW.
ASUW has paid the cost.
• “Course evaluation data are considered public information, so the faculty
can’t object to display.” Yamagishi