This document summarizes a presentation about big data analysis on student learning and course evaluation at Waseda University in Japan. It discusses how accountability in higher education has increased in Japan due to globalization and demographic changes. Waseda University established the Center for Higher Education Studies to strengthen institutional research functions and use data analytics to support decision making. As a case study, the presentation analyzes data from an integrated data warehouse on student time spent studying and course grades. It argues this type of analysis can open a dialogue about benchmarks for student learning and theories of education within higher education.

1. Big Data Analysis on Student Learning &
Course Evaluation in Waseda University, Japan
YAMAGISHI, Naoji
Center for Higher Education Studies (CHES)
Waseda University, Tokyo, Japan
May 20-21, 2016
@ Asia University, Taichung, Taiwan

2. 1
Outline
1. Accountability and its Backgrounds in Japan
• Grand Transformation in Japanese Higher Education
• Japan’s Policy Issues
• H.Ed.’s Responses
2. Waseda’s Strategies: CHES and its IR Functions
• Roles of CHES
• CHES’s IR Functions
• CHES has done and will do
3. A Case of Data Analysis
4. Issues: Chance to Open a Dialogue
Yamagishi

3. １. Accountability: Grand Transformation in Japanese H.Ed.
2
Global Competition & Knowledge-Based Economy
• Ask me my three main priorities for government…education, education and
education (Blair, Oct. 1, 1996)
 H.Ed. has become the key for economic success for nations, organizations, &
individuals.
Quality of H.Ed. has become EVERYONE’s stakes.
Japan’s Specific Background : declining in the # of 18 years old and increasing
the # of institutions falling below their quota
• 18-year-old population was about 2million in 1992, but declined to about
1.2million in 2011.
• the # of private institutions was 372 in 1990 → 605 in 2012
• among private 4-year-colleges, about 45% falls below their quota
• seating capacity (# of enrolled/ # of applicants) = 92%
• about 46% were admitted without admission tests (e.g.) recommendation
Knock, and the (college) door will be opened to you (Matthew 7:7-8 ).
Larger investment on H.Ed., but ambiguous quality →Are college graduates
sufficiently educated?
Yamagishi

4. １. Accountability: Grand Transformation in Japanese H.Ed.
3Yamagishi

5. １. Accountability: Grand Transformation in Japanese H.Ed.
4
Yamagishi
The # of applicants
(thousand)
100% & beyond
(left axis)
The # of private
institutions
• Applicant-to-seat ratio and the number of applicants (private inst.)
The # of applicants
(right axis)
Above 50% below 100%
(left axis)
Below 50%
(left axis)
Source:https://www.jri.co.jp/MediaLibrary/file/seminar/110602_359/handout_1
10602_359_01.pdf

6. １. Accountability: Japan’s Policy Stance
5
Deregulation of University Act (1991) :
• allowed H.Ed. Inst. to have considerable discretion over curriculum design,
degree-grating criteria, faculty placement, etc.
Quality Assurance through Accountability on Student Learning
• Self-assessment & evaluation was stipulated as task for each inst. (University Act,
1991)
• Third party evaluation became compulsory for all H.Ed. Inst. (2004)
 Institutional as well as program recognition
• Central Council for Education’s report to the government
 Focus on student learning (2008)
 Establish institution-wide academic management and PDCA cycle (2012)
 Annually open results of assessment of student learning to the public (2012)
• Enforcement Regulations for the School Education Law (2011)
 Mandatorily disclose 9 indicators of institution-wide information
• Public financial support (by MEXT): total budget ￥20 billion（NTD6 billion) for
300 institutions for improving education quality that includes establishing
institution-wide academic management and utilizing student course evaluation
Yamagishi

7. １. Accountability: H.Ed. Responses
6
National Survey 1 (MEXT, 2014) (n=766 institutions <response rate 100%>)
• carry out student course evaluation: 80% institutions（2008）→95％（2012)
• open the aggregated results to the public: 18%
• open the aggregated results to students and faculty only: 39%
• not open at all (1%)
National Survey 2 (The University of Tokyo, 2014) (n=555 institutions <response rate
71%>)
• institution-wide IR office (30%)
• IR related office (25%)
• function of IR: provide information to the university executive office (66%), get
ready for the third party evaluation (53%), analyze student course evaluation
(42%) , analyze GPA (39%), evaluate faculty development program (25%)
• Who can access to data:
 student course evaluation data: executive office (20%), IRer (14%), each
school/department (93%)
 student related data (e.g. GPA): executive office (18%), IRer (14%), each
school/department (98%)
Yamagishi

8. CHES
2．Waseda’s Strategies: Roles of CHES
7
Vision 150
In 2012, Waseda University established “Waseda Vision 150”, displaying a road map to be
followed until its 150th anniversary in 2032.
13 Core Strategies
encompass the entire range of topics upon which universities are operated, including the admissions process,
education, research, international initiatives, pioneering new fields, and university management.
The Center for Higher
Education Studies (CHES) was
established in February 2014,
in order to promote the
autonomous and sustainable
reform of Waseda that
contributes to its improving
the quality of education,
research and management.
Center for Higher
Education &
Institutional
Research
(CHEIR)
Center for Teaching,
Learning, &
Technology
(CTLT)
Yamagishi

9. KEY
ACTIVITIES
2．Waseda’s Strategies: Roles of CHES
8
• CHES’s director is the
university’s provost.
• CHES reports to its
management
committee.
• CHES develops various
projects around 5 key
areas that are
connected to 4 of 13
Core Strategies.
Realizes Waseda Vision 150
Strengthening
IR Functions
Higher
Education
Research
Supporting
Teaching
and
Learning
Developing
Education
Methods
Disserminating/
Developing
Good
Practice
Yamagishi

10. 2. Waseda’s Strategies: CHES’s IR has done & will do
 has done….
CHES
IR
Admissions
Center
Educational
Affairs
Management
Planning
Center for
Research
Strategy
Board
Executive Office
22 Schools
Research Institutes, Centers, Others
• built up Integrated Data Ware
House (IDWH) with IT Strategies
Division (March, 2016)
• formulated the guidelines for
managing IDWH (March,2016)
• organizes IRer liaison committee
(every other month) including
Admissions Center, Educational
Affairs Division, etc.)
Yamagishi
• support/consult with the schools
and centers on data analysis
• open the results to the university
community and the public
• provide frameworks for data
analysis
• educate IRers at each school &
center
• and so on, so on….
 will do….
9

11. 2. Waseda’s Strategies: CHES’s IR has done & will do
 Four Purposes & Roles of IR (Volkwein, 1999)
A: To describe the inst. – IR as information authority
B: To analyze alternatives – IR as policy analyst
C: To present the best case – IR as spin doctor
D: To supply impartial evidence of effectiveness – IR as scholar and researcher
Yamagishi
• CHES IR has done A, and will also intend to take roles of B, C, D
10
Purposes & Audiences
Formative &
Internal – for
improvement
Summative &
External – for
accountability
Organizational
Role
& Culture
Administrative &
Institutional
Α C
Academic &
Professional
B D
 5 Developing Stages of IR as a change agent (Swing, 2009)
1. Build Awareness
2. Develop Focus
3. Increase Knowledge
4. Resolve to Change
5. Incorporate or Replace
CHES IR

12. 3. A Case of Data Analysis
11
OBJECTIVS: The target audience was the internal stakeholders.
• internally publicized functions of IR & IR’s possible contributions to evidence
based decision making
 DOR: CHES & IT Strategies Division cooperatively worked together.
• Data Cleansing: IT Strategies Division
• Data Mining & Analysis: CHES
• NEC Corporation provided technical support.
 DATA: was retrieved from IDWH & Waseda-net Course N@vi ®.
• IDWH includes student-related data (e.g., ID, age, sex, course grade, GPA) and
course-related data (e.g., class-sizes, types, subjects, compulsory/elective
requirements, designated year)
• Waseda-net Course N@vi ® includes student course evaluation data for each
student (ID).
 Course N@vi is Waseda’s original LMS introduced in 2007
 The registered # of courses was over 13,000 (82%) , and the rate of use by
faculty was 99% in 2015.
Yamagishi

13. 3. A Case of Data Analysis
Distribution （%） of time spent in studying outside class w/n each letter grade
• Data from one of the schools
• n=9,002 (2012～2014）
Yamagishi
0%
10%
20%
30%
40%
50%
60%
70% ＜30min
30min～1hour
1～3hours
3hours＜
＜30min
30min～1hour
1～3hours
3hours＜
＜30min
30min～1hour
1～3hours
3hours＜
＜30min
30min～1hour
1～3hours
3hours＜
＜30min
30min～1hour
1～3hours
3hours＜
A+ A B C F
12

14. 4. Issues: Chance to Open a Dialogue
13
 Open a dialogue with external stakeholders
• serious concern for quality of college education
 push for more rigorous evaluation of academic achievement (GPA)
 push for increase in time for studying out of class
• First of all, what makes people invest time in studying?
 Do college grades worth something in Japan?
 Espoused Theory vs. Theory-in-Action (Argyris and Schon, 1978; Argyris, 1999)
 Espoused Theory: H.Ed. can educate people.
 Theory-in-Action: H.Ed. is merely a device for screening people.
 Open a dialogue within H.Ed. Community
• What is the espoused theory and theory-in-action in H.Ed.
• Does H.Ed. really function as a place for education?
• Does H.Ed. seriously check-in what, how, how much students learn?
• H.Ed. needs institution-wide, community-wide benchmarking data about student
learning.
 Much to work for IRer as an analyst, reporter, advocate, and communicator!!
Yamagishi

15. Bibliography
14
Argyris，C．and Schon，D．A．(1978) Organizational learning: A theory of action perspective．
Addison-Wesley．
Argyris，C. (1999) On organizational learning. (2nd ed)． Wiley-Blackwell．
Central Council for Education. (2008) Gakushikatei Toshin no kouchikunimukete
(Toshin)［Toward establishing undergraduate education (a report of an inquiry)］.
Central Council for Education. (2012) Aratana mirai wo kizukutame no daigakukyouiku no
shitsu teki tenkann ni mukete (Toshin)［Toward quality transformation of college education
for establishing a new future (a report of an inquiry)］.
Ministry of Education, Culture, Sports, Science and Technology. (2014) Daigaku ni okeru
kyouiku naiyo to no kaikaku jyoukyou nitsuite［Regarding collegeeducation reform］.
Oki, K. (2011) Nihon no shiritsu daigaku ni okeru institutional research (IR) no dokou
[The trend of institutional research in Japanese private universities]. Daigaku Hyoka
Kenkyu, 10, 37-45, 2011.
Swing, R. L. (2009) Institutional researchers as change agents. New Direction for Institutional
Research, 143, 5-16.
The University of Tokyo. (2014). Daigaku ni okeru IR no genjyo to arikata ni kansuru
cyousakenkyu [Investigation & research regarding the present and future of college IR].
Volkwein, J. F. (1999) The four faces of institutional research. New Direction for Institutional
Research, 104, 9-19.
Yamagishi