Quality Assurance of Engineering Education by Outcome Based Evaluation

1,367 views

Published on

Seminar on “Accreditation of Engineering Higher Education in Indonesia and the role of IABEE”
DIKTI, Jakarta, January 29, 2014

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,367
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
53
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Quality Assurance of Engineering Education by Outcome Based Evaluation

  1. 1. Seminar on “Accreditation of Engineering Higher Education in Indonesia and the role of IABEE” DIKTI, Jakarta, January 29, 2014 Quality Assurance of Engineering Education by Outcome Based Evaluation Mitsunori Makino, Dr. Eng. Committee Chair, Criteria, JABEE Professor Program Leader, Rubric-based Competency Development, Faculty of Science and Engineering, Chuo University Copyright © 2014 Mitsunori Makino. All rights reserved. 1
  2. 2. Self Introduction of Mitsunori Makino • Graduated from Waseda University – B.Eng. in 1987, M.Eng. in 1989, Dr. Eng. in 1992 • Faculty member of Chuo University since 1992 – 2004- : Professor – 2008- : Leader of Rubric-based Competency Development Project, Faculty of Science and Engineering, Chuo University, (www.facebook.com/ChuoCompetency) – 2009-2013: Vice Dean, Faculty of Science and Engineering • JABEE – Chairperson of Criteria Committee, JABEE (www.jabee.org) Copyright © 2014 Mitsunori Makino. All rights reserved. 2
  3. 3. Outline • • • • What is “outcomes-based” ? Quality assurance of higher education IEA graduate attributes for Washington Accord and JABEE’s criteria Rubric for outcomes-based evaluation Case study: the rubric-based competency development program in Chuo University • Conclusion Copyright © 2014 Mitsunori Makino. All rights reserved. 3
  4. 4. What is “outcomes-based” ? Quality assurance of higher education Copyright © 2014 Mitsunori Makino. All rights reserved. 4
  5. 5. What is quality assurance, especially in higher education? • The maintenance of a desired level of quality in a service or product, especially by means of attention to every stage of the process of delivery or production. from Oxford Dictionary (http://www.oxforddictionaries.com/definition/english/quality-assurance?q=quality+assurance) • In the higher education… The maintenance of a desired level or higher of quality in graduates by means of attention to every stage of the education system Copyright © 2014 Mitsunori Makino. All rights reserved. 5
  6. 6. Why do we need the quality assurance in engineering education by the third party? (1/2) • The 1st step to professional Practice Training and Experience Graduate Attributes: Indicate that program objectives are satisfied Accredited Program Observe Code of Conduct and Maintain Competence Meet Standard for Professional Competency Meet Standard for Engineering Education Copyright © 2014 Mitsunori Makino. All rights reserved. 6
  7. 7. Why do we need the quality assurance in engineering education by the third party? (2/2) • The substantial equivalence in the world: Applied to educational programs means that two programs, while not meeting a single set of criteria, are both acceptable as preparing their respective graduates to enter formative development toward registration. Signatory A Accredited Program Practice Training and Experience Substantial Equivalence Signatory B Accredited Program Copyright © 2014 Mitsunori Makino. All rights reserved. 7
  8. 8. How can we show the quality of engineering education? • Contents, environment – Book of Knowledge Benchmarks for what? • Environment – Budget – Facilities • Outcomes Necessity of “educational objective” & “performance criteria” – Skills & competencies of graduates based on professional knowledge • • • • Project/Research output (accepted papers, patents...) Enrollment numbers in professional qualification Employment as engineer … Copyright © 2014 Mitsunori Makino. All rights reserved. 8
  9. 9. Educational outcomes must be inevitably given, not happen. • Educational program is planned in advance, connected with its goal and/or objectives, about graduates’ performance and knowledge (not “what is taught” but “what is equipped”) • If so, the educational program can be trusted as assured human resource development. (Graduates are expected a certain level of their knowledge and competency) Copyright © 2014 Mitsunori Makino. All rights reserved. 9
  10. 10. Graduate Attributes & Disclosure of Three Policies of Admission, Curriculum and Diploma by MEXT Exemplified graduate attributes: 1. Knowledge and understanding i. ii. 2. Versatile skill i. ii. iii. iv. v. 3. Communication skill Quantitative skill Information literacy Logical thinking Problem solving Each university in Japan has and discloses its diploma policy regarding the attributes Behaviors and intentionality i. ii. iii. iv. v. 4. Understanding of knowledge of multiple/different cultures Understanding of knowledge of human culture, society and nature Self-control Teamwork and leadership Ethics Social responsibility as civil Ability of lifelong learning Integrated learning experiences and creative thinking Copyright © 2014 Mitsunori Makino. All rights reserved. 10
  11. 11. IEA graduate attributes for Washington Accord and JABEE’s criteria Copyright © 2014 Mitsunori Makino. All rights reserved. 11
  12. 12. IEA Graduate Attributes • The 2nd version was released in June 2009, modified in 2013 • Graduates of engineering schools should equip both of “Engineering Knowledge” and “Graduate Attributes”. • These profiles are defined for – Washington Accord (WA): Engineering education – Sydney Accord (SA): Technologist education – Dublin Accord (DA): Technician education Copyright © 2014 Mitsunori Makino. All rights reserved. 12
  13. 13. Washington Accord Membership (as of 2013) Accreditation bodies Provisional status 6 Founding Members Signatory 1989 HKIE (HK) No system at that time 1995 ECSA (South Africa) 1994 1999 JABEE (Japan) 2001 2005 IES (Singapore) 2003 2006 BEM (Malaysia) 2003 2009 ASIIN (Germany) 2003 but was removed in 2013 ABEEK (RP Korea) 2005 2007 IEET (Chinese Taipei) 2005 2007 AEER (Russia) 2007 2012 AICTE (India) 2007 IESL (Sri Lank) 2007 MUDEK (Turkey) 2010 PEC (Pakistan) 2010 COE (Thailand) Submitted in 2010 but was differed BAETE (Bangladesh) 2011 CAST (PR China) 2013 PTC (The Philippines) 2013 Indonesia Interest Peru Interest 2011 Copyright © 2014 Mitsunori Makino. All rights reserved. 13
  14. 14. IEA (Engineering) Knowledge Profile 1. Natural sciences 2. Mathematics, numerical analysis, statistics, formal aspects of computer and information science 3. Engineering fundamentals 4. Engineering specialist knowledge 5. Knowledge supporting engineering design 6. Knowledge of engineering practice 7. Comprehension of the role of engineering in society and identified issues in engineering practice in the discipline 8. Engagement with selected knowledge in the research literature of the discipline Copyright © 2014 Mitsunori Makino. All rights reserved. 14
  15. 15. IEA Graduate Attributes Profile 1. Engineering Knowledge 2. Problem Analysis 3. Design/development of solutions 4. Investigation 5. Modern Tool Usage 6. The Engineer and Society 7. Environment and Sustainability 8. Ethics 9. Individual and Team work 10. Communication 11. Project Management and Finance 12. Life Long Learning Copyright © 2014 Mitsunori Makino. All rights reserved. 15
  16. 16. Quality Assurance by JABEE Criteria Criterion 1 : Learning Outcomes A set of clear students’ performance, planned by program, enough to achieve the outcomes Criterion 4:Educational Improvement Performance Criteria Implementation Criterion 2:Educational Methods What & how high? Assurance through review and evaluation Criterion 3:Achievement of Learning Outcomes Copyright © 2014 Mitsunori Makino. All rights reserved. 16
  17. 17. Learning Outcomes in JABEE 2012 Criteria (a) An ability of multidimensional thinking with knowledge from global perspective (b) An ability of understanding of effects and impact of engineering on society and nature, and of engineers’ social responsibilities (engineering ethics) (c) Knowledge of and an ability to apply mathematics, natural sciences and information technology (d) Knowledge of the related engineering disciplines, and an ability to apply it to solve problems (e) An ability to design to respond to requirements of the society by using various sciences, technologies and information (f) Communication skills including logical writing, presentation and debating in Japanese and fundamental communication skills on the international scene. (g) An ability of independent and life-long learning (h) An ability to manage and accomplish tasks systematically under given constraints. (i) An ability to work as a team member Copyright © 2014 Mitsunori Makino. All rights reserved. 17
  18. 18. JABEE’s Outcomes-based Assessment • Instead of structural assessment, the contents of education are assessed as outcomes. • JABEE does not question the approach of the program as long as the outcomes are assured. • The program shall set up and demonstrate with benchmarks the outcomes, which the program expects. • The outcomes of the program are evaluated by the society in the era of information. Copyright © 2014 Mitsunori Makino. All rights reserved. 18
  19. 19. Rubric for outcomes-based evaluation Copyright © 2014 Mitsunori Makino. All rights reserved. 19
  20. 20. How to make students learn by themselves and evaluate appropriately graduates attributes? • “Knowledge” profile may be assured by suitable curriculum with a certain assessment (paper exam, report, …)(almost input-based). • “Ability” may be implicit and expressed as his/her action. How do we “teach” action profile? Copyright © 2014 Mitsunori Makino. All rights reserved. 20
  21. 21. Rubric: Evaluation of Performance & Level • Nobody can evaluate implicit (inside). • Rubrics check performance and level which he/she presented as some actions • For assurance of learning outcomes, behavior characteristics is important. (happened actions is not important) Therefore, rubrics of “competencies” should be considered. Copyright © 2014 Mitsunori Makino. All rights reserved. 21
  22. 22. What is rubric? • Rubric: a tool to evaluate level of performance • Three elements: – Dimensions …performance criteria – Scales …levels of performance – Descriptors • Evaluation of performance of study items • Through self-check by students and evaluation by teachers, the rubric help students’ awakening. http://www.abet.org/Linked%20Documents-UPDATE/webinars/webinar-developing-rubrics-2010.pdf Copyright © 2014 Mitsunori Makino. All rights reserved. 22
  23. 23. Rubric: Trends in Higher Education • Especially effective for PBL • Objectivity of learning outcomes and mutual understanding between students and professors • Effective of course design and FD Copyright © 2014 Mitsunori Makino. All rights reserved. 23
  24. 24. Graduate attributes also use the rubric, showing differences between engineering, technologist and technician education. Range of Problem Solving (IEA) WA requires “complex problems” Copyright © 2014 Mitsunori Makino. All rights reserved. 24
  25. 25. Rubric: Examples in U.S. Engineering Education • ABET Webinar: Developing Rubrics http://www.abet.org/webinar-developing-rubrics.shtml • WSU-EERC (Engineering Education Research Center, Washington State University) IDEALS http://ideals.tidee.org/ Copyright © 2014 Mitsunori Makino. All rights reserved. 25
  26. 26. Case study: the rubric-based competency development program in Chuo University Copyright © 2014 Mitsunori Makino. All rights reserved. 26
  27. 27. Rubric-based Competency Development Project in Faculty of Science and Engineering, Chuo University • Kick-off: Summer, 2008 • National Project “Good Practice in Higher Education” from autumn 2009 to March 2012 • Goal: – Clarification of graduate attributes (competencies) of Faculty of Science and Engineering, Chuo University – Development of systematic curriculum and teaching method of the competencies • Project leader: Mitsunori Makino • Leading department: Information and System Engineering (Computer Engineering) Copyright © 2014 Mitsunori Makino. All rights reserved. 27
  28. 28. Procedure of Competency Development in Chuo Univ. 1. 2. Definition of Educational Goal Definition of competencies as graduate attributes a. b. c. d. e. f. g. 3. Detailed competencies with levels i. ii. iii. iv. v. 4. 5. Communication ability (listening ability, reading comprehension, descriptive writing ability, ability to make proposals & discussion ability) Problem-solving ability (problem discovery, problem analysis, logical thinking, executing plans & verification) Knowledge acquisition ability (learning, application ability & information gathering ability) Organized behavior capabilities (balancing ability, role recognition, independence, cooperation, & initiative) Creative ability (conceptual ability, reasoning ability, ability to feel emotion & desire to explore) Self-realization ability (goal setting, schedule management, self-control, stress control & achievement orientation) Specialization (Basic academic ability, mathematics/natural sciences, information technology foundation, specialized knowledge & accuracy) Problematic Awaiting Standard, to be expected Original Excellent, exemplariness Application to courses Sharing information on good practices (PDCA cycle) Copyright © 2014 Mitsunori Makino. All rights reserved. 28
  29. 29. Example: Rubric for Problem-solving ability Competency Keyword Definition Level 0 1 2 3 4 Problem discovery Ascertain the current situation and goals (the ideal state) and discover Is unable to correctly understand Is able to correctly understand problems to be solved from the given issues. given issues. gap between the two. Discovers issues by Discovers issues by himself/herself, selects the best Correctly understands selfhimself/herself, and devises and solution and implements this discovered or given issues. implements solutions. Verifies systematically. Verifies the Devises and implements the results and reflects them in results from various aspects and solutions. plan reviews or subsequent reflects them in subsequent planning. planning. Ascertains the current situation Ascertains the current situation and goals, discovers problems to Ascertains the current situation and goals and discovers problems be solved from the gap between and goals and discovers problems to be solved from the gap between the two and is able to prioritize from the gap between the two. the two. these. Problem analysis Understand the cause-and-effect relationships of problems and discover the true causes (the essence). Understands the cause-and-effect Understands the cause-and-effect relationships of problems and then relationships of problems and makes efforts to discover the discovers the essence of these. essence of these. Problem-solving ability Logical thinking Executing plans Verification Correctly understand the issues. Devise and implement solutions. Is unable to correctly Verify the results and reflect understand given issues. them in plan reviews or subsequent planning. Is unable to understand or discover the cause-and-effect relationships and essence of problems. Correctly understands given issues and tries to solve them. Makes efforts to discover the cause-and-effect relationships and essence of problems in order to understand them. Having understood the cause-andeffect relationships of problems and discovered the essence of these, is aware of the directions which solutions to these should take. Be able to mentally organize the Is able to mentally organize essence of complex phenomena Is able to mentally organize complex phenomena and structure and structure them (so they are Is unable to mentally organize Makes efforts to mentally Is able to mentally organize complex phenomena and structure them. Is able to logically construct readily understandable to all). Be complex phenomena and structure organize complex phenomena and complex phenomena and structure them. Is able to logically construct and expand on his/her own able to logically construct and them. structure them. them. and expand on his/her own opinions and procedures and expand on one's own opinions and opinions and procedures. convince other people of them. procedures. Although sets goals and objectives Sets goals and objectives, selects Set goals and objectives, make Acts on things in a haphazard Sets goals and objectives, makes and makes plans, never carries out Sets goals and objectives, makes the best methods from among plans in systematic order and manner. plans and executes them as these plans and lacks solid visions plans and executes them. several methods, makes plans and execute them. planned. for executing them. executes them. Correctly assess the results of Correctly assesses the results of Correctly assesses the results of executed plans and reflect them in executed plans and reflects them executed plans from various Does not verify the results. Verifies the results on the whole. Correctly assesses the results. plan reviews or subsequent in plan reviews or subsequent aspects and reflects them in plan planning. planning. reviews or subsequent planning. Copyright © 2014 Mitsunori Makino. All rights reserved. 29
  30. 30. Two points for applying rubric to course assessment 1. Define the dimensions and scales which can be checked in the course. Then descriptor at each level can be described according to supposed students’ action. 2. If necessary, revise course which encourages students’ “awareness” and “innovation.” STUDENT Copyright © 2014 Mitsunori Makino. All rights reserved. 30
  31. 31. Introduction of Competency Development to Course (Software Design Team Project) Clarification of competencies which can be observed at each class Students’ behavior to be expected Checklist of observable competencies in expected behavior Copyright © 2014 Mitsunori Makino. All rights reserved. 31
  32. 32. Assessment of competencies • Observation of students’ behavior and/or interview are useful for assessment. • Also, self-assessment by the students contribute to their “awareness” and “innovation.” 個人評価(Max値)-平均(6名) チーム内個人評価ー平均(6名) 傾聴力 基盤となる学力 4 読解力 数学・自然科学 記述力 情報技術基礎 提案力 専門知識 議論力 3 精確性 達成指向 ストレスコントロール 自己管理能力 傾聴力 基盤となる学力4.0 読解力 数学・自然科学 記述力 情報技術基礎 提案力 専門知識 議論力 3.0 課題発見 2 1 0 スケジュール管理 目標設定 倫理 探求する意欲 感動する力 推論する力 発想する力 率先力 精確性 課題分析 達成指向 論理的思考 ストレスコントロール 計画実行 自己管理能力 検証 2010.02 2010.07 2011.07 課題分析 論理的思考 計画実行 0.0 目標設定 応用力 バランス力 役割認識 協働 主体性 1.0 スケジュール管理 学習 情報収集力 課題発見 2.0 倫理 探求する意欲 感動する力 推論する力 発想する力 率先力 検証 学習 応用力 情報収集力 バランス力 役割認識 協働 主体性 Copyright © 2014 Mitsunori Makino. All rights reserved. 2010.02 2010.07 2011.07 32
  33. 33. From Department to Faculty FD & SD through Planning of Rubric-based Competency Development • All departments in the Faculty of Science and Engineering, Chuo University have the rubrics, respectively, and apply them to some courses. • The faculty includes the competencies as the graduate attributes in its both of diploma and curriculum policies. Copyright © 2014 Mitsunori Makino. All rights reserved. 33
  34. 34. Conclusion Copyright © 2014 Mitsunori Makino. All rights reserved. 34
  35. 35. For assurance of educational quality through achieving graduate attributes • Competency development is becoming mandatory in higher education in Japan as well as global engineering education. • Combination of the rubric-based competency development with engineering education seems positive result from our project. • How do we promote the rubric-based competency development to wide area of engineering education ? – Understanding of importance – Re-organization of courses, if necessary Copyright © 2014 Mitsunori Makino. All rights reserved. 35
  36. 36. Thank you for your attention! It’s my honor to be involved in establishment of accreditation system for engineering education in Indonesia. I hope I will be here for further discussion in near future. makino@m.ieice.org Copyright © 2014 Mitsunori Makino. All rights reserved. 36

×