CityU Air forum 2013


Published on

Sabrina Crawford, the (former) VP of Institutional Effectiveness and Dr. Laura Williamson, the Director of the MBA program presented at the Association for Institutional Research (AIR) annual conference May 20, 2013. The AIR Forum is the world’s largest gathering of higher-education professionals working in institutional research, assessment, planning and related post-secondary education fields. The conference included presentations by colleagues representing all sectors of higher education and an exhibit hall that featured the latest tools and resources to support data use for decision making.

City University of Seattle created a program assessment process that utilizes Folio180’s ePortfolio to gather and track both formative feedback and summative analysis of student learning directly related to achievement of program learning outcomes. Sabrina and Laura presented on the utilization of Folio180, program assessment, and the data collection process as well as initial MBA program results.

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

CityU Air forum 2013

  1. 1. Systemizing Formative and Summative Program Data Capture in E-Portfolios City University of Seattle AIR Forum May 2013 Sabrina Crawford and Laura Williamson, PhD
  2. 2. Learning Outcomes • Participants in this session will walk through City University of Seattle’s organization process for program assessment and obtain usable templates to recreate the process at their own institutions. • Participants in this session will view how Folio180’s e- portfolio system can be configured to obtain systemized formative and summative data on student achievement of programmatic learning outcomes. • Participants in this session will discuss the benefits and challenges of using systemized formative feedback from faculty and students as a way of achieving a more thorough look at programmatic assessment.
  3. 3. Presenters • Sabrina Crawford is the Director of Institutional Effectiveness at City University of Seattle. For the first 16 years of her career, Sabrina worked in the P-12 educational arena as teacher, program coordinator, and Head of School. Her passion for assessment and curriculum has most recently brought her into the field of higher education assessment and research, where for the past 4 years she has been fortunate to work on creating systems that enable data-driven decision-making. • Dr. Laura Williamson is the Director of the MBA program at City University of Seattle. For 20 years, Laura has worked in higher education and high technology and has been responsible for a variety of instructional technology projects and program improvement initiatives with the US Armed Forces and L-3 Corporation.
  4. 4. The Basics • Founded in 1973 • WA Campuses located from Vancouver to Vancouver • 11 foreign locations • In the US, traditionally focused on working adults – Degree completion and graduate degrees • In foreign locations – More traditional student population • Practitioner faculty deliver real-world relevance to students around the world who may not otherwise have access.
  5. 5. Strengths of Existing Program Assessment • Curriculum development system (CDS) ensures connection between course learning outcomes and CityU Learning Goals (CULGs) • All course assessments linked to CULGs • Program outcomes existed for all programs • Courses had identified common assessments • Existing faculty assessment committee process • Rubrics embedded in CDS for course assignments • Discussions had begun on using e-portfolios
  6. 6. Challenges to Program Assessment • Large percentage of adjunct faculty • No direct connection between course learning outcomes and program learning outcomes • Existing process was too big to be implemented • Focus on capstone course assessment only • Rubrics focused on course or skill outcomes • Wide variety of report output between programs
  7. 7. Framework for Consistency & Flexibility Critical to be consistent • CityU Learning Goals • Program learning outcomes • Course learning outcomes • Assessments that produce direct and indirect evidence of student learning • Program level assessment of student learning – PLO rubrics Important to be consistent • Methods of assessing student learning • Collection and analysis of evidence of student learning • Example: A case study with a prescribed format, adapted to reflect a local company or organization Critical to be flexible • Teaching strategies that respect academic freedom for faculty • Learning activities that respond to different needs and populations of students • Readings, course materials, and activities that reflect local context and issues
  9. 9. CityU’s Outcomes Assessment Process The diagram to the right shows how CityU’s outcomes assessment process starts with its mission and values, and culminates in plans to achieve higher levels of student learning.
  10. 10. CityU Learning Goals ** Program Outcomes ** Course Outcomes Assessments Goal #1 – Connect the Dots
  11. 11. CityU Learning Goals The CityU Learning Goals describe exit competencies for graduates of all degree programs. Professional Competency and Identity Communication and Interpersonal Skills Critical Thinking and Information Literacy Ethical Practice and Service Diverse and Global Perspectives Lifelong Learning
  12. 12. The Importance of Program Outcomes • What specific actions should students be able to do upon completion of the program? • This is your PROMISE to your graduates • They should be current and relevant to the profession they lead to • Every faculty member should know how their course is helping students achieve these • Every student should know that their courses are supposed to be helping them develop these skills • This is an opportunity to show how your program is unique
  13. 13. Begin with Curriculum Mapping • What courses and assessments within a program are aligned to each program outcome • Identify the course work’s level of student learning: Introduction, Practice, Mastery • From all the possibilities, identify the assessments best demonstrate the three levels of indirect and direct evidence (introduction, practice, and mastery) and showcase student growth towards mastery of the outcomes? • Every course should connect to at least one outcome and every outcome should have a minimum of one course identified for each evidence level I,P, or M
  14. 14. Mapping Program Outcomes to Courses: Identifying the best location for each level of student learning MBA 500: Essentials of Business Mgmt MBA 501: Global Business Communication & Research MBA 515: Project Mgmt & Prioritization MBA 520: Managing with Technology MBA 550: Business Operations MBA 555: Business Strategy Program Outcome 3 I – Capsim Report P – Final Project Plan & Presentation P – Business Plan M – Capsim Program Outcome 4 I – Porter’s I – Individual Persuasive Paper Presentation P – Business Plan M – Value Chain Design MBA Curriculum Matrix
  15. 15. Populate the Program Design Guide • A one page spreadsheet that shows the alignments • Populate the Program Learning Outcomes (PLOs) first and connect to CityU Learning Goals (or Gen Ed) – one or two max • Populate the best assignments/courses from the matrix that show student growth towards each PLO • Identify the course learning outcome that align to the identified assessment – the set of 3 should read as a progression of student learning towards the PLO • This becomes the template for the e-portfolio
  16. 16. CityU Learning Goals Program Learning Outcomes Course Learning Outcomes Courses Required Assessments Level Which CityU Learning Goals are supported by program outcomes? What abilities must the learner successfully demonstrate as a result of this program? Which course learning outcome is tied to the required assessment and supports the programmatic outcome? List the courses being used for assessment What graded assessment(s) provide evidence that the learner can demonstrate proficiency in this program outcome? I=Introduce P=Practice M=Mastery Professional Competency and Professional Identity Critical Thinking Skills 9. Apply socially responsible and sustainable business practices to an organization Document strategic decision making. MBA500 Porter Paper I Conduct a SWOT analysis MBA525 Sustainability Case Analysis P Utilize business simulations to make executive management decisions for every aspect of leading and managing a complex, multidimensional business. MBA555 Business Plan M Program Design Guide
  17. 17. Create a PLO Rubric • Mastery assignments to be assessed against program learning outcomes • PLO rubric designed to describe each student learning level: Introductory, Practice, Mastery • Ensure student awareness of these levels and their connection to their assignments • Mastery assignments to be evaluated by external evaluators within e-portfolio
  18. 18. Annual Outcomes Report Rubric Introductory Standard Practice Standard Mastery Standard 1. Leverage managerial effectiveness through recognition of individual strengths, values and leadership strategies. Student has a good understanding of his/her personal strengths, a real appreciation of the difference in others, and is beginning to understand how these characteristics influence team development. Recognizes the value of working within diverse, cross-functional teams but not quite sure on how to leverage good team management strategies or practices. Commits to achievement of common goals when working on a team. Student starts to leverage thier own strengths and those of team members to better develop team communication, high ethical working standards, trust, and a climate of cooperation and collaborative problem solving amongst team members in order to acheive objectives of the project. Interacts and cooperates productively and maturely with others. Accomodates the protocols and expectations of teams. Selects appropriate media fo dissemination or accumulation of information. Places information in appropriate context when listening, reading, writing, and speaking. Accepts suggestions and guidance of team leaders and other members. Student leverages their own strengths and turns their weaknesses into opportunities for team development to manage projects, co-located teams, and/or virtual teams. Student also leverages their teammates' strengths and weaknesses into opportunities for coaching and mentoring others in appropriate circumstances. Accepts both leadership and participation roles gracefully. Addresses challenges with the team as they occur and works to an acceptable and ethical resolve with input from each team member. Realizes and communicates the value each member brings to the team.
  19. 19. WAIT!! That’s a lot of assignments! How is it Sustainable?
  20. 20. Conceptual Framework Emphasis on Indirect Evidence of Student Learning • Evidence at the first two levels – Introductory and Practice – tracks student growth towards program learning outcomes • Course-level assessments and their aligned course outcomes build toward the Mastery summative assessments and achievement of program learning outcomes • Assessed formatively, receiving direct feedback from students and faculty within e-portfolio
  21. 21. Conceptual Framework Emphasis on Direct Evidence of Student Learning • Each program identifies major summative assessments that provide direct evidence of students Mastering program learning outcomes • Mastery assessments can be used to assess multiple program learning outcomes • Assessed by review team using program learning outcome rubrics embedded in e-portfolio
  22. 22. FOLIO180 – CITYU’S E-PORTFOLIO SOLUTION Student Learning Outcomes Assessment
  23. 23. Analysis of CityU’s Outcomes Assessment Approach • Strengths: – Emphasis on authentic assessment and relevance for students – Integration of CityU, program, and course outcomes – Widespread faculty support and engagement – Embeds student feedback and focus on outcomes • Challenges: – Increased complexity with new e-portfolio system – Multiple changes to procedure within a short time frame – Embed what is learned during pilot year to enhance process for all programs