More Related Content
What's hot
What's hot (11)
Viewers also liked
Viewers also liked (20)
Similar to Bb world 2012 Suzan Harkness It’s all about the data increasing student success via partnerships ...
Similar to Bb world 2012 Suzan Harkness It’s all about the data increasing student success via partnerships ... (20)
Recently uploaded
Recently uploaded (20)
Bb world 2012 Suzan Harkness It’s all about the data increasing student success via partnerships ...
- 1. It’s All About The Data: Increasing Student Success Via Partnerships (Quality Matters) And Professional Development S.SUZAN J. HARKNESS Assistant Dean & Director Center for Academic Technology University of the District of Columbia
- 2. Institutional Statistics Spring 2011 Student population – 5,667 (FTE – 3819) 48% attending Flagship Institution 44% attending Community College 8% attending Law School Faculty population (Fulltime) – 253 (Adjunct)- 262 Courses per semester – 1,700 (traditional) Helpdesk Support – 24-7-365 + on-campus Center for Academic Technology – 7 FT Staff 2
- 3. This Session Will Cover: How the University of the District of Columbia used a strategic triangulated approach (research and data, professional development initiatives, and partnerships) to impact: Blackboard use across campus; Faculty capacity to expand functional use and applications; A quality approach to distance education and; Student outcomes and success. 3
- 4. Three Drivers • Strategic Planning • Needs Assessment • Bench Marking- early online experiences • 360 Assessment to understand capacity and motivation • Professional Development – data driven • Data - Content Analysis (2010, 2011, 2012) • Design and Findings– Targeted Training • Partnerships • Quality Matters, Blackboard Learn/Analytics 4
- 5. Strategic Planning •Needs Assessment •Gap Analysis - Survey Faculty skills, Needs, Motivations •360 Assessment Capacity •Benchmark early online experiences •Content Analysis- Design and implement Nominal in scale/binary Pretest Spring 2010 Posttest Spring 2011 Posttest II Spring 2012 •Data Driven Professional Development •Skills Workshops •Online Instruction Certification (OLA & Track II) •Continuous Assessment 5
- 6. Professional Development F09 S10 F10 S11 F11 S12 Totals Total 25 24 55 36 50 47 237 Workshops Total Hours 53 149 151 127.5 118.5 79.5 678.5 Webcasts 2 0 0 0 1 2 5 Gross seat 147 313 356 180 176 257 1429 occupancy Unique 91 64 181 105 79 125 403 Faculty attending Cost per $319.00 $163.00 $65.00 $77.00 $157.00 $62.00 $197.00 faculty/term /faculty
- 7. Assessment Feedback •83% response rate on faculty assessments • 89% concluded that professional development workshops were effective in providing tools and applications to assist them •100% reported that would deploy the new application(s) •100% reported that the workshops were a valuable use of time •77% reported that the workshops were excellent training •88% reported that the workshop instruction was excellent •89% felt that the trainings provided them sufficient comfort and experiences to use the new applications 7
- 8. Content Analysis 2010 Examination 2011 Examination 2012 Examination Banners Banners Banners Buttons Buttons Buttons Color Scheme Color Scheme Color Scheme Announcements Announcements Announcements Course Info Course Info Course Info Syllabus Syllabus Syllabus Faculty Info Faculty Info Faculty Info Course Course docs/content Course docs/content docs/content Assignments Assignments Assignments Discussion Boards Discussion Boards Discussion External Links External Links Boards Additional Tools Additional Tools External Links Grade center Grade center Groups Additional Tools Groups Additional Resources Additional Resources
- 9. Blackboard Use Among Faculty S2010 S2011 Percent Change S2012 Percent Change Total Users 209 270 + 29 % users 225 -17% users (58%) (55%) -5 proportion of (47%) -15 % users proportion of users Heavy 131 (35%) 180 (37%) 6% 141 (29%) -22% Users Moderate 78 (21%) 90 (18%) -14% 84 (18%) null Users Non-Users 153 (41%) 223 (45%) 10% 253 (53%) 18% Total 362 493 36% 478 -3% Faculty FT 241 231 -4% 182 -21% Adjunct 121 262 117% 296 13%
- 11. 11
- 12. Let’s Drill Down •Bimodal distribution –tells us where we need to focus •11% of faculty who attended workshops shifted their categorization (2010 to 2011) •3% shifted from a nonuser to a moderate user (2010-2011) •4% shifted from a nonuser to a heavy user (2010-2011) •4% shifted from a moderate to a heavy user (2010-2011) •<1% shifted from a minimum user to a moderate user (2010-2011) 2012 & 2013: New Challenges New Opportunities 12
- 13. Partnerships: QM, Bb Learn, Bb Analytics, Bb Student Services QUALITY MATTERS • Beyond the pilot - 17 Courses Peer Reviewed • Mean QM score 74.0 (met QM standards). • 73 percent increase over QMR of courses pre QM • Standard deviation 7.05 (Increased rater reliability) • More importantly – Positive Student impact • 1 Master Reviewer and a Train the Trainer on staff BLACKBOARD: Learn, Analytics, Support Services • Partner approach to support and services • Analytics – data mining
- 14. Descriptive Overview of Online Courses 2007- 2012 Pre/Post QM Partnership 14
- 15. Statistical Impact 2007-2012 Percent Change Pre/Post QM 2007-2012 Pre/Post QM % Change (N=1570) WDL -23.53 PASS Class +19.74 FAIL CLASS -66.66 15
- 16. Statistical Impact 2011-2012 Percent Difference: Traditional: Online 2011-2012 Data Percent Percent QM Percent points – Traditional All Classes Online/Hybri Difference v online/hybrid (N=5716) d Classes Traditional v courses (n=935) Online WDL .071 .13 58.7 PASS Class .80 .91 12.9 FAIL CLASS .20 .09 75.9 16
- 17. Lessons Learned • Faculty who engaged in PD increased and expanded Blackboard use • Upward drift of expanded use and engagement • Triangulated data-driven approach yields better student experiences • Fewer students WDL from online courses that meet QM standards • Students in peer-reviewed courses earn higher grades/more likely to pass courses • New policies and procedures ensure consistency • Quality and integrity over quantity – instrumentalism • Collegiality and collaboration sustain our strategic plan, drive institutional effectiveness, and expand capacity across campus 17
- 18. We value your feedback! Please fill out a session evaluation. 18
Editor's Notes
- Good Afternoon, My name is Suzan Harkness and I am the Assistant Dean in the Learning Resources Division at the University of the District of Columbia and the Director of the Center for Academic Technology.
- In 2009, The Center for Academic Technology was reorganized to lead the institution in expanding its approach toward distance education, assess and build faculty capacity to support an increased demand for flexible and mobile learning, and expand the functional use of Blackboard. An early analysis found that less than 50% of the faculty used Blackboard and of those, many used it as a repository to hold course documents and execute tests. Moreover, an analysis of early online courses found that many suffered poor design principles and lacked student engagement and alignment to desired outcomes.
- A preliminary self-study provided the institutional impetus to partner with Quality Matters (QM) and use the QM Rubric and course design standards as a backbone to build capacity, ensure course design integrity, expand instructional pedagogy, and to infuse a research-based best practices approach toward faculty professional development while advancing Blackboard use. Our triangulated approach impacted student experiences. We find that fewer students withdraw from courses that have been peer reviewed and meet standards. Additionally, students enrolled in courses that have undergone peer review earn higher grades and are more likely to pass the class.
- Initial analytics found that early online courses had an overall Quality Matters Rubric (QMR) mean of 42.75 and a standard deviation of 24.4, indicating that they did not meet informal review standards. Retention and pass rates in the online courses prior to our QM partnership and professional development initiatives were lower than for traditional courses. These preliminary findings supported the forethought to implement a rigorous assessment plan, quality assurances, and faculty professional development initiatives to improve instructional capacity, course design, and student outcomes. An aggressive strategy was drafted and implemented to provide support and tools to enable pedagogical innovation in the traditional and online learning environments and improve the educational experience of students. The Center for Academic Technology organized professional development opportunities on the use of resources and tools available in Blackboard and external to Blackboard, partnered with Quality Matters as well as other organizations (NBC Learn, Café Scribe, Wimba, Respondus, Barnes & Noble Nook Study…) and worked with faculty to ensure pedagogy and technology were effectively integrated and aligned to learning outcomes. To further the strategic plan, newly proposed policies and procedures for online and hybrid course processes and assessments were drafted and adopted by the University Academic Senate (April 2010 and April 2011). These processes employ the Quality Matters standards in course design and assessment and serve as a guiding principle for a new online instruction certification initiative. To operationalize the strategic plan the Center for Academic Technology grew from a staff of two to a staff of seven, including instructional designers, a 24/7/365 helpdesk, on-campus technical support staff, and faculty dedicated to instructional pedagogy.Developed an assessment instrument to capture the degree to which Bb was being used on campus.The instrument captured data using a nominal scale instrument where data was recorded in a binary fashion – present or not present.There were some variables where we counted or estimated the number of links or resources such that we could capture them in an ordinal scale.It was a common assumption that faculty did not use the system and that it might be a waste of money. We needed data to back-up our assumptions that the system was being used but could be used more efficiently. Created an instrument to capture depth of use by drilling down into each course and looking at how faculty used the system Excel spreadsheet – binary check system – present not present About 8 staff one –two week period (Spring Break) Compile the data into descriptive statistics Use the data to target users and build skillsWe conducted some Basic Bb workshops to a smashing success Goals were to determine where faculty were in their skills and abilities Prime them for the upgrade to Bb 9.1 Conduct a needs assessment to see what they needed to expand use of the system Be available to meet faculty where they were and build skills (Brown bags and Just-in-time training)
- Had a well subscribed professional development campaign. Faculty were excited about what was available and kept coming.By the gross occupancy numbers and unique faculty – we can see that faculty attended more than one workshop each term. A positive indicator.If we look at the expenditures – we are not spending much on professional development but reaping good returns. Per seat occupancy, we spend $56.00 on each BIC (butt in seat); If we break that down into how much each faculty member is consuming, they consume on average about $257.00 of PD a year. – Again, the returns are far greater.Number of QM Workshops: (26 in total since Spring 2010) We have 1 master reviewer who is also a train the trainer. We have over 35 faculty who may serve as peer reviewers and 17 internally trained and certified online instructors.We also have two online certification paths, 0ne 6-week Summer academy and one more flexible option where we have partnered with Sloan-C.
- (236 out of 286 attending workshops completed our assessment instruments. – an 83% response rate.
- These were the basic categories where we examined content.We drilled down a bit deeper in the areas of Discussion boards, External links, Additional tools, and additional resources.
- 36% increase of total users impacted by hiring 141 adjunct faculty to fill teaching slots. The largest increases among hiring adjunct faculty – can be found at our Community college.Total Users decreased nearly in proportion to the number of FT faculty who departed the University. Some FT faculty in the CC took on administrative positions.Heavy users increased by 6% - a positive signModerate users decreased by 14% - again a positive sign IF some of these shifted into the heavy user categoryAnd Nonusers Increased by 10% as a proportion of faculty. Not what we want to see, but with a large hiring influx it is to be expected.
- This is a more visual descriptive view of the data from Spring 2011 broken out by FT and adjunct and 4 year and community college campus.
- Findings: Since we have begun tracking Blackboard use, we have noticed a couple interesting trends: Faculty who participate in Bb professional development are more likely to use Bb and over-time move from less-use classifications to heavier-use classifications. Such faculty are prime foci for online instruction.This database provides the Center for Academic Technology (CAT) targeted groups to promote specific levels of professional development.Permanent fulltime faculty are more likely to use Bb than adjunct faculty; therefore, as more fulltime permanent faculty retire and or leave the institution and more adjuncts are hired, we see an increase in the number of faculty who do not use Bb for instruction. Greater efforts need to be made to help adjuncts who are often hired late and have limited access to resources get up and running in Bb. The Center for Academic Technology (CAT) will work with adjuncts to provide them a course template in Bb and to pre-load syllabi or other instructional materials in coordination with the colleges.
- Seventeen courses have been proposed and reviewed for online delivery and added to the roster of flexible learning options for students. The peer reviewed QM Rating for newly proposed courses (after the implementation of a QM partnership and faculty professional development) had an overall mean of 74.0 (courses met standards). This represents a 73 percent increase over peer review courses prior to the implementation of QM standards and professional development initiatives. This is a significant accomplishment.The standard deviation of newly reviewed courses (after QM partnership and professional development) was 7.05. This tells us that newly proposed courses have undergone significant improvement in course design. Moreover, it demonstrates that the partnership with QM has been successful in helping our institution build capacity and meet national standards. It also signals the success and outcomes of intense professional development. Another important finding revealed a smaller standard deviation. This corroborates less rater variation, which informs us that our QM peer reviewers demonstrate rater validity and reliability.
- This table is a longitudinal view of online courses at the University of the District of Columbia and our efforts to increase capacity to implement a strategic portfolio of online courses and programs for our students. Pre-QM, we can see that courses on average had a .17 percent WDL rate, .76 pass rate (A-D), .24 Failure rate. After implementing Quality Matters and an internal peer review process for online courses, see immediate impact: We see that we have 24% fewer WDL’s, a 22% increase in the number of students who pass the course and a 70% decrease in the number of students who fail the course. These are significant impacts of the impact to student success as a direct result of QM and our PD efforts. Interesting Data Points1) UDC has on average 13 students enrolled per class - Online/hybrid courses average 15 students per class.2) 62% of online/hybrid courses enrolled 16 students or more/section.3) On average, 7% of students drop traditional courses; in the online/hybrid courses, 13% of students drop the course.4) High student success rates are aligned to online courses. 91% of students in online classes pass the class whereas 80% of students enrolled in all classes pass the class.5) 20% of students enrolled in courses at UDC fail the class; in online courses 9% of enrolled students fail the course.
- Since the implementation of Quality Matters and targeted professional development efforts to train faculty to design collaborative online courses that meet nationally benchmarked standards, we see significant impact upon student outcomes. The data shows a 23 percent reduction in the number of students who drop online courses once enrolled; a 67 percent reduction in the number of students who fail online/hybrid courses; and a 20 percent increase in the number of students who pass the class earning an A-D grade. This tells us that faculty professional development, online course design standards, and peer review have had a significant impact upon the quality of online course design and implementation. Many early efforts to teach online courses (pre Spring 2010) did not meet quality standards or produce such positive student outcomes.
- Findings: When analyzing student outcomes in peer reviewed online and hybrid courses that meet QM standards to students in all courses, we find that students in online/hybrid courses are more likely to pass the course (earning grades A-D) and significantly less likely to fail the class. Students enrolled in online/hybrid classes are more likely to drop the class than those enrolled in traditional instruction courses. The reason may be influenced by online instructors who present very clear expectations to students and advise students lacking online competencies to take a traditional class and come back to the online arena once ready.