Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
Upcoming SlideShare
Loading in …5
×

It Takes a Village: Collborating for success with high-volume assessments

5,589 views

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
5,589
On SlideShare
0
From Embeds
0
Number of Embeds
831
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

It Takes a Village: Collborating for success with high-volume assessments

  1. 1. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. 2013 Users Conference Baltimore | March 3rd – 6th It Takes a Village Collaborating for Success with High-Volume Assessments
  2. 2. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Rio Salado College is part of the Maricopa Community College District  Rio Salado offers over 600 unique courses online  Our model – one course, many sections  Development/Production of courses is centralized  Perception users since 2003 A Little About Us Slide 2
  3. 3. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Penny Freuler, Director of Instructional Technology and Course Production  Liz Burnett, Coordinator of Distance Learning Services  Heather Tyler, Testing Coordinator  Lance Cummins, Networking Manager Introductions Slide 3
  4. 4. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  How we use Perception with our LMS  How we use Perception Data outside of the application  How we work as a team to support assessment delivery  What we have learned to do/what we have learned not to do What We Will Cover Slide 4
  5. 5. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Building Assessments  Delivering Assessments  Scoring Results  Supporting Services Course Production Slide 5
  6. 6. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Topics and questions  Outcomes Generator  Assessment controls  Password protection  Time limits  Randomization  Open access for self-assessment  Outcomes  Feedback on question, topic, or assessment level  Branch to another assessment Building Assessments Slide 6
  7. 7. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  LMS Integration  Offline versions available for Testing Services  Schedule control  Sequencing  Device support  Perception 5.2 inclusion of device settings file Delivering Assessments Slide 7
  8. 8. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Entry points from LMS to Scoring Tool  Scoring Tool updates have not kept pace with the rest of the Perception application  Building, delivering, and scoring work together  Post Perception 5.4 Deployment  Integrating scoring functions in LMS Scoring Results Slide 8
  9. 9. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Work with helpdesks, testing services, students, faculty to troubleshoot issues  Greatly improved in Perception 5.2, 5.4  Coordinate resolutions with Programming, Networking, Technical Support, Instructional Design  Evaluate coaching and transcript reports  Work with Questionmark staff on system issues beyond our control and best practice advice Supporting Services Slide 9
  10. 10. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Exam Administration  Proctoring Services  Academic Integrity  Data for Content Owners - Reports Testing Services Slide 10
  11. 11. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  In-Person Exams  Midterm and Final exams delivered through QuestionMark  Password protected  Identification verification  Monitoring during exam  Close working relationship with Course Production and Networking allows issues to be identified sooner Testing Services- Exam Administration Slide 11
  12. 12. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Same level of testing security required for all out of area students.  Proctoring staff dedicated to vetting and communicating with proctors to assure same testing standards are met  Creates the challenge for Networking and Course Production that there is potentially a student taking a high stakes exam 24 hours a day 7 days a week.  Issues encountered  Transcript report research  Branching profile workaround Testing Services – Proctoring Services Slide 12
  13. 13. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Working with Faculty  Supply supporting evidence for a reported academic integrity issue • Turn-It-In Reports • Peer-to-Peer Reports • Adjunct Suspected (similar course work submitted) • AI Team suspected (data pulled from reports)  Working with Financial Aid  Supply supporting evidence for a suspected FA issue  Working with the Student Academic Progress process  AI team works to alert FA to potential issues before they happen Testing Services – Academic Integrity Slide 13
  14. 14. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Testing Services – Academic Integrity Slide 14 What actions are taken by the Judicial Affairs Department? What does Academic Integrity look to identify? Who/What can instigate an inquiry? Financial Aid SAP Process Sufficient Evidence to confirm less than honest behavior New Students: Send Internal Review Letter to gather further documentation Current Students: Actions necessary to include suspension and expulsion Insufficient Evidence to confirm wrong doings Record Research This database will house all confirmed, suspected and cleared student research Staff Referrals Advisement Counseling Admission and Records Financial Aid Academic Integrity Reports
  15. 15. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Testing Services – Academic Integrity Slide 15 •Questionable Documents submitted for SAP. FA staff marks student record with ‘Referral’ Service Indicator and emails XXX@riosalado.edu with student information, description and supporting documentation. •Examples of questionable documents •Falsified obituaries •Altered id cards Financial Aid •Reviews and researches all referrals. •Sufficient Evidence •Compile all documentation necessary to support recommendation for further action. •Insufficient Evidence •Remove Referral service indicator •Alert FA that student is not considered a risk •Forward all research to Judicial Affairs for storage. AI Team •Review all records and recommendations •Confirmed Sufficient Evidence •Pre-Student Enrollment •Send Internal Review Letter •Current Student •Actions necessary •Insufficient evidence •Return to AI for further research or clarification or file with other insufficient evidence files Judicial Affairs
  16. 16. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Item Analysis  Allows Content Owners to review Assessments based on • Question Difficulty – P-Value Proportion Correct • How did students perform on the questions • What can this tell us about the question or the content Testing Services – Reports Slide 16
  17. 17. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Testing Services – Reports Slide 17
  18. 18. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Testing Services – Reports coming in 5.4 Slide 18
  19. 19. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Team approach  Programming  Networking  Production  Support Services  Work closely with Questionmark staff Version Testing Slide 19
  20. 20. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Some lessons learned  Previously patched items  Greatly improved with Perception 5.4  Items created in previous versions  Items created that employ irregular practices  Test, retest, test again, and go back and test  If you think users won’t use it that way, they will. Version Testing Slide 20
  21. 21. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Server Builds  Install and configure Perception Servers  Test and Production environments  Maintain multiple versions of Perception throughout testing and development cycle  Integrate into Rio Salado’s custom LMS  Coordinate testing and development cycle with other departments such as Course Production Network Administration Slide 21
  22. 22. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  New versions of Perception require testing and development by multiple teams  We delivered over 2.3 million assessments in 2012  Networking test builds– Build and configure new servers, copy database (~40-50GB).  Course Production – Test features, look for bugs, ensure that previously patched items are retained in the new version  Programming – Ensure code works with potential QM changes, modify as needed. Version Testing Slide 22
  23. 23. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Course Production  Takes initial reports of issues and troubleshoots to determine if there is an issue with the Perception servers, the application itself, or some other issue that needs to involve Networking  Cuts down greatly on the amount of issues that come to Networking. Since we don’t use the application ourselves, it takes much longer for us to determine where the issue lies.  Now, if it comes to us we know there’s been a great deal of groundwork done and we know more about the sort of problem people are having. It also weeds out things like user error and problems with the course content. Collaboration Slide 23
  24. 24. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Testing Services  With students taking exams all day in our testing centers, and proctored exams possible at any time, server downtime must be kept to a minimum  Perception servers are load balanced, so we can reboot them one at a time without affecting students.  SQL– On the rare occasion that there is an issue with our main SQL cluster, Perception and other applications may experience a few moments of downtime while SQL is moved over to another cluster node. This will cause problems for students currently taking exams.  We notify Testing as soon as possible before or during outages so that they can be aware of issues and take appropriate action in our testing centers to minimize lost work. Collaboration Slide 24
  25. 25. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Questionmark – If/when issues are discovered, we work with QM support to see if it’s a configuration issue or a bug. If it’s a bug then they work on getting it fixed and either hotfixed or rolled into their next update, depending on severity.  For previous versions we’ve had weekly or semi-weekly meetings with multiple teams at QM (e.g. Customer Care, Tech Support, and QM developers) to go through our list. Some issues were bugs, and some were unique ways we were using Perception that didn’t work as expected, but weren’t necessarily bugs. Testing/Support Slide 25
  26. 26. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Several years ago Questionmark began offering Enterprise support. It’s a paid service, but one I highly recommend  This has saved us a great deal of time  Previously we’d have to bring multiple techs up to speed on issues we were having, had to send the same files/logs, etc. repeatedly  Now we have one point of contact, and he deals with us often enough to have a grasp on our environment without having to start at square one each time we contact him. Enterprise Support Slide 26
  27. 27. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  For 5.4, QM has instituted many changes to their development process, and we’ve been very pleased with our testing  In fact, for the first time we’ve not needed anything hotfixed, and are preparing to deploy 5.4 to production at the end of the month. We would have gone sooner, but we had some code changes on our end that held it up, not directly QM related. Perception 5.4 Slide 27
  28. 28. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Best Practices – read them  Review and understand server settings  Use system as designed  Questions? Remember Slide 28

×