This presentation discusses using the Qualtrics Research Suite as a training learning management system (LMS) at Kansas State University. Some key features that make Qualtrics useful as a training LMS include its multimedia capabilities, logic functions, branching logic, scoring features, integration with Google Translate, and data analytics dashboards. The presentation provides an overview of the basic needs for automated trainings on a university campus and how Qualtrics meets these needs through its various features. While additional features could improve Qualtrics as a training LMS, it demonstrates potential as an alternative to traditional LMS platforms.

1. Using the Qualtrics®
Research Suite as a
Training LMS
SIDLIT 2017 | LMS PRECONFERENCE
COLLEAGUE 2 COLLEAGUE
AUG. 2, 2017

2. Presentation
While learning management systems (LMSes) are a particular technology class, plenty of other
technologies have been harnessed and retrofitted for learning purposes. Alternate LMSes
include social media platforms like Twitter and survey systems. This presentation summarizes
the uses of an online research (survey) suite, Qualtrics, for large-scale compliance-based trainings
at Kansas State University. Some affordances of Qualtrics as a training LMS include the following:
a multimedia approach (including mobile),
a file upload approach,
logic functions,
branching logic for versioning a training (based on profiles or behaviors),
2

3. Presentation (cont.)
a scoring feature for grading performance,
logic tools to set standards for pass/fail,
integration with Google Translate,
security features,
built-in data analytics tools,
a dashboard for versioning different reports for different data clients,
an API for recording training completions and performances in information systems,
easy data export for external analytics,
and unique “skins”.
3

4. Presentation Order
Basic Needs for Automated Trainings on a Campus
Qualtrics Research Suite Features
How Qualtrics is a Training LMS
Additional Desired Features
4

5. Basic Needs for
Automated Trainings on
a Campus
5

6. Automated Trainings
An institution of higher education requires…
◦ a number of policy and safety compliance trainings for the entire campus as well as specialized trainings
for specific populations (ability to roll out trainings to tens of thousands annually)…with a dynamic
workforce and dynamic student population
◦ the ability to verify that the individual took the training and passed or failed (sufficient to the level of
legal standards) (ability to respond to courts in lawsuits and to regulatory and oversight agencies)
◦ the ability to quickly update contents to a training to ensure that it is as accurate as possible for the
particular topic (ability to ensure accurate training…to the minute, such as on issues of safety and
security for lab workers, etc.)
◦ the ability to deliver trainings in an accessible way given the requirements from Section 508 and other
federal accessibility requirements
6

7. Why Automated?
Trainings have to be delivered to scale (to the full campus)
◦ Macro-level scale
◦ Micro-level scale
◦ All points in-between
Trainings have to be delivered to recurring high quality for pedagogical and training purposes
There have to be accurate records of the contents of the trainings, to ensure that they were
designed to documented requirements
The trainings may be delivered synchronously or asynchronously, face-to-face or at a distance,
and in various other permutations… The most convenient approach is to have online contents
that may be versioned in different ways for different uses.
7

8. Why Automated? (cont.)
Trainings have to be delivered so that records are fully accurate and stand up to legal and
regulatory agency scrutiny
◦ Legal requirements include protection of intellectual property, privacy protections, accessibility, and
others
Regulatory agencies require three main things:
◦ Delivery of the proper information and skills, without inaccuracies
◦ Delivery of the proper trainings to the proper individuals at the proper times
◦ Clear record-keeping of all trainings taken and the performance of the individuals (in a way that stands up to courts)
The privacy of all involved has to be protected
8

9. About Qualtrics, Inc. and the Qualtrics
Research Suite
ABOUT THE COMPANY
Based in Provo, Seattle, Dallas, and Washington,
D.C.; Dublin; London; Munich; Melbourne,
Canberra, and Sydney
“Powers over 8,000 of the world’s leading brands
and 99 of the top 100 business schools”
Features a range of tools, with Qualtrics Research
Suite as one
◦ Many be instantiated with a range of features
(some of which are not in the K-State instance and
some of which will not be included in this
slideshow)
ABOUT QUALTRICS RESEARCH SUITE
Offers a cloud-based research suite (software
as a service / SAAS)
Survey tool typically used for qualitative,
quantitative, and mixed methods research
9

10. Qualtrics Research Suite
Features
10

11. Feature: Multimedia Integration
Variety of question types (including visual ones like hotspot and heat map)
Ability to integrate audio, video, and other forms of multimedia
Ability to integrate external apps and websites through inline frames (iframes)
Ability to integrate external application programming interfaces (APIs) and applications with
Qualtrics API
Ability to use scripts and codes to access external APIs (like Google Maps)
Strengths for Automated Training:
Ability to create a seamless multimedia-rich experience
Ability to enrich ways of interacting within the training
11

12. Feature: A File Upload Approach
Ability to request learners / trainees to take a photo and upload it…or create a digital file (of
various types) and upload those
Strengths for Automated Training:
Broadens the range of possible assessment types
Enables the use of smartphone and mobile device functionalities such as built-in cameras
12

13. Feature: Logic Functions
Ability to set up notifications when particular events occur (email triggers)
Ability to populate various Contact Lists based on particular preconditions (contact list triggers)
Ability to set quotas for responses or particular occurrences (quotas)
Ability to set up math logic for variable questions (like for story problems) and with the use of
Embedded Data (to capture the dynamic data)
Ability to use prior responses of the learner in a future customized question (piped text)
13

14. Feature: Logic Functions (cont.)
Strengths for Automated Training:
Ability for trainers to be aware of particular event occurrences
Ability to collect information based on particular preconditions
Ability to set ceilings on quotas
Ability to customize the training experience
Ability to add some non-deterministic elements (such as through math logic)
14

15. Feature: Branching Logic
Ability to create branching logic based on particular conditions (like performance, like selection
of a particular answer to a question, like identity-selection, or other feature)
Ability to customize training based on performance (or learner profile)
Ability to create different training experiences for people based on their level of understanding
Strengths for Automated Training:
Ability to make trainings unique to the individuals (based on their profiles, their performance,
their selections, and other mixes of features)
Ability to build one training for a wide range of learners with differing learner needs
15

16. Feature: Scoring
Ability to set values for particular questions
Ability to sum scores for a particular training (or segment of training)
Ability to change scoring levels after data collection for accurate processing
◦ For example, if a Likert scale is applied, and the valuing is opposite of what is desired, the scoring may
be reset to the correct values
Strengths for Automated Training:
Ability to add point values to each question…and to set granular conditionals for the earning of
particular points within that question
16

17. Feature: Standards for Pass/Fail
Ability to set a score threshold for pass / fail
Ability to set scoring tiers for grouping or classification
Ability to set defined granular conditions for passing (and / or)
Strengths for Automated Training:
Ability to set a threshold for “passing” and then sending learners off in different directions
(based on performance)
17

18. Feature: Integration with Google
Translate
Ability to machine-translate an entire training into a number of different languages using Google
Translate (with dozens of languages available and representable using UTF-8)
◦ But need vetting by native speakers to ensure the accuracy of the translations!
Easy ability to update the translations manually
Strengths for Automated Training:
Ability to tailor the language of the training to trainee’s first or preferred languages
18

19. Feature: Security Tools
Access protections:
◦ CAPTCHA (Completely Automated Public Turing test to tell Computers an Humans Apart) requirements
(to prove respondent’s humanity against ‘bot)
◦ Password protections
◦ Going to a training site from a given URL (uniform resource locator) but disallowing other accesses
◦ Making a training not findable by web browser spiders and crawlers
◦ No ballot box stuffing based on Internet Protocol (IP) address (so a person may not continuously take a
training by returning to a particular link from a particular computer on the network)
◦ Setting time limits on any training
◦ Enabling access to a training based on an Authenticator feature or email invite (with unique links)
19

20. Feature: Security Tools (cont.)
Strengths for Automated Training:
◦ Striving to ensure that whomever is signed up to take the training is the one actually doing so
◦ Protecting training contents against unintended downstream other uses
20

21. Feature: Built-in Data Analytics
Data visualizations from online trainings
In-tool data analytics
◦ Cross-tabulation analysis (contingency table)
◦ Text analysis (rudimentary tool)
Ability to study results data (such as pass/fail) as a dummy variable
Strengths for Automated Training:
Ability to learn about learner performances
21

22. Feature: Dashboard for Reportage
Dashboard for reporting summary data about trainings to different data clients
Dashboard enabling access to raw granular data (including single learner-level data, depending
on privacy settings—but should never unmask learners)
Ability to conduct item analysis on respective questions in a survey
Strengths for Automated Training:
Can enable others to access the summary performance data (but will need to turn off “View
Personal Data” feature at the global level, so personal identifiers and data are not accidentally
leaked
22

23. Feature: Dashboard for Reportage (cont.)
Strengths for Automated Training (cont.):
Ability to capture an overall sense of learner performance
◦ Can run a decision tree learning algorithm from the data to see if there are indicators of whether a
learner will “pass” or “fail”
Ability to capture the training efficacy
Ability to study question effectiveness through an item analysis
◦ Can know when learners stop a training and drop out and at which part of the training
23

24. Feature: API for Grades Recording
Ability to conduct a closed invite for a training…and then auto-capture learner performance for
recording in external information systems
◦ Requires developer work through the Qualtrics API
Strengths for Automated Training:
Ability to handle grade recording to various information systems for college- or university-wide
trainings and performance without the need for a human intermediary (human intermediaries
may advertently or inadvertently introduce inaccuracies into the grade records)
24

25. Feature: Data Export
Ease of data export for external analysis (such as in Excel and / or SPSS, or others)
An integration with NVivo 11 Plus (integration)
Strengths for Automated Training:
Ability to leverage the automated online training data to greater effect
Ability to create more data visualizations from the automated online training data
25

26. Feature: Unique “Skins”
A variety of design looks-and-feels for laptops, desktops, and mobile devices
Strengths for Automated Training:
Branding trainings to particular learner groups based on look-and-feel
Branding trainings to particular “use cases”
26

27. About Features
A tool’s functions and features should not be considered in isolation but as part of a system.
As with all technology systems, there are tradeoffs with different parameter settings. People
who would design to complex systems would do well to
◦ Understand the full implications of going with one setting over another (even the hidden implications)
◦ Test and retest all programming of the training
◦ Test and retest all programming of the training whenever certain changes are made because a cloud-based service may
occasionally have coding errors
◦ Keep open channels so that respondents can let a trainer know if something is not looking right
27

28. How Qualtrics is a
Training LMS
28

29. Required Features of a Training LMS:
Basics
Learner enrollment
◦ Light learner tracking
◦ Light learner profiling
Learner and user identity protections
Delivery of digital contents
◦ Easy updatability
◦ Easy customizability
◦ Ability to deliver broadly
◦ Ability to target learners specifically
Data protection and storage
29

30. Required Features of a Training LMS:
Basics (cont.)
Question creation and assessment
◦ Ability to provide real-time feedback
Accurate scoring per question and “grade” totaling
◦ Setting thresholds for passing
◦ Setting paths for training re-study and retakes
Accessibility features
◦ Accommodations for vision (visual acuity, color blindness, contrast, and others), hearing, symbolic
processing (dyslexia, and others), mobility, and other challenges
Mobile friendliness to enable broad device uses
Learner aids: cognitive scaffolding, effective feedback, access to review, access to downloadable
learning objects
30

31. Required Features of a Training LMS:
Basics (cont.)
Analysis of training efficacy / inefficacy (with downloadable data tables and effective data
visualizations)
Data analysis
System robustness (to legal standards and legal defensibility)
Automated and accurate data populating to university information systems (for recordkeeping)
Seamless linking to other learning resources, through inline frames (iframes), links, uploads, and
other integrations
31

32. Required Features of a Training LMS:
Advanced
Branching logic
Customizations for learners
◦ Based on learner profiles
◦ Based on learner performance (prior or current)
Select email contacts (through triggers)…to enable further human support or interventions
Select contact list populating (through triggers)
Built-in data analytics
◦ Cross-tabulation analysis
◦ Text analysis
32

33. Takeaways
Technology tools may be designed for particular target purposes, but there are other
applications that may be applied. It helps to decouple a tool from its original stated purposes
and to look at functionality alone.
A tool has to be tested from beginning-to-end to make sure everything functions as required.
It helps to have developers look at API (application programming interface) functions, too, to
add capabilities to a particular tool instance.
33

34. Additional Desired
Features
34

35. Additional Desired Features in Qualtrics
as a Training LMS
Social profiles of learners
◦ Possible to build using {a} piped text feature
Heightened learner awarenesses of each other (for learning communities)
◦ Possible with third-party tool integrations
Ways to extract indicators that are correlated with learning
◦ Possible with decision tree learning in an external machine learning tool
More item analysis and learner performance features
Suggested ways to improve trainings
Easier integration of off-campus individuals to collaborate on the design of trainings
35

36. Additional Desired Features in Qualtrics
as a Training LMS (cont.)
Ways to create learning communities
Some third-party content integration for learning
User-created content sharing (for training)
Ability to create course sequences that are linked and auto-sequenced
36

37. Contact and Conclusion
Dr. Shalin Hai-Jew
◦ iTAC
◦ 212 Hale / Farrell Library
◦ Kansas State University
◦ shalin@k-state.edu
◦ 785-532-5262
The presenter has no formal tie to Qualtrics, Inc.
37