Results of 47 interviews with 54 educational technology decision makers in higher education institutions on their research practices and decision making for edtech. This presentation was presented on Wednesday, May 3, 2017 at the Ed Tech Efficacy Symposium in Washington, DC.

1. EdTech Decision-making in Higher Education:
Working Group B Findings
Fiona Hollands
Kristin Palmer
Stephanie Moore
Whitney Kilgore

2. Thanks to Digital Promise, Curry/UVa, JEA, our
interviewees, sponsors, and Working Group B members:
Alison Griffin
Amy Bevilacqua
Bill Hansen
Bror Saxberg
David Kim
Deborah Quazzo
Emily Kinard
Jerry Rekart
Kristin Palmer
Matt Chingos
Maya Escueta
MJ Bishop
Phil Hill
Stephanie Moore

3. A. Role of Research in K-12 EdTech Decision Making
B. Role of Research in Higher Ed EdTech Decision Making
C. Research Spending and Popular EdTech Products
D. Evidence and Quality of Efficacy in Research Approach
E. Institutional Competence in Evaluating Efficacy Research
F. Investors and Entrepreneurs
G. Goals and Roles of Federal Funding for EdTech Research
H. Education Philanthropies
Where do we fit?

4. https://www.edtechdecisionmakinginhighered.org/

5. What did we do?
Conducted 47 interviews from 43 colleges/universities with 52 decision-
makers representing different types of higher education institutions to
understand EdTech decision-making
Asked about sources and networks and had interviewees describe one
particular EdTech decision in detail
Interviews were recorded or annotated and then coded for themes

6. Why did we do it?
Understand what factors and information - including research -
influence EdTech decisions
Provide transparency regarding decision making
Identify and showcase best practices

7. Needs first or EdTech solutions first?
Ron Hutchins, VP – IT at University of Virginia:
“Technology in and of itself is useless unless it's in support of the business of the
university. To me this is one of the things that we miss on a continual basis: we don't start
with the business of education, we start with, ‘Here's a tool.’ It's kind of like the old
adage, ‘Bring me a rock.’ ‘No. This isn't the rock I want.’ ‘Well, what's wrong with it?’ ‘I
don't know. Bring me another rock.’ We keep looking for new rocks until we find
something and somebody says, ‘Oh yeah. That's the rock I want.’”

8. What is Higher Ed trying to accomplish with EdTech?
41% Support Teaching and Learning
36% Gain Operational Efficiencies/Decrease Costs
30% Increase capacity to serve students online
27% Improve user experience/modernize existing system

9. Unpacking: 41% Support Teaching and Learning
Collaboration
Individualization
Active learning
Virtual reality
Authentic assessments
Competency-based education
Flipped classrooms and blended learning
Feedback

11. Peer to peer information exchange
Adrian Sannier, Chief Academic Technology Officer at Arizona State University
“I talk to CIOs at other universities, especially if they have tried some EdTech that we are
considering ourselves. Higher ed is a ‘near follower industry.’ People like to see whether
things have worked for someone else first before they jump in.”

12. “I get bombarded with stuff”
Interviewees reported that their major sources of information include:

13. Where do EdTech decision-makers get their
information?
93% Network Events/Conferences
91% Written Publications
89% Social Media and Online Communications

14. Types of publications referenced

15. Social media/online communications mentioned

16. All interviewees stated they conduct research:
38% Review student outcomes after implementing a strategy/product
29% Asked other IHEs for feedback about products
24% Administered own student /faculty/staff surveys
22% Ran Pilots
18% Read scholarly articles and journals

17. Thoughts from the interviewees on research:
There is a lack of efficacy research
It’s hard to do...
a. Pace of EdTech change
b. Context differences
Results of pilot studies are rarely shared outside of an IHE

18. Thoughts from the interviewees on research:
78% indicated they do their own research
Studies varied widely in goals and methodological rigor with few resulting in
peer-reviewed publications and most not being shared publicly
Results were often used for:
continuous improvement of instruction
deciding whether to continue or scale up pilots
Others lack “bandwidth” to do research

19. Use of Research in Decision Making
Judy Giering, Director, Learning Design and Technology at UVa: “We don't necessarily
look for research – and I mean real research – on a specific tool. Frankly, I'm not really
aware of a lot of research on tool X that's been done with a real methodology. … But we
do turn to the research to really help build a rationale for things like active learning or
digital portfolios. ... We've been looking at research into whether this type of tool impacts
learning. But in terms of the three or four or five tools that we've become aware of, we're
kind of just saying, "Okay, let's get a demo. Let's start asking our questions." A lot of times
we have to connect dots between what we know about pedagogy and what the tools can
do, for example, what we know about active learning and
whether specific tools enable us to do active learning better.”

20. POLL: Which type of EdTech product/strategy was most
often being considered by decision-makers?
Virtual reality
Student advising
Gamification
Simulations
LMS

21. EdTech products interviewees were considering
27% Learning Management System (LMS)
9% Switching from textbooks to digital content
7% Online/Blended course design
7% Assessment tools

22. Decision Making Processes
At public and non-profit, tends to be protracted and inclusive with
strong faculty voice. Input is gathered in meetings, committees,
and surveys with the final decision most often being made by one
or more administrators.
For-profits tend to have swifter, more centralized decision-making
processes with faculty and student buy-in sought after decision is
made.

23. What criteria mattered for decision making?
Feasibility of implementation
Features and functionality
User experience/usability
Cost/ROI considerations
Vendor capacity and relationship

24. Evaluating EdTech options
73% Decisions involved cost assessment
64% Decisions informed by vendor demos
59% Decisions involved IT vetting
47% Involved formal RFP/RFI/RFQ
39% Involved piloting solution

25. The pitfalls of decentralized decision-making
Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer at the University of
Texas at Austin:
“Independence of the faculty to make their own decisions about technology without a
consultation or discussion has in some instances led to the circumstance where a student
can, for example, be taking four courses, all of which require a different brand of clicker,
because the independent choice of the faculty was to choose one that they liked. Now,
the student is spending an extra $200 for that term and juggling carrying around four
different clicker brands.”

26. For profit vs. nonprofit decision timelines
Andrew Shean, Chief Academic Learning Officer, Bridgepoint Education:
“I'd say it's probably seven months from soup to nuts, to make the [LMS] selection. Our
previous president was the Chancellor of University of Maine's system. When he came
here, he said the difference was like between driving a cruise ship and driving a sports car.
Kind of good and bad. You could make bad decisions really quickly.”

27. Solving decentralized EdTech procurement
Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU:
“We've tried to now funnel all of the first looks at EdTech products through the Sandbox--
not that we dictate who gets to buy what, but it's more of a centralization of channels, so
that we're not duplicating each other’s efforts and we're not having three different license
agreements with the same vendor.”

28. POLL: What was the hard part?
Change management on campus after implementation
Identifying suitable EdTech options
Sorting through too much information on the technology
Supporting the technology

29. Advice for EdTech Decision Makers:
Talk to people outside of higher education
Focus on functionality that faculty and students use
Consider ancillary costs such as training
Standardize the EdTech procurement process
Involve stakeholders (faculty, staff, students) early in process
Plan change facilitation strategies

30. Advice for Researchers:
Focus on pedagogical strategies the technology supports and how to
use it better to improve student outcomes (What works, how, and in
what conditions)
Use cases and studies of real-world implementations are most useful
to decision-makers
Differentiate findings by context and types of students

31. Advice for Vendors:
Do due diligence for clients and tailor your pitch for their specific needs
Transparency around product and functionality is critical
Aim for mutually beneficial relationships with “partners”
Consider pro bono exchanges with schools - product for research
Be prepared to customize your product

32. Advice for Funders:
Fund a hub for sharing results of EdTech pilots for higher education
Incentivize grantees to build in time and resources to share their internal
study findings with peers
Develop a tiered system of funding to support edtech research at
amounts appropriate to the level of higher education investment in the
product/strategy

33. What recommendations and
next steps are you interested in
discussing further now?