The EdTech Efficacy Research Symposium was held in May 2015. This presentation reviews lessons learned from the research in higher education working group. Results included networking with peers, conducting pilots and decision making strategies differ by organization type.

1. EdTech Decision-Making in Higher Education:
Review of Findings
Dr. Kristin Palmer
Dr. Whitney Kilgore

2. Expectations
I see value in networking.
I am an introvert.
SO
You will talk with one another.

3. Virtual Attendees Collaborate on Google Doc
● https://goo.gl/KiMwyx

4. http://symposium.curry.virginia.edu/reports/

5. Cut to the Chase
● We do pilots to evaluate edtech
● We’ll share information if asked
● We’re inundated with information
● We talk to our peers at conferences
● We read newsletters and follow social media
● We propose creating a template and repository

6. What role are you?
 Administrator
 Faculty
 Staff
 Researcher
 Venture Capitalist
 Ed Tech Business Owner

7. What happened in this study?
Conducted 47 interviews from 43 institutions with 52 decision-makers
Had interviewees describe one particular EdTech decision in detail
Interviews were recorded or annotated and then coded for themes

8. Why did we do this study?
Understand what factors and information influence EdTech decisions
Provide transparency regarding decision making
Identify and showcase best practices

9. What is Higher Ed trying to accomplish with EdTech?
41% Support Teaching and Learning
36% Gain Operational Efficiencies/Decrease Costs
30% Increase capacity to serve students online
27% Improve user experience/modernize existing system

10. Support of Teaching and Learning
Collaboration
Individualization
Active learning
Virtual reality
Authentic assessments
Competency-based education
Flipped classrooms and blended learning
Feedback

11. 3-2-1 with your Neighbor
3) What are 3 things you try to do with edtech?
2) What are two edtech misconceptions?
1) What is the one thing you want to walk out of
this session with?

12. Where do EdTech decision-makers get their
information?
93% Network Events/Conferences
91% Written Publications
89% Social Media and Online Communications

13. “I get bombarded with stuff”
Interviewees reported that their major sources of information include:

14. Types of publications referenced

15. Social media/online communications mentioned

16. Share with your OTHER neighbor!
What’s a source of information you find super valuable and
follow regularly?
Share these with us: #OLCAccelerate2017

17. All interviewees stated they conduct research:
38% Review student outcomes after implementing a strategy/product
29% Asked other IHEs for feedback about products
24% Administered own student /faculty/staff surveys
22% Ran Pilots
18% Read scholarly articles and journals

18. Thoughts from the interviewees on research:
There is a lack of efficacy research
It’s hard to do...
a. Pace of EdTech change
b. Context differences
Results of pilot studies are rarely shared outside of an IHE

19. Thoughts from the interviewees on research:
78% indicated they do their own research
Studies varied widely in goals and methodological rigor with few resulting in
peer-reviewed publications and most not being shared publicly
Results were often used for:
● continuous improvement of instruction
● deciding whether to continue or scale up pilots
Others lack “bandwidth” to do research

20. Time to share with the person in front/behind you!
What is an edtech solution you were a part of implementing?
What worked? What didn’t?
How did you share that out?

21. EdTech products interviewees were considering
27% Learning Management System (LMS)
9% Switching from Textbooks to Digital Content
7% Online/Blended Course Design
7% Assessment Tools

22. What criteria mattered for decision making?
Feasibility of implementation
Features and functionality
User experience/usability
Cost/ROI considerations
Vendor capacity and relationship

23. Evaluating EdTech options
73% Decisions involved cost assessment
64% Decisions informed by vendor demos
59% Decisions involved IT vetting
47% Involved formal RFP/RFI/RFQ
39% Involved piloting solution

24. Decision Making Processes
At public and non-profit, tends to be protracted and inclusive with
strong faculty voice. Input is gathered in meetings, committees,
and surveys with the final decision most often being made by one
or more administrators.
For-profits tend to have swifter, more centralized decision-making
processes with faculty and student buy-in sought after decision is
made.

25. Group share: How does this work for you?
● How do you make decisions – centralized, decentralized, top-down or bottom up?
● What do you use to make decisions – ROI, RFP, peer results, pilot?
● How do you conduct research – ask peers, pilot, vendors, papers?

26. Advice for EdTech Decision Makers:
Talk to people outside of higher education
Focus on functionality that faculty and students use
Consider ancillary costs such as training
Standardize the EdTech procurement process
Involve stakeholders (faculty, staff, students) early in process
Plan change facilitation strategies

27. Advice for Researchers:
Focus on pedagogical strategies the technology supports and how to
use it better to improve student outcomes (What works, how, and in
what conditions)
Use cases and studies of real-world implementations are most useful
to decision-makers
Differentiate findings by context and types of students

28. Advice for Vendors:
Do due diligence for clients and tailor your pitch for their specific needs
Transparency around product and functionality is critical
Aim for mutually beneficial relationships with “partners”
Consider pro bono exchanges with schools - product for research
Be prepared to customize your product

29. Advice for Funders:
Fund a hub for sharing results of EdTech pilots for higher education
Incentivize grantees to build in time and resources to share their internal
study findings with peers
Develop a tiered system of funding to support edtech research at
amounts appropriate to the level of higher education investment in the
product/strategy

30. https://www.edtechdecisionmakinginhighered.org/

31. Where do we go from here?
What does a repository look like?
What is on the template?
What does maintenance look like?

32. Some quotes for reference…

33. Use of Research in Decision Making
Judy Giering, Director, Learning Design and Technology at UVa: “We don't necessarily
look for research – and I mean real research – on a specific tool. Frankly, I'm not really
aware of a lot of research on tool X that's been done with a real methodology. … But we
do turn to the research to really help build a rationale for things like active learning or
digital portfolios. ... We've been looking at research into whether this type of tool impacts
learning. But in terms of the three or four or five tools that we've become aware of, we're
kind of just saying, "Okay, let's get a demo. Let's start asking our questions." A lot of times
we have to connect dots between what we know about pedagogy and what the tools can
do, for example, what we know about active learning and
whether specific tools enable us to do active learning better.”

34. Peer to peer information exchange
Adrian Sannier, Chief Academic Technology Officer at Arizona State University
“I talk to CIOs at other universities, especially if they have tried some EdTech that we are
considering ourselves. Higher ed is a ‘near follower industry.’ People like to see whether
things have worked for someone else first before they jump in.”

35. Solving decentralized EdTech procurement
Michelle R. Weise, Chief Innovation Officer of the Sandbox ColLABorative at SNHU:
“We've tried to now funnel all of the first looks at EdTech products through the Sandbox--
not that we dictate who gets to buy what, but it's more of a centralization of channels, so
that we're not duplicating each other’s efforts and we're not having three different license
agreements with the same vendor.”

36. For profit vs. nonprofit decision timelines
Andrew Shean, Chief Academic Learning Officer, Bridgepoint Education:
“I'd say it's probably seven months from soup to nuts, to make the [LMS] selection. Our
previous president was the Chancellor of University of Maine's system. When he came
here, he said the difference was like between driving a cruise ship and driving a sports car.
Kind of good and bad. You could make bad decisions really quickly.”

37. The pitfalls of decentralized decision-making
Prof. Phillip Long, Associate Vice Provost & Chief Innovation Officer at the University of
Texas at Austin:
“Independence of the faculty to make their own decisions about technology without a
consultation or discussion has in some instances led to the circumstance where a student
can, for example, be taking four courses, all of which require a different brand of clicker,
because the independent choice of the faculty was to choose one that they liked. Now,
the student is spending an extra $200 for that term and juggling carrying around four
different clicker brands.”

38. Needs first or EdTech solutions first?
Ron Hutchins, VP – IT at University of Virginia:
“Technology in and of itself is useless unless it's in support of the business of the
university..we start with, ‘Here's a tool.’ It's kind of like the old adage, ‘Bring me a rock.’
‘No. This isn't the rock I want.’ ‘Well, what's wrong with it?’ ‘I don't know. Bring me
another rock.’ We keep looking for new rocks until we find something and somebody says,
‘Oh yeah. That's the rock I want.’”