Presentation at the annual conference of the Higher Education Learning and Teaching Association (HELTASA) in Bloemfontein, South Africa, 20 November, 2014

1. Evidence-based
education as paradox:
A critique
By Paul Prinsloo
Annual conference of the Higher Education Learning and Teaching
Association of South Africa (HELTASA), 18-21 November, Bloemfontein,
South Africa

2. I do not own the copyright of any of the images in this
presentation. I hereby acknowledge the original
copyright and licensing regime of every image and source
I’ve used. Images used in this presentation have been
sourced from Google labeled for non-commercial reuse,
or from Flickr published under a CC license. Where no
ownership or license could be established, I indicate the
hyperlink address.
This work is licensed under a Creative Commons
Attribution-NonCommercial 4.0 International License

3. “In an age of advanced technology, inefficiency
is the sin against the Holy Spirit” (Morozov,
2013b, quoting Aldous Huxley, p. ix).
“ The strong desire for proof burns bright in
education” (Wagner & Ice, 2011, p. 36)
“…we optimise the measurable at the risk of
neglecting the immeasurable” (Richardson,
2012)

4. Purpose of this presentation
My purpose is not to debunk evidence-based
education as such but to assemble a number
of voices and histories in a reflective caring but
critical space (e.g. Latour, 2004). No one
disagrees that evidence is important.

5. … a first and necessary step in counteracting the force
of any discourse is to recognise its constitutive power,
its capacity to become hegemonic, ‘to saturate our very
consciousness, so that the … world we see and interact
with, and the commonsense interpretations we put on
it, become the world tout court, the only world’ (Apple,
1979, p. 5)”
(Davies, 2003, p. 102)

6. The elephant in the room…
(Denzin, 2009)
http://wrexhamfan.wordpress.com/2014/03/23/the-small-elephant-in-the-room/

7. The signal and the noise
“Most of it is just noise, and the noise is increasing
faster than the signal”
(Silver, 2012, p. 13)
The “problem with predicting the future is rarely the
predictions themselves, but rather the base
assumptions that make it the logical progression”
(Tweet: InfoSec Taylor Swift, 2014)

8. Situating evidence-based education (1) – the
higher education landscape: looking for a center
that holds
Disruption
Technosolutionism
Innovation
Rankings
Unbundling and
unmooring
Revolution
Crisis
Increasing
casualisation
of faculty
Privatisation of
higher education
Doing more
The data deluge
We need
evidence of
what works…
Accountability with less
Quantification fetish
Disaggregation

9. Situating evidence-based education (2) – the
higher education landscape: data
Student data as “the
new oil” (Watters, 2013)
“…the claims about big data and education are
incredibly bold, and as yet, unproven” (Watters,
2013, par. 17)
Learning analytics as the “new black”
(Booth, 2011)
Learning analytics as “the
new black” (Booth, 2011)
“Most of it [the data] is just noise, and the noise is
increasing faster than the signal” (Silver, 2012, p.
13)
Image credits: https://flic.kr/p/dSHr87

10. Hartley (1995) - McDonaldisation of
higher education
• Impact of external scrutiny,
inspection, increasing emphasis on
efficiency, calculability, predictability,
and control
http://commons.wikimedia.org/wiki/File:
Mcdonalds-90s-logo.svg
• Doing more with less
• Funding following performance
rather than preceding it
Commentators “claim that we [higher education] cost too
much, spend carelessly, teach poorly, plan myopically and
when questioned, act defensively (Lagowski 1995, p. 861)

11. Overview of the presentation
1. Problematising
evidence/data/accountability/quantification
2. Roots of evidence-based education
• Colonialist roots
• Neoliberalism and managerialism
• The ‘gold standard’ of evidence in medicine
• Concerns about educational research
3. Problematising evidence-based education
• Epistemological concerns
• Ontological concerns
• Evidence and power
4. Towards value-based education

12. Problematising evidence
evidence/data/accountability/quantificati
on
Data cannot and do not speak for itself (Boyd &
Crawford, 2013; Gitelman, 2013)
“…data are political in nature – loaded with values,
interests and assumptions that shape and limit
what is done with it and by whom” (Selwyn, 2014,
p. 6)

13. Roots 1: Colonialist roots (Shahajan, 2011)
http://en.wikipedia.org/wik
i/British_Empire
http://en.wikipedia.org/wiki/Slave_breeding_in_the_
United_States

14. From a “masculinist epistemology of science”
and epistemologies of control it is necessary
to “systematise everything, reducing them to
manageable questions and subjects, and then
find some causal links between them”
(Shahjahan, 2011, p. 188)

15. “Students are reduced to test scores, future slots
in the labor market, prison numbers, and possible
cannon fodder in military conquests. Teachers are
reduced to technicians and supervisors in the
education assembly line – ‘objects’ rather than
‘subjects’ of history. This system is fundamentally
about the negation of human agency, despite the
good intentions of individuals at all levels”
(Lipman, 2004, as quoted by Shahjahan, 201, p.
196).

16. Evidence-based education therefore becomes
synonymous with the “thingification” of
education (Shahjahan, 2011, p. 197).

17. Roots 2: Neoliberalism & managerialism
Neoliberalism and
managerialism heralds
the most significant shift
in “the discursive
construction of
professional practice that
any of us will ever
experience”
(Davies, 2003, p. 91)
https://openclipart.org/detail/170056
/they-are-watching-you-by-asrafil

18. “ … as long as the objectives have been specified and
strategies for their management and surveillance put
in place, the nature of the work itself is of little
relevance to anyone.”
“If the auditing tools say that the work has, on
average, met the objectives, it is simply assumed
that the work has been appropriately and
satisfactorily tailored according to the requirements
of the institution (and often of the relevant funding
body).”
(Davies, 2003, p. 92; emphasis added)

19. Those who resist “the terms of auditors and
economists” are identified for re-education (Davies,
2003, p. 93).
https://www.flickr.com/photos/13476480@
N07/8680070938

20. Roots 3: The ‘gold standard’ of evidence in
medicine
We need to question the “homology between
education and medicine” and Biesta points to “the
different meanings of evidence in these fields”
(Biesta, 2007, p. 4).
Students are not patients or ill and education is not a
cure (Biesta, 2007, 2010)
(Also see Simons, 2003)

21. Roots 4: Concerns about educational
research
Current educational research and professional
practices don’t provide the answers government is
looking for (Biesta, 2007, 2010)
Experimental research is seen as “the only method
capable of providing secure evidence about ‘what
works’ (Biesta, 2007, p. 3; emphasis added)

22. Problematising evidence-based education
1. Epistemological concerns
2. Ontological concerns
3. Concerns re the relationship between evidence and
power

23. Epistemological concerns
“… research knowledge is always fallible, even if it is
more likely to be valid than knowledge from other
sources” (Hammersley, 2001, par. 7).
https://flic.kr/p/4932J1

24. Our increasing assumptions about and reliance on
algorithms resemble a possible “gnoseological turning
point” in our understanding of knowledge, information
and faculties of learning where bureaucracies
increasingly aspire to transform and reduce “ontological
entities, individuals, to standardised ones through
formal classification” into algorithms and calculable
processes (Totaro & Ninno, 2014, p. 29).

25. Experimentation and interventions are not
disentangled but embedded and part of the system
under investigation(Barad, 2007; Biesta, 2010, ).

26. Ontological concerns
Education is not a closed system or a “causal
technology” but an “open and recursive system”
(Biesta, 2007, p. 8)
Research and evidence “can tell us what worked but
cannot tell us what works (Biesta, 2007, p. 16).
In an open and recursive system it is highly
improbable that the same solution will work again…

27. http://commons.wikimedia.org/wiki/File:C
ynefin_framework_Feb_2011.jpeg
The Cynefin
framework (Snowden
& Boone, 2007)

28. Data, vidence and power
https://flic.kr/p/6hv3zZ

29. Data, vidence and power
http://www.websophist.com/Profiling_TSA_Toon.jpg

30. “… data are political in nature – loaded with values,
interests and assumptions that shape and limit what is
done with it and by whom” (Selwyn, 2014; p. 6).
As data sets are increasingly combined and reused, it is
important to acknowledge that “data itself can take on
its own life” where the original context and intention
of harvesting, as well as the original assumptions
informing the parameters are lost (Selwyn, 2014, p. 7).

31. Towards a value-based education
We accept that data and evidence are
• Anything but neutral or speak for itself (Gitelman,
2003)
• Embedded in historical and present socio-economic,
ideological and geopolitical power
relations (Apple, 2010; Castells, 2009; Henman,
2001; Selwyn, 2014)
• Not the whole/real picture(Mayer-Schönberger,
2009)
• Precariously temporary, fragile and mutable
(Fenwick & Edwards, 2014)

32. Noddings (1999) moots the interesting point
that “when a just decision has been reached,
there is still much ethical work to be done” (p.
16).
So while it is important to ask whether an
intervention is effective, it is even more
important to ask whether it is appropriate,
caring and interrupt cycles of inequity and
injustice

33. Thank you. Baie dankie. Ke a leboga
Paul Prinsloo
Research Professor in Open Distance Learning
3-15, Club 1
P O Box 392
Unisa
0003
prinsp@unisa.ac.za
http://opendistanceteachingandlearning.wordpress.com
Twitter: 14prinsp
Office: +27 12 433 4719

34. References and reading list
Altbach, P.G. (1999). The logic of mass higher education. Tertiary Education and
Management, 5(2), 107—124.
Altbach, P.G., Reisberg, L., & Rumbley, L.E. (2009). Trends in global higher education:
tracking an academic revolution. A report prepared for the UNESCO 2009
World Conference on Higher Education. Retrieved from
http://graduateinstitute.ch/files/live/sites/iheid/files/sites/developpement/
shared/developpement/cours/E759/Altbach,%20Reisberg,%20Rumbley%20
Tracking%20an%20Academic%20Revolution,%20UNESCO%202009.pdf
Andrejevic, M. (2014). The Big Data divide. International Journal of Communication,
8, 1673—1689 .
Apple, M.W. (2010). Global crises, social justice, and education. New York, NY:
Routledge.
Barad, K. (2007. Meeting the universe halfway. Quantum physics and the
entanglement of matter and meaning. London, UK: Duke University Press.
Beer, C., Jones, D., & Clark, D. (2012). Analytics and complexity: Learning and
leading for the future. In Proceedings of the 29th Annual Conference of the
Australasian Society for Computers in Learning in Tertiary Education
(ASCILITE 2012) (pp. 78-87). Australasian Society for Computers in Learning
in Tertiary Education (ASCILITE).

35. Berliner, D.C. (2002). Comment: Educational research: The hardest science of all.
Educational Researcher, 31, 18—20. DOI: 10.3102/0013189X031008018.
Biesta, G. (2007). Why “what works” won’t work: evidence-based practice and the
democratic deficit in educational research. Educational Theory, 57(1), 1–22. DOI:
10.1111/j.1741-5446.2006.00241.x.
Biesta, G. (2010). Why ‘what works’ still won’t work: from evidence-based education to
value-based education. Studies in Philosophy of Education, 29, 491–503. DOI
10.1007/s11217-010-9191-x.
Boyd, D., & Crawford, K. (2013). Six provocations for Big Data. Retrieved from
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431
Carr, N. (2012). The crisis in higher education. Technology Review. Retrieved from
http://www.technologyreview.com/featuredstory/429376/the-crisis-in-higher-education/
Castells, M. (2009). Communication power. Oxford, UK: Oxford University Press.
Christensen, C. (2008). Disruptive innovation and catalytic change in higher education.
Forum for the Future of Higher Education. EDUCAUSE,[Online].Retrieved from
https://net.educause.edu/ir/library/pdf/ff0810s.pdf

36. Clegg, S. (2005). Evidence-based practice in educational research: a critical realist
critique of systematic review. British Journal of Sociology of Education, 26(3):
415-428. DOI: 10.1080/01425690500128932
Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6),
683—695. DOI: 10.1080/13562517.2013.827653
Davies, P. (1999). What is evidence-based education? British Journal of Educational
Studies, 47(2): 108-121.
Davies, B. (2003). Death to critique and dissent? The policies and practices of new
managerialism and of 'Evidence-based Practice', Gender and Education, 15(1),
91-103, DOI: 10.1080/0954025032000042167
Elliot, J. (2001). Making evidence-based practice educational. British Educational
Research Journal, 27(5): 555-574. DOI: 10.1080/01411920120095735
Eynon, R. (2013). The rise of Big Data: what does it mean for education, technology,
and media research? Learning, Media and Technology, DOI:
10.1080/17439884.2013.771783
Fenwick, T., & Edwards, R. (2010). Actor-network theory in education. London, UK:
Routledge.
Fenwick, T., & Edwards, R. (2014). Networks of knowledge, matters of learning, and
criticality in higher education. Higher Education, 67: 35-50. DOI:
10.1007/s10734-013-9639-3

37. Francis, B. (2006) Heroes or zeroes? The discursive positioning of ‘underachieving boys’ in
English neo‐liberal education policy, Journal of Education Policy, 21(2),187-200,
DOI: 10.1080/02680930500500278
Gitelman, L. (Ed.). (2013). “Raw data” is an oxymoron. London, UK: MIT Press.
Hammersley, M. (2001). Some questions about evidence-based practice in education.
Retrieved from http://www.leeds.ac.uk/educol/documents/00001819.htm
Hargreaves, A., & Braun, H. (2013). Data-driven improvement and accountability. National
Education Policy Centre. Retrieved from http://co.chalkbeat.org/wp-content/
uploads/sites/2/2013/10/PB-LB-DDIA-POLICY-FINAL-EMBARGOED.pdf
Hartley, D. (1995). The ‘McDonaldisation’ of higher education: food for thought? Oxford
Review of Education, 21(4): 409—423.
Henman, P. (2004). Targeted!: Population segmentation, electronic surveillance and
governing the unemployed in Australia. International Sociology, 19, 173-191. DOI:
10.1177/0268580904042899
InfoSec Taylor Swift. [@SwiftOnSecurity] (2014, 8 November). Problem with predicting the
future is rarely predictions themselves, but rather the base assumptions that make
it the logical progression. [Tweet]. Retrieved from
https://twitter.com/SwiftOnSecurity/status/530831217227497472

38. Kitchen, R. (2014). The data revolution. New York, NY: SAGE.
Knight, S., & Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment,
pedagogy: where learning meets analytics in the middle space. Journal of
Learning Analytics. (In press). Retrieved from http://oro.open.ac.uk/39226/
Lagowski, J.J. (1995). Higher education: a time for triage? Journal of Chemical
Education, 72(10): 861.
Lather, P. (2004). This is your father's paradigm: government intrusion and the case of
qualitative research in education, Qualitative Inquiry 10: 15-34. DOI:
10.1177/1077800403256154
Latour, B. (2004). Why has critique run out of steam? From matters of fact to matters
of concern. Critical Inquiry, 30: 225-248.
Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. LAK '11
Proceedings of the 1st International Conference on Learning Analytics and
Knowledge, pp. 153—156. Retrieved from
http://dl.acm.org/citation.cfm?id=2090140.
Mayer-Schönberger, V. (2009). Delete. The virtue of forgetting in the digital age.
Princeton, NJ: Princeton University Press.
Mayer-Schönberger, V., & Cukier, K. (2013). Big data. A revolution that will transform
how we live, work, and think. New York, N.Y.: Houghton Miffling Harcourt Publishing
Company
Morozov, E. (2013). To save everything, click here. London, UK: Penguin Books.

39. Nutley, S., Walter, I., & Davies, H. T. (2003). From knowing to doing a framework for
understanding the evidence-into-practice agenda. Evaluation, 9(2): 125-148.
Oliver, M., & Conole, G. (2010). Evidence-based practice and e-learning in higher
education: can we and should we? Research Papers in Education, 18(4): 385-397.
DOI: 10.1080/0267152032000176873
Olssen, M., & Peters, M.A. (2005) Neoliberalism, higher education and the knowledge
economy: from the free market to knowledge capitalism, Journal of Education
Policy, 20(3): 313-345, DOI: 10.1080/02680930500108718
Pirrie, A. (2001). Evidence-based practice in education: The best medicine? British Journal
of Educational Studies, 49(2), 124—136. DOI: 10.1111/1467-8527.t01-1-00167.
Prinsloo, P. (2014). A brave new world: student surveillance in higher education. Paper
presented at the 21st Southern African Association for Institutional Research
(SAAIR), Pretoria, 16-18 September. Retrieved from
http://www.slideshare.net/prinsp/a-brave-new-world-student-surveillance-in-higher-
education.
Prinsloo, P., & Slade, S. (2014). Educational triage in higher online education: walking a
moral tightrope. International Review of Research in Open Distance Learning
(IRRODL), 14(4): 306-331.
http://www.irrodl.org/index.php/irrodl/article/view/1881

40. Reeves, T. (2011). Can educational research be both rigorous and relevant? Educational
Designer. Journal of the International Society for Design and Development in
Education. Retrieved from
http://www.educationaldesigner.org/ed/volume1/issue4/article13/index.htm
Richardson, W. (2012, July 14th). Valuing the immeasurable. [Web log post]. Retrieved
from http://willrichardson.com/post/27223512371/valuing-the-immeasurable
Schildkamp, K., Lai, M.K., & Earl, L. (eds). (2013). Data-based decision making in
education. Challenges and opportunities. London, UK: Springer.
Selwyn, N. (2014). Data entry: towards the critical study of digital data and education.
Learning, Media and Technology, 1-19. DOI: 10.1080/17439884.2014.921628
Simons, H. (2003). Evidence-based practice: panacea or over promise? Research Papers
in Education, 18(4): 303-311.
Silver, N. (2012). The signal and the noise. The art and science of prediction. London, UK:
Penguin.
Shahjahan, R.A. (2011). Decolonizing the evidence‐based education and policy
movement: revealing the colonial vestiges in educational policy, research, and
neoliberal reform, Journal of Education Policy, 26(2): 181-206, DOI:
10.1080/02680939.2010.508176

41. Siemens, G. (2011). Learning analytics: a foundation for informed change in higher
education. Retrieved from
http://www.educause.edu/library/resources/learning-analytics-foundation-informed-
change-higher-education
Simons, H. (2003). Evidence-based practice: panacea or over promise? Research Papers
in Education, 18(4): 303-311. DOI: 10.1080/0267152032000176828
Snowden, D.J., & Boone, M.E. (2007). A leader’s framework for decision making.
Harvard Business Review, November: 1-9.
Solove, D.J. (2001) Privacy and power: Computer databases and metaphors for
information privacy. Stanford Law Review, 53(6), 1393—1462.
Swain, H. (2013, August 5). Are universities collecting too much information on staff and
students? [Web log post]. The Guardian. Retrieved from
http://www.theguardian.com/education/2013/aug/05/electronic-data-trail-huddersfield-
loughborough-university
Sykes, G., Schneider, B., & Plank, D.N. (2009). Handbook of education policy research.
New York, NY: Routledge.
Totaro, P., & Ninno, D. (2014). The concept of algorithm as an interpretive key of
modern rationality. Theory Culture Society 31, pp. 29—49. DOI:
10.1177/0263276413510051

42. Wagner, E., & Ice, P. (2012, July 18). Data changes everything: delivering on the promise
of learning analytics in higher education. EDUCAUSEreview, [online]. Retrieved
from http://www.educause.edu/ero/article/data-changes-everything-delivering-promise-
learning-analytics-higher-education
Waitere , H.J., Wright, J., Tremaine, M., Brown, S., & Pausé, C.J. (2011). Choosing
whether to resist or reinforce the new managerialism: the impact of
performance‐based research funding on academic identity, Higher Education
Research & Development, 30(2): 205-217, DOI: 10.1080/07294360.2010.509760
Watters, A. (2012). Unbundling and unmooring: technology and the higher ed tsunami.
EDUCAUSE review, [online]. Retrieved from
http://www.educause.edu/ero/article/unbundling-and-unmooring-technology-and-
higher-ed-tsunami
Watters, A. (2013, October 13). Student data is the new oil: MOOCs, metaphor, and
money. [Web log post]. Retrieved from
http://www.hackeducation.com/2013/10/17/student-data-is-the-new-oil/

43. Wiley, D., & Hilton III, J. (2009). Openness, dynamic specialization, and the
disaggregated future of higher education. International Review of Research in
Open and Distance Learning, 10(5), 1—16. Retrieved from
http://www.irrodl.org/index.php/irrodl/article/view/768/1414
Wishon, G.D., & Rome, J. (2012, 13 August). Enabling a data-driven university.
EDUCAUSEreview, [online]. Retrieved from
http://www.educause.edu/ero/article/enabling-data-driven-university