Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Truth, Justice, and Technicity: from Bias to the Politics of Systems


Published on

Keynote at the Digital Methods Summer School, University of Amsterdam, July 2, 2018

Published in: Education
  • Be the first to comment

  • Be the first to like this

Truth, Justice, and Technicity: from Bias to the Politics of Systems

  1. 1. Truth, Justice, and Technicity: from Bias to the Politics of Systems Bernhard Rieder Mediastudies Department Universiteit van Amsterdam Amsterdam, July 2, 2018
  2. 2. Introduction Terms like data mining, data science, big data, machine learning, or algorithmic decision- making point toward a set of practices that have come to play important roles in a variety of domains. These roles are increasingly under scrutiny. What is the role of the humanities scholar in this?
  3. 3. computer says no… Areas of decision-making like hiring, criminal justice and policing, access to credit and insurance, dynamic pricing, and information ordering are some of the most emblematic problem areas.
  4. 4. Data mining is concerned with assessing differences and similarities between entities in a dataset in the context of some task or decision. The desire to discriminate – in "decentered, non-traditional societies" (Giddens 1994) that complicate "signaling" (Spence 1973) – lies at the heart of the practice, raising specific issues in areas like hiring, credit, or criminal justice. "We have stipulated that the employer cannot directly observe the marginal product prior to hiring. What he does observe is a plethora of personal data in the form of observable characteristics and attributes of the individual, and it is these that must ultimately determine his assessment of the lottery he is buying." (Spence 1973) Data mining is often used to realize "interested readings of reality" that detect patterns in data as they relate to desired operational outcomes. Interested readings
  5. 5. Interested readings Machine learning, in particular, shifts the normative "core" of decisions towards the empirical made data, the target variable, and some process of labeling. Signals become meaningful in relation to a distinction. This indicates a further shift from "metrological realism" to "accounting realism" where the "'equivalence space' is composed not of physical quantities (space and time), but of a general equivalent: money" (Desrosières 2001). Inductive techniques allow for a deep embedding of "interestedness" into practices and infrastructures.
  6. 6. This presentation The "dominant perspective" on the normative dimension of algorithmic decision-making is highlighting important problems, but the phenomenon merits broader interrogation. This presentation suggests two directions for critical expansion: 1) interrogating and engaging notions of justice; 2) thinking in terms of systems rather than individual algorithms; These lines of reasoning lead toward: 3) three ideas for possible digital methods projects;
  7. 7. Common critiques distinguish between "epistemic concerns" (~truth) and "normative concerns" (~justice) raised by algorithmic decision-making. (cf. Mittelstadt et al. 2016) Expansion 1: from truth to Justice Epistemic concerns include the transformations of probability assessments into binary decisions, opacity, and data problems such as the over- or underrepresentation of a group in a dataset. Normative concerns include disparate impact, loss of autonomy, and challenges to privacy.
  8. 8. "An action can be found discriminatory, for example, solely from its effect on a protected class of people, even if made on the basis of conclusive, scrutable and well-founded evidence." (Mittelstadt et al. 2017) This more difficult problem arises because social reality is biased: "Data mining takes the existing state of the world as a given, and ranks candidates according to their predicted attributes in that world." (Barocas & Selbst 2015) Decisions are based on the traces from societies characterized by centuries of inequality and domination. Data mining can make these inequalities actionable and lead to disparate impact on protected classes of people. Proposed solutions focus on "disparate impact detection" (Barocas & Selbst 2015) and (technical) compensation strategies that implement "fair affirmative action" (Dwork et al. 2011) Disparate impact
  9. 9. These strategies raise complicated questions: ☉ How to define and delimit protected classes? ☉ What are the risks of collecting data containing these class attributes? ☉ What are the broader societal effects of such classifications? ☉ Can these strategies be transferred to different national contexts? "Any discrimination based on any ground such as sex, race, colour, ethnic or social origin, genetic features, language, religion or belief, political or any other opinion, membership of a national minority, property, birth, disability, age or sexual orientation shall be prohibited." (EU Charter of Fundamental Rights) Disparate impact
  10. 10. "This paper discusses several fairness criteria that have recently been applied to assess the fairness of recidivism prediction instruments. We demonstrate that the criteria cannot all be simultaneously satisfied when recidivism prevalence differs across groups." (Chouldechova 2017)
  11. 11. Justice is NP-hard Different – and (possibly) incommensurable – notions of justice are embedded in different moral and political philosophies.
  12. 12. Moral and political narratives The deep entanglement between algorithmic data analysis and organizational practice in business and government suggests an examination of the specific normative commitments made. Which understandings of justice (and other values) are being put forward explicitly or implicitly? The narrative informing much of the procedural fairness movement is that of meritocracy. "Meritocracy has become an idea as uncontroversial and as homely as 'motherhood and apple pie'." (Littler 2013)
  13. 13. Meritocracy "Indeed, nowadays meritocracy seems to be simply another version of the inequality that characterises all societies." (Dahrendorf 2005) While Dahrendorf still criticized academic achievement as central measure, data mining can take almost everything as an indicator for "merit", performing opaque readings of the inequalities the certificate system hoped to reduce. "Meritocracy has shifted from impersonal technology to a situation where the relation between abilities and rewards has been deeply personalised." (Allen 2012) True or not, the narrative remains a powerful legitimizing myth. "Meritocracy, as a potent blend of an essentialised notion of 'talent', competitive individualism and belief in social mobility, is mobilised to both disguise and gain consent for the economic inequalities wrought through neoliberalism." (Littler 2013)
  14. 14. Competition The notion of meritocracy points to the central role of competition in our cultural and moral imaginaries. Foucault (2004) argues that the key difference between classical liberalism and neoliberalism is not the belief in markets, but whether specialization and exchange or competition are the main source of wealth creation. "Competition is important primarily as a discovery procedure whereby entrepreneurs constantly search for unexploited opportunities that can also be taken advantage of by others." (Hayek 2002 [1968]) The extension of competitive constellations (e.g. into the public sector) multiplies opportunities for data-based decision-making. "Why do we believe in competition? Why do we, at least many of us, think of it as a beneficial societal institution? Which particular kind of competition is at the heart of this belief?" (Werron 2015)
  15. 15. If data mining provides new "levers on 'reality'" (Goody 1977), new forms of designating winners and losers, is the focus on procedural fairness and non-discrimination enough? Beyond procedural justice
  16. 16. Another example: information diversity Information diversity is often seen as a desirable good for democratic life. "Personalisation algorithms reduce the diversity of information users encounter by excluding content deemed irrelevant or contradictory to the user's beliefs. Information diversity can thus be considered an enabling condition for autonomy." (Mittelstadt et al. 2017) A critical perspective should interrogate concept of the a "marketplace of ideas" as a "legitimizing myth" (Ingber 1984). "In our complex society, affected by both sophisticated communication technology and unequal allocations of resources and skills, the marketplace's inevitable bias supports entrenched power structures or ideologies. […] A diversity of perspectives first requires a corresponding diversity of social experiences and opportunities." (Ingber 1984)
  17. 17. Breaking the Cycle (Lindh & Stark 2017)
  18. 18. The singular focus on the isolated agency of individual algorithms runs against recent understandings of technology as infrastructure, assemblage, or actor-network rather than as artifacts without causal coupling or history. "Concepts like 'algorithm' have become sloppy shorthands, slang terms for the act of mistaking multipart complex systems for simple, singular ones." (Bogost 2015) "[T]he competitive advantage really comes from the hard work of what you do with the algorithm and all the processes around making a product, not from the core algorithm itself." (Norvig 2018) "Technical invention consists in rendering a system of disparate elements coherent." (Chabot 2003) This can go (far) beyond questions of data collection/construction. Expansion 2: from algorithms to systems
  19. 19. Grammars of action "Specific configurations of code, or programs, enable some actions over others, reflecting the choices of programmers; these choices structure users’ experiences, what they can and cannot do or say with or through a program. This structuring is not only political, but can be considered expressive: a sort of embedded expression." (Ratto 2005) "Software configures friendship online by encoding values and decisions about what is important, useful, and relevant and what is not. Software restricts certain activities by making others possible or impossible. As I have shown, this becomes apparent when considering the multifarious ways in which software." (Bucher 2013)
  20. 20. Revealed preference To move beyond individual algorithms we can look at the whole range of "ontologies" and "functional expressions" (Petersen 2013) at work. We can also inquire into the principles and practices informing design. "[A]gile programming practices allow developers across services to continuously tweak, remove, or add new features using 'build-measure-learn feedback loops'." (Gürses & van Hoboken 2017) At the center of the guiding rationale lies the theory of "revealed preference", which holds that "the individual guinea-pig, by his market behaviour, reveals his preference pattern" (Samuelson 1948). This fuels and justifies the use of feedback signals as "votes".
  21. 21. "[W]e're making a major change to how we build Facebook. I'm changing the goal I give our product teams from focusing on helping you find relevant content to helping you have more meaningful social interactions. […] Now, I want to be clear: by making these changes, I expect the time people spend on Facebook and some measures of engagement will go down. But I also expect the time you do spend on Facebook will be more valuable. And if we do the right thing, I believe that will be good for our community and our business over the long term too." (Zuckerberg 2018) Here, values seem to clash directly with the ad-driven business model and the IPO logic. Data mining calls for more descriptive ethics! Values in design
  22. 22. The expansion of market forms By lowering transaction cost, information technology has facilitated the organization of many activities around market forms (Ciborra 1985). "[B]y imposing a mathematically precise form upon previously unformalized activities, capture standardizes those activities and their component elements and thereby prepares them […] for an eventual transition to market-based relationships." (Agre 1994) Some heatedly debated instances of algorithmic structuring concern platforms that enact some kind of market structure – e.g. Facebook News Feed (posts), Google Search (documents), Uber (transportation), etc. Data and algorithms are used to optimize transactions, often with explicit appeals to democratic values or consumer benefit. Algorithmic coordination affords interested optimization.
  23. 23. From a purely economic framing of platforms as "intermediaries" to a wider understanding as "mediators" and "curators of public discourse" (Gillespie 2010). "New operators such as Google, Microsoft, Yahoo! and Apple, as well as the new, rising social media firms, such as Facebook or Twitter, should by now be included in the list of the most powerful media organisations worldwide." (Centre for Media Pluralism and Media Freedom 2013) Do we accept "winner takes all" dynamics and cross-sector ownership / expansion in the media sector? There is a tradition of limiting concentration and foster diversity in "media-like" domains. Shifts in media power
  24. 24. Conclusions If data mining is concerned with assessing differences and similarities, in relation to a desire to distinguish, we can interrogate more than the methodologies and outcomes of decisions. What are the normative commitments made in particular contexts and how do they connect to wider systems of value? Thinking in terms of larger ensembles and distributed causality points toward many instances of technical and institutional design that have to be taken into account.
  25. 25. Breaking the Cycle (Lindh & Stark 2017)
  26. 26. Three interconnected ideas for possible digital methods projects (or project components): 1) descriptive assemblage; 2) engaging techniques; 3) engaging values (in design); Three ideas for digital methods projects
  27. 27. RankFlow for [gamergate], [syria], [trump]. (Rieder, Matamoros, Coromina, 2018)
  28. 28. Correlating Change (Rieder, Matamoros, Coromina 2018) p = 1 p = 0.8 Rank-Biased Distance (Webber, Moffat, Zobel 2010)
  29. 29. Investigating Spotify (Rieder 2016, Creamer 2018)
  30. 30. Three directions: description "[S]ocial scientists might seek to elaborate a set of social scientific inscription devices, borrowing from their colleagues in natural sciences, in market research, information technology, etc., or they may prefer to champion description in the form of unique narratives, much as it has been deployed in the humanities and cultural disciplines." (Savage 2009) Combining "fairness forensics" (Crawford 2017) with other forms of analysis, e.g. "discursive interface analysis" (Stanfill 2015) and forms of "descriptive ethics". Critique can identify layers of integrated forms and functions, discuss their politics and identify "points of intervention"? First idea: descriptive assemblage
  31. 31. Second ideas: engaging techniques Many of the techniques in use are available and ready for experimentation. Digital methods is a great space to experiment in relation with actual cases.
  32. 32. Third idea: engaging values (in design) Engaging "values in technical design" (Nissenbaum 2005) can take different forms, one is to think about alternatives. What should the "politics" of YouTube be? How would that look like? Thinking about alternatives can deepen critique and help us confront our own politics.
  33. 33. "[W]e're making a major change to how we build Facebook. I'm changing the goal I give our product teams from focusing on helping you find relevant content to helping you have more meaningful social interactions. […] Now, I want to be clear: by making these changes, I expect the time people spend on Facebook and some measures of engagement will go down. But I also expect the time you do spend on Facebook will be more valuable. And if we do the right thing, I believe that will be good for our community and our business over the long term too." (Zuckerberg 2018) What more is there to say about alternative modes and possibilities? Third idea: engaging values (in design)
  34. 34. Issue Crawler, 1999-today
  35. 35.
  36. 36. Thank you! @RiederB