Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Can We Make Algorithmic News More Transparent?

508 views

Published on

A proposed framework for making news algorithms more transparent, presented at the http://www.algorithmic.news/ conference #AlgorithmicNews

Published in: Technology
  • Be the first to comment

How Can We Make Algorithmic News More Transparent?

  1. 1. How Can We Make Algorithmic News More Transparent? Stuart Myles Director of Information Management, Associated Press www.ap.org @smyles Algorithms, Automation and News, 22nd May 2018
  2. 2. News Algorithm Transparency • Automation of production, distribution and consumption of news • Such as performing analyses, ranking search results, generating news reports • A framework for making news algorithms more transparent • Four levels of transparency - from simple “disclosure” to full “reproduction” • Three sets of stakeholders - technicians, journalists and readers • Some transparency examples from the Associated Press • Including where we could do better • Suggestions for further transparency topics to explore www.ap.org @smyles
  3. 3. Transparency Stakeholders (1/3): Technicians • Technical transparency is the main focus of algorithmic transparency research • Is the algorithm working correctly? • Accuracy, bias, semantic drift • Automatic decisions • Ranking, classification, filtering • Synthetic news • Fill-in-the-blank templates, summaries, video-from-text, text-from- images www.ap.org @smyles
  4. 4. Transparency Stakeholders (2/3): Readers • Transparency often cited as a means of (re)building trust in journalism • It seems likely that lack of transparency undermines trust • How can algorithmic news be more transparent for consumers of news? • Readers • Viewers • Listeners • Why am I seeing / not seeing this news item? www.ap.org @smyles
  5. 5. Transparency Stakeholders (3/3): Journalists • Most algorithms used in news originate outside journalism • News algorithms should reflect an organization’s editorial voice • (Do platforms which deliver news have an editorial voice?) • Journalists should be involved in crafting the explanations for readers • Readers need understandable explanations • Journalists themselves should have some understanding of the algorithms which power and mediate their work • Journalists and editors • Lawyers, archivists, other professionals within a news organization www.ap.org @smyles
  6. 6. Levels of Transparency (1/4): Disclosure • Reveal algorithms were used in creating or making decisions about news items • Could be a general statement about a set of items • Might be attached to individual items • Could identify which algorithms were used and in what ways • Disclosure is the minimal level of transparency • May be the only form of transparency available if the algorithm is handled by a 3rd party? • Disclosures are most useful for readers www.ap.org @smyles
  7. 7. Disclosures by Associated Press • AP includes a disclosure on automatically synthesized text stories AP created this story using Automated Insights' Wordsmith Platform (http://automatedinsights.com/ap) and data from Zacks Investment Research. • Indicates that AP journalists created the story • Journalists designed template • Template is automatically populated per our rules • Identify 3rd party data sources and tools • Included on every automatically created story www.ap.org @smyles
  8. 8. Levels of Transparency (2/4): Justification • Justifications aim to show that the results of the algorithmic news are reasonable in a particular instance • A step up from disclosure - provides a degree of transparency • Offers some reasons for the algorithmic result in a particular instance • Not a comprehensive set of reasons • Discusses a particular decision, rather than the general use of an algorithm • A complete set of reasons may not be appropriate • To keep proprietary information confidential • Recipient cannot reasonably be expected to understand the full scope of the workings of the news algorithm www.ap.org @smyles
  9. 9. Facebook Advert Justification • Facebook’s “Why am I seeing this ad?” One reason you’re seeing this is [specific campaign criteria]. This is based on [tracking techniques]. There may be other reasons you’re seeing this ad, including [broad targeting criteria]. This is based on [other types of tracking techniques]. • Offers a potential model for news algorithm justifications • Justifications would be most helpful for readers and journalists • Avoids the excuse of “it is too complex to fully explain” www.ap.org @smyles
  10. 10. Levels of Transparency (3/4): Explanation • Why was a particular decision, categorization or arrangement of news selected and not some other? • Explain the outcome of a specific instance of a news algorithm • I haven’t found algorithmic explanations “in the wild” • There is active research into how to generate algorithmic explanations • Algorithmic approximations, generate counterfactuals, generate rules • AP’s rule based system provides explanations suitable for technicians • What makes an explanation useful? www.ap.org @smyles
  11. 11. Useful Explanations (1/2) • A useful explanation allow you to take an action in response • Alter an algorithm which has made an incorrect decision • Alter a news item to conform to an algorithm’s criteria • Such as adding missing metadata • Alter other metadata, not on the item, to get a different result • Such as user preferences • Useful explanation can include confidence scores, as well as narrative • Is editorial review required, due to low confidence? www.ap.org @smyles
  12. 12. Useful Explanations (2/2) • Useful explanations of multiple decisions can reveal systematic issues • Biased decisions which favour or penalize particular groups • Where an algorithm is suitable and where it will not be applicable • Are the training data or assumptions out-of-date? www.ap.org @smyles
  13. 13. Levels of Transparency (4/4): Reproduction • Sufficient information to allow the news algorithm to be independently replicated • Provide underlying data and code directly • Describe in a “nerd box” • It may not be possible to provide all the data • Time-sensitive like trending topics or there’s just too much of it • Algorithms may be proprietary • Running algorithms and handling data can require a lot of technical wherewithal • Most useful for technicians www.ap.org @smyles
  14. 14. Rule based classification at AP: Explanation and Reproduction • AP auto-categorizes all English-language content • Tags for people, places, companies, organizations and subjects • AP’s automated rules engine has about 200,000 classification rules • Hand-crafted by a team of specialists • Sophisticated strategies for disambiguation and precision • “Classification Admin” application to develop and test rules • Evaluate a news item against one or more rules • Highlights why a text matches the rules • Both reproducibility and a form of explanation • Explanations are not designed to be shared with journalists or readers www.ap.org @smyles
  15. 15. News Algorithm Transparency at AP • In writing the paper, I felt AP has a good transparency story • But I see there is room for improvement • We use disclosure on machine-generated stories • We use rules-based classification • Transparency for internal technicians • We make algorithms and data for data journalism stories available to AP members • Requires a certain amount of know how • We could do better at transparency for journalists and news consumers • With justifications and explanations www.ap.org @smyles
  16. 16. Other Areas to Explore • Difficulty of effectively conveying explanations of news algorithms • Visual symbols and icons for disclosure of the use of algorithms? • Narratives for justification or explanation of algorithmic outcomes, rather than statistical readouts, using Natural Language Generation? • Are narratives helpful for photo, video or audio? • Transparency for algorithmic errors and corrections? • Transparency for time-dependent algorithms? • Trending stories or collaborative filtering • All transparency all the time? Or only on demand? • In the pre-algorithm days, we didn’t have full transparency • Equivalent to responding to letters to the Editor • Consumers don’t require transparency from other algorithmic systems – e.g. cars www.ap.org @smyles
  17. 17. News Algorithm Transparency • A framework for making news algorithms more transparent • Three sets of stakeholders • Technicians, journalists and readers • Four levels of transparency • Disclosure, justifications, explanations, reproduction • Active research into how to generate justifications and explanations • Journalists should be involved in crafting useful justifications and explanations www.ap.org @smyles

×