Artificial intelligence breaks into our lives. In the future, everything will probably be clear, but so far, some questions have arisen, and increasingly these issues affect aspects of morality and ethics. Which principles do we need to keep in mind while surfacing machine learning algorithms? How the editorial team affects the day to day development of applications at BBC? Place: Kharkiv National University of Radio Electronics, Ukraine When: 17th November 2019.

1. Kharkiv National University of Radio Electronics
17 November 2019
@tati_alchueyr
Ethical Machine Learning
building recommendation engines with
editorial support

2. мне приятно быть здесь с тобой
большое Вам спасибо

3. About me
● Brazilian living in London since 2014
● Senior Data Engineer at the BBC Datalab team
● Graduated in Computer Engineering at Unicamp
● Passionate software developer for 16 years
● Experience in the private and public sectors
● Developed software for Medicine, Media and Education
● Loves Open Source
● Loves Brazilian Jiu Jitsu
● Proud mother of Amanda

4. BBC
● British Broadcasting Corporation
● Values
○ Independent, impartial and honest
○ Audiences are at the heart of everything we do
○ We take pride in delivering quality and value for
money
○ Creativity is the lifeblood of our organisation
○ We respect each other and celebrate our diversity
so that everyone can give their best

5. BBC
● Founded in 1922
● Purpose
○ Inform
○ Educate
○ Entertain
● “Our organisation exists in order to serve individuals and
society as a whole rather than a small set of stakeholders.”
Reference: Gabriel Straub (BBC)

6. bbc.stats()
➢ BBC TV reaches 91% UK adult population
➢ BBC News reaches 426 million global audience weekly
Reference 1: BBC
Reference 2: BBC
Image Credit: BBC

7. BBC. .
“Bring the BBC’s data together
accessible through a common platform,
along with flexible and scalable tools to
support machine learning to enable
content enrichment and deeper
personalisation”

8. Some of the Datalab team members (15 August 2019)
BBC. .

9. BBC. .
● Multi-disciplinary team
○ Editorial
○ Data scientists
○ Engineers
○ Product Manager
○ Project Manager

10. BBC
machine learning

11. BBC Machine learning applied to the audiences
Image credit: BBC

12. BBC Machine learning applied to content creation
Image credit: BBCMade by the Machine: when AI met the archive (BBC 4)

13. Machine learning overview

14. BBC+
experimental personalised app

15. BBC+ app experiment
● Fully personalised experience on short videos, on Android & iPhone
● Allow users to find gems that they didn’t know at a time that suits them

16. Content-based recommendations content

17. Content-based recommendations content

18. Content-based recommendations content
We create a content representation (*):
{
"genres": {
"science": 0.8,
"nature": 0.2,
}
}
(*) simplified for didactic purposes

19. Content-based recommendations user
We learn about the user indirectly
● news you read
● videos you watch
● things you search
● quizzes you answer
● things you like
● things you comment

20. Content-based recommendations user
We create a user representation (*):
{
"genres": {
"science": 0.4,
"folk-music": 0.5,
"judo": 0.1,
}
}
(*) simplified for didactic purposes

21. Content-based recommendations prediction
We use the user representation to search for content similar to it,
using Elasticsearch. As an output, we have a ranked list of content.

22. BBC+ app experiment
● How to get from algorithm to product
○ Start with content-based recommendations
○ Apply business rules

23. Legal, editorial, GDPR, business values
https://www.bbc.com/editorialguidelines/

24. Legal Policies
Programme: BBC
Contempt of court
● The recommendations should not affect the
outcome of a legal case
● The BBC can be held accountable for
influencing the jury’s opinion
Action
● Create a “contempt of court risk” label by
detecting keywords such as arrest, assault,
allegation etc
● Avoid items with this label

25. Legal Policies
Electoral law
● During elections we should not surface
political content that could influence the vote
Action
● Create a “political risk” label by detecting
political content sources
● Avoid items when appropriate

26. Editorial Policies
Quality criteria
● Avoid content that shows little care has been
taken in the metadata
Action
● Avoid content with poor titles and descriptions

27. Editorial Policies
Under 16 audience
● Provide children-safe content
● BBC’s 9PM watershed
Action
● Avoid items with warnings of sex, violence,
strong language

28. Cold start: human curation alongside automation

29. GDPR
Explainability
● Choose simple models over complex ones
● UI features to provide explanations
Agency
● UI features for users to interact with the algorithm
● Eg. delete history items, like, dislike, report

30. Curation values
● Affection
● Authenticity
● Compelling
● Fresh
● Warm
● Quirky
● Relatable
● Aspirational
● Entertaining
● Reassuring
Reference: Anna McGovern
“Website editor, manager, analyst and
digital nurturer” at the BBC
Much more than click rates

31. Business values & objectives
Quantitative offline evaluation
● NDCG, hit rate, diversity, recency, surprisal
● Prioritise diversity and recency over accuracy
Qualitative offline evaluation
● Prioritise content for young audiences
● Prioritise content of editorial importance

32. BBC+ app experiment

33. BBC+ app experiment
Takeaways
● The editorial partnership is key to how we work
● The company’s principles are at the heart of all of our decisions
● There is a significant path between implementation and
production ready

34. BBC Sounds Recommendations
Challenging existing recs provider

35. Current external provider
Content-based recommendations

36. ● 9 to 12 items on native apps and web
● Current provider: content-based algorithm
○ Poor metadata, poor recommendations
○ Popularity biases towards heritage audience
○ Cold start using editorially curated lists
○ Opportunity for improvement of performance
We decided to try a different approach: Factorisation Machines
Recommended for you

37. Recommendation strategy content-based
How it works
● Given a user, find similar content to their preferences
● Characterising item using genres, masterbrand, etc.
● Based on user’s historical data and content metadata
Challenges
● Potential lack of diversity and relies on good content description
Where can we find this?
● “You may also be interested in …”

38. How it works
● Given a user, find similar users and the content they watched
● Based on all users’ historical data
● Uses implicit feedback (user-item interactions)
Challenges
● Sparse matrix
○ SVM very efficient except in sparse settings where
not enough data to estimate interactions
● Cold start
Where can we find this?
● “Customers who viewed this item, also viewed...”
Recommendation strategy collaborative filtering

39. How it works
● Hybrid content-based and collaborative filtering
● SVM and factorisation techniques
● Based on all users’ historical data and content metadata
● Based on reliable information (latent features)
● Linear time complexity
Recommendation strategy factorisation machine
Reference: Academic Paper

40. Example
● Estimate interaction between Alice and
Star Trek
a. No case where A and ST > wA,ST= 0
b. Use factorized interaction parameters
{vA, vST}
c. Dot product of the factor vectors of A and
ST will be similar to the one of A and SW
Recommendation strategy factorisation machine
User Item Rating
Alie (A) Titanic (T) 5
Alice (A) Notting Hill (NH) 3
Alice (A) Star Wars (SW) 1
Bob (B) Star Wars (SW) 4
Bob (B) Star Trek (ST) 5
Charlie
(C)
Titanic (T) 1
Charlie
(C)
Star Wars (SW) 5

41. Qualitative Experiment
Who
● ~30 test users recruited
○ From non-editorial and editorial
teams from BBC audio networks
○ Under 35
How
● Two sets of recommendations
displayed
● Users have to pick either the best list,
or “both”, or “neither”
● And explain why

42. Qualitative Experiment Feedback
● “Need to categorize speech vs music,
background listening vs ‘serious’
content”
● “Need to consider the age of the item”
● “Looking for diverse content durations
…”
Reducing item/user biases helped to
generate more personalised
recommendations than the current state
Neither Content-
Based
Hybrid
approach
Both
2 8 17 1
7% 28.5% 61% 3.5%

43. Quantitative Experiment (MVT or A/B test)

44. Machine Learning Principles

45. The BBC Machile Learning Values
1. Audiences at the heart of everything we do. We celebrate diversity
○ Good value for money and focusing on using the audience-based
data to improve their experience
3. Our algorithms serve our audiences equally and fairly, so that the
full breadth of the BBC is available to everyone
6. Algorithms form only part of the content discovery process for our
audiences, and sit alongside (human) editorial curation
Reference: Gabriel Straub (BBC)

46. Flourishing
in the age of AI

47. Flourishing in the age of AI
● Research
● 11,000 people
● 7 markets
● What people want from their lives
● How technology might enable that
Reference: Flourishing in AI report

48. Flourishing in the age of AI
“(...) people in the UK don’t think technology is being
developed with their best interests at heart”
Reference: Flourishing in AI report

49. Flourishing in the age of AI
Reference: Flourishing in AI report
● How satisfied are you with
your life?
● To what extent the thing
you do in life is
worthwhile?
● How anxious did you feel
yesterday?
Base: 5432, May 2019

50. Flourishing in the age of AI
Reference: Flourishing in AI report

51. Flourishing in the age of AI
Reference: Flourishing in AI report

52. Flourishing in the age of AI
Reference: Flourishing in AI report

53. Flourishing in the age of AI
Reference: Flourishing in AI report

54. Empower the user

56. Bonus
BBC Radio 1 studios tour

60. Link to video from Jacob Rickard

61. http://datalab.rocks
We are hiring

62. Further Reading

63. Ethical Machine Learning
● How do you make decisions about what is fair?
● What metrics can you use?
● How to achieve an ethical machine learning in your work?
Reference: Avoiding the Fate of Icarus
Medium

64. дуже тобі дякую
Спасибо
Image credit: Wikipedia Commons
@tati_alchueyr