Talk at Understandable AI, SummerPIT 2019, Aarhus University, 14th August 2019 https://alandix.com/academic/talks/PIT-2019-bias-and-explanation/ In this talk I will pick up threads of research dating back to early work in the 1990s on gender and ethnic bias in black-box machine-learning systems, as well as more recent developments such as deep learning and concerns such as those that gave rise to the EPSRC human–like computing programme. In particular I will present nascent work on an AIX Toolkit (AI explainability): a structured collection of techniques designed to help developers of intelligent systems create more comprehensible representations of the reasoning. Crucial to the AIX Toolkit is the understanding that human–human explanations are rarely utterly precise or reproducible, but they are sufficient to inspire confidence and trust in a collaborative endeavour.

1. sources of
bias and explanation
Alan Dix
Computational Foundry
Swansea
http://alandix.com/academic/talks/PIT-2019-bias-and-explanation/

2. Tiree
Tiree Tech Wave
3-7 October
Computational Foundry
Swansea University

3. the foundry
building
mission
community

7. types of algorithms …
rules and regulations
ordinary code
classic AI
machine learning and neural nets
increasing
opacity

8. when things go wrong – deliberate
misuse
hacking
bad use
cyberwarfare – Stuxnet, etc.
autonomous weapons

9. when things go wrong – well meaning
accidents
autonomous car crashes
unintended consequences
bias (gender, ethnicity)
disproportionate social effects
https://www.bbc.co.uk/sounds/play/m00017s4 (report @ 1:41:00 in)

10. 25 years back …

12. warns of the danger of gender and ethnic bias in
black-box machine learning systems
gives example: database queries using ID3
offers (partial) solution: Query-by-Browsing
and even some broader heuristics
inter alia …

13. yes, 25 years ago!

15. Query-by-Browsing
creating scructable
internal representations

16. Query by Browsing
user chooses records of interest
 tick for those wanted
 cross for those not wanted
system infers query
web version uses rule induction
variant of Quinlan’s ID3
www.meandeviation.com/qbb

17. Query by Browsing
what it looks like
user asks
system to
make a query
system infers
SQL query
query results
highlighted

18. Query by Browsing
dual representation
query (intensional)
for precision
listing (extensional)
for understanding

19. Query by Browsing – how it works
examples
machine
learning
SQL query
cond
cond
decision
tree

21. it is not just about
being accurate
not just right
but also upright

22. learning
past bias in
training data
training
data
learnt
rules
objective
function
societal
bias in goals
‘best’ may
be biased

23. learning
past bias in
training data
training
data
learnt
rules
objective
function
societal
bias in goals
‘best’ may
be biased

24. algorithms reflect society

25. mimicking human behaviour and choices

26. learning
past bias in
training data
training
data
learnt
rules
objective
function
societal
bias in goals
‘best’ may
be biased

27. pandering to human bias
(effective outcomes?)
• dating sites using ethnicity (CHI 2018!)
• young pretty waitresses sell more drinks
• Trump (reportedly) hiding black employees at
casino when certain rich customers arrived
• BBC (& others) paying male presenters more
because they are more popular

28. ‘good’ business
but is it good?

29. learning
past bias in
training data
training
data
learnt
rules
objective
function
societal
bias in goals
‘best’ may
be biased

30. reinforcing societal/cultural norms
at school
boys more likely to study STEM subjects
girls more likely to study humanities
so, on average, with no other information
gender is an (albeit poor) predictor
of communication skills
and engineering knowledge

31. as a society we choose
to use other (and better)
predictors

32. innate (but largely irrelevant) differences
men are (on average) larger and stronger
so gender is a Bayesian predictor of strength
this may explain gender differences in some jobs
but …
it does NOT justify employment discrimination

33. bias is not about
algorithmic correctness
it is about social choice

34. the choice of input features
often critical in
creating or controlling bias
more data not always better!

35. Note:
human reasoning is
poor at ignoring low quality cues
even when we have better ones

36. algorithms may be better?

37. however …
not sufficient to remove explicit indicators:
gender/ethnicity/disability/religion
potential correlating factors e.g. clothing
algorithms need to actively avoid discrimination

38. and how do we know our
algorithms are OK?

39. Not just bias
safety – e.g. autonomous cars
democracy – e.g. social media, fake news
health and well being – e.g. soft-drink adverts
social issues – e.g. credit ratings

40. we need to ask
Why?
algorithmic transparency
c.f. court judgment

42. an AIX Kitbag
AI explainability
how to make sense of
black-box machine-learning algorithms

43. crucial insight …
human–human explanations
rarely utterly precise or reproducible
but are
sufficient to inspire confidence and trust

44. white-box black-box
grey-box
creating scructable
internal representations
analysing and
understanding
from the outside
peeking within
understanding
internal representations

46. but … this was all evident
25 years ago
why didn’t I do more?
if it is important
not sufficient to publish
you need to transform into
publicity and policy

48. white-box methods
creating scructable
internal representations

49. WB0. choose a white box classifier!
training set
scrutable
rules
white-box
algorithm
unseen data white-box classifier outputs

50. WB1. black-box generation of white box
classifier
training set
scrutable
rules
black-box
algorithm
unseen data white-box classifier outputs

51. WB2. Adversarial examples for white-box
learning
case-base of
behaviour scrutable
rules
black-box
adversarial learning
white-box
learning

52. WB3. Simplification of rule set
scrutable
rules
black-box
learning
training
set
inscrutable
rules tweak

53. black-box methods
analysing and understanding
from the outside

54. BB1. exploration analysis for human
visualisation
black-box
learning
training
set
inscrutable
rules
lots of
examples
black-box
classifier
visualise
input-output

55. BB2. perturbation/exploration analysis for
key feature detection
black-box
learning
inscrutable
rules
randomly vary
feature values
black-box
classifier
hotspot
visualisation

56. BB3. perturbation analysis for central and
boundary cases
lots of
examples
black-box
classifier
central and
boundary
cases
user
visualisation
white-box
learning

57. BB3. close up
central cases
perturbations
do not change class
boundary cases
small perturbations
change class
penumbra
larger perturbations
change class

58. BB4. black-box oracle – white-box learning
input
examples
black-box
classifier
scrutable
rules
white-box
learning
input–output
pairs as
training set
output
classes

59. grey-box methods
peeking within

60. GB0a. sensitivity analysis – weights
perturb parameters in
the inscrutable rules
lots of
examples
black-box
classifier
hotspot analysis
on parameters

61. GB0b. sensitivity analysis – activation
input
example
black-box classifier
(low level)
extract
intermediate
activation
black-box classifier
(high level)
perturb
activations
hotspot analysis
of nodes

62. GB0c. sensitivity analysis – algorithmic
apply black-box
algorithm
inverse
algorithm

63. GB1. high level model generation
input
examples
black-box
classifier
extract
intermediate
activation
scrutable
rules
white-box
learning
activations with
output class
as training set
output
classes

64. GB2. Clustering and comprehension of
low level
input
examples
black-box
classifier
extract
intermediate
activation
clusters
various
algorithms
activations
as input
MDS
SOM

65. GB3. triad distinctions
input
examples
black-box classifier
(low level)
A
B
C
hotspot analysis
of nodes
compare

66. GB4. apply generatively
output to input
activation to input
output to activation
between layers