Richard Neill Horie discusses issues and eccentricities within AI Assistants, and how we should compensate for and overcome them.

1. LEEDS / LONDON / GIBRALTAR
WWW.HOMEAGENCY.CO.UK
Do you know if alexa is lying
to you?Neill Horie
Head of Artificial Intelligence Optimisation
https://bit.ly/lyingalexa

2. ARTIFICIAL INTELLIGENCE
OPTIMISATIONhttps://bit.ly/europeanswallow

3. Flaws/experiments
We have been continually playing and experimenting with the AI
agents with a variety of questions over the last year to see how they
fare.

5. This is an apple.
You probably
recognize it, even
if like me, you
don’t like fruit. If
you ask the AI
agents what an
apple is, they will
tell you.

6. “apple”, the round
fruit of a tree of the
rose family, which
typically has thin
green or red skin
and crisp flesh.
Google will give this answer even
if you ask “what is apple”,
expecting the US company,
because Google prioritises quick
answers/SCRBs over Knowledge
Graph. Definitions always win.

9. Similarly, if you were Mars the
company, you’d want Google to
tell customers about you when
you ask “what is mars”. Based on
the previous slide, you may think
it’ll talk about the planet or the
Roman god of war, though.

10. But you’d be wrong! Definitions
always win.
Google has actually changed this
now, but for most of the last year,
this was the response.
If we come up with a new brand or
product, we need to consider how
assistants will describe us – if at
all.
“mar”; impair the
quality or
appearance of;
spoil.

12. If you ask about the Last King of Prussia, only Cortana and Google answer.
Cortana is based on a web result, and only Google understands that the last
entry in the series of Kings of Prussia on Wikipedia is therefore the last King
of Prussia.

14. Having gone through a few of my
favourite examples, here are
some more top level stats.
Take these stats with a pinch of
salt, as it’s based on our eclectic
set of questions which we have
going back a year so we can keep
a track of what’s changing, but it
should give you a flavor of things.

16. a) 46% b) 66%
c) 76% d) 96%

17. a) 46% b) 66%
c) 76% d) 96%
Amazon Alexa is able to answer 66% of our questions correctly. I don’t
count search results as answers, as they’re not that precise, and obviously
wrong answers aren’t correct. This stat is so bad for Alexa because Alexa
says it doesn’t know 25% of the time.

20. Cortana can answer correctly 76% of the time, but actually responds about
98% of the time. Often these results are search results, though.

22. a) 38% b) 48%
c) 78% d) 98%

23. a) 38% b) 48%
c) 78% d) 98%
Google answers correctly 78% of the time, but when it does answer, those
answers are 97% correct, so it’s mainly ones where it doesn’t know which
let it down. This used to be worse, but they’ve fixed more of the situations
like “what is mars”.

25. a) 34% b) 54%
c) 64% d) 84%

26. b) 54%
c) 64% d) 84%
Siri only answers correctly 34% of the time, which is awful. Almost half of
the time it responds with a search result, and almost half of actual answers
are wrong ,which is a big problem.

28. For a little whilst in 2016/2017, if you asked Google who the King of the
United States was, it would tell you it was Barack Obama. This was from a
Brietbart source used as a quick answer.

30. Let’s talk about these sources. Wikipedia is an obvious one, where lots of
information comes from – either directly in Google’s quick answers, or
indirectly via Bing or Google Knowledge Graphs.

32. This is Wolfram Alpha, and is used by a lot of Siri’s answers as a knowledge
base. We can query Wolfram Alpha’s website ourselves and verify lots of it,
but it’s still a source most normal people don’t think of.

34. This is Evi, which was a knowledge company bought by Amazon a few
years ago. In hindsight, it was kind of embarrassing that we had to figure
out that this was where Alexa got some answers from, but it is.
Unfortunately, you can no longer query it – the answers are dark and
impossible to interrogate.

36. This leads to interesting issues, where Alexa claims that King Ecgbert (of
Wessex) was the first King of England…

37. Image Source: Odeja from Wikipedia

38. Image Source: Odeja from Wikipedia
…when actually it was this chap, Æthelstan.
These might seem to be irrelevant examples, but if they can be wrong about
even these basic facts, don’t assume that they can understand your website
and product immediately.

40. Similarly, if you ask who the next King of England will be, sometimes Google
used to tell you that it would be Prince William, because Queen Elizabeth
said so. (Based on a quick answer derived from a YouTube comment.)

42. On a different note, if you ask if Brexit was a good thing you’ll probably get
told yes, because people who think it was tend to ask that question rather
than another question. As marketers, we need to think carefully about our
phrasing, almost like old-school SEO.

44. KNOW QUESTIONS
ANSWER QUESTIONS
BUILD ASSISTANTS

45. KNOW QUESTIONS
ANSWER QUESTIONS
BUILD ASSISTANTS
We are the gatekeepers
of content.
From all of our work on
SEO, we should know
what content our users
need, and they’re trying
to find out and what AI
should tell them.
Make sure all of it is
consistent and accurate.

48. Anecdotally, users often
trust what they’re told
rather than what they
see in search results. If
an AI assistant tells the
user the wrong thing
about your brand, the
problems are obvious.

49. Appendix

50. appendix
Artificial Intelligence Optimisation
https://bit.ly/europeanswallow - Presentation
https://bit.ly/aiogoldrush - Blog post
AI Assistant Experiments
http://bit.ly/aioexperiments - Blog post
Can Google Tell Us The Truth?
http://bit.ly/nonfacts - Article
Alexa Versus Google
https://bit.ly/alexavgoogle - Video
Alexa Versus Accents
https://bit.ly/alexavaccents - Video