Speaker: Daniël Heres Genre & level: Backend, Way of working, Medior Currently, chatbots are stupid; they only get what you want when you ask them in the exact way the system understands. At bol.com, we are improving our chatbot using Machine Learning and AI. To give users better answers more quickly, in addition to analyzing sentences from users, we also use information about the current situation of a customer, such as information about recent orders, returned items and status of package transport. Using this context to predict what the customer wants leads not only to improvements in our chatbot, but can also be used to proactively give customers and customer service agents information they need. We can make much better decisions by learning from our data instead of hardcoded rules, so we have lots of plans to be smarter in the future. Through machine learning, we give customers better service, detect and prevent problems in the shopping and ordering process, and help agents solve problems. It’s a win-win-win! This talk will inspire you to make the most of your own data and provide an approach for using that data to improve your services and decision-making.

1. Billie, Grow Up! AI for Customer Service
Daniël Heres

2. INTRO

3. This Talk
START
V
.----------'------------.
| INTRO / DATA-DRIVEN |
'----------,------------'
v
.----------'------------.
| BILLIE |
'----------.------------'
v
.----------'------------.
| AI/ML FOR BILLIE |
'----------,------------'
V
.----------'------------.
| CS 2.0 USING AI |
'----------,------------'
v
FIN
1

4. Personal Info
MAIN SIDE JOBS RELEVANT PROJECTS
.--------------------.
| TI-83 Programmer |
'---------,----------'
v
.---------'----------.
.--------------------------------,--------------| NLP Chatbot | .---(2008)
| | '---------,----------' /
v | | /
.----------'------------. .---------'----------. | (Yes, this is true. It was part
| Computer Science BSc | | SE Job 1 Untitled | | of our work ”Kunstmatige
'----------,------------' '---------,----------' | intelligentie als toepassing
| v v op de communicatie tussen mens
| .---------'----------. .---------'----------. en computer”.)
| | Customer Insights | | ML for Programs |
| '---------,----------' '---------,----------'
v v v
.----------'------------. .---------'----------. .---------'----------.
| Computing Science MSc | | Computer Vision | | ML Plagiarism Det. |
'----------,------------' '---------,----------' '---------,----------'
| / /
v / /
.----------'------------. / /
| Bol.com CS |<--------------' /
'----------,------------' /
| /
v .--'
.----------'------------. .--------------------. / .-------------------.
| Bol.com CS + AI | | AI News Letter |<'>| Experiments | (2018)
'-----------------------' '--------------------' '--------------------'
2

5. Goals 42DT
88D8888888888O8O8888OO888D88DD88OOO8888888888888888888OZO7778D8888888888O8ZZO$ZZ ......................,.....,,....,..,,,,,,,,,,,,,,,,,,,,,,,,,,ZZZZOZOZZZOO88O88
88888888888888O888888O888D88D8D88OZ788DZ88888+?$Z7$7$7$ZZ$$7III7O88O$$Z$7Z$$$ZZZ ........................,..,...........,,,,,,,,,,,,,,,,,,,,,$ZIZZZZZOZZOOOO88888
88888888D888D8O88888O888888O8D8O8D888O$8ZO8D8??77I7777I7777$$7$77$7Z7$$Z$Z$$OZ88 $,.............................,.......,..,,.,,.,,,,,,,,,.,$$ZZOOZOOOZOOOO8O88OO
888888888888888888888888O88Z$788D88DD8887ZO88O7$7I77777I777I7$77777Z7$$Z$$$Z$$ZZ $77......................................,....,.,.,.,,.,,,7ZZZZZOOO8OOOOOO8OOO88
8888888D888888O88O8Z88$8888888D88DDDDDOOD8D8D88O8OOO8888888D88888888$7$Z$Z$$77$$ $$7II?.................................................,.:ZZZOZOZOOO8OOOO88OOO88
IIIIIII888888888O8OOOO88888D88888D8888888O888888OOO8O88DOOD$OZO88888O$7$$$$7$7$Z $$77III?=..............................................,.$ZZZOOOOOOOO888O8OO8888
II7IIII88888888888Z88O888D88D888O88O8OOO888OO88888O8888D???$777$7$$$77$77$7$77$$ $$7$I$77I?+:.................. .......,....,............,ZZZOOOOOOOOOO8888888888
IIIIIIIO888888O888OO88888OOOOZ7O?++++=++I$Z8OOO888OO88O87??ZI7I7777I7$7$7777777$ $7$$$77III??++~..................=~~~...,?I+~:,.........$ZZZZOZ888O8888O8O888888
IIIIIIIO888888OO88888O7?II+=+++I==~~~~===+??II$ZO8OO888OO+?OZOOOZZZZZOZZ$ZZZ$$$Z $$$$$$7I7III???+?=............::..,,I.~..:I$7=+........=$ZZZOOOOOOO8OOO88O88D88D
IIIIIIII??I??O8O8888I?++++?=~+I=~~~~~~~===+?++I7ZOO8OOOOZ+?N877$$88ODD78DD888888 $$$$7$77777IIII?+?++=.......:......:.,I~.~,OO7,.,....,,ZOZZZOOOOO888888888888888
I?IIII?III?Z$O88887I?+=====~=~~=~~~~~~~~~======??I8ZOOOOZ+?II?I????I?IIII7888888 $7$$7$7$77$7777III????I+:........=~....+ZOO$??7=+....?$ZZZZOOOO8OO88888888888D88
III?III7?I$O8OO8OIII?===~~~~~=~~~~~~~~~~~~~==+=++?$ZO888O7?IIIII???IIIIII8D88888 $77?II7I$7$7$777$$7III??,:~+:,~?7?$7$IOZ$$=~:.=:?=+,.=$OZZZOOOOOO8O8888888888888
IIIIIIIII7I88D88$I??====~=~~~~~~~~~~~~~~~=~====++?IZZO88ZI?I??IIII?II??IID88D888 ???I77I$777$77$$$$$777=..,IZ?~~ZOO88OZZZZI.,$$Z?$Z77++?$ZZOOOOOOO8OOO88888888888
I?IIIIIIIIZZ8O8OII?+====~~~=~~~~~~~~~~~~~~~~~==+++?7$88887??Z$$II?IIIII7?8D8D888 7$7$$I?I?I???II7$7$$7,.,.+?7?O8OOO88OOOOO8OZZZ7$$ZZ$Z?~$OOOZOOOO8OOOOO8888888888
?I???I?I??$O8OO7I??+=====~~~~~~~~~~~~~~~~~~~===++++IOO8888+?$OZ8888OOOOOOZ888888 Z$$$$7777I$77$$$I$7?+.,:+??O888OOO888OO888OZZ$ZZZOOZZII+7ZOOOOOOOOOO888888888888
I??II???I??OOOOZI??++===~~:~~~~:~~~~:~~~~~=+++??I??I$O88D87?N8D888DDDDDOOZ88D8DD $Z$$$Z$$$777$7$$7$77,:77+ZOO8OOOOOOO88O8OOZZ$ZZZZOZZ7ZII=$OOOOOOOOOO88888O888O88
??IIII?III?Z88OZ7?+++=====~~~~~:~~:~~~~~??I7?++?I7++?O88D88?8DD8DDDDDDDD8O8D88DD $$$$$$$$$$$77$$$$$I=+7$OOO88OOOZZZOOOOOOOZZZZZ$$$ZOOOZO7=~I$OOOO8O8OO888888O8O88
III???IIIII?8O8O7?+?+?++?I????=~~~~~~~===+?II???+=77?O8OD88?$888O888D8DD8OODDDDD $7I7$$Z$$$$$$$$$77=?~$ZO8O88OOOZZZZO8OOZOZZZ$$$$$$ZOZOOZI+:?~OOOO8OOO888O8888888
IIIII?IIIIIIOO8ZI???7I7I??+=~===+~=+I++=~+:=++++++?+=IO888ZI+?7+7$$$$$$$Z$O88DDD III77$$$$$ZZI????++~I7=8O8Z$OZZZZ$$ZOOOOZ$$$$$Z$$$ZZZZZZZ=,=Z8OOO88OO88888888888
????+????II?OOO$II$?$?++===+====+~===+?~++=8OZ~I++=+=I88DOI$Z$$ZZO88888D887DD8DD I77$7III$$$$7?+???I??O88OOZ$ZOOOZ$$$$ZZOZOOOOOZZZ$ZOZZZZZ7=$ZOOO8OO8888888888888
?????????II?OO88$?+?I?=+7$I7===?I+===+Z?===~===++++?=+$O=====ZO888DDDD8DDD7DD8DD II7I??II??7???III???IZOOOO$ZOOZZZZ$$77$$$$$$$$777$$OZOZZ$+ZO8OOOOO8888888O8D88DD
+????????II?7O88I+=?++??++====++?++===+Z=++++=++==++=?I$~=~~=Z8DDDDDDDDDDD$?88D8 $ZZ$7I7Z$7I777I$$Z7+?$ZOZOZ777$ZOOZ$$$$7$ZOO88OO$$7ZOZO$$$$OOO8OOO88O8OOD8888ODD
$++??????III?88O7++I++??======++++==~=++===========+++II~~~~~??8DDDDDNDDDD$?D8DD ZZZ$Z$7$II7777777$$7?ZOOZZ$$$OD88O8OZ$$$ZOOZ$$$$$$$$OZZZOZ77$8Z888O8O88888O88888
I$+??????I???+8O??==++++==~~===+=++===++=~=~~~~~~~===?I+=~~==?O?I7Z77OZOO8$??88D ZOOZZOZZ$I7$$I7II$$Z$7OOOZ$$$$$$$OZZZ$ZZZOZZZZZ$$$$$ZZ$ZO$$77OOO8OD888O88D88888O
II7777I?II?==+7O7??++===~==~=~===?+==~=+++=~~~~~~~~==+I~=:~~=7ZII?I???I???7??DDD ZOOO$8OO$Z$ZZ7I$$ZZZ7=O8Z$77$$ZZZZ$$ZZZZ$$Z$ZZ$$$777$7I7ZOOZZZ88O888O88888OOOO88
III777777?=~+==77I?++===~~::~~=+=++=~~~+=+=====~~~~==+?=~~~~=$888D888DD888Z+$+Z7 OOZOZ888OZZ$O$$Z$ZZ7$7ZZZ$77$$$$ZZ$$ZZZZZ$$$$$$$$7777$$$O8Z$7Z8O888OOO8OOOO88877
I7II77I777Z=+=~+II??==~~~~:~~~+++$ZI+++I=++==~~==~===+?~=====O888D8DDDDDDDZI$$$? OZZZZO88OOZZOZZZOZ$ZZ$ZO$777777$$$ZZ$$$$$$$Z$777777777Z8888O$O88888OO8OOOOO8O7IZ
II7?$7II77$=?~~=+77?+==~=~==~==+++++++============~==+=~~~==+88DDD8D8D8DDDO$ZOZZ OOOOZZOZ8OOZZZZOZZZZ$$Z7I77777777$777777$$7$$$77777$77ZOO88888888O8O8O88O77$I$$O
II??7I7777I=?===+77I+==~~==~==+==+==~~~~~========+~=++~~~=~+788DDD8DDDDDDDDOZ7OO 888OOOZOZOZZZZ$ZZOZO$77777$7777$$$$$$$7$$$Z$$$$$777$8OOOO888O888OOOO87I$?$7$$ZZZ
???III777IIZ~++++~7I++=============~~~~====++====~~=++I?==+?788DDD8DDDDDD8DOOZOO 8OZZOZZZOZZOZOZZZ77I$7$$77I7I7$$$$$$$$$$777777$$77$$OOO888O8ZZ8ZO8OO$7?I77ZZOZ$O
????IIIIIIII?=+===?7??+=~+=~=++++++==+=====+===~~:~+=?IIIIII$888888888888DDZ8?I7 8OZOZOZZZOOZZZO$Z$$$$ZIZ7$7777Z7$Z$$$$$$$$$$O8$$777ZOO8OOOO8ZO8OO87O88OI7I77?77Z
?I?IIIIIIII7$?====~?7=+=~~~==~+=====~~~~~=~~===~~~=++?IIIIII$888888888888DDOOIII OOOZOOZOOZOOZOZOZZZ$$Z$$Z7ZZ77I$Z8O7II?I?I78O$$$77$8O88OOO8ZZ8OZO$8O7$7I77IIZ$7O
IIIIIIIIIIIIIO====~I$?=+=~:~======~~~~===~=====~~===+??IIIII$88DDDDDDDDDD8D8ZIII OZOZO8OZZ8ZZOOZZZZOOZZ$OZZZZ$77$$$ZZOOO8OOZZ$$$777$888ZOO88ZO88ZOO$$I7I?7I$Z7$$$
IIIIIIIIIIIIII88$Z8OO7+=+==~=============~~~~======+??777$77$ZOOO7OO77$O8NDDOI$Z 8ZOOOZOZ8OOO88OZOOOOOO7$OZOZZ$77$$$7$$Z$$$77$$$$7$OZO8Z888O$Z888$$I?I7?7$IZO8ZZZ
IIIIII7IIIII7II7$7II78I?========~~===~===~~~~~==+==?I+7I?IIIIIIIIIIII?$$$7II???? 88OOOOZZ8O8OOO8OOZOOOZOOZOOZO$$$$$$$7$$77$$$$$$$Z$ZO$$OD77OO8Z7ZI7I77$I$77Z$O88O
7I777IIIIIIII77II77778Z$?========~~~=~~=~==~~~~++=+?++II?II???III?+I??Z$$$I?I7+? 88OO8O88OO8ZOOZOOO88O$ZOOZO8ZZ$$$$$Z$$ZZZZZ$$$$$$$ZDZ$8OZOO8O$8887IZ7OZ8O8ZOOO88
IIIIIIIIIIII7II7$777I7$IZ?+=++=~=====~~==~~====+++?+=+I??==+?I777II77~~~~Z7II7?I 8O888OOOD88D8ZOOO8OOOZZ88O8O$~77$Z$$ZZZZZZZ$$$ZZ$$$,D,O88$$88$II$?$IZ$888O8DD8OO
IIIII77IIII77$777IIII78?IOI++?+=====~~~~~~====+?+7++==I+===?IIIIIII7=:::~~~~?+++ O8D88O8ODD88O8Z888OOZZO88Z,+8.:$7$$$$$$$$$$$ZZZ$$$ZO+??=78Z8Z$OZ$$ZOOOOZ8ODDD$ZO
777II7777777777IIIIZ888+++?OI???+=====~~=~====+?I+=====Z8+++7IIIIII7::::::~+==== 888888OOOOOO88ZOO8DOOZ7,.7~Z,=7$7$$$$$$$$$ZZZZZ$$$ZNDZ8ZZ.:==IOZZZOD8888ZOOZZZOO
77I7777777777IIIII88888?++++?$7???+=======++??$++=====+O88D?IIIIIII:::,::~~~:~~~ OZD8O88888OOOO$...=~=+,II:8$$$Z77$$$$7$7$$$$$$$$$$DNN8888$OI?=~:~~+88DD8O8OZO8OZ
7I7IIIIIIIIII7II8888888I+++=???IZIIII???+7$$??+++======788888OIII?I:::,:~::::::: O88O8888OO:...,.,I7I$+88?,8ZZ$8$77$$$$$7$$$$$$$$$$NNMMD888O7$7III?I?+..O8D88O8OO
I7777$Z$$7I7778888OD888Z?=++==+??++??++===========~===+?D8888888II:::::::::::::: 88OOOO8$..,+++=~78787DNIO=8ZZ8DN7$7$$7$$777$$$$$$ZNMNNN88DDOOZOODNNNNI$=..ZD8OO8
777$O$+???I$888D88888$$O?+====++++=======~~=======~====?7Z88888888:::::::~~::::: 8O88O8..~+I$Z7$Z8DND$NNO$?8OZDDND$$$$77777$$$$$$Z8MMMDMD8DNNNNNNNNNNN88O??$,OOZ$
777777?+??8D888888DOZO$O+=======++====~~~~~~=======~===+:I88888888O8::::~~:::::: 888O$,:~IZOOZO8ONNNODNNI++8OODNNN8Z$$$$$77$$$$$ZO8MMDMMM88MNNMNNNNMMMDI8OOO$?,O8
77777$??+88888888DO$$I$O?=======++===~~~~~~~======~=====?I7D88888888888:~~:~~~~: 8D8~~=?$8OOO888OMNNDNNDOOO8OODDNNDDZ$Z$$$$$$$$ZODDMMNNNMD8MMNMMNNMMMMD88O8OOO$~+
777$Z$I+8DD8888D8877O~8Z?+======+====~~~~~~====+===~~==+$O7?888888888888I:~~~:~~ DD$+7ZOO8O8O888ONNNNNN8OO888ODMDMDD8ZZZZZ$$$Z8D8DNNMMDNMN8NMMNMMMMMNMN8O888OOOO$
7777$$788DDD888888IIO=Z$+=============~~~~~=======~====+$Z$I?8888888888888===~~= DDZ$OOOO8OO88888NNNNMN88O88ODNMMNNDD88D888888DDD8DNMM8DMM8NMMMNNNMMMMN8O8888OOOO
I?????O888O88888O8I7Z=?Z?+==~=~====~=====~~========~===?8?87I$88888888888888?+?? NOOZOOOZ88O8888ONNNNNN88OOO8DNNNMNNDDD88D8D8888DDDNMMDNMM8NMMNMMMMNMMM8O88888OOO
?I?+$8D88888O8888OI7I8I?I====~==~~=====~~========~====+I$?887II88888888888888OD~ D$OZOOZO8OOO8888DNNNMDDDNDDD8DMDMND88888888888DDDDNMMNMMMDDMMNNNMMMMMNOO88OO8OOO
+++D88888OO888888877I8$III=~~=~~~======~~=============+II7O8DII$8888888888888888 D+8ZOOOO8Z88888NDNDNNDD88D888DMNMNNNDD88888D8DD8DDDMMDMMMDDNMNNNMMMMMNDO88888OOO
================================================================================ II77III????????II??++?II7IIIIII7ZZ8888D88ODOO$$$77?++?+?$$7+7$$$$$$$ZZOOOOZ$$$$Z
================================?OZZ7=+=?I?O7+?=?+============================== IIIIII?+????+????????IIII??II7$Z8DNNN88DOOOOZOZ8ZZ7?+???$$$7$$$$$$$$ZZZOOOZZ$$ZZ
=============================++I8DDDDOO888D88888DDZ+============================ I?IIIII?+?+??III???++?IIIIII7$O8Z8OZO8DDDODZOOZZZZOO????$$$Z$$$$$$$$ZZZOOOZ$$$ZZ
===========================~ZOODDDDDD88DDDD88D8DDD88Z?========================== IIIIIII???????II???????IIII?IZ$8O88O88Z888D88ZO8OOO8O8??$$Z$$$$$$$$$ZZZOOOZ$$$$Z
===========================O8DDDD8DDDD8888OODO8DD8DD88O?======================== ??IIIII????????????+?I?III+?$7O8O8OOO88DO88888ZO8D88D88I$$ZZZZ$$$ZZZZZOOOOZZZZZZ
=========================?ODDDDDDDO8OOZ7II7$ZO888888D8NZ$======================= +++I777I???????I???=?IIII?+I$7$ODDD8OZO8DO88D88ZDDDD8D8O$$$$$$$$$$$$ZZZZZOZZ$$$$
=======================~~ODDD8OZ$7I?++++++=+??IIIIII8DDD8Z====================== +==?III????????I???????+?7III??I$ZZ8$8OO8DODDDDDODDDDDN8Z$$$$$$$$$$$ZZZZZZZ$$$$$
======================~=88DDD7II?+?+============+++??7888D8===================== ===?III????????I???++++?II?~::::~~~=+I$O8DD8DDDDDODDDDDDD$$Z$$$$$$$$ZZZZZZZ$7777
====================~~~+8D8$??+++++===============++++?$DD8Z==================== ===?III??????????+++==?+7+:,,,,,:::~~~==$8D88DDDDDODNDDDD8$$$$$$$$$$$ZZOZZZ$$$$$
==================~~~~=8887???+++++++=============+++++?7888O=================== ===?I7I?????+??+======??$:,,,,,,,:::~~~~+$8DODDD8D8DDNDDDD777777I????I7$ZZZ$$$$$
================~~~~~~O8O?+??++++=+=======~========++++++I8O8=~~================ ===?III???+?+++~,.,~=~=O+,,,,,,,,,,::::~=+Z8MONNDDD8NNDDDND?+?++=======++??I7$$$
==============~~~~~~~IOO?+?+++++======~~~~~~~~~~~====+++++?OZZ=~~=============== ===?III+???+++=,::.,:~+N~,,,,,,,,:,::::~~=I8NNDNNDDD8NNNDDD7???++==~~~~~=++?I7$$
=============~~~~~~~~ZO??+++++++==~~~~~~~~~~~~~~~~===++++++?OO~~~~============== +++?III+==+=++=,,:,,,:I8~,,,,,,,,,,:::~~~~=7DNONNNDNNDNDNDN877??+++????IIIII77$$
============~~~~~~~~~8O????+++===~~~~~~~~~~~~~~~~~~====+++++$Z~~~~~============= ???III7????++++~..,,,=$8~,,,,,,,,,,,::::~~=+ZDM8NNNMMNNNDD8D$7?+====++?I7ZZOZOZZ
===========~~~~~~~~~:87???+++=====~~~~~~~~~~~~~~~~~=====++++IZ+~~~~============= I777777II??++++=:,,::+IN~,,,,,,,:::::~==???=+ONNNMMNMMNNDNDDZ7+=~:::::~=?7ZZZZZZ
==========~~~~~~~~~:~OI???++++=====~~~~~:~~~~~~~~~~~====++++78?~~~~~~=========== +II777II???+=+=~::,,:+?M:,=7I+~::~~=?$OZI?+=+IDMNMMMMMMNDNDD877I++=~~~:~~~+I7$$$
==========~~~~~~~~~::87??++++=====~~~~~::~~~~~~~~~~~====++++$D?~~~~~~=========== =?I77III??+=~,...::,:~IN::~?7$?~:~~+III7$8DZ$?$DMNMMMMMND8ND8Z$$7I?==~~~==+?I7ZZ
==========~~~~~~~~::~O$??++=====~~~~::::::~~:~:~~~~~~~==++++Z8+~~~~~~~========== ~???III??+=,......,:~=ZN,=Z$7$O?,,~=?I+7:I7$77I$NMNMMMMNMDDD8O$$I?+=~~~:::~==+I7
==========~~~~~~~~:::IZ+++===~~~~~~:::::::::::::::~~~~===+++O8?~~~~~~~~========= ~=+III???+=:........,~8D:~:=?++:,:~=++==========OMMMNMNNNNDDD8$$7I?++=~::::~==?I
=========~~~~~~~~:::::O++++++=++==~~~:~~~~~~~~~~=+++==+===++ZO+:~~~~~~~========= =+II7III?+=.........,=88,:,:::,,,~~==~~~~~~~~~~=+NMMMMNMDDNDD8ZZ7II??+===~====++
=========~~~~~~~:::::$Z++?7?+++?7$$I+===~~==+?I$7?+=~==+?+++Z7+~~~~~~~~========= =II77II??+:,........,+ZZ,,,,,,,,,~=+++~::::::~~=+INMNMMDN8DDDDZ$7I??++======++?+
========~~~~~~~::::::7?+?II??+$O$OZ7$7+~~:~=IIIIZ?O$I$??+?++Z=?~~~~~~~~========= ~?77II??+:,,........,+IZ,,,,:~:,:~====I?=~:::~==+?OMMNNNDD8DDD8I???++==~~~~=+??I
========~~~~~~~~:::::I7+?I7$O?788:~?~??=~~~==~++~~Z8=?77?+++7+?~~~~~~~~========= ~I7III??+,~:.......,~+IZ,,,:+?:+=?ZZZ7++=+====++?I7NMMNNDD8DD88ZI?++====~~~==+??
========~~~~~~~~:::::I+++?I??????+===?+~~~~==~~=+??++==++===I==~~~~~~~~========= IIII????:,=:........~?IZ::~+I=:::=+======++????II??8MM8DD8DOD8ZOIII?+=+=~~~~=+??
========~~~~~~~~:::::?++++++===~~~=~+++==~=~=~~==~~~=======+I+~~~~~~~~~========= 7II?I??+,~=,........~??7I:=??~::~~=?II777I7$?=+?I+?ONMO8D8DO88ZZ777II???+====+??
========~~~~~~~~:::::++++==+=~~~~~~=+++=~~==~~~~~~~~~~~~~==+?==~~~~~~~~========= 77I???+:,=,.......,,:~?IO,::==OZ,,..,,~==7I~~~+?==IDDD8ODD8DODO$7IIIIIII++++???I
========~~~~~~~~:::::=+?====~~~~~~~=+?++~~====~:~:::~~~~~==+?+=~~~~~~~~========= =?I??~..,::,..,,,,:~==I7ZI,,:,,~~+:+II+++=~=~=+=~?$N88OODD888O877??II7II????????
========~~~~~~~~::::::++++====~~~~=++++=~~=====~~~:~~~~~==+?+==:~~~~~~~~======== ,:~~,......,..,,,:==++7$$Z?,,,,,:~=++=~~==~~~===+$NDOOOODD888Z8O~:,:=III??????I+
=========~~~~~~~~::::+++++===~~~~==I??I====?+===~~~~~~~===+?===~~~~~~~========== ,:~,......,,,,..,.:=+?7$7ZD:,,,,,::~~~~==~====++$ND8OOOODDD8DNO8::,:=+=+==+++??=
=========~~~~~~~~:::::?+++++===~~=++ZO87I7$OZ?~~==~~~~~==++?=+~~~~~~~~========== ~=,.....,,.,....,:~~:+I7$ZDM:,,,,,,:::~~~==++?7ZD8DO8O8DDND888O8?~::~~~~~==+?I+~
=========~~~~~~~~:::::::+??++=====++?IIIII++==~~=+======+++~~:~~~~~~~~========== ~:,,...,..,,.....,:,,~?$$$DNMN:,,,,::~~~=+?I$OOZZO8OO888DN888D88?~::~~~~==+?I777
==========~~~~~~~~~:::::=???+++++++++++=?I?++===+=+=+++++++~~~~~~~~~~=========== ~:,,,,,..,::.....~,,,:?7$$ODNNM?:~~~==+?7$$Z$7$ZO8OND8D8DNDOO888?:,,~I7777777777
==========~~~~~~~~~~:::::?????++??I??++=~=~===+??===+=++?+?~~~~~~~~~============ ::,,,.,,,:~:,....:,:~~I77$Z8NNDMM?++?I777III?I7$O$ZNNDDN8DN88O8O7:::~?$$ZZZZZ$$7
===========~~~~~~~~~::::~??????++$I$7??=====??+==+++==++++?~~~~~~~============== ,:~++?=~:=I+:,,..:,~=~II7Z$O8DDNM$?~==+++??++?I7ZZDNNNDDNDNZO8O8ODI:~=?+?I$Z$$$+
============~~~~~~~~~:::~$I???+++++???IIIII?++=+==+++++++?~~~~~~~=============== ,,:~+???=?I?+:,...~~=,I7$O$ZO8ZDMZI=~~~======?I$78ZNDDDDDNDOOOO88NDZZ,:~=+7$777~
=============~~~~~~~~:~:DD$?II?++?++++++++======++++??+?I?~~~~~================= ,,,,,,,::,:+?+:....=:~7I$ZI7ZOZ88Z7+~:~~~~=~~+I7Z88NN8DD8NNOZ8ODNND8877?I7$$777:
==============~~~~~~~~:DDND7I??++?7?+++++======++=++++?IZ$~~==================== ::::::,,,,,:~,...,::,:77I8$$Z88ZI77?==~~~~~:~=?$88DNDOD8DND8O8DOMN8D$I?+??$O7I?:
===============~~~~~~ODDDDD7$II?+?+?+++=======++++++?I$I8D~~~================+== .,,,:~~==~~:....,~+~.,7ZI8Z7$OO$?III+=~~~~::~=IO8DNNO8DDNDD8ZD8DND8O7+??+?+II$I:
=================~~~IDDDDDDI7$Z$III??++==+=+++++++I7Z7?IDDD~==================== .,,,,::~~~~,..,,:::,,~7O7DD7$Z8O??II+=~~::::~=OO8DDD88DDD8NDONDDN8Z7++==+++?+???
==================~~DDDDDDDII7$$Z$$IIII7?II?III77ZO$II?IDDD8~===========++++++== ::,,,,,,,,:,,,,,,,,,~~?ZODD7ZOO8+III+~:~::,::$O888DD8DDND8NNODD8DZ$======+++++++
==================~DDDDDDDDIII7$ZZOO88888888D88OZ$77IIIIDD8DDZ=======+++++++++++ ~~:::::::,::~========?IZD8D$Z$ZZIIII+=~:::,,+O88888DDNDDDDNDD8888I=============+
=================8DDDDDDDZZIIII7$ZZOOOO888888O$$$III???IDDDDDD88I+++++++++++++++ :::::::::,:=??+=+??+??78ND8O7$$7????+~::::::OZO8OODDNDDDDDNNDZOD7=+===========++
++============DDDDDDDDDNDZ$7IIII777$$OOOOOZZ777III?????ID8DD8DDDDD8==+++++++++++ ,,,,,,,,,:~==~:::+I$7?ZDNNDZ$77I?+++~~::,,:+$ZZ$O88DDD8DDNNDO8OO====+=========++
=+++++=7DD?8DDDDDD8DDDDD$$Z77I77777777I7I7I?I????I??+??IDDDDDNDDDDDDD88D8O++++++ ~~~~==:,~=++~::,:~7Z$IDDDNZO$$7I7?+=~:::,,:O$$$ZZO888DO8DNNDOOZ+=+=?+========+++
+++++8DNNDDDDDDDDDDD8DDD$ZZ$77777$$7II???+?????III?+???8DDDDDDDDDDDDNDDDZ$DDD+++ =+??+==++?++=~::~~=$Z$NDNOZOZ77?==~~::::,,+7777$Z88OO8O8DDD8OO++==+?========++++
+888DDDNDDDDDDDDDDDDDDDDZOO$$7777$$$$7I??+??IIIII?????IDDN8DDDDDDDDDDDDDDZODDDO8 =??+~~+?++++=:::~=~I$Z8D88OZ$7I+=~~:::,,,=77I7Z$Z88ZOOO8DNN8OI====??=====++=++++
8DNNDDDDDDDDDDDDDDDDDDDDZOOZZ77I77$$$$$7I?IIIIII?????IDND8DDDDDDDDDDDDDDDDD8DDD8 +?I7I?~:,,,,,:::~~~=I7I7777I+I+~~:::,,,,:??I7$$7ZOZZZZZ8DNDOI+++=+?+====?+=+++++
DDDDDDDDDDDDDDDDDDDDDDDDDDOOZ$77I7$$$$Z8OZ$IIIII????$DDDDDNDDDDDDDDDDDDDDDDDD8DD III77?=~~~:::::~~=~~~===+====+~~=~::,,,:=+I?$O7ZOO$OZ$$888D7?+++=??=+==I+===++++
DDDDDDDDDDDDDDDDDDDDDDDNDODOZZ$7777$$$$$$77IIIIIIIIODD8DDDDDDDDDDDDDDDDDDDDDDDDN ??III?=:,,,,::~~~~~~~~~~==~====~=======????$ZZ7ZO8$O7Z7$O8OI?++++I?+==I+==+++=+?
DDDDDDDDDDDDDDDDDDDDDDDDDDZD8ZZ$77$$Z$$$$77777IIIIDDDDDDDDDNDNDDDDDDDDDDDDDDDDDD ??????+~:::::~~~=~~~~~~~~~=~~~~====~~=++++7Z$Z$$Z87O7$IO$O8I+++?+?+==I?==++==?+I
DDDDDDDDDDDDDDDDDDDDDDDDDNDDDDDZ$7$$$$$7$$777IIIIDDDD8DDDDDDDDDDDDDDDDDDDDDDDDDD ????++~::::,:~:~=~:~~~~~~~~~~~~~~~~~~~+++7$$OZ7$Z87Z7Z$7$Z8I?+??+?+=?I==+=~?8N?.
λ Improving Customer Service...
λ ...by improving processes
λ ...and enabling better decision making
λ Better (automated) actions for proactive customer service
λ Better insights for bol.com employees, sellers
3

6. Data-driven Development / Feedback Loop
START
|
| 1
v
.---------'----------.
| DATA |<-------------------------.
'---------,----------'
|
| 2 | .----------.
v | 3 | ???????? | 4 .-----------.
.---------'----------. .---------------. |---->| ???????? |---->| PROFIT!!! |
| PREDICTIONS |----->| ACTIONS |-----| | ???????? | '-----------'
'---------,----------' '---------------' | '----------'
v |
.---------'----------. .---------------. /
| INSIGHTS |----->| DECISIONS |---'
'--------------------' '---------------'
4

7. Data-driven Development / Feedback Loop
START
|
| 1
v
.---------'----------.
| DATA |<-------------------------.
'---------,----------'
|
| 2 | .----------.
v | 3 | FEEDBACK | 4 .-----------.
.---------'----------. .---------------. |---->| REVENUE |---->| PROFIT!!! |
| PREDICTIONS |----->| ACTIONS |-----| | SAVINGS | '-----------'
'---------,----------' '---------------' | '----------'
v |
.---------'----------. .---------------. /
| INSIGHTS |----->| DECISIONS |---'
'--------------------' '---------------'
5

8. Billie, our Customer Service Assistant
Figure: This is baby Billie
λ About 10 years old
λ Knowledge system with manual written patterns & rules & heuristics & synonyms...
λ Created by external party
λ 2M conversations / year and going UP
6

9. DATA ANALYSIS

10. Data Analysis
8

11. Analysis Tool
9

12. Some Statistics
Conversations 4412
Solved 31%
Customer leaves 33%
Unsolved 18%
To human agent 17 %
Table: Conversation Level Statistics
10

13. Some Statistics (Answer Level)
Answers (from Billie) 28044
Good 75%
Incorrect 17%
Repetition 6%
Other 2 %
Table: Sentence Level Statistics
11

14. Let’s Go Deeper...
Of the conversations...
λ ...52% contain ≥ 2 confirmation questions (”Do you mean ...?”)
λ ...44% contain ≥ 1 incomprehension answers (”I don’t know what you mean”)
12

15. Clustering
Figure: Clustering of answers from Billie, as a directed graph (left=unknown cluster, middle=start of
conversation, right=end of conversation). Uses DSBSCAN, character tri-grams.
13

16. Clustering 2.0
Wil je weten wanneer je bestelling afgeleverd wordt?
Ben je op zoek naar (informatie over) een product?
Wil je uitleg over het retourneren van een bestelling?
niet herkend
Je wilt graag weten waar je bestelling blijft, begrijp ik...
Wil je weten wat te doen met een beschadigd (of niet goed...
Oke, met mij gaat het goed. Waarmee kan ik je helpen?
Zeg, ik ben geen rekenmachine! (Maar het antwoord is 19)....
Om je te kunnen vertellen wat de status van je bestelling...
Wil je weten of je bestelling doorgekomen is?
Wil je informatie over kortingscodes van bol.com?
Hallo, je hoeft geen aanhef te gebruiken hoor ;-) Ik ben ...
Wil je informatie over het inwisselen van cadeaubonnen, c...
Gaat het om je meest recente bestelling met daarin de art...
Goedemorgen, je mag wel gewoon Billie zeggen hoor... Je k...
anders
start
Figure: Interactive flow-diagram of conversations, based on answers from Billie
14

17. Why Improving Billie?
λ 24/7, no delay for customers
λ Automated solutions have lowest cost
λ Escalating to live chat, our most efficient channel (both for customer / bol.com)
λ By solving most frequent, easiest problems, our agents can focus on complex cases
λ By integrating we can make more decisions ourselves
15

18. Current Results
λ ML-based model that recognizes intents (topics) of customer
λ Gives suggestions to external party
λ Context-based prediction of intent (predicting relevant FAQs for customers, agents,
etc)
λ A/B test soon, measure impact on interactions, nr. escalations, etc.
16

19. Comprehender Is Live
We are LIVE
textbfThe service Comprehender
Handling ±1 input sentences per second, around 43000 a day.
Mostly about ”where is my order”, ”Contact” and ”I search for a product”
Response times ±10 ms per call.
17

20. How does it work?
P(y = i | x) (1)
λ Dataset of input sentences + labels (intents)
λ {(”I want to return my phone”,RETURN), (”Does my phone arrive
on time”,WHERE_IS_MY_ARTICLE), ...}
λ Character-level Convolutional Neural Network (after selection of word-level, LSTMs,
etc)
λ Gives suggestions to external party
λ Context-based prediction of intent (predicting relevant FAQs for customers, agents,
etc)
λ Analysis and AB-test planned
18

21. Neural Networks
Neural Network ̸= Anything like a human
brain
19

22. Neural Networks
A learnable function f : X → y, using set of parameters Θ.
y = f(x; Θ) (2)
Applied in sequence (function composition), also called layers:
f = fn ◦ ... ◦ f2 ◦ f1 ◦ f0 (3)
20

23. Neural Networks
Hidden layer:
hi = g(WT
x + b) (4)
Convolutional Neural Networks: similar, but only connected locally.
λ Images: 2D kernel
λ Text / Audio / Time-series ...: 1-dimensional (over time)
λ Recent successes: residual connections, attention, transfer learning...
21

24. Convolutional Neural Networks for Text
Some Text
Convolutions Max-pooling
Length
Feature
Quantization
...
Conv. and Pool. layers Fully-connected
Figure: Character-level Convolutional Networks for Text classification, Zhang et al.
22

25. Context-based intent prediction
λ Historically available context (time since last order, type of order, numper of
packages, time of day, time since last return
λ Experiment using limited variables, Gradient Boosting model
λ 29.0% vs 23.6% (baseline) top-1 accuracy
λ We expect we can do much better using more relevant data
23

26. CUSTOMER SERVICE 2.0

27. Modular Approach
.--------------------. .---------------. .--------------------.
| Anomaly Detection | | Agent Assist | | Agent Forecast |
'---------,----------' '-------,-------' '---------,----------'
| | |
.---------'----------. .------'------. .---------'----------.
| Trend Analysis |-----| |-----| Intent Recognition |
'--------------------' | | '--------------------'
.--------------------. | CUSTOMER | .--------------------.
| Speech To Text |-----| |-----| Intent Prediction |
'--------------------' | SERVICE | '--------------------'
.--------------------. | | .--------------------.
| Sentiment Analysis |-----| 2.0 |-----| Search |
'--------------------' | | '--------------------'
.--------------------. | | .--------------------.
| Auto Routing |-----| |-----| Feedback Analysis |
'--------,-----------' '------,------' '---------,----------'
| | |
.--------'-----------. .-------'-------. .---------'----------.
| Auto Channel | | Microbots | | Insights |
'--------------------' '---------------' '--------------------'
25

28. Conversational UI
λ Adding new agents, e.g. for routing the customer to correct channel, returning
process ...
λ ContactBot: More conversations directly with seller, better channel selection
λ Better for the customer, better for us
λ Understanding user input (free text, speech): classification, entity recognition
26

29. Text & Trend Analytics
λ Getting insights out of data
λ Seeing what important issues to fix, problems with products, etc.
λ Feedback to sellers, service agents, IT teams...
27

30. Auto Routing
λ Directly getting the best possible agent
λ Based on (historical) customer feedback, handling time, agent info, other metrics
28

31. Summary
λ Make Customer Service + bol.com better using AI
λ A way of working for data-related projects
λ Analysis + development of AI systems
λ Modular approach for further developing the future of customer service
29

32. End
30