Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Future is here or how to
test NLP in chatbots
Iryna Yaroslavtseva
● 5 years of testing;
● 3+ years of testing chatbots;
Cherkasy, Ukraine
IRYNA
YAROSLAVTSEVA
of businesses want chatbots by 2020
Rule based AI based
Rule based AI based
Current project tech stack
NLPStructure
Messenger
“Show me yesterday’s financial news”
Utterance
“Show me yesterday’s financial news”
Utterances
● Delivery
● Thanks!
● Book me a flight to Rio next week.
● I already have...
“Show me yesterday’s financial news”
Intent: showNews
Verb Noun
“Show me yesterday’s financial news”
● checkCoverage
● buyIphone
● findBus
Intents
● bookFlight
● changeLine
● bookAppoint...
“Show me yesterday’s financial news”
Entity Entity
Intent
● bookFlight
● checkCoverage
● buyIphone
Entities
● City, Date
● Location, Country
● Model, Color, Capacity
DISCOVERY MODEL
TRAINING
TESTING LEARNING
ON
PRODUCTIO
N
DISCOVERY
Key requirements
● Bot language
● Feature scope
● Target audience (user’s profile)
Key requirements
● Bot language
● Feature scope
● Target audience (user’s profile)
Key requirements
● Bot language
● Feature scope
● Target audience (user’s profile)
MODEL
TRAINING
TESTING
Challenges
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
Solutions
● Risk analysis
● Dat...
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Dat...
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Dat...
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Dat...
Challenges Solutions
● Unlimited inputs quantity
● No definition of quality
● No clear exit criteria
● Risk analysis
● Dat...
Structured response
Turning on / Turning
off NLP
Lost in the flow
Business restrictions
Profanity words
Similar intents
De...
● Order pizza = Order piza
● Change day to Monday = Change day to Mobday
● Send picture = Senf picture
Misspellings
● New offers = New deals
● Some help = Some support = Some assistance
● Order iPhone = Purchase iPhone
Synonyms
● What benefits do you have? = Show me the advantages
● Where is my parcel? = Track my order
● Make a reservation on Frida...
Can I order bundle of
two headphones, one
for me and one for
my dad?
Intent + other words
I saw your promo,
how to order t...
Hello, find my bus
station please.
Intent + small talk
findStation
Morning! Nearest
bus station. Thanks.
My inbox is a dumpster! I'm like flooded with all the
messages. How can I unsubscribe ?
Intent + other sentences
unsubNews
● Lol - laugh out loud
● Asap - as soon as possible
● Np (No Problem), But how can I send my
booking confirmation
Acronyms...
Informal shortening
● Gonna - going to
● Lil’ - little
● Wanna - want to
Formal shortening
● I’m - I am
● Smth. - something
● I’ve - I have
Top:
👍 😀
😂 ❤️
Emoji
Banned:
🔪 🖕 🍑 🍆 💩
🖕🖕🌈 + 🚫 ⛪️ +
● Kudos! = praise and honor received for an
achievement
● You rock! = You're awesome (at something)
● On fleek = smooth, n...
● I’d like to order dress
● I’d like to know where
is my dress?
Similar intents
trackOrder
makeOrder
● Sorry, I didn’t get that. Try again
● If you didn’t find what you wanted feel
free to see information in the Main Menu
o...
Score for None intent
● If utterance score < 0.3
● Avoid repetitions
none
Dogs suffer = stop animal testing
Business restrictions
getInfo
● Bullshit
● Dammit
● Go to hell
Profanity
● Friday, 29 May 2015 05:50:06
● Test@test.com
● $ 1,234.60 = 1 234,6
Structured response
● Feedback forms
● Free form surveys
Turning off/ turning on NLP
Lost in the flow
Lost in the flow
● Intents - 3
● Utterances - 30
● Checklist based testing ≈ 8 hours
Checklist execution time
LEARNING ON
PRODUCTION
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● ...
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● ...
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● ...
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● ...
● Gathering the stats
● Analyzing what to add/change/delete in NLP model
● Training of the new version of NLP
● Testing
● ...
New project roles
NLP model training
Statistics
Bot flow
Copy
NLP model training
Statistics
CONVERSATIONAL
DESIGNER
CONTEN...
● Chatbot project = regular project
● NLP is the only unique component
● Analyze risks and statistics to enforce
effective...
● Chatbot project = regular project
● NLP is the only unique component
● Analyze risks and statistics to enforce
effective...
● Chatbot project = regular project
● NLP is the only unique component
● Analyze risks and statistics to enforce
effective...
NLP testing checklist
Thank you!
Questions
About me
Facebook LinkedIn
QA Fest 2019. Ірина Ярославцева. Майбутнє вже тут, або як тестувати систему обробки природньої мови у чатботах
QA Fest 2019. Ірина Ярославцева. Майбутнє вже тут, або як тестувати систему обробки природньої мови у чатботах
QA Fest 2019. Ірина Ярославцева. Майбутнє вже тут, або як тестувати систему обробки природньої мови у чатботах
QA Fest 2019. Ірина Ярославцева. Майбутнє вже тут, або як тестувати систему обробки природньої мови у чатботах
Upcoming SlideShare
Loading in …5
×

QA Fest 2019. Ірина Ярославцева. Майбутнє вже тут, або як тестувати систему обробки природньої мови у чатботах

19 views

Published on

Ще якихось 3 роки назад важко було уявити, що роботи зможуть увійти в життя людей і полегшити виконання повсякдених речей. Сьогодні штучний інтелект частково замінив працю людей, допомагаючи бізнесу досягати своїх цілей і стаючи пріорітетним напрямом розвитку.
Зі збільшенням попиту на чатботи, збільшується кількість інструментів для їх розробки, змінюються технології та ускладнюються задачі, які робот повинен виконувати. Розмови клієнтів з чатботом заощадять близько $8 мільярдів до 2022 року завдяки системам NLP (обробка природньої мови), яка є основною складовою частиною чатботів. В залежності від задач, які виконує NLP, ціна помилки може бути дуже вагомою.
Доповідь розкриє тему обробки природньої мови, як частину чатботів та побудови стратегії тестування моделей: яким чином розподілити пріорітети, які задачі можна автоматизувати і як успішно випускати чатботи в умовах нескінченної гонки за інноваціями.

Published in: Education
  • Be the first to comment

  • Be the first to like this

QA Fest 2019. Ірина Ярославцева. Майбутнє вже тут, або як тестувати систему обробки природньої мови у чатботах

  1. 1. Future is here or how to test NLP in chatbots Iryna Yaroslavtseva
  2. 2. ● 5 years of testing; ● 3+ years of testing chatbots; Cherkasy, Ukraine IRYNA YAROSLAVTSEVA
  3. 3. of businesses want chatbots by 2020
  4. 4. Rule based AI based
  5. 5. Rule based AI based
  6. 6. Current project tech stack NLPStructure Messenger
  7. 7. “Show me yesterday’s financial news” Utterance
  8. 8. “Show me yesterday’s financial news” Utterances ● Delivery ● Thanks! ● Book me a flight to Rio next week. ● I already have a phone 9. What the plan going to cost me ulimited switching over from crickety eyes you have anything without 1st month.
  9. 9. “Show me yesterday’s financial news” Intent: showNews Verb Noun
  10. 10. “Show me yesterday’s financial news” ● checkCoverage ● buyIphone ● findBus Intents ● bookFlight ● changeLine ● bookAppointment
  11. 11. “Show me yesterday’s financial news” Entity Entity
  12. 12. Intent ● bookFlight ● checkCoverage ● buyIphone Entities ● City, Date ● Location, Country ● Model, Color, Capacity
  13. 13. DISCOVERY MODEL TRAINING TESTING LEARNING ON PRODUCTIO N
  14. 14. DISCOVERY
  15. 15. Key requirements ● Bot language ● Feature scope ● Target audience (user’s profile)
  16. 16. Key requirements ● Bot language ● Feature scope ● Target audience (user’s profile)
  17. 17. Key requirements ● Bot language ● Feature scope ● Target audience (user’s profile)
  18. 18. MODEL TRAINING
  19. 19. TESTING
  20. 20. Challenges ● Unlimited inputs quantity ● No definition of quality ● No clear exit criteria Solutions ● Risk analysis ● Data analysis
  21. 21. Challenges Solutions ● Unlimited inputs quantity ● No definition of quality ● No clear exit criteria ● Risk analysis ● Data analysis
  22. 22. Challenges Solutions ● Unlimited inputs quantity ● No definition of quality ● No clear exit criteria ● Risk analysis ● Data analysis
  23. 23. Challenges Solutions ● Unlimited inputs quantity ● No definition of quality ● No clear exit criteria ● Risk analysis ● Data analysis
  24. 24. Challenges Solutions ● Unlimited inputs quantity ● No definition of quality ● No clear exit criteria ● Risk analysis ● Data analysis
  25. 25. Structured response Turning on / Turning off NLP Lost in the flow Business restrictions Profanity words Similar intents Default answer Acronyms Informal shortening Shortening in writing Emojis Slang words Intent + other words Intent + small talk Intent + other sentences Misspellings Synonyms Similar phrases NLP testing checklist
  26. 26. ● Order pizza = Order piza ● Change day to Monday = Change day to Mobday ● Send picture = Senf picture Misspellings
  27. 27. ● New offers = New deals ● Some help = Some support = Some assistance ● Order iPhone = Purchase iPhone Synonyms
  28. 28. ● What benefits do you have? = Show me the advantages ● Where is my parcel? = Track my order ● Make a reservation on Friday = Book appointment, Friday Similar phrases
  29. 29. Can I order bundle of two headphones, one for me and one for my dad? Intent + other words I saw your promo, how to order the headphones orderHeadphones
  30. 30. Hello, find my bus station please. Intent + small talk findStation Morning! Nearest bus station. Thanks.
  31. 31. My inbox is a dumpster! I'm like flooded with all the messages. How can I unsubscribe ? Intent + other sentences unsubNews
  32. 32. ● Lol - laugh out loud ● Asap - as soon as possible ● Np (No Problem), But how can I send my booking confirmation Acronyms confirmBooking
  33. 33. Informal shortening ● Gonna - going to ● Lil’ - little ● Wanna - want to
  34. 34. Formal shortening ● I’m - I am ● Smth. - something ● I’ve - I have
  35. 35. Top: 👍 😀 😂 ❤️ Emoji Banned: 🔪 🖕 🍑 🍆 💩 🖕🖕🌈 + 🚫 ⛪️ +
  36. 36. ● Kudos! = praise and honor received for an achievement ● You rock! = You're awesome (at something) ● On fleek = smooth, nice, sweet Slang words
  37. 37. ● I’d like to order dress ● I’d like to know where is my dress? Similar intents trackOrder makeOrder
  38. 38. ● Sorry, I didn’t get that. Try again ● If you didn’t find what you wanted feel free to see information in the Main Menu or type ‘show me main menu’ Default (None) intent
  39. 39. Score for None intent ● If utterance score < 0.3 ● Avoid repetitions none
  40. 40. Dogs suffer = stop animal testing Business restrictions getInfo
  41. 41. ● Bullshit ● Dammit ● Go to hell Profanity
  42. 42. ● Friday, 29 May 2015 05:50:06 ● Test@test.com ● $ 1,234.60 = 1 234,6 Structured response
  43. 43. ● Feedback forms ● Free form surveys Turning off/ turning on NLP
  44. 44. Lost in the flow
  45. 45. Lost in the flow
  46. 46. ● Intents - 3 ● Utterances - 30 ● Checklist based testing ≈ 8 hours Checklist execution time
  47. 47. LEARNING ON PRODUCTION
  48. 48. ● Gathering the stats ● Analyzing what to add/change/delete in NLP model ● Training of the new version of NLP ● Testing ● Monitoring the results Learning on production
  49. 49. ● Gathering the stats ● Analyzing what to add/change/delete in NLP model ● Training of the new version of NLP ● Testing ● Monitoring the results Learning on production
  50. 50. ● Gathering the stats ● Analyzing what to add/change/delete in NLP model ● Training of the new version of NLP ● Testing ● Monitoring the results Learning on production
  51. 51. ● Gathering the stats ● Analyzing what to add/change/delete in NLP model ● Training of the new version of NLP ● Testing ● Monitoring the results Learning on production
  52. 52. ● Gathering the stats ● Analyzing what to add/change/delete in NLP model ● Training of the new version of NLP ● Testing ● Monitoring the results Learning on production
  53. 53. New project roles NLP model training Statistics Bot flow Copy NLP model training Statistics CONVERSATIONAL DESIGNER CONTENT MANAGER
  54. 54. ● Chatbot project = regular project ● NLP is the only unique component ● Analyze risks and statistics to enforce effective and efficient testing QA takeaways
  55. 55. ● Chatbot project = regular project ● NLP is the only unique component ● Analyze risks and statistics to enforce effective and efficient testing QA takeaways
  56. 56. ● Chatbot project = regular project ● NLP is the only unique component ● Analyze risks and statistics to enforce effective and efficient testing QA takeaways
  57. 57. NLP testing checklist
  58. 58. Thank you!
  59. 59. Questions
  60. 60. About me Facebook LinkedIn

×