Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

#ATAGTR2019 Presentation "Performance testing of Chatbot" By Sarah Lovely and Raja RajKaliappan

38 views

Published on

Sarah Lovely who is a Performance Tester at Cognizant Technology Solutions along with Raja RajKaliappan who is a part of Non Functional Testing CoE at Cognizant Technology Solutions took a Session on "Performance testing of Chatbot" at Global Testing Retreat #ATAGTR2019

Please refer our following post for session details:
https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-sarah-lovely-as-our-esteemed-speaker/

https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-raja-rajkaliappan-as-our-esteemed-speaker/

Published in: Technology
  • Be the first to comment

#ATAGTR2019 Presentation "Performance testing of Chatbot" By Sarah Lovely and Raja RajKaliappan

  1. 1. #ATAGTR2019 Performance Testing of Amazon Lex Chatbot & Alexa enabled Smart Speakers Raja Kaliappan (Raja.RajaKaliappan@cognizant.com) & Sarah Lovely S (SarahLovely.S@cognizant.com), Cognizant 14th-15th Dec 2019
  2. 2. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Market Predictions for 2020 Chatbots & Smart Speakers Key Market Players Benefits Customer service interactions will be powered by Chatbots85% Businesses expected to have some Chatbot automation implemented 80% US households will have Voice Assistant Smart Speakers75% 138 Million Voice Assistant Smart Speakers will be in use with an audience of 258 million 138MM Amazon Lex IBM Watson Microsoft LUIS Google DialogFlow Amazon Echo devices enabled by Alexa Google Home 24/7 Service Faster Service Business Automation Enhanced UX Cost reduction Scalability
  3. 3. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Amazon Lex - Chatbot service Amazon Lex  An AWS service that enables building conversational interfaces using voice and text  Provides deep functionality and flexibility of Natural Language Understanding (NLU) and Automatic Speech Recognition (ASR) to build highly engaging user experiences  A fully managed service that scales automatically with no need of infrastructure management  Pay-as-you-use model and charged based on number of requests FEATURES Natural Language Understanding Automatic Speech Recognition Seamless Deployment Auto Scaling Cost Effective Interoperability with AWS services Text & Voice Support Ease of Development KEY BENEFITS Simple Bot development through conversation flow definition Auto scalable platform eliminating infrastructure overheads No upfront cost and cost for idle time
  4. 4. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Amazon LEX / Alexa – Interaction Model Wake word Launch Invocation Name Utterance Alexa, tell Domino Pizza, I need a pizza Lex identifies the user intent as Order Pizza Slot Sure, what size pizza do you want? Lex prompts to elicit the Slot Size Prompt Slot Value I want a medium sized Pizza Lex captures the Slot value Medium I have placed your order and you will be receiving your pizza shortly Lex fulfills the Intent by placing the order
  5. 5. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) AWS Lambda – Code hooks for Lex AWS Lambda Serverless platform letting one run code without provisioning (or) managing servers Execution of Lambda functions (Your code) is event driven and can be triggered from other AWS services like Lex, S3 events, DynamoDB, API Gateway etc. Automatically scales and launches as many copies of the code as and when triggered and executes them in parallel Cost effective with Pay-as-you-use model and you are charged only for number of requests and the duration of the execution LEX Chatbot Architecture Lambda Role in a Lex Chatbot  Initialization & Validation code hooks (E.g. Validation of data inputs from the user – Claim Number format validation)  Fulfilment code hooks (E.g. Calls to On-premise Claims application for getting Claims Status)
  6. 6. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Lex Chatbot Performance – Key Considerations Critical Intents  Identify business critical Intents, test for maximum number of possible utterances so AI feature of the Bot is put to test under load Peak Workload  Maximum concurrent connections to Lex and maximum request rate for the critical intents End User Response Time  Define & validate SLA for End user response time Lex Latency  Validate & set baseline for Lex Latency Lambda Execution Time  Monitor and improve Lambda Execution time with code and/or memory optimization Lambda Memory Configuration  Configure Optimal memory for Lambda function considering performance & cost
  7. 7. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Lex Chatbot Performance Test Approach Performance Testing & Monitoring Approach for a Lex Chatbot Performance & Scalability Assurance High Availability Optimized Cloud Cost Best End User Customer Experience Key Benefits Lex API Tests Lex Monitoring Lambda Monitoring Enterprise Services Monitoring Load test Lex API for different throughput & Lambda memory configurations and optimize both performance & cost Leverage CloudWatch dashboard to monitor Lex metrics like Lex Latency, Lex Throughput etc. Monitor built-in Lambda metrics in CloudWatch and build & monitor custom metrics like memory consumption by extracting CloudWatch logs Leverage Enterprise APM tools like Dynatrace, AppDynamics for monitoring on premise Infrastructure & Application E2E Tests Simulate load from UI to capture end user performance for different throughputs & validate against SLA
  8. 8. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us)  Chatbot Initial Connection for first few initial requests observed to take higher time  End User & Lex API Response time affected by user proximity / Request origin  Lex Latency observed to be consistent for different test user loads  Lambda Cold Starts causing higher response time for initial few utterances  Lambda Execution Time improves with higher memory configuration but at cost of higher price  Lambda Overbilled Duration can be a factor in considering for cost reduction Lex Chatbot Performance - Observations
  9. 9. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Lex Chatbot Performance – Best Practices Balance both Lambda Performance & Cost by right memory configuration Maintain proximity between User base location, Bot orchestration & AWS resources to avoid network latency Best Coding practices to optimize Lambda Execution Time and optimize Overbilled duration Implement plugins to keep Lambda containers warm at right intervals to avoid Cold starts
  10. 10. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Alexa Architecture Request Voice Stream Response Voice Stream Request Text Response Text Alexa Voice Service Custom Alexa Skill + Alexa Skill Kit AWS Lambda Enterprise Services Other AWS Services Alexa enabled device User API test for Alexa Skill API Performance E2E Response Time ? Speech Recognition Natural Language Processing Interaction model definition Logic Execution Backend Calls
  11. 11. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Web Speech API Web Speech API Converts input text in the web page into speech using device’s speech synthesis system and plays it out of a device’s speaker Speech Synthesis (Text-to-Speech) Receives speech through a device’s microphone and converts into text through either device’s or server based recognition system & outputs the text in the web page Speech Recognition (Speech-to-Text)
  12. 12. #ATAGTR2019 As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial channels(Provided due credit is given to me/us) Alexa End User Performance Evaluation  Automates real world voice based interactions and assesses end-to-end performance of Smart Speakers like Amazon Echo (Alexa) & Google Home devices for a single user  Automated reports with Response time metrics and SLA validation highlighting outlying utterances  Identification of performance issues of smart speakers at the end user level  Elimination of human effort required for voice based testing with automation of test execution & reporting  Need for automation of voice based testing for Smart speakers  Used in unison with load testing of Alexa Skill API / Lex API to capture end user response time of Smart speakers Features BenefitsWhere can it be used? In-browser Speech Synthesis for pre-defined user utterances In-browser Recognition of voice responses from Smart Speakers Powered by Web Speech API Alexa End User Performance Evaluation Solution using Web Speech API How it works
  13. 13. Thank You

×