Downloaded 39 times
Uploaded byKalilur Rahman
Artificial Intelligence in testing - A STeP-IN Evening Talk Session Speech by Kalilur Rahman
The document covers the advancements in artificial intelligence (AI) and robotics, emphasizing the impact of AI on testing processes and the complexities involved in AI development. It discusses various AI methods, notable AI achievements, and highlights significant concerns regarding safety and ethical implications. Additionally, it presents statistics on software failures and examines how AI can aid testing through improved defect detection and predictive modeling.
More Related Content
Similar to Artificial Intelligence in testing - A STeP-IN Evening Talk Session Speech by Kalilur Rahman
![[TestWarez 2017] Testing Artificial Intelligence](https://cdn.slidesharecdn.com/ss_thumbnails/nogapawetestwarez2017-180114163331-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DevDay2019] How AI is changing the future of Software Testing? - By Vui Nguy...](https://cdn.slidesharecdn.com/ss_thumbnails/vuinguyenhowaiischangingthefutureofsoftwaretesting-190412021635-thumbnail.jpg?width=640&height=640&fit=bounds)
Artificial Intelligence in testing - A STeP-IN Evening Talk Session Speech by Kalilur Rahman
- 1.
- 2. • Progress ofAI and Robotics • What’s the need for Artificial Intelligence? • What will happen at singularity? • Some AI Concepts High level Intro to AI • Is it an Intelligent Activity? • Are we testing at the heights of Augmented General Intelligence? • AI in Testing - Is it augmented or Artificial / Is anything artificial about it? • How will AI evolve Testing? • Some Examples of AI Testing AI in Testing Agenda
- 3. - ANDREW NG Founderof Coursera, Stanford Adjunct Professor Ex. Chief AI Scientist of BAIDU
- 4.
- 5. Source : https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/ AGood Explanation of Progress of AI E a r l y A I Basic Turing Test Style Use of Memory and Knowledge Post John McCarthy’s Conceptualization Basic Robotics and Degrees of Freedom D e e p L e a r n i n g Rapid Infrastructure Growth Advanced Algorithms Big Data Explosion Quantum Computing M a c h i n e L e a r n i n g Algorithms Centric Statistics Driven Supervised and Unsupervised Learning
- 6. Perhaps the greatest ComputerScientist ever predicting on Machine Intelligence We have clearly passed the TURING TEST We are seeing Leaps and Bounds in advances of Technology! Let’s hear!
- 7. Neal Creative |click & Learn moreNeal Creative © Singularity or transcendence is around the corner – Aided by AI
- 8. https://youtu.be/zatL4uFRpC0 Fast Learning –Download and Fly an helicopter Can AI Take us to this stage? How about Fast Testing – Hey – Can I test this brand new “thing” in 2 minutes?
- 9. From a Leaderin AI – AI or an Algorithm Writing itself
- 10. Top Human WorldChampions Royally defeated by AI! 2011 2016 1996/97 IBM’s Deep Blue IBM Watson Google Deepmind AlphaGo won 60–0 rounds on two public Go websites including 3 wins against World Go champion Ke Jie.
- 11. But... AI isnot without controversies though! Facebook Researchers shut down an AI engine at the Facebook AI Research Lab (FAIR), discovering that the AI created its own unique language undecipherable by humans - Simultaneous glimpse of both the awesome and horrifying potential of AI Elon Musk - “AI isotentially more Dangerous than Nukes” sets up a $1 billion (£770M) OpenAI.org to try and promote safe development of AI Vladimir Putin - “Whoever masters AI will rule the world!” ISAAC ASIMOV’s Laws of Robotics Law 1: A tool must not be unsafe to use. Law 2: A tool must perform its function efficiently unless this would harm the user. The safety of the user is paramount. Law 3: A tool must remain intact during its use unless its destruction is required for its use or for safety.
- 12. Japan is usingAI and Robots in Multiple ways
- 13. https://quickdraw.withgoogle.com/ , https://www.autodraw.com/& https://g.co/aiexperiments Google is using Deep Learning to make Simple things – Better!
- 14. https://oxism.com/thing-translator/ Take it tonext level How can you easily integrate globally. Language will no longer be a barrier.
- 15. AI Startups aretaking it to next Level – in all areas Source: https://www.cbinsights.com/research-ai-100 Bots Automobile Computer Vision Core / Functional AI Commerce IIoT/IOT Healthcare Fintech Robotics Analytics Cyber Security Sales & Marketing
- 16.
- 17. “ Let’s getto TESTING • How is AI helping Testing? • How can we test better with AI? • How can we test AI systems Better? 17
- 18. “Be a yardstickof QUALITY. Some people aren't used to an environment where EXCELLENCE is expected” 18 Steve Jobs
- 19. Business Agility -Some Statistics 19 Google - refactors code by 50% each month* Netflix - 5 Billion+ API Calls per Day (and increasing daily) ~75% of Corporates to have bi-modal IT ~63% all projects are not aligned to Business Strategy ~79% organizations using CI/CD/DevOps practices in one form or the other 52% of Fortune 500 companies have disappeared from the list & Average S&P500 span reduces from 61 Years to 17 Years in 60 years In 2020, 100 million consumers will shop via augmented reality By 2020, 30% web browsing will be done without a screen by 2022 - $1 Trillion a year to be saved through IoT Source: Gartner, Inc. Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption, 14-Oct-2016 * - Google runs on ~2 Billion LOC Source: CA Workshop on Modern Software Factory Source: CB Insights * AR Market $143 Billion by 2020 - HW/SW/Apps/Consulting & SI
- 20. Is the TestingIndustry ready for testing the following innovations? 21
- 21. Tip of theIceberg seen in 2016 2016 A Year in Review – Software Failures 22Source: Tricentis Software Fails 2016 Report - https://www.tricentis.com/wp-content/uploads/2017/01/20161231SoftwareFails2016.pdf Over 4.4 Billion people got affected by a Software Fail (Up from 4.3 Billion in 2015) > 50% Global Population $1,062,106,142,949 - Assets Affected (Up from $4.2 Billion in 2015) 315 years, 6 months, 2 weeks, 6 days, 16 hours, & 26 minutes - Accumulated time- lost due to Bugs 2.66 Billion Mobile Phones impacted with Malware 12% Year on Year Increase in impactful Software Bugs British Airways lost $20 Billion (3%) in Market Cap within a few days after a failed software upgrade More Than 21 Million Automobile recalls as a result of Glitches / Bugs $5.7 Billion Impact in Failed Government Software Projects due to Bugs 2.2 Billion people live on less than $2 a day
- 22. One School ofThought on Testing – By Tricentis Source: TRICENTIS webinar on Future of Testing WhereAIcanhelp Legacy Firms Bi-model Firms Technology Leaders
- 23. Test Coverage GAP Years Months Weeks Days Hours Seconds TestingDuration Challenge of Complexity,Less time and more Tests Where we are in time Testing Complexity Time
- 24. 26 Resources Effort CostsCoverage #of Test Cases When youhave less time overall
- 25. Some Algorithms makingMachine Think! Source: https://futurism.com/predicting-2017-the-rise-of-synthetic-intelligence/ - Some of the artificial intelligence (AI) algorithms currently helping machines think. Credit: CIO Journal/Narrative Science
- 26. Approaches used forAI, Machine Learning and Deep Learning Reinforcement Learning • Passive Reinforcement Regression Algorithms • Linear Regression • Gaussian Process Supervised Learning • Neural Networks Unsupervised Learning • Independent Component Analysis • Principle Component Analysis Natural Language Understanding • Morphological, , semantic, syntactic , Discourse analysis Natural Language Generation • Deep planning • Syntactic generation Clustering Algorithms • K-Means Clustering • KPCA – Kernel Analysis Statistical Algorithms • Support Vector Machines • K-Nearest Neighbor • Native Bayes Classifier • Maximum Entropy Classifier Pattern Recognition • Statistical , Syntactic approach • Template Matching • Neural Networks Other Techniques • Spanning Trees and Graphs • Neural Network – Multi- Level Perceptron's Other Techniques • Labeling • Hidden Markov Model • Maximum Entropy MM Other Techniques • Conditional Random Fields • Parsing Algorithms
- 27. 29 Some methods tobuild AI Data Models for Testing 1 2 3 4 5
- 28. What are thefeasibilities with AI Driven Testing? 30 Automated Defect Detection Automated Exploratory Testing Test Coverage Heat map Self Healing Automation Predictive Modeling Self Adjusting Regression Pattern Recognition Risk & Coverage Optimization Diagnostic, Prescriptive and Predictive Analysis Deep Learning Root-Cause Analysis Sentiment Analysis
- 29. 31 AI Models Algorithms ApplicationUnder Test Designer Developer Business UserTesterBots / Agents AI Engine Testing Outcomes Test Cases Production Logs Requirements Defect Logs Source Code Traceability Matrix Root Cause Analysis Test Data Specifications Functional Logic Sample AI Model for Testing Historical & Real-time Data
- 30.
- 31. Example: Candy CrushSaga’s AI Strategy https://www.youtube.com/watch?v=wHlD99vDy0s • Use of AI engine for continuous Feedback Loop • Use of BOTS to perform Testing • Continuous Feedback Loop • Deep Artificial Neural Network • Use of Monte Carlo Tree Simulation • Use of Advanced Automation by BOTS • Hybrid Test team (150-200+ Testers) with unique skills • Use of Data Scientists for Domain Knowledge, Fun (using historic info and user behavior, Game Balancing) • Regular Crash Testing, Performance Testing, Regression Testing • Regular Upgrade of AI Bot for Testing v
- 32.
- 33.
Editor's Notes
- #6 Since John McCarthy invented AI in 1956 – Progressed by Marvin Minsky Etc.
- #8 Ray Kurzweil – Highlights that Singularity will happen by 2045
- #9 https://youtu.be/zatL4uFRpC0 Fast Learning – Download and Fly an helicopter – Can AI be so advanced that we can scan an image and learn faster, deeper and efficiently?
- #10 Let’s hear it from one of the leaders in the AI Space – AI or an Algorithm Writing Code – Watch out Developers and Testers!
- #11 Garry Kasparov Lee Seoul Ken Jennings and Randy Burr Google DeepMind's AlphaGo won 60–0 rounds on two public Go websites including 3 wins against world Go champion Ke Jie. AI induced Algorithms have been winning a tough game of Texas Hold’em poker where majority of the information is hidden – against world’s leading Poker Players as well.
- #12 AI is definitely not without controversies – It could potentially start the WW3 soon and a lot of countries are embarking on hegemony and superiority of AI – Just like the Cold-war era SPACE RACE that resulted in a lot of brilliant inventions, discoveries and humankind’s progress. But will the new AI war be different – Let’s wait and see..
- #13 AI is definitely not without controversies – It could potentially start the WW3 soon and a lot of countries are embarking on hegemony and superiority of AI – Just like the Cold-war era SPACE RACE that resulted in a lot of brilliant inventions, discoveries and humankind’s progress. But will the new AI war be different – Let’s wait and see..
- #14 Google is using machine learning and deep learning principles in a simple method. Let’s see a video and try it out!
- #15 Take it to next level – How can you easily integrate globally. Language will no longer be a barrier.
- #19 A perfectionist of sorts, Steve Jobs quoted - “Be a yardstick of quality. Some people aren't used to an environment where excellence is expected” Without a focus on quality, simplicity and efficiency, APPLE wouldn’t have become the most valuable company on earth, a brilliant turn around from a company that was almost dead before Jobs 2.0 began.
- #20 Source: CA Workshop on Modern Software Factory Source: Gartner, Inc. Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption, 14-Oct-2016 While the defects and bugs are making a dramatic impact, the world is leaping ahead. Business is expecting agility in business delivery... Take these for some stats Google – which supposedly has a single code repository, refactors code by upwards of 50% each month. They have ~2 Billion LOC (and counting). Even taking a 75-95% test coverage taken up by empowered teams (as claimed by some of the engineers in published artefacts), this is a humongous testing effort. If you have a backlog of code to be verified, it could be a disaster exceeding the size of a titanic by all means. When the Cyclomatic complexity of testing is so huge, how can you test the entire code base and application flawlessly? This is a brilliant example of how one can run an efficient test strategy. Take NetFlix that currently has over 5 Billion API Calls per day (up from Billion+ a few years ago). How would you do effective Load, Stress, Performance Testing and ensure Availability , Redundancy and Reliability of service is not impacted? A lot of firms are moving towards a bi-modal IT (doing a transformation while running the legacy apps running) and doing continuous delivery and Testing all the time, leveraging all the fancy words such as Agile, DevOps DevQAOps etc. etc. Additionally, nearly 41% of Global corporate workload is shifting to cloud, to ease out on Capital Expense and controlled Operational Expense strategies. By some means, Augmented Reality, Gestural Computing, IoT is expected to take the world by storm. How are we going to test all these permutations? AR Market $143 Billion by 2020 - HW/SW/Apps/Consulting & SI
- #21 Source: http://www.kurzweilai.net/single-molecule-level-data-storage-may-achieve-100-times-higher-data-density
- #22 If you take the Gartner’s Hype Cycle for Emerging Technologies for 2017 – You see a pattern. Some are in the slope of enlightenment but majority in the curve of inflated expectations and disillusionment. For the technologies emerging stronger, we need to have some solid test approach / strategy to deliver high quality outcomes Artificial Intelligence Internet of Things (or Everything) Machine Learning / Deep Learning AR/VR & Wearables Block Chain Drones & Vehicles Gestural Computing Connected Devices Human Augmentation Robotics Algorithms Smart Assistants Are we ready for these?
- #23 Carrying on Quality - Some statistics or a tip of the ice-berg Over 4.4 billion people got affected by a software fail – which is greater than 50% of Human Population. It is almost a number similar to the people not having access to a Toilet – But less than the number of mobile phones in use in the world!. More than a $Trillion worth of assets affected and a cumulative impact of 315 years. A leading airlines lost 3% market cap due t a botched software upgrade infested with bugs
- #25 Broadly speaking – New Age Test Innovation focuses on the following needs with Intelligent Testing Rapid High Quality and Innovative Test Delivery Test Suite Creation and Optimization (Risk Coverage ) Useful Automation – Test Smart, Self-Healing, Script less, Purposeful Predictive and Cognitive Testing – Foresee issues reduce reactive time, resolve rapidly Rapid Impactful Defect Finding - Intelligent Defect Detection, Pattern Analysis, Predictive Modeling Intelligent Environment Provisioning Management with Intelligent Metrics and Dashboard Are we capable of building intelligent automated frameworks and leverage cognitive models to optimize our test strategy and test suites to do proactive application health analytics via rapid defect finding and scale up rapidly to do niche and special areas of testing? That remains the key
- #26 https://www.linkedin.com/pulse/ai-software-testing-jason-arbon Explore user experience, by analyzing text from social media feeds (sentiment analysis) to spot feedback trends about what has already been released Cluster similar bugs together by data visualization heat mapping, for easier attack by Development (via the Pareto Approach, theorizing that bugs like to nest together) Reduce test cases that can be determined to be unnecessary before execution? Predict if specific follow-up DevOps sprints require specific tests cases to be run, vs. being omitted because there’s no chance that the problem got addressed yet. Reviewing specifications tell us what a program should do and how it should work. I.’s pattern matching helps us eliminate unneeded “too close’ test cases by seeing which ones are too similar. As we mentioned earlier, this may mean focusing on boundary value analysis (edge cases, literally), emphasizing state transition, or ensuring all-pairs testing. Exploration testing session logs, via pattern recognition of verbose logging, seek activity patterns of specific warnings tracking to specific user actions, modules, forms, etc. Known product issues, once analyzed, can have A.I. cluster similar bugs through pattern recognition, suggesting likely duplicates. Bugs from automated test cases can be auto-run on previous builds to find the causal build to help pinpoint root cause code changes. Discussions with knowledgeable personnel (product owners, developers or Marketing, etc.) may determine code danger areas. White box-driven test design targets the actual revised code, hunting for specific code level problems. Factors may include the coder, change date, functions referenced, or specific non-standard notations. A.I. pattern-matching techniques help pinpoint applicable code based on your search parameters. End user analysis applies to two different areas. The first is studying the frequency of specific user feedback words to help the most popular concerns bubble to the top of a list for further research. The second is end user usage analysis, where log file statistics (based on A.I. pattern searching) show how much time each type of user spends in different program areas on different actions. Early focus on these heat mapped areas concentrate attention where the most user time is spent.
- #31 Test suite optimization - Identifies duplicate/similar and unique test cases Predicting the next - To help predict the key parameters of software testing processes based on historical data. Log Analytics - Identifies hotspots and automatically execute test cases Traceability - Identifies complex scenarios from the requirements traceability matrix (RTM) and extract keywords to achieve test coverage Customer sentiment Analytics - Analyzes data from social media and provides an interactive visualization of feedback trends Defect analytics - Identifies high-risk areas in the application which helps in risk-based prioritization of regression test cases Its benefits include: Improved quality – Prediction, prevention, and automation using self-learning algorithms Faster time to market – Significant reduction in efforts with complete E2E test coverage Cognitively – Scientific approach for defect localization, aiding early feedback with unattended execution Traceability – Missing test coverage against requirement as well as, identifying dead test cases for changed or redundant requirement One integrated platform – Adaptable to client technology landscape, built on open source stack
- #36 Leave the exhaustive testing to AI. Leave tapping every button, inputting obvious valid and invalid data into text fields, etc. to the machines. Focus on the qualitative aspects of software testing that is specific to their specific app and customer. Focus on creative and business-specific test inputs and validations. Be more creative and think of email address values that a machine with access to thousands of possible email test inputs wouldn’t think to try. Verify that cultural- or domain-specific and expectations are met. Think of test cases that will break the machine processing for your specific app (e.g., negative prices, disconnecting the network at the worst possible time, or simulating possible errors). Record these human decisions in a way that later helps to train the bots. Schematized records of input and outputs are better than English text descriptions in paragraph form