The approaches and techniques that worked yesterday may not be optimum for the next generation of enterprise AI platforms. This session will cover how to prove Artificial Intelligence (AI) platforms by leveraging Cognitive, Reliability, and Chaos Engineering heuristics.

2. 2 | DevOps Next 2020 perforce.com
Confidentiality Statement
The information contained in this document is strictly confidential, privileged, and
only for the information of the intended recipient. The information contained in this
document may not be otherwise used, disclosed, copied, altered, or distributed
without the prior written consent of Perforce Software, Inc.

3. © 2020 Perforce Software, Inc.
Cognitive Engineering
Shifting Right (Digital Twin)
feat. GATED.AI Testing
J O N AT H O N W R I G H T

4. 4 | DevOps Next 2020 perforce.com
- COGNITIVE ENGINEERING

5. 5 | DevOps Next 2020 perforce.com
• Cognitive Engineering
• Digital Experiences (DX)
• Data visualization / stewardship
• Complex Event Processing
• Blockchain / Event Driven
• Context Brokering Platforms
• Natural language processing
• Situation Awareness
• People-Literate Technology
• Internet of Everything (IoE)
• Human Augmentation
• Ambient Experiences
• Gesture Control
• Brain Computer Interface (BCI)
• Emotion Detection
• Head-Mounted Displays
• Virtual Worlds
• Digital Offers
• City Data Exchange
• Atomic Level Mass Personalization
• Connected Vehicle
• Autonomous Vehicles (C2X)
• Enterprise Artificial Intelligence (AI)
• Ultra-Intelligence (Singularity)
• Artificial Intelligence Foundations
• Deep Reinforcement Learning
• Learning / Selfaware Software
• Neural Networks / Conversational
• Quantum / Fog / Edge Computing
• Cognitive Reckoning / Abstract Thinking
• Humanoid robots [Sophia]
• Vision / Sensors / Knowledge Replication
• Neuromorphic Hardware
• Shift Right / Digital Twins
• Cyber-physical systems
• Cryptocurrency Lawful hacking
• Cyber Threat / Zerotrust security
• Location-based authentication
• Growth / Performance hacking
• Flexible displays / Screenless interface
• Smart Dust / Workspace / Robots
• Nanotube Electronics
• Accumulated Reality
• Self-healing & aware systems
• GATED.AI
• Augmented Intelligence
• Chaos Engineering / Dark Canary
• Immersive / Human Augmentation
• Virtual / Augmented / Mixed Reality
• Intelligent / Enterprise of Things (EoT)
• Virtual Personal Assistants (VPA)
• Enterprise CrowdTesting
• Device Mesh / Sensorization
• Business / Digital transformation
• Tri-Modal / Shadow IT
• Interactive digital signage / Ambient UX
• Omni channel engagements
• Adaptive risk, trust & learning
• Microcontainerization [Unikernels]
• BI / Big Data [NoSQL]
• Wearables / Haptics / Gesture tech
• 4D / 3D printing
• Uberfication of services
• Object based storage
• Smart building technologies
• Gamification
COGNITIVE ENGINEERING + DIGITAL TWINS

6. 6 | DevOps Next 2020 perforce.com
WHAT IS COGNITIVE ENGINEERING?

7. 7 | DevOps Next 2020 perforce.com
COMPLEX EVENT PROCESSING (2015) – PREDICITVE CRIME

8. 8 | DevOps Next 2020 perforce.com
DIGITAL QUALITY - SMART CITY DATA EXCHANGE (2015)

9. 9 | DevOps Next 2020 perforce.com
CONTACT TRACING (2015) – LOCATION DATA (GPX)

10. 10 | DevOps Next 2020 perforce.com
CONTACT TRACING (2020) – GPS & GAENS

11. 11 | DevOps Next 2020 perforce.com
CONTACT TRACING (2020) – GPS & GAENS

12. 12 | DevOps Next 2020 perforce.com
The recent announcement of
Apple and Google’s work will
enable Path Check to
accelerate our efforts by
providing a critical piece of
the technology that would
have been harder to do
without them. Path Check
technology is open source. –
We are the integration point
for public health agencies at
all levels to bring the best
open, free, privacy-by-design
technologies together.
GOOGLE APPLE EXPOSURE NOTIFICATIONS (GAENs)

13. 13 | DevOps Next 2020 perforce.com
HOW DOES GAENS WORK?

14. 14 | DevOps Next 2020 perforce.com
• Exploratory Testing
• Functional Testing
(Manual /
Automated)
• Performance Testing
• Security Testing
• Localizations Testing
• UX Design
• Multi-Language
CALL TO ACTION – GAENS NEEDS YOU!

15. 15 | DevOps Next 2020 perforce.com
CALL TO ACTION – TESTERS ASSEMBLE!

16. 16 | DevOps Next 2020 perforce.com
CALL TO ACTION – TESTERS AROUND THE WORLD
https://leanpub.com/AroundTheWorldWith80SoftwareTesters/

17. 17 | DevOps Next 2020 perforce.com
CALL TO ACTION – HELP SAVE LIVES!

18. 18 | DevOps Next 2020 perforce.com
- SHIFTING WRIGHT

19. 19 | DevOps Next 2020 perforce.com
SHIFTING RIGHT, OVER SHIFTING LEFT

20. 20 | DevOps Next 2020 perforce.com
COGNITIVE LEARNING
Build
Test
TrainEvolve
Design
Model
Evolve
Learn
Think
Ideas
Beliefs,
Values,
Culture
Heuristics
Semantics
Behaviors
Bias
Ethics
Solutions
Agile
Lean
Methods
and Skills

21. 21 | DevOps Next 2020 perforce.com
COGNITIVE THINKING
Insight
Observe
CognitiveThinking
Research
Interpreting
Modelling
ChallengingPredicting
Observing
Insight
Experiment
Hypothesis
Research
CognitiveThinking
Experiment
Interpreting
Modelling
Lean Thinking (Definition) - is a business methodology that aims to provide a new way to think about how to organize human activities to
deliver more benefits to society and value to individuals while eliminating waste.

22. 22 | DevOps Next 2020 perforce.com
DESIGN
SHIFT LEFT SHIFT RIGHT
DEVELOP DELIVER DISRUPT
COGNITIVE ADAPTIVE TESTING
SHIFTING RIGHT THEN LEFT

23. 23 | DevOps Next 2020 perforce.com
COGNITIVE INTELLIGENCE

24. 24 | DevOps Next 2020 perforce.com
COGNITIVE TESTING
BUILD TEST

25. 25 | DevOps Next 2020 perforce.com
COGNITIVE LEARNING
• Dark (Launching) Canary (Rollout)
• Smart anomaly detection (“Hypercube baselining”)
• System Learning
• Failure Prediction
• Predictive analysis, feedback loops
• Relationships predicted
• Root cause analysis of failures
Discover
InnovateAdapt
Disrupt
Focus
Experience
Evolve
Divide
Behaviours

26. 26 | DevOps Next 2020 perforce.com
- GATED.AI METHODOLOGY

27. 27 | DevOps Next 2020 perforce.com
GATED.AI - METHODOLOGY
• GOAL – CAPABILITY to identify and correctly categorise images of products.
• ACCURACY – REQUIREMENT be able to successfully categorise women’s
fashion. (1,000+ subcategories on the Amazon channel) with a
CLARIFICATION RATE of over 70%.
• TIME – TIMEFRAME per day be able to process over 10,000 product images.
• ENRICHMENT – SEMANTIC MODELS applying Data Engineering (Enhance,
Transform and Load) heuristics for mining ecosystem to enable A.I. Test
Data Lake.
• DATA – TRAINING DATA mining for clustering (Percentage Split) i.e.
Development Training Set (60%), Testing Training Set (30%) and Proving
Training Set (10%) of the training set sizes (5/10 million).

28. 28 | DevOps Next 2020 perforce.com
GATED.AI - GOALS
• A.I. CAPABILITY to identify and correctly categorise images

29. 29 | DevOps Next 2020 perforce.com
GATED.AI - ACCURACY
• A.I. REQUIREMENT be able to successfully categorise fashion.
• A.I. CLARIFICATION RATE of over 70%
“The ROC curve (receiver operating characteristic
curve) is a graph-based approach for measuring
the performance of a classification process.
Then the total successful predictions made divided
by the total predictions made (True Positive Rate
(TPR) vs. False Positive Rate) helps us establish
the training AUC (Area Under the ROC
Curve) performance against each of the training
data sets”

30. 30 | DevOps Next 2020 perforce.com
GATED.AI - TIME

31. 31 | DevOps Next 2020 perforce.com
DSSTNE Semantic Maps (ETL (Crawl / Harvest)+ RPA (Vision & Meta)) AI-Supervised ML (TensorFlow (CV) + Kaggle (SWARM) + mTurk)
GATED.AI - ENRICHMENT
• A.I. SEMANTIC MODEL applying Data Engineering (Enhance,
Transform & Load) heuristics.
CATEGORY
MATCHING
SIMILARITY
INDEX
ENRICH
TRAINING
DATA
SET

32. 32 | DevOps Next 2020 perforce.com
Mining Test Training Data (CRAWLER / SPIDER / SUBSET)
GATED.AI – TESTDATA.AI
• A.I. TRAINING DATA mining for clustering, number of
parameters and associated test training data set size.
Harvested Test Training Data (MindCube.io - Domain-Driven)
HAVESTED
TRAINING
DATA
SET
MINING
TRAINING
DATA
SET

33. 33 | DevOps Next 2020 perforce.com
GATED.AI – SIMPLE TEST EXAMPLE
• GOAL - A.I. CAPABILITY - Image category
mapping
• ACCURACY - A.I. CLASSIFICATION RATE = > 70%
• TIME - A.I TIMEFRAME = < 1 day for 10,000
• ENRICHMENT - A.I SEMANTIC - Harvest
manufactures website and internal ERP
platform.
• DATA - A.I TEST TRAINING DATA –
Training data set size
(5M/10M), semantic
model parameters (500)
& training nodes (4/8).
A B
RESULTS
MODEL 2 = PASS

34. THEQALEAD.COM/PODCASTS