The document discusses Performance as Code (PAC). Key points include: - PAC aims to treat performance testing like code through tools like PerfDriver that allow defining performance tests as code. - The concept of Minimum Viable Performance (MVPx) seeks to establish basic performance testing capabilities. - Continuous Performance (CPx) and microcontainerization with tools like Kubernetes allow ongoing performance testing. - The Digital Performance Lifecycle (DPL) aims to systematize performance testing approaches. - Test Data as Code (TDaC) and Robotic Process Automation (RPA) aim to automate data creation and parameterization. - Lifecycle Virtualization aims to apply AI to further automate

1. Performance as Code (PAC)
Jonathon Wright

2. PerformanceEngineeringCommunity
https://www.linkedin.com/in/Automation/
https://github.com/PerfDriver
https://twitter.com/Jonathon_Wright
jlwright@iee.org
https://www.slideshare.net/Jonathon_Wright

3. Performance as Code

4. Performanceas Code (PAC)
• Background – World Digital Report 2018
• Performance as Code (PaC)
• Minimum Viable Performance (MVPx)
• Continuous Performance (CPx)
• High Volume Performance (HVPx)
- Microcontainerization (Unikernal + Kubernetes)
• Digital Performance Lifecycle (DPL)
• Test Data as Code (TDaC)
- Model transactions with powerful data creation engine (GDPR) and databases look-up
• Robotic Process Automation (RPA)
- Expose all the parameters, attributes and options as parameters (CLI)
• Lifecycle Virtualization

5. WORLD DIGITAL REPORT 2018

6. WORLD DIGITAL REPORT 2018
https://www.neotys.com/resources/world-digital-report-boxing-day-edition

7. WORLD DIGITAL REPORT 2018
RETAIL FOCUSED
200,000+
TOP RETAILERS
500
GLOBAL WEBSITES
1 Million
SITES MONITORED
1,000+
REGIONS
4
http:///www.WorldDigitalReport.com

8. WORLD DIGITAL REPORT 2018
Ready to use, DOM
interactive, visually and
render complete times.
Speed
Digital assessment methodology for assessment of current digital
solution with a focus for independently assessing opportunities for
improvement within performance engineering.
Digital Maturity Model index (DMMi)
Monitored pre / post
Boxing Day & January
Sales (48hrs)
Stability
Measured average
response times for both
Desktop (WIFI) and Mobile
(4G / 3G / 2G).
Response
Performance index report
covering, W3C, R/R, DOM,
CDN and DNS
benchmarking.
Index
30
INDEX
RESPONS E
STABILIT Y
SPEED
DIGITAL
MATURITY
MODEL
INDEX
(DMMi)

9. WORLD DIGITAL REPORT 2018
BENCHMARK
Internet rankings came from Alexa.com and is a
indication of global internet traffic to the site, this years
weightings to derive the overall score was performance
index (30%), response (15%), stability (20%) and speed
(15%) which was real end user experience.
MYLOADTEST.COM
Monitored pre / post
Christmas & Boxing Day
(48hrs).
(result A indicates between
-1% & +1% change of
response over time)
STABILITY
Measured average response
times for both Desktop (WIFI)
and Mobile (4G / 3G / 2G).
(result of less than 5 (Desktop)
/ 10 (Mobile) seconds
response time people tend to
typically navigate away)
RESPONSE
Performance index report
covering, W3C, R/R, DOM,
CDN and DNS
benchmarking.
(result of less than the
average of 70 really needs
help with optimization)
INDEX
Company Twitter Index Traffic
Response
Stability Issues Overall
Desktop Mobile Spike
1 Dominos.co.uk @Dominos_UK 98 8,113 0.54s 1.81s 5.81s A N/A A

10. Company Twitter Index Traffic
Response
Stability Issues Overall
Desktop Mobile Spike
81 watchshop.com @watchshop 37 16,473 7.0 14.2 14.5 C+ D+ E
82 harrods.com @harrods 35 13,256 7.0 12.4 22.3 A F E
83 maplin.co.uk @maplintweet 35 15,362 4.5 24.0 42.6 E- E E
84 whsmith.co.uk @WHSmith 35 49,846 13.1 15.5 35.2 F D- E
85 topman.com @topman 34 11,733 5.2 10.6 11.3 A E E-
86 jessops.com @jessops 33 61,237 7.2 30.6 38.9 F D- E-
87 dorothyperkins.com @dorothy_perkins 33 27,140 5.3 10.6 19.9 A- D E-
88 dfs.co.uk @DFS 33 61,650 9.3 16.6 20.0 C- D- E-
89 waterstones.com @waterstones 31 26,529 10.2 20.4 24.4 A D- E-
90 lipsy.co.uk @LipsyLondon 30 12,133 9.8 14.8 30.0 A E F
WORLD DIGITAL REPORT 2018

11. Performanceas Code (PAC)
• PerfDriver.io (https://github.com/PerfDriver)
execution:
- scenario: perfdriver
scenarios:
perfdriver:
script: VIP.pac

12. Performanceas Code (PAC)
<?xml version="1.0" encoding="UTF-8"?>
<pacTestPlan version="1.2" properties="2.3">
<hashTree>
<HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="/" enabled="true">
<boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
<elementProp name="HTTPsampler.Arguments" elementType="Arguments">
<collectionProp name="Arguments.arguments">
<elementProp name="" elementType="HTTPArgument">
<boolProp name="HTTPArgument.always_encode">false</boolProp>
<stringProp name="Argument.value"></stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
</collectionProp>
</elementProp>
<stringProp name="HTTPSampler.domain">curiositysoftware.ie</stringProp>
<stringProp name="HTTPSampler.port"></stringProp>
<stringProp name="HTTPSampler.connect_timeout"></stringProp>
<stringProp name="HTTPSampler.response_timeout"></stringProp>
<stringProp name="HTTPSampler.protocol">http</stringProp>
<stringProp name="HTTPSampler.contentEncoding"></stringProp>
<stringProp name="HTTPSampler.path">/</stringProp>
<stringProp name="HTTPSampler.method">GET</stringProp>
<boolProp name="HTTPSampler.follow_redirects">true</boolProp>
<boolProp name="HTTPSampler.auto_redirects">false</boolProp>
<boolProp name="HTTPSampler.use_keepalive">true</boolProp>
<boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
<boolProp name="HTTPSampler.monitor">false</boolProp>
<stringProp name="HTTPSampler.embedded_url_re"></stringProp>
</HTTPSamplerProxy>
<hashTree />
https://github.com/PerfDriver/PerfDriver/tree/master/samples

13. Minimum ViablePerformance(MVPx)– Selenium.io
sudo apt-get update sudo apt-get install
• python
• default-jre-headless
• openjdk-8-jre-headless
• python-tk
• python-pip
• python-dev
• zlib1g-dev
• net-tools
• selenium
• Appium-Python-Client
• brew/apt§/yum
• npm/node/js_tests/mocha
• bzt

14. Minimum ViablePerformance(MVPx)- NeoLoad
• uname –a (check build x86_64)
• neoload_6_4_0_linux_x86.sh

15. Minimum ViablePerformance(MVPx)– PerfDriver.io
Load generator agent
• Geo-realistic location
• Network Virtualization
• High volume performance
(HVPx) + mobile emulation
• Low CPU / memory usage
(1.4GHz Quad-Core / 1GB DDR2)
• Network Function Virtualization
(Gigabit Ethernet / 802.11ac)
• Low Cost @ £32 GBP per agent

16. Minimum ViablePerformance - Microcontainerization
• docker
https://hub.docker.com/r/neotys/neoload-loadgenerator
• unikernel
https://hub.docker.com/r/unikernel/mirage
• windows 2019
http://aka.ms/ws2019previewbuildsreleasenotes
• openshift for red hat
https://www.openshift.com
• kubernetes
https://kubernetes.io
• suse CaaS

17. Continuous Performance - Microcontainerization- Kubernetes
• https://portal.azure.com
• devops project (preview)
• docker
docker pull neotys/neoload-loadgenerator
docker pull perfdriver/perfdriver.io

18. Continuous Performance - Microcontainerization- HVPx
• High vCPU
(16 cores)
• High Memory
(432GB)
• Nodes
(5,000)
• vUsers
(2.5million)
• £3.70 per hr

19. Continuous Performance - Microcontainerization- HVPx

20. Continuous Performance - Microcontainerization- HVPx

21. DigitalPerformanceLifecycle – Testinsight.io
systemised ordered cognitive thinking represented in an algorithmic approach
INSIGHT-DRIVEN – HOLISTIC MODELLING
http://leanpub.com/digital

22. DigitalPerformanceLifecycle – Quantum Téléportation
https://www.dynatrace.com/
platform/artificial-
intelligencehttps://www.contrastsecurity.com/devops

23. DigitalPerformanceLifecycle – Shift Right (PAL)
• Activating event
• Encapsulate interactions
• Cause & effect modelling
• Learn behaviours
• Automate workflows
• Adaptive difficulty
• Generate schematics
• GAP Analysis
• Process validation
• Behavioural experiment
• Visualize blueprint
• Self monitoring

24. DigitalPerformanceLifecycle – GenerateDigitalBlueprint
• Generate Business Process Model (BPMn v2.2)
• Transactions (BPTx)
• Swimlanes (ecosystems)
• Generate Digital Blueprint (xPDL)

25. DigitalPerformanceLifecycle – Shift Right (PAL)
• WebDriver (Headless) • PerfDriver (Headless)

26. DigitalPerformanceLifecycle – ReverseEngineering
• Top Layer
- End Users (Prod) / User Acceptance (UAT)
- Manu’mation Tests via Browser / App
- Automated Tests via Selenium (DOM)
- Automated Tests via ArtOfTesting (.NET)
• Middle Layer
- Manu’mation Tests via Postman (API)
- Automated Tests via Selenium (Headless)
- Automated Tests via RESTSharp (Swagger)
- Performance Tests via PerfDriver.io (R/R)
- Load Tests via NeoLoad Agent (Raspberry)
• Bottom Layer
- IntelliTest Generated (VS2017)
- Code Analyzers (SonarCube)
- Unit Testing Frameworks (xUnit)
fiddler2.com/r/?reverseproxy
IIS or
Apache
(Port 81)
WinINET
System
.NET
.NET APP
Internet
Explorer
Selenium
(Headless)
Firefox
RESTSharp
WinHTTP
Proxy
Fiddler
(Port 80)

27. RoboticProcess Automation – CLI for TestingPerformance
Sentiment Analysis

28. RoboticProcess Automation – MicrosoftCognitiveServices
Trigger examples
• Conversational AI (ChatOps)
• Recurrence MS Flow control
Action examples
• Computer vision (OpenCV)
• CustomVision.AI (Microsoft)
https://southcentralus.api.cognitive.microsoft.com/customvision/v2.0/Prediction/b5cece8c-ea39-4b7d-bfe5-8ff4d3852b58/image?iterationId=bd8d8b99-559b-4c48-8372-3f3cb92f50f1

29. RoboticProcess Automation – Test Dataas Code

30. RoboticProcess Automation - jidoka.io
Robot Type
Business Value
Driver
Robot
Activities
Robot Interactions Robot Characteristics Robot Technology
Basic
Operational
Efficiency
• Do
• Robot to
Applications
• Structured Data Processing
• Semi-structured Data Processing
• Rule-based RPA
• Unattended RPA
• UI, API
Smart
Operational
Efficiency
• Do
• Think
• Robot to
Applications
• Robot to Ops
Employees
• Structured Data Processing
• Semi-structured Data Processing
• Unstructured Data Processing
• Advanced Exception Handling
• Complex Unattended RPA
• UI, API, OCR/Computer Vision
• Event based Pause/Resume
• Email/Text Alert Triggers
Collaborative
Speed of
Business
Transaction
• Do
• Think
• Manage
• Robot to
Applications
• Robot to Ops
Employees
• Robot to Robot
• Structured Data Processing
• Semi-structured Data Processing
• Unstructured Data Processing
• Interactive Exception Handling
• Executing other robots
• Unattended RPA
• UI, API, OCR/Computer Vision
• Event based Pause/Resume
• Email/Text Alert Triggers
• NLP, Virtual AI Assistant
• API Triggered Robot Execution
• IoT device triggered Robot
Management
Autonomous
Digital
Experience
(DX)
• Do
• Think
• Manage
• Learn
• Robot to
Applications
• Robot to Ops
Employees
• Robot to Robot
• Robot to
Customer
• Structured Data Processing
• Semi-structured Data Processing
• Interactive Exception Handling
• Unstructured Data Processing
• Managing other robots
• Decision Making
• Unattended RPA
• UI, API, OCR/Computer Vision
• Event based Pause/Resume
• Email/Text Alert Triggers
• NLP, Virtual AI Assistant, Chatbot
• API Triggered Robot Execution
• Machine Learning, Data Analytics

31. Lifecycle Virtualization(AI Ready) – Dynamic MVC (R/R)
var mdarray = [];
for (i = 0; i < excounter; i++)
{
mdarray.push(context.variableManager.getValue("MaterialArray_"+ i));
}
for (loop = 1; loop < excounter; loop++) {
response += "<column><id>" + mdarray[loop] + "</id></column>";
}
context.variableManager.setValue("MasterData", response);
Controller
Request
Model
Builds
View
RendersResponse

32. Lifecycle Virtualization(AI Ready) – (SV+ NV + NFV)
BRINGS TRUE ARTIFICIAL INTELLIGENCE
(AI) TO LIFECYCLE VIRTUALIZATION
▪ Virtualize services without requiring any
knowledge or decoding of the service protocols
▪ Applies a genome sequence alignment algorithm,
discovers byte-level patterns in message protocols
▪ Now virtualize a much wider range of protocols
without requiring a new DPH
HIGH ACCURACY: 99.6 – 100%
▪ Increased speed and accuracy with Entropy
Weighting + Message Clustering
▪ The more data a service observes the data, more
intelligent it becomes
▪ Perfect for performance testing where we deal
with tons of data

33. DigitalAssurance – CognitiveAdoption Platforms
• Cognitive Learning Services (CLS)
• Virtual Personal Assistants (VPA)
• Smart Advisors
• Natural language processing
• Situation Awareness
• People-Literate Technology
• Digital Experiences (DX)
• Internet of Everything (IoE)
• Human Augmentation
• Ambient Experiences
• Gesture Control
• Brain Computer Interface (BCI)
• Emotion Detection
• Head-Mounted Displays
• Virtual Worlds
• Context Brokering Platforms
• Digital Offers
• City Data Exchange
• Complex Event Processing
• Atomic Level Mass Personalization
• Connected Vehicle
• Autonomous Vehicles (C2X)
• Vehicles to Infrastructure (I2X)
• Mood Recognition
• Cognitive Adoption - The AI Imperative
• Ultra-Intelligence (Singularity)
• Artificial Intelligence Foundations
• Deep Reinforcement Learning
• Learning / Selfaware Software
• Neural Networks / Conversational
• Quantum / Fog / Edge Computing
• Cognitive Reckoning / Abstract Thinking
• Humanoid robots
• Vision / Sensors / Knowledge Replication
• Neuromorphic Hardware
• Digital Twins / Hyperconverged
• Cyber-physical systems
• Cryptocurrency Lawful hacking
• Cyber Threat Prediction / Zerotrust security
• Location-based authentication
• Growth / Performance hacking
• Flexible displays / Screenless interface
• Smart Dust / Workspace / Robots
• Nanotube Electronics
• Accumulated Reality
• Self-healing & aware systems
• Digital Out-of-Home (DOOH)
• Additive manufacturing
• Cognitive Adoption Platforms (GAP)
• Virtual / Augmented / Mixed Reality
• Immersive / Human Augmentation
• Intelligent / Enterprise of Things (EoT)
• Blockchain / Event Driven
• Mobilegeddon / Intelligent Apps
• Device Mesh / Sensorization
• Data visualization / stewardship
• Business / Digital transformation
• Tri-Modal / Shadow IT
• Interactive digital signage / Ambient UX
• Omni channel engagements
• Adaptive risk, trust & learning
• Microcontainerization [Unikernels]
• BI / Big Data [NoSQL]
• Wearables / Haptics / Gesture tech
• 4D / 3D printing
• Uberfication of services
• Object based storage
• Smart building technologies
• Cloud aggregator / broker
• Chaos Monkey
• Gamification
• Commercial drones (UAVs)

34. DigitalEvolution- CognitiveLearning

35. DigitalAssured – Evolution,over Revolution