The document discusses using AI and OpenAI APIs to assist with software testing. It provides an overview of AI types like narrow, general and super intelligence. It also discusses generative AI and common tools for text, image, video and code generation. The document demonstrates how to use OpenAI APIs with Postman, including exploring ideas for API fuzzing and test result analysis. It shares the current state of AI in testing tools and how AI could help with other testing activities beyond automation.

1. Thanks
Ministry of Testing
Poppulo
1

2. @MinistryCork
2

3. Upcoming Events
@MinistryCork
TestBash UK, Liverpool Wednesday, the 20th of September 2023, for two days.
3

4. AI assisted testing using
postman and OpenAI
Sivaganesh Sivakumar
16-May-2023
Ministry of Testing Cork
@sivaganesh_siva 4

5. About me Sivaganesh Sivakumar
Senior software test engineer
India (originally), currently in Cork
Experience: 13+ years in multiple industries
Passionate about technology
Enjoys food and movies
First public speaking engagement
@sivaganesh_siva 5
https://www.postman.com/aviation-meteorologist-80909569

6. Overview
By the end of the session, we will be able to
• Understand the current state of AI and AI in software testing.
• Learn the basics of OpenAI APIs and their potential in testing.
• Develop a simple test use-case using postman and OpenAI APIs.
• Discover interesting AI-assisted testing implementations.
@sivaganesh_siva 6

7. AI – Types
Artificial Intelligence (AI), a term coined by Stanford Professor John
McCarthy in 1955, was defined by him as “the science and engineering of
making intelligent machines”
Artificial Narrow
Intelligence
Artificial General
Intelligence
Artificial Super
Intelligence
Based on capabilities Based on functionalities
Reactive
machine
Theory of mind Self-aware
Limited
memory
@sivaganesh_siva 7

8. Artificial Narrow
Intelligence
• Focusses on one narrow task
• Targets a single subset of cognitive
abilities and advances in that spectrum
@sivaganesh_siva 8

9. Artificial General
Intelligence
Autonomous system
that surpasses human
capabilities in the
majority of
economically valuable
tasks
@sivaganesh_siva 9

10. Artificial Super Intelligence
Skynet begins to learn at a geometric rate.
An intellect that is much
smarter than the best human
brains in practically every field,
including scientific creativity,
general wisdom and social
skills
https://nickbostrom.com/superintelligence
@sivaganesh_siva 10

11. Generative AI
https://github.blog/2023-04-07-what-developers-need-to-know-about-generative-ai/
Category of AI models and tools designed to create new content,
such as text, images, videos, music, or code.
@sivaganesh_siva 11
Text Generation -
ChatGPT
Image Generation –
DALL-E2,
Midjourney,
Microsoft Designer
Video Generation –
Stable Diffusion
Programming Code
Generation –
OpenAI Codex,
PaLM 2
Data Generation -
MostlyAI
Language translation
- Google Neural
Machine Translation,
ChatGPT, PaLM 2

12. Generative AI - implementations
• AlphaCode
• Google DeepMind
• Competitive programming
• Amazon Codewhisperer
• GitHub Copilot X
• openAI Codex
• ChatGPT
• GPT4
• GPT3.5
• Tabnine
Programming / Coding
@sivaganesh_siva 12

13. Generative AI - implementations
Issue Tracking / Communication / Observability
Atlassian Intelligence
New Relic
SlackGPT
@sivaganesh_siva 13

14. State of AI in software testing - Tools
Unit tests
Contract tests
Integration/API
tests
UI
• Majority of the AI assisted testing tools focus on UI/Visual validation and self healing features
• All the AI features are built within the tools and need subscriptions
@sivaganesh_siva 14

15. Where else can AI help us ?
We spend a lot of time outside of the core testing activities like test
automation, execution, test planning etc.
Automation test failure/report analysis
Bug prioritization
Monitoring and Observability
Code reviews
Complex test data generation
Performance report analysis
Documentation
@sivaganesh_siva
15

16. Where to get started ?
OpenAI APIs
• Pay-as-you-go pricing of the APIs
• Ability to choose between multiple frameworks
• python
• node
• curl
• postman etc.
• Freedom to explore with smaller experiments
• Cons include complexity with prompts and don’t have any
reference materials
@sivaganesh_siva 16

17. OpenAI APIs - models
Offers multiple models which can be used for content generation,
image generation, speech recognition etc
GPT-4 : Improved natural language and code generation
GPT-3.5: Advanced natural language and code generation
DALL·E: Image generation and editing using natural language prompts
Whisper: Audio-to-text conversion
Embeddings: Text-to-numerical conversion
Moderation: Fine-tuned model to detect sensitive/unsafe text
GPT-3: Natural language understanding and generation
Codex: Natural language to code translation
@sivaganesh_siva 17

18. OpenAI APIs – Usage fee
5$ Free API usage for the first month
@sivaganesh_siva 18

19. openAI APIs – Completions
• Fundamental API which is extremely flexible and powerful
• Given a prompt it returns a text completion as per the provided instructions
POST https://api.openai.com/v1/completions
POST https://api.openai.com/v1/chat/completions
Supports multiple models.
text-davinci-003/gpt-3.5-
turbo
Max tokens to limit the
response text.
Values between 0 to 2. 0
being more focused and
deterministic and 2 being
very random and creative
@sivaganesh_siva 19

20. Leveraging openAI using postman
1) Generate openAI API key
2) Fork openAI API collection in postman
3) Update the API key from step 1 in
postman
4) Start working with the APIs
Key steps to get started
@sivaganesh_siva 20

21. Leveraging openAI using postman
LIVE DEMO
@sivaganesh_siva 21

22. Ideas to explore further
API Fuzzing with OpenAI APIs and Traditional Fuzzing Engines
Traditional fuzzing engines:
- Examples: AFL, libFuzzer, honggfuzz
- Generate test inputs using random mutations and code coverage monitoring
- Effective for discovering vulnerabilities in software
AI-driven fuzzing with OpenAI APIs:
- Leverages natural language understanding and reasoning capabilities
- Generates context-aware and complex test cases
- Can target specific functionality and edge cases
Benefits of a hybrid approach:
- Combines the strengths of both traditional fuzzing engines and AI-based methods
- Increases efficiency and likelihood of discovering vulnerabilities
@sivaganesh_siva 22

23. Ideas to explore further
Leveraging OpenAI APIs for Test Automation result analysis
Objectives
- Summarize test automation reports
- Analyse AWS CloudWatch logs to identify issues
Benefits
- Accelerate issue identification and resolution
- Enhance team productivity through automated test result analysis
@sivaganesh_siva 23

24. Interesting implementations
@sivaganesh_siva 24

25. Interesting implementations
@sivaganesh_siva 25

26. Questions ?
@sivaganesh_siva 26