SlideShare a Scribd company logo
*in 8 minutes
Assurances in Software Testing
presented by Marcel Böhme
Assurances in Software Testing
• 2017: Preparing for a meeting with a company 

• Provides security assessment services

• Over 30 customers in industry and governments

• Over 20 technology partners

• Over 10 standards/compliance partners

• Main product: a blackbox protocol fuzzer

• Used for security cert. of medical devices (IEC 62443-4-2)

• Fuzzer’s task: discover vulns, exploited over the network

• Other tasks: Secure dev. process, impl. security policies
We don’t have them
Assurances in Software Testing
presented by Marcel Böhme
Assurances in Software Testing
We don’t have them
Assurances in Software Testing
presented by Marcel Böhme
Assurances in Software Testing
• Rigorous cyber security assessment

• Powerful mitigator of cyber risks

• Inspires trust
• Violation of this trust would be disastrous

• Reputation of the company and the certification authority,

as well as the credibility of the certificate depend on

• assurances which the fuzzer provides.
We don’t have them
Assurances in Software Testing
presented by Marcel Böhme
• Which assurances are derived for the certificate

from applying the company’s fuzzer?
Assurances in Software Testing
We don’t have them
Assurances in Software Testing
presented by Marcel Böhme
• Which assurances are derived for the certificate

from applying the company’s fuzzer?

• How to assess residual risk of a fuzzing campaign

that finds no vulnerabilities?
Assurances in Software Testing
We don’t have them
Assurances in Software Testing
presented by Marcel Böhme
• Which assurances are derived for the certificate

from applying the company’s fuzzer?

• How to assess residual risk of a fuzzing campaign

that finds no vulnerabilities?

• When to stop the fuzzer and proceed with certification?

• Not specified in the certification procedure.

• How to specify allowable residual risk?
Assurances in Software Testing
We don’t have them
Assurances in Software Testing
presented by Marcel Böhme
• Which assurances are derived for the certificate

from applying the company’s fuzzer?

• How to assess residual risk of a fuzzing campaign

that finds no vulnerabilities?

• When to stop the fuzzer and proceed with certification?

• Not specified in the certification procedure.

• How to specify allowable residual risk?
Assurances in Software Testing
We don’t have them
Decision is based 

on experience.
Assurances in Software Testing
presented by Marcel Böhme
• Security researcher should be able to

• systematically assess and quantify 

• uncertainty

• residual risk

• cost-benefit trade-off

• Certification authority should be able to

• Provide concrete guidance (i.e., threshold values)

• Specify allowable residual risk
Assurances in Software Testing
But we need them
Assurances in Software Testing
presented by Marcel Böhme
Assurances in Software Testing
presented by Marcel Böhme
We found a preliminary 

statistical framework
borrowed from ecology.
Assurances in Software Testing
presented by Marcel Böhme
https://fuzzingbook.org
Don’t want to read the 52 page TOSEM article?

More the hands-on kind of researcher?
Interactive book chapter: “When to stop fuzzing”
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• In software testing, we are often most interested in program
behaviours that are both extreme and rarely observable.
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• In software testing, we are often most interested in program
behaviours that are both extreme and rarely observable.

• Hot topics in applied statistics:

• Rare event analysis 

• Extreme value theory

• Black swan theory

• Good-Turing theory

• Adaptive sampling strategies to estimate probabilities 

of extremely rare (or rather extreme) events.
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• In software testing, we are often most interested in program
behaviours that are both extreme and rarely observable.

• Hot topics in applied statistics:

• Rare event analysis 

• Extreme value theory

• Black swan theory

• Good-Turing theory

• Adaptive sampling strategies to estimate probabilities 

of extremely rare (or rather extreme) events.
We should study 

these techniques!
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• Oracle problem: Machine can’t tell correct from incorrect.
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• Oracle problem: Machine can’t tell correct from incorrect.

• Uncertainty problem: Human can’t tell correct from incorrect.

• Spotify recommends next song from “Justin Timerlake”

• Goodreads suggests next book Sartre’s “Being and Nothingness”

• GPS is 5 meter off. (Still more correct than one that is 10m off)
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• Oracle problem: Machine can’t tell correct from incorrect.

• Uncertainty problem: Human can’t tell correct from incorrect.

• Spotify recommends next song from “Justin Timerlake”

• Goodreads suggests next book Sartre’s “Being and Nothingness”

• GPS is 5 meter off. (Still more correct than one that is 10m off)

• Statistics is well-suited to handle this uncertainty

• Example from ecology: Do 2 images of bumblebees show the same
species? Even taxonomic experts were incorrect 60% of time.
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• What if not all inputs can be generated?

• An Android fuzzer that

cannot generate system-level events (e.g., change battery level)

cannot exercise the corresponding program / app behaviour.
(Restricted search space)
Assurances in Software Testing
presented by Marcel Böhme
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• What if not all inputs can be generated?

• An Android fuzzer that

cannot generate system-level events (e.g., change battery level)

cannot exercise the corresponding program / app behaviour.
(Restricted search space)
• We need to develop statistical methodologies that can

• extrapolate from restricted to entire input space.

• integrate estimates from several fuzzers with

overlapping restricted search spaces.
Assurances in Software Testing
presented by Marcel Böhme
(Restricted search space)
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• What if not all inputs can be generated?

• What about Search-Based Software Testing?

• The fuzzer “gets better” at finding bugs during fuzzing

• Breaks our assumption that generated executions are IID.

• It is easy to show that estimation bias reduces over time.
(Adaptive bias)
Assurances in Software Testing
presented by Marcel Böhme
(Restricted search space)
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• What if not all inputs can be generated?

• What about Search-Based Software Testing?

• The fuzzer “gets better” at finding bugs during fuzzing

• Breaks our assumption that generated executions are IID.

• It is easy to show that estimation bias reduces over time.

• How to correct this adaptive bias?

• a-priori (statistical theory) or a-posteriori (engineering).
(Adaptive bias)
Assurances in Software Testing
presented by Marcel Böhme
(Restricted search space)
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• What if not all inputs can be generated?

• What about Search-Based Software Testing? (Adaptive bias)
Assurances in Software Testing
presented by Marcel BöhmeAssurances in Software Testing
presented by Marcel Böhme
https://fuzzingbook.org
Interactive book chapter: “When to stop fuzzing”
Conclusion
Assurances in Software Testing
presented by Marcel Böhme
We found a preliminary 

statistical framework
borrowed from ecology.
Assurances in Software Testing
presented by Marcel Böhme
Assurances in Software Testing
We don’t have them
Assurances in Software Testing
presented by Marcel Böhme
(Restricted search space)
Challenges and Opportunities
• What about extremely rare & rather extreme behaviours?

• What if we are uncertain whether a behaviour is correct?

• What if not all inputs can be generated?

• What about Search-Based Software Testing? (Adaptive bias)

More Related Content

Recently uploaded

Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
ARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdf
ARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdfARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdf
ARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdf
Inglês no Mundo Digital
 
Comparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdfComparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdf
Andrey Yasko
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
Edge AI and Vision Alliance
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Kunal Gupta
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Mydbops
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
ChristopherTHyatt
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Applying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAI
Applying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAIApplying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAI
Applying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAI
ssuserd4e0d2
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Bert Blevins
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
LINUS PROJECTS (INDIA)
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
Emerging Tech
 

Recently uploaded (20)

Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
ARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdf
ARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdfARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdf
ARTIFICIAL INTELLIGENCE (AI) IN MUSIC.pdf
 
Comparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdfComparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdf
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Applying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAI
Applying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAIApplying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAI
Applying Retrieval-Augmented Generation (RAG) to Combat Hallucinations in GenAI
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
 

Featured

2024 Trend Updates: What Really Works In SEO & Content Marketing
2024 Trend Updates: What Really Works In SEO & Content Marketing2024 Trend Updates: What Really Works In SEO & Content Marketing
2024 Trend Updates: What Really Works In SEO & Content Marketing
Search Engine Journal
 
Storytelling For The Web: Integrate Storytelling in your Design Process
Storytelling For The Web: Integrate Storytelling in your Design ProcessStorytelling For The Web: Integrate Storytelling in your Design Process
Storytelling For The Web: Integrate Storytelling in your Design Process
Chiara Aliotta
 
Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...
Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...
Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...
OECD Directorate for Financial and Enterprise Affairs
 
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
SocialHRCamp
 
2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
Marius Sescu
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
Expeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
Pixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
marketingartwork
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
Skeleton Technologies
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
SpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Lily Ray
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
Rajiv Jayarajah, MAppComm, ACC
 

Featured (20)

2024 Trend Updates: What Really Works In SEO & Content Marketing
2024 Trend Updates: What Really Works In SEO & Content Marketing2024 Trend Updates: What Really Works In SEO & Content Marketing
2024 Trend Updates: What Really Works In SEO & Content Marketing
 
Storytelling For The Web: Integrate Storytelling in your Design Process
Storytelling For The Web: Integrate Storytelling in your Design ProcessStorytelling For The Web: Integrate Storytelling in your Design Process
Storytelling For The Web: Integrate Storytelling in your Design Process
 
Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...
Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...
Artificial Intelligence, Data and Competition – SCHREPEL – June 2024 OECD dis...
 
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
How to Leverage AI to Boost Employee Wellness - Lydia Di Francesco - SocialHR...
 
2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 

Assurances in Software Testing

  • 2. Assurances in Software Testing presented by Marcel Böhme Assurances in Software Testing • 2017: Preparing for a meeting with a company • Provides security assessment services • Over 30 customers in industry and governments • Over 20 technology partners • Over 10 standards/compliance partners • Main product: a blackbox protocol fuzzer • Used for security cert. of medical devices (IEC 62443-4-2) • Fuzzer’s task: discover vulns, exploited over the network • Other tasks: Secure dev. process, impl. security policies We don’t have them
  • 3. Assurances in Software Testing presented by Marcel Böhme Assurances in Software Testing We don’t have them
  • 4. Assurances in Software Testing presented by Marcel Böhme Assurances in Software Testing • Rigorous cyber security assessment • Powerful mitigator of cyber risks • Inspires trust • Violation of this trust would be disastrous • Reputation of the company and the certification authority,
 as well as the credibility of the certificate depend on • assurances which the fuzzer provides. We don’t have them
  • 5. Assurances in Software Testing presented by Marcel Böhme • Which assurances are derived for the certificate
 from applying the company’s fuzzer? Assurances in Software Testing We don’t have them
  • 6. Assurances in Software Testing presented by Marcel Böhme • Which assurances are derived for the certificate
 from applying the company’s fuzzer? • How to assess residual risk of a fuzzing campaign
 that finds no vulnerabilities? Assurances in Software Testing We don’t have them
  • 7. Assurances in Software Testing presented by Marcel Böhme • Which assurances are derived for the certificate
 from applying the company’s fuzzer? • How to assess residual risk of a fuzzing campaign
 that finds no vulnerabilities? • When to stop the fuzzer and proceed with certification? • Not specified in the certification procedure. • How to specify allowable residual risk? Assurances in Software Testing We don’t have them
  • 8. Assurances in Software Testing presented by Marcel Böhme • Which assurances are derived for the certificate
 from applying the company’s fuzzer? • How to assess residual risk of a fuzzing campaign
 that finds no vulnerabilities? • When to stop the fuzzer and proceed with certification? • Not specified in the certification procedure. • How to specify allowable residual risk? Assurances in Software Testing We don’t have them Decision is based 
 on experience.
  • 9. Assurances in Software Testing presented by Marcel Böhme • Security researcher should be able to • systematically assess and quantify • uncertainty • residual risk • cost-benefit trade-off • Certification authority should be able to • Provide concrete guidance (i.e., threshold values) • Specify allowable residual risk Assurances in Software Testing But we need them
  • 10. Assurances in Software Testing presented by Marcel Böhme
  • 11. Assurances in Software Testing presented by Marcel Böhme We found a preliminary 
 statistical framework borrowed from ecology.
  • 12. Assurances in Software Testing presented by Marcel Böhme https://fuzzingbook.org Don’t want to read the 52 page TOSEM article?
 More the hands-on kind of researcher? Interactive book chapter: “When to stop fuzzing”
  • 13. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • In software testing, we are often most interested in program behaviours that are both extreme and rarely observable.
  • 14. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • In software testing, we are often most interested in program behaviours that are both extreme and rarely observable. • Hot topics in applied statistics: • Rare event analysis • Extreme value theory • Black swan theory • Good-Turing theory • Adaptive sampling strategies to estimate probabilities 
 of extremely rare (or rather extreme) events.
  • 15. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • In software testing, we are often most interested in program behaviours that are both extreme and rarely observable. • Hot topics in applied statistics: • Rare event analysis • Extreme value theory • Black swan theory • Good-Turing theory • Adaptive sampling strategies to estimate probabilities 
 of extremely rare (or rather extreme) events. We should study 
 these techniques!
  • 16. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • Oracle problem: Machine can’t tell correct from incorrect.
  • 17. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • Oracle problem: Machine can’t tell correct from incorrect. • Uncertainty problem: Human can’t tell correct from incorrect. • Spotify recommends next song from “Justin Timerlake” • Goodreads suggests next book Sartre’s “Being and Nothingness” • GPS is 5 meter off. (Still more correct than one that is 10m off)
  • 18. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • Oracle problem: Machine can’t tell correct from incorrect. • Uncertainty problem: Human can’t tell correct from incorrect. • Spotify recommends next song from “Justin Timerlake” • Goodreads suggests next book Sartre’s “Being and Nothingness” • GPS is 5 meter off. (Still more correct than one that is 10m off) • Statistics is well-suited to handle this uncertainty • Example from ecology: Do 2 images of bumblebees show the same species? Even taxonomic experts were incorrect 60% of time.
  • 19. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • What if not all inputs can be generated? • An Android fuzzer that
 cannot generate system-level events (e.g., change battery level)
 cannot exercise the corresponding program / app behaviour. (Restricted search space)
  • 20. Assurances in Software Testing presented by Marcel Böhme Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • What if not all inputs can be generated? • An Android fuzzer that
 cannot generate system-level events (e.g., change battery level)
 cannot exercise the corresponding program / app behaviour. (Restricted search space) • We need to develop statistical methodologies that can • extrapolate from restricted to entire input space. • integrate estimates from several fuzzers with
 overlapping restricted search spaces.
  • 21. Assurances in Software Testing presented by Marcel Böhme (Restricted search space) Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • What if not all inputs can be generated? • What about Search-Based Software Testing? • The fuzzer “gets better” at finding bugs during fuzzing • Breaks our assumption that generated executions are IID. • It is easy to show that estimation bias reduces over time. (Adaptive bias)
  • 22. Assurances in Software Testing presented by Marcel Böhme (Restricted search space) Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • What if not all inputs can be generated? • What about Search-Based Software Testing? • The fuzzer “gets better” at finding bugs during fuzzing • Breaks our assumption that generated executions are IID. • It is easy to show that estimation bias reduces over time. • How to correct this adaptive bias? • a-priori (statistical theory) or a-posteriori (engineering). (Adaptive bias)
  • 23. Assurances in Software Testing presented by Marcel Böhme (Restricted search space) Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • What if not all inputs can be generated? • What about Search-Based Software Testing? (Adaptive bias)
  • 24. Assurances in Software Testing presented by Marcel BöhmeAssurances in Software Testing presented by Marcel Böhme https://fuzzingbook.org Interactive book chapter: “When to stop fuzzing” Conclusion Assurances in Software Testing presented by Marcel Böhme We found a preliminary 
 statistical framework borrowed from ecology. Assurances in Software Testing presented by Marcel Böhme Assurances in Software Testing We don’t have them Assurances in Software Testing presented by Marcel Böhme (Restricted search space) Challenges and Opportunities • What about extremely rare & rather extreme behaviours? • What if we are uncertain whether a behaviour is correct? • What if not all inputs can be generated? • What about Search-Based Software Testing? (Adaptive bias)