The document discusses risk based testing and random testing approaches. It outlines the challenges of time and resource constraints when testing software. Risk based testing uses risk analysis and metrics to focus testing on high risk areas in order to save time and money while maintaining quality. Metrics are developed to manage and organize large test projects. Random testing involves automatically generating random inputs and scenarios to stress test software in ways not covered by nominal testing. It can be used with differential and fault injection testing against a reference implementation to automatically check test results.
Whether you are new to testing or looking for a better way to organize your test practices and processes, the Systematic Test and Evaluation Process (STEP™) offers a flexible approach to help you and your team succeed. Dale Perry describes this risk-based framework—applicable to any development lifecycle model—to help you make critical testing decisions earlier and with more confidence. The STEP™ approach helps you decide how to focus your testing effort, what elements and areas to test, and how to organize test designs and documentation. Learn the fundamentals of test analysis and how to develop an inventory of test objectives to help prioritize your testing efforts. Discover how to translate these objectives into a concrete strategy for designing and developing tests. With a prioritized inventory and focused test architecture, you will be able to create test cases, execute the resulting tests, and accurately report on the quality of your application and the effectiveness of your testing. Take back a proven approach to organize your testing efforts and new ways to add more value to your project and organization.
Whether you are new to testing or looking for a better way to organize your test practices and processes, the Systematic Test and Evaluation Process (STEP™) offers a flexible approach to help you and your team succeed. Dale Perry describes this risk-based framework—applicable to any development lifecycle model—to help you make critical testing decisions earlier and with more confidence. The STEP™ approach helps you decide how to focus your testing effort, what elements and areas to test, and how to organize test designs and documentation. Learn the fundamentals of test analysis and how to develop an inventory of test objectives to help prioritize your testing efforts. Discover how to translate these objectives into a concrete strategy for designing and developing tests. With a prioritized inventory and focused test architecture, you will be able to create test cases, execute the resulting tests, and accurately report on the quality of your application and the effectiveness of your testing. Take back a proven approach to organize your testing efforts and new ways to add more value to your project and organization.
Test Management as Chapter 5 of ISTQB Foundation 2018. Topics covered are Test Organization, Test Planning and Estimation, Test Monitoring and Control, Test Execution Schedule, Test Strategy, Risk and Testing, Defect Management
Building a Test Automation Strategy for SuccessLee Barnes
Choosing an appropriate tool and building the right framework are typically thought of as the main challenges in implementing successful test automation. However, long term success requires that other key questions must be answered including:
- What are our objectives?
- How should we be organized?
- Will our processes need to change?
- Will our test environment support test automation?
- What skills will we need?
- How and when should we implement?
In this workshop, Lee will discuss how to assess your test automation readiness and build a strategy for long term success. You will interactively walk through the assessment process and build a test automation strategy based on input from the group. Attend this workshop and you will take away a blue print and best practices for building an effective test automation strategy in your organization.
• Understand the key aspects of a successful test automation function
• Learn how to assess your test automation readiness
• Develop a test automation strategy specific to your organization
- Risks you'll encounter while throughout software development and testing.
- Cataloguing and managing you risks
- Adding value to your QA by managing your risks.
Full webinar recording:
https://www.practitest.com/qa-learningcenter/webinars/testing-risk-management-webinar/
Join Sauce Labs’ Automation Specialist and Selenium project contributor, Leo Laskin, as he discusses the value of open source in testing. He will also share his personal experience in moving from manual to automated testing, the lessons he has learned, and the steps he took to build a powerful, international test coding army.
View the recording at https://saucelabs.com/resources/webinars/automation-best-practices
Test Automation
Test automation is the use of test automation software like Selenium or self-developed testware to execute test cases.
Test automation is mostly used to automate repetitive testing tasks in a formalized way. It is also used to execute tests that would be difficult to perform manually like performance testing.
There are many advantages of test automation that are mostly related to the repeatability of the tests and the speed of test execution. There are a lot of commercial and open source tools available which can be grouped into two main categories; Code-Driven and Graphical User Interface Testing. Thus the key success factor in test automation is selecting the right tool and have a specialized test automation team.
It is to use test automation tools by considering ROI (return on investment). Otherwise it is quite easy to waste big amount of energy, commitment and definitely money.
With more than 500 clients, Keytorc is the leading software testing company in EMEA region that have competencies of automating any kind of software in diverse industries.
For more information about test automation tools and Keytorc’s test automation service you can contact with our performance test engineers: www.keytorc.com or blogs.keytorc.com
Test Otomasyonu:
Test otomasyonu özellikle tekrarlayan ve manuel olarak yapılması zor olan testlerin Selenium gibi test otomasyon araçları kullanılarak ya da özel olarak geliştirilen test otomasyon scriptleri ile yapılmasıdır. Bu bakımdan:
- Regresyon testleri
- Performans testleri
- Yük ve Stres testleri
- Test yönetimi
test otomasyonuna en uygun test tipleri ve aktiviteleridir.
Test otomasyonunun en büyük faydaları:
- testlerin hıznının artırılması
- testlerin kapsamının artırılması
- testlerin doğruluğunun artırılması
- testlerin raporlama kalitesinin artırılmasıdır.
Test otomasyonu doğru araç seçimi yapılmaması, ya da test otomasyonunu bilen uzman bir ekip tarafından yapılmaması durumunda faydadan çok zarar getirmektedir.
EMEA bölgesindeki lider yazılım test firması olan Keytorc’un test otomasyon ekibiyle iletişime geçmek için: www.keytorc.com ya da blogs.keytorc.com
This is chapter 5 of ISTQB Advance Technical Test Analyst certification. This presentation helps aspirants understand and prepare the content of the certification.
Agile Testing: The Role Of The Agile TesterDeclan Whelan
This presentation provides an overview of the role of testers on agile teams.
In essence, the differences between testers and developers should blur so that focus is the whole team completing stories and delivering value.
Testers can add more value on agile teams by contributing earlier and moving from defect detection to defect prevention.
Test Management as Chapter 5 of ISTQB Foundation 2018. Topics covered are Test Organization, Test Planning and Estimation, Test Monitoring and Control, Test Execution Schedule, Test Strategy, Risk and Testing, Defect Management
Building a Test Automation Strategy for SuccessLee Barnes
Choosing an appropriate tool and building the right framework are typically thought of as the main challenges in implementing successful test automation. However, long term success requires that other key questions must be answered including:
- What are our objectives?
- How should we be organized?
- Will our processes need to change?
- Will our test environment support test automation?
- What skills will we need?
- How and when should we implement?
In this workshop, Lee will discuss how to assess your test automation readiness and build a strategy for long term success. You will interactively walk through the assessment process and build a test automation strategy based on input from the group. Attend this workshop and you will take away a blue print and best practices for building an effective test automation strategy in your organization.
• Understand the key aspects of a successful test automation function
• Learn how to assess your test automation readiness
• Develop a test automation strategy specific to your organization
- Risks you'll encounter while throughout software development and testing.
- Cataloguing and managing you risks
- Adding value to your QA by managing your risks.
Full webinar recording:
https://www.practitest.com/qa-learningcenter/webinars/testing-risk-management-webinar/
Join Sauce Labs’ Automation Specialist and Selenium project contributor, Leo Laskin, as he discusses the value of open source in testing. He will also share his personal experience in moving from manual to automated testing, the lessons he has learned, and the steps he took to build a powerful, international test coding army.
View the recording at https://saucelabs.com/resources/webinars/automation-best-practices
Test Automation
Test automation is the use of test automation software like Selenium or self-developed testware to execute test cases.
Test automation is mostly used to automate repetitive testing tasks in a formalized way. It is also used to execute tests that would be difficult to perform manually like performance testing.
There are many advantages of test automation that are mostly related to the repeatability of the tests and the speed of test execution. There are a lot of commercial and open source tools available which can be grouped into two main categories; Code-Driven and Graphical User Interface Testing. Thus the key success factor in test automation is selecting the right tool and have a specialized test automation team.
It is to use test automation tools by considering ROI (return on investment). Otherwise it is quite easy to waste big amount of energy, commitment and definitely money.
With more than 500 clients, Keytorc is the leading software testing company in EMEA region that have competencies of automating any kind of software in diverse industries.
For more information about test automation tools and Keytorc’s test automation service you can contact with our performance test engineers: www.keytorc.com or blogs.keytorc.com
Test Otomasyonu:
Test otomasyonu özellikle tekrarlayan ve manuel olarak yapılması zor olan testlerin Selenium gibi test otomasyon araçları kullanılarak ya da özel olarak geliştirilen test otomasyon scriptleri ile yapılmasıdır. Bu bakımdan:
- Regresyon testleri
- Performans testleri
- Yük ve Stres testleri
- Test yönetimi
test otomasyonuna en uygun test tipleri ve aktiviteleridir.
Test otomasyonunun en büyük faydaları:
- testlerin hıznının artırılması
- testlerin kapsamının artırılması
- testlerin doğruluğunun artırılması
- testlerin raporlama kalitesinin artırılmasıdır.
Test otomasyonu doğru araç seçimi yapılmaması, ya da test otomasyonunu bilen uzman bir ekip tarafından yapılmaması durumunda faydadan çok zarar getirmektedir.
EMEA bölgesindeki lider yazılım test firması olan Keytorc’un test otomasyon ekibiyle iletişime geçmek için: www.keytorc.com ya da blogs.keytorc.com
This is chapter 5 of ISTQB Advance Technical Test Analyst certification. This presentation helps aspirants understand and prepare the content of the certification.
Agile Testing: The Role Of The Agile TesterDeclan Whelan
This presentation provides an overview of the role of testers on agile teams.
In essence, the differences between testers and developers should blur so that focus is the whole team completing stories and delivering value.
Testers can add more value on agile teams by contributing earlier and moving from defect detection to defect prevention.
Test Environment Management Maturity Modelcrawforn992
Test Environments are considered to be a leading cause of low Development & Test productivity. This slide provides you with a mechanism of reviewing your current capability against the 8 Dimensions of Test Environment Management. Copyright of www.enov8.com.
Key performance indicators to determine that an organisation has an effective, efficient, result oriented TEM function or is close to having one, based on best practise and ITIL/ISO standards.
Test Environment Management (TEM) is a function in the software delivery process which aids the software testing cycle by providing a validated, stable and usable test-environment to execute the test scenarios or replicate bugs.
Testing- Fundamentals of Testing-Mazenet solutionMazenetsolution
For Youtube Videos: bit.do/sevents
Why testing is necessary,Fundamental test process, Psychology of testing, Re-testing and regression testing,
Expected results,Prioritisation of tests
The recent revolution in software development, including agile / iterative development, cloud computing, continuous integration, and many more, opened new opportunities for performance testing and affected its role in performance engineering. For example, early and continuous performance testing is becoming the new norm. However, performance testing in general and specific performance testing techniques should be considered in full context: including environments, products, teams, issues, goals, budgets, timeframes, risks, etc. The question is not what technique is better, the question is what technique (or what combination of techniques) to use in particular case (or, in more traditional wording, what should be performance testing strategy).
Drastic changes in the industry in recent years significantly expanded the performance testing horizon: agile development and cloud computing probably the most. Basically, instead on single way of doing performance testing (and all other were considered rather exotic), we have a full spectrum of different tests which can be done at different moments; so deciding what and when to test became a very non-trivial task heavily depending on the context. We need to create and run different tests mitigating different performance risks.
So the art of performance engineering is to find out the best strategy of combining different performance tests and other approaches to mitigate performance risks to optimize risk mitigation / costs ratio for, of course, the specific context.
This session will provide you the insights of understanding various types of performance risks, which test techniques and practices to use in a specific context to measure and evaluate system performance, and how to interpret the data derived from these tests in order to drive performance engineering excellence.
This slide is special for master students (MIBS & MIFB) in UUM. Also useful for readers who are interested in the topic of contemporary Islamic banking.
The simplified electron and muon model, Oscillating Spacetime: The Foundation...RitikBhardwaj56
Discover the Simplified Electron and Muon Model: A New Wave-Based Approach to Understanding Particles delves into a groundbreaking theory that presents electrons and muons as rotating soliton waves within oscillating spacetime. Geared towards students, researchers, and science buffs, this book breaks down complex ideas into simple explanations. It covers topics such as electron waves, temporal dynamics, and the implications of this model on particle physics. With clear illustrations and easy-to-follow explanations, readers will gain a new outlook on the universe's fundamental nature.
A workshop hosted by the South African Journal of Science aimed at postgraduate students and early career researchers with little or no experience in writing and publishing journal articles.
Acetabularia Information For Class 9 .docxvaibhavrinwa19
Acetabularia acetabulum is a single-celled green alga that in its vegetative state is morphologically differentiated into a basal rhizoid and an axially elongated stalk, which bears whorls of branching hairs. The single diploid nucleus resides in the rhizoid.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
How to Build a Module in Odoo 17 Using the Scaffold MethodCeline George
Odoo provides an option for creating a module by using a single line command. By using this command the user can make a whole structure of a module. It is very easy for a beginner to make a module. There is no need to make each file manually. This slide will show how to create a module using the scaffold method.
Normal Labour/ Stages of Labour/ Mechanism of LabourWasim Ak
Normal labor is also termed spontaneous labor, defined as the natural physiological process through which the fetus, placenta, and membranes are expelled from the uterus through the birth canal at term (37 to 42 weeks
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
A review of the growth of the Israel Genealogy Research Association Database Collection for the last 12 months. Our collection is now passed the 3 million mark and still growing. See which archives have contributed the most. See the different types of records we have, and which years have had records added. You can also see what we have for the future.
2. Risk Based Testing
and
Random Testing
• Use of Risk Analysis and Metrics
for Software Testing
• Focus Testing to Save Time and
Money while maintaining quality
• How to develop metrics to manage
and organise large test projects
4. Risk Analysis and Testing
Test Plan
Test Item Tree
Risk
Strategy
Risk
Identification
Testing,
Inspection etc.
Risk
Assessment
Matrix: Cost
and Probability
Risk
Mitigation
Test Metrics
Risk
Reporting
Risk
Prediction
5. Risk Based Testing - Theory
• The Formula
Re(f) P(f)*C(f)
– Re(f) Risk Exposure of function f
– P(f) - Probability of a fault in function f
– C(f) - Cost related to a fault in function f
6. Risk Based Testing - Approach
• Plan: Identify Elements to be Tested
– Logical or physical Functions, Modules etc.
• Identify Risk Indicators
– What is important to predict the probability of faults?
• Identify Cost of faults
• Identify Critical Elements
– I.e. functions, tasks, activities etc. based on Risk Analysis
(Indicators and Cost)
• Execute: Improve the Test Process and Organization:
Schedule and Track
7. Simple Test Metrics
• Test Planning
– Number of test cases per function
– Number of hours testing per function
• Progress Tracking
–
–
–
–
Number of tests planned, executed and completed
Number of faults per function
Number of hours used for test and fix
Estimated to Complete
• Probability of faults - Indicators
–
–
–
–
New functionality
Size
Complexity
Quality of previous phases and documents
• Cost of Faults
8. Risk Based Testing - Metrics
• Identify Areas with “High Risk Exposure”
– Probability and Cost
• All functions/modules should be tested to a
“minimum level”
• “Extra Testing” in areas with high risk exposure
• Establish Test Plan and Schedule
– Monitor Quality
• Number of Faults per function and time
– Monitor Progress
• Number of hours in test and fix -> ETC
9. Risk Based Testing - Example
Ranking the functions based on Risk Exposure
The Probability of a Fault
The Cost of a Fault
C(c) C(s)
Re(f) P(f)*
2
Example:
Cost
Func.
Probability
New Design
Func. Quality Size
5
5
1
Risk
Com- Weighted Exp.
plexity Averag func.
3
f
C( s )
C(c)
Avr.
Interest
Calc.
3
3
3
2
3
3
3
37
111
Close
Account
1
3
2
2
2
2
3
31
62
Cust.
Profitab.
2
1
1,5
3
3
2
3
41
61,5
Other Probability Factors might include: Function Points, Frequency of Use etc.
10. High
Probability
Risk Based Testing
- Reporting
1
2
3
4
TECHNICAL
INTERFACE
RISK
10
1
Low
Low
440
Low Medium
Probability
High
510
439
11
2
2
370
369
5
302
Low
Low
Medium
Medium
BUSINESS RISK
High
High
Consequence
Consequence
High
11. Risk Based Testing - Practice
Prior to test execution:
identify critical transactions
1
Test Execution
identifies
“bad” transactions
“Top-20”
2
Extra Testing:
- Additional testing by product specialist
- Automated regression testing
3
12. Planning and Progress Tracking
Number of Test
Cases
On-line Test Cases Completed
Planned
Executed
QAed
Date
Started
Planned
Actual
13. Progress Indicators - “To be vs. Actual”
“To be
Retested”
vs.
“Actually
Retested”
To Be Fixed
Actually fixed
To Be Restested, Actually Retested and Rejected
Number of Faults
• “To be fixed”
vs.
“Actually fixed”
Number of Faults
To Be Fixed and Actually Fixed
To be retested
Act. retested
Rejected
14. Progress Indicators - Hours Used
Number of hours
for finding one
fault and for
fixing one
Hours per Fault for Test and Fix
Hours per Fault
Online
Batch
Fix
Hours per Fault for Test and Fix
Hours per Fault
Number of hours
for finding one
fault and for
fixing one
Test
Fix
Test
Date
15. “Estimated to Complete”
• ETC for system test based on:
Hours ETC
– Number of hours testing per fault found
– Number of hours fixing per fault
– Number of
faults found
Calculated ETC and Actual Hours
per function
Actual to Complete
– Number of
at Time t
fixes being
rejected
– Number of
remaining
tests
(functions
to be tested)
Date
Estimated to Complete at Time t
16. Benefits of Risk Based Testing
• Improved Quality?
– all critical functions tested
• Reduced Time and Money in Testing
– effort not wasted on non critical or low risk
functions
• Improved customer confidence
– due to customer involvement and good reporting
and progress tracking
17. Test Process Work Flow
Risk Identification
Risk Assessment
Basic Test
Data
PD
LD
Case Quality
Standards
Test Exec.
Procedure
Test Case
Case Build
Procedure
Test
Completed
QC / QA
Good
Test Exec
Risk Mitigation
Good/
Bad
Bad
PTDs
Raised
Risk Reporting
Retest
Risk Prediction
Problem Mngmnt.
Procedure
Fix
Fix Procedure
CR
Change Mngmnt.
Procedure
Regression
Test
ProAte
18. Summary
• Risk Based Test Approach
– Focused Testing
• Reduced Resources
• Improved Quality
– Metrics are fundamental
• Process and Organization must support the
new strategy
– Metrics must support the organization and
process
19. Random testing
– Start off with a practical look, and some useful
ideas to get you started on the project: random
testing for file systems
– Then take a deeper look at the notion of feedback
and why it is useful: method for testing OO
systems from ICSE a couple of years
• Then back out to take a look at the general idea of
random testing, if time permits
20. A Little Background
– Generate program inputs at random
– Drawn from some (possibly changing) probability
distribution
“Throw darts at the state space, without drawing a
bullseye”
– May generate the same test (or equivalent tests)
many times
– Will perform operations no sane human would
ever perform
21. Random Testing
• Millions of operations and scenarios, automatically
generated
• Run on fast & inexpensive workstations
• Results checked automatically by a reference oracle
• Hardware simulation for fault injection and reset
simulation
(x 100000)
A day (& night) of testing
(x 100000)
(x 100000)
(x 100000)
22. The Goals
• Randomize early testing (since it is not possible
to be exhaustive)
– We don’t know where the bugs are
Nominal
Scenario Tests
Randomized
Testing
23. The Goals
• Make use of desktop hardware for early testing – vs.
expensive (sloooow) flight hardware testbeds
– Many faults can be exposed without full bit-level
hardware simulation
24. The Goals
• Automate early testing
– Run tests all the time, in the background, while continuing
development efforts
• Automate test evaluation
– Using reference systems for fault detection and diagnosis
– Automated test minimization techniques to speed
debugging and increase regression test effectiveness
• Automate fault injection
– Simulate hardware failures in a controlled test
environment
25. Random testing
• Simulated flash hardware layer allows random
fault injection
• Most development/early testing can be done
on workstations
• Lots of available compute power – can cover
many system behaviors
• Will stress software in ways nominal testing will
not
26. Differential Testing
• How can we tell if a test succeeds?
– POSIX standard for file system operations
• IEEE produced, ANSI/ISO recognized standard for file
systems
• Defines operations and what they should do/return,
including nominal and fault behavior
File system
/
POSIX operation
Result
mkdir (“/eng”, …)
mkdir (“/data”, …)
creat (“/data/image01”, …)
creat (“/eng/fsw/code”, …)
mkdir (“/data/telemetry”, …)
unlink (“/data/image01”)
SUCCESS
SUCCESS
SUCCESS
ENOENT
SUCCESS
SUCCESS
/eng
/data
image01
/telemetry
27. Differential Testing
• How can we tell if a test succeeds?
– The POSIX standard specifies (mostly) what correct
behavior is
– We have heavily tested implementations of the
POSIX standard in every flavor of UNIX, readily
available to us
– We can use UNIX file systems (ext3fs, tmpfs, etc.) as
reference systems to verify the correct behavior of
flash
– First differential approach (published)
was McKeeman’s testing for compilers
28. Random Differential Testing
Choose (POSIX) operation F
Perform F on NVFS
Perform F on Reference
(if applicable)
Compare return values
Compare error codes
Compare file systems
Check invariants
(inject a fault?)
29. Don’t Use Random Testing for Everything!
• Why not test handing read a null pointer?
– Because (assuming the code is correct) it guarantees
some portion of test operations will not induce failure
– But if the code is incorrect, it’s easier and more
efficient to write a single test
– The file system state doesn’t have any impact (we
hope!) on whether there is a null check for the buffer
passed to read
• But we have to remember to actually do these
non-random fixed tests, or we may miss critical,
easy-to-find bugs!
30. Principles Used
•
•
•
•
Random testing (with feedback)
Test automation
Hardware simulation & fault injection
Use of a well-tested reference implementation
as oracle (differential testing)
• Automatic test minimization (delta-debugging)
• Design for testability
– Assertions
– Downward scalability (small model property)
– Preference for predictability
31. Synopsis
• Random testing is sometimes a powerful
method and could likely be applied more
broadly in other missions
– Already applied to four file system-related
development efforts
– Part or all of this approach is applicable to
other critical components (esp. with better
models to use as references)