Justification Factory : From justification requirements elicitation to their continuous production

Justification Factory :
From justification requirements
elicitation to their continuous
production
PhD Defense - 16/11/2018
Clément DUFFAU

How to build a medical device ?
16/11/18 - duffau@i3s.unice.fr PhD Defense 2
legally

Critical domains
Standards
Internal practices
Regulatory
authority
Company’s
experts
3. produce
1. produces
4. reviews
2. contextualize
5. develop
Product
6. V&V
Deliverable product
7. audits
certified
Textual and interlaced
Anticipation
Deviation
‘No conformance’
risk
Costly rollback on
previous documents
Novelty identification
‘No conformance’
risk
Costly document
production, overquality
and intrusive

Medical technologies and justifications
Various disciplines
Clinical Electronics Software Mechanics
Standards
Software Lifecycle IEC 62304 Risk Management ISO 14791 Usability IEC 62366
Internal Practices
Software Development Plan Risk Matrix Analysis Usability criteria
Justifications
Sprint Report Risk mitigation Report
Agility
⇒ disciplinary practices but also transversal
⇒ interlaced
⇒ informal practical know how
⇒ differences between the expected and the produced
⇒ from ‘last day’ concern to justification tsunami

Some vocabulary
Justification requirement : a proof element that must be reached to be compliant
with a law. It can be set through a standard or a guide
Elicitation : action to help experts to formalize their knowledge to keep and share
them
Justification : an argumentation attesting that a justification requirement is fulfilled
Justification artefact : a data (e.g. document, result) to establish in order to
construct a justification

Objectives
Standards
Internal practices
Regulatory
authority
Company’s
experts
3. produce
1. produces
4. reviews
2. contextualize
5. develop
Product
6. V&V
Deliverable product
7. audits
certified
justification requirement
justification
conformance
conformance
O1 Elicitate
Help to produce
O2
O3
Check and analyze
Check and analyze

Outline
State of the Art
Contributions
Validation
Conclusion and Perspectives

Outline
State of the Art
Contributions
Validation

Derived from Toulmin argumentation schema ²
Comprehensive notation to explain why a result is trustable
Captures the rationale logical structure of all evidence that leads to the
acceptance of a high-level property
+ Proved to be useful ³
- Only a graphical notation
⇒ A good starting point but need to be formalized
1. Polacsek, T.: 2016, Validation, accreditation or certification: a new kind of diagram to provide
Confidence
2. Toulmin, S.E.: 2004, The Uses of Argument
3. Polacsek, T., S. Sharma, C. Cuiller et V. Tuloup. 2018, The need of diagrams based on toulmin schema application :
an aeronautical case study, EURO Journal on Decision Processes
Justification Diagrams¹

Historically Goal Structuring Notation¹ (GSN) and Claim-Argument-Evidence² (CAE)
Used for safety case
New OMG standard : Structured Assurance Case Metamodel ³
SACM focus on 3 aspects :
● Argumentation
● Artifact
● Assurance Case
+ A formal model
- A minimal conformance relationship
⇒ A good semantics starting point
1. Kelly, T., Weaver, R.:2004, The goal structuring notation /- a safety argument notation
2. Emmet, L., Cleland, G.: 2002, Graphical notations, narratives and persuasion: a pliant systems approach to hypertext tool
design
3. OMG: 2018, Structured assurance case meta-model (sacm)
Assurance Case

Justification production
● Traceability from requirements to production
● Specifics methodologies and tools for each stage
○ Requirement framework (ECHO ¹, SEMIOS ², ...)
○ Architecture framework (AREL ³, ...)
○ Development framework (Issue-based system, Continuous Integration ⁴, …)
+ Give ways to collect justifications
- No conformance with justification requirements
1. Lee , C., L. Guadagno et X. Jia . 2003, «An agile approach to capturing requirements and traceability», 2nd International
Workshop on Traceability in Emerging Forms of Software Engineering
2 Mezghani , M. , J. Kang , F. Sedes ; 2018, « Industrial requirements classification for redundancy and inconsistency detection in
SEMIOS
3. Tang, A., Y. Jin et J. Han . 2007, «A rationale-based architecture model for design traceability and reasoning», Journal of
Systems and Software
4. Fowler , M. et M. Foemmel . 2006, «Continuous integration»

State of the Art matrix

Outline
State of the Art
Contributions
Validation

A multi-level conformance
Justification Pattern
Diagram for
standards
Diagram for internal
practices
Justification Diagram
for a product
conformance conformance

Conformance at the baseline
DUFFAU , C., T. POLACSEK M. BLAY -FORNARINO . 2018, Une sémantique pour les patrons de justification,
INFormatique des ORganisations et Systèmes d’Information et de Décision 2018 (INFORSID)
O3 Check and analyze conformance

A flexible conformance for justification step
O1
O3
O1
DUFFAU , C., T. POLACSEK M. BLAY -FORNARINO . 2018, Une sémantique pour les patrons de justification,
INFormatique des ORganisations et Systèmes d’Information et de Décision 2018 (INFORSID)
Check and analyze conformance
Elicitate justification requirements

Evolution and conformance
for a product
feedbacks
from the field
Diagram for
standards
practices

Evolution characterizationO1
e’34 R e3
e’34 R e4
O3

Comparison and alignment operation
O1
O3
e’34 R e3
e’34 R e4

Bridge the gap with production
for a product
artifactsA lot of
Diagram for
standards
practices

Justification FactoryO2
Help to produce justifications

Outline
State of the Art
Contributions
Validation

Challenge
● New products
● Changing practices to Agility
● A small team
=> Capitalization on our practices while cost effective
quality

study conduct for JPD
Stakeholders:
1 researcher/practitioner, 2 quality system managers, 3 technical leaders
Study guideline for a stage:
1. The researcher designs a JPD according to quality management team and technical
leader requirements
2. The technical leaders
a. Identify justifications items that must be produced
b. Develop tools to produce them
3. During the development step, the technical leaders
a. Possibly define new activities
b. Produce the necessary justifications
4. During the deployment step, the quality managers and the researcher analyze the
differences between the original JPD and the JDs
DUFFAU , C., T. POLACSEK M. BLAY -FORNARINO . 2018, Support of justification elicitation: Two industrial reports,
Advanced Information Systems Engineering - 30th International Conference, CAiSE 2018
JPD
JD
JPD/JD revision

JPD for prototyping - IEC 62304 Software Lifecycle
JPD design to elicitate
medical standards

JPD for system validation - Internal practices
Interview on the
usefulness ⇒
Acceptable
Understanding between
experts ⇒ Acceptable
Feedbacks from auditor
⇒ Acceptable
O1
Elicitate justification
requirements

Justification Factory with
DUFFAU , C., B. GRABIEC M. BLAY-FORNARINO . 2017, Towards embedded system agile development challenging
verification, validation and accreditation: Application in a healthcare company, ISSRE 2017-IEEE International
Symposium on Software Reliability Engineering
AXONIC
environment
Automate justification
construction based on
existing environment
Continuous Integration ⇒
79/82 automated
construction
Redmine ⇒ 184/184
automated validation
O2
Help to produce
justifications

● Control lifecycle and automate output document
● Justifications Verification ( ~ 1 time /week /employee)
○ Error rate reduced from 33.9% to 16.3%
○ Timing reduced from 60 minutes to 1 minute
● Justifications Validation (~ 1 time / week)
○ Error rate reduced from 13.8% to 0%
○ Timing reduced from 30 minutes to 2 minutes
Metrics, feedbacks on
O3
Check and analyze
conformance

Outline
State of the Art
Contributions
Validation

Standards
Internal practices
Regulatory
authority
Company’s
experts
3. produce
1. produces
2. contextualize
5. develop
Product
6. V&V
Delivered product
7. audits
4. reviews
elicitate
elicitate
Help to produce
O1
O2
O3
check and analyze
certified
JF
Back to objectives

Perspectives
● Co-evolution of justifications (hidden dependencies, hierarchy impacts, ...)
● Study the relation between business process and justification process (BPMN)
● Capture all the semantics in our meta-model (OCL)
● Extending Requirement Engineering with justification approaches (KAOS, i*)
● Validation on different contexts

Justification and Experimental Systems
1. DUFFAU , C., C. CAMILLIERI M. BLAY-FORNARINO. 2017, Improving confidence in experimental systems through
automated construction of argumentation diagrams, ICEIS 2017
2. DUFFAU , C., C. CAMILLIERI M. BLAY-FORNARINO . 2017, Vers l’argumentation automatique d’expérimentations :
application à un portfolio de workflows

Thanks !
Questions, Comments ?

Justification Factory : From justification requirements elicitation to their continuous production

Recommended

Recommended

More Related Content

Similar to Justification Factory : From justification requirements elicitation to their continuous production

Similar to Justification Factory : From justification requirements elicitation to their continuous production (20)

Recently uploaded

Recently uploaded (20)

Justification Factory : From justification requirements elicitation to their continuous production