Leveraging Natural-language Requirements for Deriving Better Acceptance Criteria from Models
•
0 likes•220 views
Here are the key elements extracted: A = "line" B = "File" contains “line.ISIN”. Rule Condition Structure CS1 14 Given order does not exist in System_A And Is data correct? is equal to Yes When System_A performs Create Order Then order exists in System_A CS1: If a requirement contains the keywords “Given”, “When”, “Then” followed by a condition/action/result, extract the condition, action, result. R: Given order does not exist in System_A And Is data correct? is equal to Yes When System_A performs Create Order Then order exists in System
![Context: Acceptance Testing in
Industry
2
Acceptance
Criteria (AC)
Test Engineers
P:Participant T2S: Settlement Platform
InterruptibleActivityRegion1
Merge2
ActivityInitial
Sendsettlement
Instruction
Inx:T2SSettlementIns
State =ToValidate
Receive and
Generate InstructionpInx:Participant
SettlementIns
Validate Ins:Validate
Instruction
Inx:T2S
SettlementIns
Settle Instruction
Inx.SettlementDate >
T2S.CurrentDate
Inx.SettlementDate
starts
Merge1
Inx:T2SSettlementIns
State =Settled
Send
Notification
Inx:T2S
SettlementIns
RunMatchingProcess
X
days
passed
Inx:T2SSettlementIns
State =Matched
ProcessInstruction
Rejection
Inx:T2S
SettlementIns Inx:T2SSettlementIns
State =Valid
Inx:T2SSettlementIns
State =Rejected
«localPostcondition» Lp1:....
FlowFinal
Receive
notification
notif:
Participant
Notification
[No]
[Inx.State ==
Valid]
[Yes]
DomainModel(Classdiagram)
Participant Settlement Ins
T2SSettlement Ins
State:T2SInstruction State
Participant Notification
Reason:String [0..1]
Message:String
Settlement Instruction
SettlementDate:Date
«enumeratio...
T2SInstruction
State
ToValidate
Valid
Settled
Rejected
Matched
0..*Participant
Instruction
ucActors
Participant
«actor»
SettlementPlatform
isInitialised:Boolean
«Pre-condition»
{SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
Natural language (NL)
requirements
Business Analysts
Requirements Analysis Acceptance Testing
Requirements
models](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
Similar to Leveraging Natural-language Requirements for Deriving Better Acceptance Criteria from Models
Similar to Leveraging Natural-language Requirements for Deriving Better Acceptance Criteria from Models (20)
More from Lionel Briand
More from Lionel Briand (20)
Recently uploaded
Recently uploaded (20)
Leveraging Natural-language Requirements for Deriving Better Acceptance Criteria from Models
- 1. .lusoftware verification & validation VVS Leveraging Natural-language Requirements for Deriving Better Acceptance Criteria from Models Alvaro Veizagaa, Mauricio Alfereza, Damiano Torrea, Mehrdad Sabetzadehba, Lionel Briandab aUniversity of Luxembourg, Luxembourg bUniversity of Ottawa, Canada October 22nd, 2020 Elene Pitskhelauri Clearstream, Luxembourg
- 2. Context: Acceptance Testing in Industry 2 Acceptance Criteria (AC) Test Engineers P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} Natural language (NL) requirements Business Analysts Requirements Analysis Acceptance Testing Requirements models
- 3. Our Earlier Work on Requirements Specification 3 AGAC • Supports the automated generation of AC in Gherkin • AC automatically generated from models Rimay • A language for writing functional requirements • Helps write more precise requirements P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 4. AGAC: Automatic Generation of Acceptance Criteria 4 P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} Models Business Analysts Requirements Analysis Acceptance Testing NL Requirements Generation of AC Acceptance Criteria (AC) Test Engineers
- 5. AGAC Example Requirements Model Acceptance Criteria 5 Create Alert Send Alert Receive Alert Create Order Is data correct? [Yes] [No] Given order does not exist in System_A And Is data correct? is equal to Yes When System_A performs Create Order Then order exists in System_A Given alert does not exist in System_A And Is data correct? is equal to No When System_A performs Create Alert Then alert exists in System_A //Gherkin scenarios for the send and receive actions not shown AC1AC2 Generation of AC
- 6. Requirements Written in Rimay 6 ACTOR For each “line of the File”, System must check that Share_CLass_Identifier.Value contains “line.ISIN”. SYSTEM RESPONSE SCOPE (Optional) SYSTEM RESPONSE ACTOR When Transfer_System receives a File, Transfer_System must forward the File to System. SYSTEM RESPONSE CONDITION STRUCTURE (Optional)
- 7. NL Requirements to Generate AC Natural-language Requirement: Acceptance Criteria 7 When System_A creates an alert, then System_A must set the priotity of the alert to “high”. Given order does not exist in System_A And Is data correct? is equal to Yes When System_A performs Create Order Then order exists in System_A Given alert does not exist in System_A And Is data correct? is equal to No When System_A performs Create Alert Then alert exists in System_A And the property priority of alert is equal to high //Gherkin scenarios for the send and receive actions not shown AC1AC2
- 8. Main Goal 8 Acceptance Criteria (AC) Test Engineers P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} NL Requirements Business Analysts Requirements Analysis Acceptance Testing Requirements models Rimay AGAC
- 9. 9 Enrich models with information extracted from NL requirements in order to generate better AC • Define a set of 13 information extraction rules (RQ1) • Propose a systematic method that generates recommendations (RQ2) • Verify that the recommendations are relevant to AC (RQ3) Main Goal
- 10. Identify Model Elements to Enrich Enriched Model Acceptance Criteria 2 3 Create Recommendations 1 Extract Information Enrich Model4 Generate Acceptance Criteria 5 Recommendations NL Requirements Model Requirements Specification P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} Our Approach 10
- 11. Our Approach 11 Identify Model Elements to Enrich Enriched Model Acceptance Criteria 2 3 Create Recommendations 1 Extract Information Enrich Model4 Generate Acceptance Criteria 5 Recommendations NL Requirements Model Requirements Specification P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 12. Extract Information 12 13 rules to extract AC relevant information content from NL requirements • Derived from manual analysis of overlaps between meta-models element and the element types in Rimay Category # Scope 1 Condition Structure 7 Actor 2 System Response 3 RQ1: How can we extract AC-related information from NL requirements?
- 13. Rule Scope S1 13 Check ISIN more elements … more elements … : File line S1: If a prepositional phrase starts by “for each”, and further mentions: the type A of the collection that will be iterated over and an item B in the collection, then extract A and B. R: For each "line of the File”, System must check that Share_Class_Identifier.Value contains “line.ISIN”. Type A Type A Item B Item B
- 14. Rule Condition Structure C1 14 C1: If the verb phrase A in a When structure does not match the name of any of the actions preceding the traced action, then extract A. R: When Transfer_System receives a File, Transfer_System must forward the File to System. Forward File f: File more elements … Receive File more elements … f: File
- 15. Rule Actor A1 15 A1: If an actor A in an NL requirement does not match the name of any UML actor linked to the activity partition of the traced action, then extract A. Create Alert : System …more elements follow more elements precede… R: Before "8:00 am", every "calendar day”, if System does not receive the File, then System must create an "Alert”.
- 16. Rule System Response SR1 16 SR1: If a system response creates data A (e.g., Report, Instruction, Alarm), then extract A. Create Alert …more elements follow more elements precede… : Alert R: Before "8:00 am", every "calendar day”, if System does not receive the File, then System must create an “Alert" .
- 17. Our Approach 17 Identify Model Elements to Enrich Enriched Model Acceptance Criteria 2 3 Create Recommendations 1 Extract Information Enrich Model4 Generate Acceptance Criteria 5 Recommendations NL Requirements Model Requirements Specification P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 18. Identify Models Elements to Enrich 18 Requirement. When the Order_Issuer (hereafter known as OI) creates an Order of type Subscription_Order, then the OI must set the settlement_method of the Order to “FOP”. Compare text sequences Activity Diagrams Create Order … …more elements Class Diagrams (Domain Model) Use Case Diagrams (Actors) … actor Order_Issuer … settlement_date : date Subscription_Order act Create subscription order
- 19. Identify Models Elements to Enrich 19 6 model elements to enrich R1. When the Order_Issuer (hereafter known as OI) creates an Order of type Subscription_Order, then the OI must set the settlement_method of the Order to “FOP”. SR1 actor actor alias object type property name property valueobject name A1 A2 SR2 SR3 SR2 Activity Diagrams … Class Diagrams (Domain Model) act Create subscription order Order : Subscription_Order settlement_date : date settlement_method : string Subscription_Order Create Order …more elementssettlement_method = “FOP” Use Case Diagrams (Actors) … A2 A1 SR1 SR2 SR2 SR3 «actor» Order_Issuer OI : Order_Issuer …
- 20. Our Approach 20 Identify Model Elements to Enrich Enriched Model Acceptance Criteria 2 3 Create Recommendations 1 Extract Information Enrich Model4 Generate Acceptance Criteria 5 Recommendations NL Requirements Model Requirements Specification P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 21. Create Recommendations 21 ID Description Rule Rec. 4 Add the property “settlement method” to the object node of type “Subscription Order" SR2 Rec. 5 Set the “settlement_method” property’s value to “FOP” SR2 … … … Recommendations on how to enrich the model elements Activity Diagrams … Class Diagrams (Domain Model) act Create subscription order Order : Subscription_Order settlement_date : date settlement_method : string Subscription_Order Create Order …more elementssettlement_method = “FOP” Use Case Diagrams (Actors) … A2 A1 SR1 SR2 SR2 SR3 «actor» Order_Issuer OI : Order_Issuer …
- 22. Our Approach 22 Identify Model Elements to Enrich Enriched Model Acceptance Criteria 2 3 Create Recommendations 1 Extract Information Enrich Model4 Generate Acceptance Criteria 5 Recommendations NL Requirements Model Requirements Specification P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 23. Enrich Model 23 Enrich the model elements following recommendations Activity Diagrams … Class Diagrams (Domain Model) act Create subscription order Order : Subscription_Order settlement_date : date settlement_method : string Subscription_Order Create Order …more elementssettlement_method = “FOP” Use Case Diagrams (Actors) … A2 A1 SR1 SR2 SR2 SR3 «actor» Order_Issuer OI : Order_Issuer … RQ2: How can we systematically enrich models with the (AC-related) information from NL requirements? ID Description Rule Rec. 4 Add the property “settlement method” to the object node of type “Subscription Order" SR2 Rec. 5 Set the “settlement_method” property’s value to “FOP” SR2 … … …
- 24. Our Approach 24 Identify Model Elements to Enrich Enriched Model Acceptance Criteria 2 3 Create Recommendations 1 Extract Information Enrich Model4 Generate Acceptance Criteria 5 Recommendations NL Requirements Model Requirements Specification P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 25. Generate Acceptance Criteria 25 Generation of AC Activity Diagrams act Create subscription order Order : Subscription_OrderCreate Order …more elementssettlement_method = “FOP” … A2 A1 SR1 SR2 SR3 OI : Order_Issuer @Intent Create @Requirement_Id: R1 Scenario: Create an Order Given an Order of type Subscription_Order does not exist in OI of type Order_Issuer When OI Create Order, Then Order exists in OI And the property settlement_method of Order is equal to FOP
- 26. Empirical Evaluation 26 • Assess whether financial analysis benefit from our solution • We conducted a case study in collaboration with Clearstream Luxembourg Investment Fund Services RQ3: Are our recommendations for model enrichment useful in practice?
- 27. Case Study Preparation 27 Q1) Are the recommendations to enrich the model useful to generate better AC? Requirements Specification 27 Recommendations AC from the Model Enrich Model AC from Enriched Model Compare AC and answer Q1 5 Domain Experts P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 28. Results 28 Answer to Q1: the experts found 89% of the recommendations (24 out of 27) relevant for generating better Acceptance Criteria Question Yes No Q1 24 3 TPs FPs FNs Precision % Recall % 24 3 0 89 100
- 29. Results – Model Elements 29 Model Element Original Enriched %Increased Actions 22 24 9.1% Events 1 3 200% Objects 11 15 36.4% Decision nodes 8 9 12.5% Fork and join nodes 2 3 50% Property values 0 75 N/A The number of instances of all element types increases in the enriched model P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)} P:Participant T2S: Settlement Platform InterruptibleActivityRegion1 Merge2 ActivityInitial Sendsettlement Instruction Inx:T2SSettlementIns State =ToValidate Receive and Generate InstructionpInx:Participant SettlementIns Validate Ins:Validate Instruction Inx:T2S SettlementIns Settle Instruction Inx.SettlementDate > T2S.CurrentDate Inx.SettlementDate starts Merge1 Inx:T2SSettlementIns State =Settled Send Notification Inx:T2S SettlementIns RunMatchingProcess X days passed Inx:T2SSettlementIns State =Matched ProcessInstruction Rejection Inx:T2S SettlementIns Inx:T2SSettlementIns State =Valid Inx:T2SSettlementIns State =Rejected «localPostcondition» Lp1:.... FlowFinal Receive notification notif: Participant Notification [No] [Inx.State == Valid] [Yes] DomainModel(Classdiagram) Participant Settlement Ins T2SSettlement Ins State:T2SInstruction State Participant Notification Reason:String [0..1] Message:String Settlement Instruction SettlementDate:Date «enumeratio... T2SInstruction State ToValidate Valid Settled Rejected Matched 0..*Participant Instruction ucActors Participant «actor» SettlementPlatform isInitialised:Boolean «Pre-condition» {SettlementPlatform.allInstances()->forAll (t/t.isInitialised=true)}
- 30. Results – AC Details 30 Enriched model leads to the generation of more precise and complete AC AC Details Original Augmented % Increase Pre-conditions 432 535 22.1% Post-conditions 325 2262 596% Gherkin scenarios 156 191 22.4%
- 31. Summary 31
- 32. Conclusions 32 • Generating AC exclusively from models would miss critical information that is available only in NL requirements • Our industry partner confirmed that the AC resulting from our approach are more precise and complete • Take-home message: We need to simultaneously consider both models and NL requirements to be able to generate good AC
- 33. Future Work 33 • Including semantic analysis for better tracing NL requirements to models • Investigating how our approach can be applied to other domains and information systems that are commonly modeled using UML
- 34. .lusoftware verification & validation VVS Leveraging Natural-language Requirements for Deriving Better Acceptance Criteria from Models Alvaro Veizagaa, Mauricio Alfereza, Damiano Torrea, Mehrdad Sabetzadehba, Lionel Briandab aUniversity of Luxembourg, Luxembourg bUniversity of Ottawa, Canada October 22nd, 2020 Elene Pitskhelauri Clearstream, Luxembourg