MODELS 2018 paper presentation

1. .lusoftware verification & validation
VVS
Model-driven trace diagnostics
for pattern-based
temporal specifications
Wei Dou, Domenico Bianculli, Lionel Briand
SnT Centre - University of Luxembourg

2. .lusoftware verification & validation
VVS
Model-driven trace diagnostics
for pattern-based
temporal specifications
Wei Dou, Domenico Bianculli, Lionel Briand
SnT Centre - University of Luxembourg

3. Trace Diagnostics Problem

4. Trace Checking
!4
“An automatic procedure for evaluating a
specification over a trace of
recorded events produced by a system”

5. Verification Verdict
• A truth value from some truth domain stating whether the
specification holds on the trace
• It the verdict is false, a software engineer needs to diagnose
the property violations found in the trace
!5

6. How useful is a verdict
for trace diagnostics?
!6

7. 7
Example of Verdict
!7
MarQ
First violating trace element
Incomplete information
Textual output

8. 8
Example of Verdict
!8
MonPoly
All violating trace elements
Limited information
Textual output

9. 9
Example of Verdict
!9
Breach
Checking Simple Properties with Nominal Parameters
We can plot the satisfaction function again to interpret the result.
AFC_w_Specs.PlotRobustSat(AF_ev_not_ok2);
0 5 10 15 20 25 30
-1
0
1
controller_mode[t]==0
false
true
Quant. sat
Bool. sat
0 5 10 15 20 25 30
-20
0
20
AF[t]- AFref[t] > 0.01*14.7
false
true
0 5 10 15 20 25 30
-1
0
1
(controller_mode[t]==0) and (AF_not_ok)
false
true
time
0 5 10 15 20 25 30
-1
0
1
ev ((controller_mode[t]==0) and (AF_not_ok))
false
true
Binary results
Limited information

10. 10
Example of Verdict
!10
SOLOIST-ZOT
True/False
No violation information

11. 11
Example of Verdict
!11
RV-Monitor
User-defined error messages
No systematic guidance for
trace diagnostics

12. Understanding a Verdict
!12
Property:
(when it evaluates to false)
“it is always the case that
if event A occurs then it
should stimulate, within 5
time units, the sequential
occurrence of events B
followed, within 2 time
units, by C”

13. Understanding a Verdict
!13
Property:
(when it evaluates to false)
“it is always the case that
if event A occurs then it
should stimulate, within 5
time units, the sequential
occurrence of events B
followed, within 2 time
units, by C”
Reason for violation:
1) at least an occurrence of
A not followed by B-C

14. Understanding a Verdict
!14
Property:
(when it evaluates to false)
“it is always the case that
if event A occurs then it
should stimulate, within 5
time units, the sequential
occurrence of events B
followed, within 2 time
units, by C”
Reason for violation:
2) at least an occurrence of
A that is followed by B-C
but B occurs after more
than 5 time units since the
last occurrence of A

15. Understanding a Verdict
!15
Property:
(when it evaluates to false)
“it is always the case that
if event A occurs then it
should stimulate, within 5
time units, the sequential
occurrence of events B
followed, within 2 time
units, by C”
Reason for violation:
3) at least an occurrence of
A that is followed by B-C
but C occurs after more
than 2 time units since the
occurrence of B

16. Understanding a Verdict
Reasons for violations
1) at least an occurrence of A not
followed by B-C
2) at least an occurrence of A that is
followed by B-C but B occurs after
more than 5 time units since the last
occurrence of A
3) at least an occurrence of A that is
followed by B-C but C occurs after
more than 2 time units since the
occurrence of B
!16
Property:
(when it evaluates to false)
“it is always the case that
if event A occurs then it
should stimulate, within 5
time units, the sequential
occurrence of events B
followed, within 2 time
units, by C”

17. A boolean verdict is not enough
for trace diagnostics!

18. Project Context

19. Centre des Technologies de l'Information de l'Etat
CTIE
!19

20. Model-driven Engineering
at CTIE
!20
Level%1
1b.%Function%Tree
A"Function"Tree"
provides"a"structured"
inventory"of"all"activities,"
as"well"as"a"view"of"
Credit: Centre des Technologies de l'Information de l'Etat

21. Collaborative Project Topics
• Problem: Compliance of business processes with respect to
their specifications
• Technique: Run-time verification - trace checking
!21

22. Requirements of the solution (1)
• To be viable in the long term, any solution shall rely on standard MDE
technology (tools implementing OMG specifications)
• Why?
• our industrial partner has adopted MDE in practice
• requires any software solution added to the development process to
adhere to OMG specifications and rely on the corresponding tools
• It can be generalized to other contexts in which solutions have to be
engineered by using standard MDE technologies
!22

23. Requirements of the solution (2)
• Any solution shall:
• scale linearly with respect to the length of the trace
• complete within practical time limits
• e.g., a trace with millions of events should be processed within
seconds
• to enable real-time log analysis, to promptly detect violations
!23

24. Model-driven Trace Checking of
Pattern-based Temporal Properties
!24
Eclipse OCL
Trace
Properties
to check
Trace
Instance
Properties
Instances
OCL Constraints
on Trace Class
TemPsy-Check
MODELS 2017

25. Model-driven Trace Checking of
Pattern-based Temporal Properties
• TemPsy-Check supports
properties expressed in
TemPsy, a pattern-based
specification language
• Extended version of
Dwyer et al.’s
specification patterns
(SCOPE + PATTERN)
!25

26. TemPsy Example
!26
“Once a card request is approved, the applicant is notified
within three days; this notification has to occur before the
production of the card is started.”
(Before + Response)

27. !27
temporal R1:
before ICM.issueCard
ICM.notifyApproval responding at most 3*24*3600 tu
ICM.approveRequest
“Once a card request is approved, the applicant is notified
within three days; this notification has to occur before the
production of the card is started”
TemPsy Example

28. !28
temporal R1:
before ICM.issueCard
ICM.notifyApproval responding at most 3*24*3600 tu
ICM.approveRequest
“Once a card request is approved, the applicant is notified
within three days; this notification has to occur before the
production of the card is started”
TemPsy Example

29. The Trace Diagnostics Problem
for TemPsy-Check
• TemPsy-Check yields
boolean verdicts!
!29

30. Goal of this Work
• To develop a practical and scalable solution for the trace
diagnostics problem
• in the settings of model-driven trace checking
• for pattern-based temporal properties
!30

31. Three Ingredients for
Trace Diagnostics
!31
characterization of
property
violations
Credit: otogestoeber/Shutterstock Credit: charc-concepts.org/James Rofus
collection of
diagnostics
information
visualization of
diagnostics
information

32. Characterization of
TemPsy Violations

33. Eight Types of TemPsy Violations
• NSOC (No-Show OCurrence)
• UNOC (UNexpected OCcurrence)
• NSOR (No-Show ORder)
• WTO (Wrong Temporal Order)
• WTC (Wrong Temporal Chain)
• WTOC (Wrong Temporal Order and Chain)
• LVRI (Left Valid and Right Invalid)
• LIRV (Left Invalid and Right Valid)
!33
Universality
Absence
Response
Precedence
{
Existence

34. NSOC (No-Show OCcurrence)
!34
event b b b a b b b b b b
timestamp 5 15 20 30 35 40 45 50 60 80
eventually at least 2 a

35. UNOC (UNexpected OCcurrence)
!35
event a b b a b b a b b a
timestamp 5 15 20 30 35 40 45 50 60 80
eventually at most 2 a

36. NSOR (No-Show ORder)
!36
event c c c c c c b c a b
timestamp 5 15 20 30 35 40 45 50 60 80
a preceding b

37. WTO (Wrong Temporal Order)
!37
event b a b c a a a b c a
timestamp 5 15 20 40 42 43 45 50 58 80
a, b preceding at most 10 tu c
>10 tu

38. WTC (Wrong Temporal Chain)
!38
a, #at least 10 tu b preceding c
<10 tu
event b a b c a a a b c a
timestamp 5 15 20 40 42 43 45 50 58 80

39. Model-driven
Trace Diagnostics
of TemPsy Properties

40. Model-driven Trace Diagnostics
!40
Eclipse OCL
Trace
TemPsy
Properties
Trace
Instance
TemPsyExpression
Instances
OCL Queries
on Trace Class
TemPsy-Report

41. Trace Meta-Model
!41
Trace TraceElement
event: String
timestamp: Integer
*
properties:
Elist<TemPsyExpression>
traceElements

42. Simplified Template of
OCL Queries
let subtraces:Sequence(Tuple(begin:Integer, end:Integer)) = applyScope*S*(scope)
in subtraces->iterates(subtrace:Tuple((begin:Integer, end:Integer));
result:Sequence(Tuple((begin:Integer, end:Integer, violations:OclAny)))
= Sequence{} |
let v:OclAny = reportPattern*P*(subtrace, pattern) in
if v->notEmpty() then
result->append(Tuple{
begin:Integer = subtrace.begin,
end:Integer = subtrace.end,
violations:OclAny = v})
else result endif)
!42
compute sub-traces
based on scope
collect diagnostics
based on pattern
save diagnostics
information

43. reportPattern*P*
43
*P*
Universality
Existence
Absence
Response
Precedence{

44. Algorithm 1: reportPatternUniversality
Input: begin, end: the boundaries of a sub-trace;
pattern: an instance of the universality pattern
of the form “always E”
Output: result: a list of locations at which violations
occur
1 E event name in pattern
2 result {}
3 for i begin to end do
4 if self.traceElements[i] 6= E then result.append(i)
5 return result
Existence. As shown in Table I, according to the number
of occurrences of the event specified in an existence pattern,
with r
event
(line 1
or “ex
to col
at line
thresho
or “ex
and th
a 2-tu
type.
Abs
specifi
E”, in
of eve
the tw
event a a a a a a b a a b
timestamp 5 10 25 30 35 40 45 50 60 80
“globally always a”
NSOC
!44
NSOC

45. TemPsy-Report
http://weidou.github.io/
TemPSy-Report/
!45

46. Visualization of
Diagnostics Information

47. Requirements of a
Trace Diagnostics Visualization Tool
• User-friendly navigation of traces
• Easy access to violations
• Support for understanding violations
!47

48. mark out the locations of violations as well
as the relevant data points and intervals
!48
Card.isReturned
ICM.notifyCardReturned
globally ICM.notifyCardReturned responding at most 24 tu Card.isReturned

49. Highlight the violated portion of
a TemPsy property
!49

50. Evaluation of the Scalability
of TemPsy-Report

51. Research Questions
RQ1) What is the relation between the execution time of
TemPsy-Report and the length (number of logged events) of a
trace?
RQ2) What is the relation between the execution time of
TemPsy-Report and the number of violations (with respect to a
property) contained in a trace?
!51

52. Experiment Results RQ1
Averageexecutiontime
3s
6s
9s
Trace length
100k 200k 300k 400k 500k 600k 700k 800k 900k 1000k
p1 p2 p3 p4 p5.nsor
p5.wto p6.nsor p6.wto p7.nsor p7.wto
p8.nsor p8.wto p9.nsor p9.wto p10.nsor
p10.wto p11.nsor p11.wto p12.nsor
!52
(fixed number of violations: 1000)

53. Scalability (trace length)
• TemPsy-Report scales
linearly with respect to the
trace length
• execution time ranges from
about 1.5s to 8.2s
!53
Averageexecutiontime
3s
6s
9s
Trace length
200k 400k 600k 800k 1000k

54. Experiment Results RQ2
Averageexecutiontime
3s
6s
9s
Number of violations
1k 2k 3k 4k 5k 6k 7k 8k 9k 10k
p1 p2 p3 p4 p5.nsor
p5.wto p6.nsor p6.wto p7.nsor p7.wto
p8.nsor p8.wto p9.nsor p9.wto p10.nsor
p10.wto p11.nsor p11.wto p12.nsor
!54
(fixed trace length: 1 million)

55. Scalability (# violations)
!55
Averageexecutiontime
3s
6s
9s
Number of violations
1k 2k 3k 4k 5k 6k 7k 8k 9k10k
• The number of violations
contained in a trace makes no
tangible impact on the execution
time of TemPsy-Report
• execution time ranges from
about 3.8s to 8.2s

56. Summing up

57. !57

58. .lusoftware verification & validation
VVS
Model-driven trace diagnostics
for pattern-based
temporal specifications
bianculli@svv.lu