Nesma Autumn Conference 2017 - presentation Euro Project Office

1. Thomas Fehlmann, Eberhard Kranich
Euro Project Office, Zürich & Duisburg
NESMA Autumn Meeting, Soestduinen, 7th November 2017

2. Speaker & Authors

3. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

4. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
Test Metrics
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

5. Bad Mathematics in Software Testing
▪ What is the Size of Software?
▪ Lines of (undocumented?) Code?
▪ What is a Software Defect?
▪ An entry in a bug tracking system??
▪ What is a Defect Density?
▪ Number of bug entries per line of
code in a bug tracking repository????
What Defect
Density has this
Google Car’s
Software?
Compared to
Nissan’s Software?
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

6. Testing refers to Functionality not Code
▪ Code is not the object of testing
of software or systems
▪ Test metrics refer to
functionality
▪ Test metrics cannot refer to
code
▪ Most code is open source
▪ Well code-tested by Daily Builds
& JUnit’s Green Bar
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#include "mbed.h"
DigitalIn button1( SW2 ); // Right Button on ARM Board
DigitalOut led( LED1 );
int main()
{
led = 1; // red (RGB LED use inverse logic
// 1 = OFF, 0 = ON)
while ( true )
{
if ( button1 == 0 ) // Button pressed
led = 0;
else
led = 1;
}
}

7. Software Models
▪ Modeling Software by
▪ ISO/IEC 20926 IFPUG
▪ ISO/IEC 19761 COSMIC
▪ Others….
▪ Sizing Software according
▪ Functionality & Test Intensity
▪ Non-functional Characteristics
▪ Privacy
▪ Safety
▪ Other constraints
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

8. The IFPUG Model of Software (ISO/IEC 20926)
▪ Transactions
▪ EI: Elementary Input
▪ EO: Elementary Output
▪ EQ: Elementary Query
▪ Data Elements
▪ ILF: Internal Logical Files
▪ EIF: External Logic Files
Software Boundary
ILF
User
(Person or application)
EIF
EI EO EQ
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

9. The IFPUG Model of the Employee Database
▪ Transaction Map
▪ 4 EI, 1 EO, 1 EQ
▪ 2, 12 and 13 DET
▪ 9 FTR
▪ 1 ILF, 1 EIF
▪ 3 and 12 DET
▪ For Test Size, we need sizing parts
▪ IFPUG Counting Rules are not
compliant to the VIM and the GUM
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
T001 T003 T005 T006
T002
T004
D002 D001
BoundaryIFP=33
ILF
12 / 1
Employee Data
EIF
3 / 1
Active Directory
EI
13 / 1
Add Employee
EQ
12 / 2
View Employees
EI
13 / 1
Merge Employees
EO
12 / 2
Weekly Report
EI
13 / 1
Update Employee
EI
2 / 2
Terminate Employee

10. The COSMIC Model of Software (ISO/IEC 19761)
▪ Functional
Processes
▪ Devices & Applications
▪ Entry and eXit
▪ Persistent Data
▪ Read and Write
▪ Data Movement
▪ Moves Data Groups
▪ Trigger triggers a functional process
Software Boundary
Functional
Process
Persistent Data
Trigger
Entry
Entry
eXit
eXit
Write Read
Device User Application User
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

11. The COSMIC Model – Data Movement Map
▪ Data Movement Map
▪ One object per functional process
▪ It has six Functional Processes
▪ Add Employee
▪ Update Employee
▪ Merge Employees
▪ View Employee
▪ Weekly Report
▪ Terminate Employee
▪ 31 Data Movements yields 29 CFP
▪ Some data movements move
identical data groups
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
9 Entry (E) + 11 eXit (X) + 5 Read (R) + 4 Write (W) = 29 CFP
HR User Add New Employee Update Employee Merge Employees View Employee Weekly Report Terminate Employee Employee Database Active Directory
1.// Add Employee
Add
2.// Get Employee ID
3.// Write Employee Data
4.// Show Employee ID
5.// Confirm Transaction
6.// Update Employee
Update
7.// Read Employee Data
8.// Write Employee Data
9.// Confirm Update
10.// Select Employee 1
Merge
11.// Read Data for Employee 1
12.// Select Employee 2
13.// Read Data for Employee 2
14.// Show Data for Merge
15.// Select Data to Merge
16.// Consolidate Merge
17.// Confirm Merge
18.// Request Employee Data
View
19.// Get Employee Data
20.// Read Access Rights
21.// Show Employee Data
22.// Confirm Data Retrieval
23.// End of Week
End of Week
24.// Get Data for Weekly Report
25.// Read Access Rights
26.// Weekly Report
27.// Confirm Report
28.// Terminate Employee
Terminate
29.// Block IT Access
30.// Terminate Employment
31.// Confirm Termination

12. Visualizing Software Testing
▪ Tester sees selected sequences in the
Data Movement Map
▪ Tester can “walk” the data movements
when planning or executing tests
▪ Makes functionality visible to the
development team
▪ Localizes defects that impact
functionality
▪ Supports communication between
testers, users, and developers
Functional
Process
Other
Application
Some
Device
8.// Move some data
9.// Move some data
10.// Move some data
11.// Move some data
Other
Device
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

13. Functionality, Defect Size, and Defect Density
▪ What happens if data movements
don’t work as expected, have defects
instead?
▪ Testers mark and count data
movements where defects have been
detected
▪ One Size Metric:
▪ ISO/IEC 19761
COSMIC
Functional
Process
Other
Application
Some
Device
8.// Move some data
Move some data
10.// Move some data
11.// Move some data
Other
Device
Test Size = 4
Defect Count = 1
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
 Functional Size
 Number of Data Movements needed to implement required functionality
 Test Story
 Collection of Test Cases aiming at certain functionality
 Test Size
 Number of Data Movements executed in Tests
 Defect Count
 Number of Data Movements affected by some defect detected in a Test Story

14. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

15. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
Privacy Metrics
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

16. Current Approach in Assessing Security
▪ The Center for Internet Security (CIS) presents
the CIS Controls for Effective Cyber Defense
Version 6.0, a recommended set of actions
that provide specific and actionable ways to
stop today's most pervasive and dangerous
cyber attacks
▪ The CIS Security Benchmarks program provides
well-defined, un-biased and consensus-based
industry best practices to help organizations
assess and improve their security
▪ https://www.cisecurity.org
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

17. Critique at Current Approaches to Security
▪ This is not software security but
system security
▪ Assessment concerns a system
as a whole
▪ Cyber defense must start at the
component level
▪ Where is the system as a whole in
the Web of Things?
▪ The Web of Things changes
equally fast as requirements in
agile software development
The
Traditional
Approach
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

18. How to Measure Privacy for Software?
▪ Privacy Value
▪ Value = 0: No privacy. It’s public.
▪ Value = 1: Disclosure is inconvenient
▪ Value = 2: Disclosure can be harmful
▪ Value = 3: Disclosure costs money
▪ Value = 4: Disclosure makes guilty
▪ Value = 5: Disclosure threatens lives
▪ Public Exposure
▪ Value = 0: No encryption. It’s public.
▪ Value = 1: Weak encryption
▪ Value = 2: Strong encryption
▪ Value = 3: Two-way encryption
▪ Value = 4: Data never leaves system
▪ Value = 5: Computer-internal data
▪ The principle for Each Model Element is
Privacy Protection = Privacy Value * Public Exposure
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

19. 𝑃𝑟𝑖𝑣𝑎𝑐𝑦 𝐼𝑛𝑑𝑒𝑥 =
𝑃𝑟𝑖𝑣𝑎𝑐𝑦 ∗ 𝐸𝑥𝑝𝑜𝑠𝑢𝑟𝑒 + 1 − 1
26 − 1
∗ 5
The Privacy Index
▪ The product
Privacy Value * Public Exposure
is in the range 0 to 25
▪ Privacy Index is range 0 ⋯ 5
▪ Five 5 is the index for
maximum privacy
▪ Zero 0 privacy means public data
▪ The Privacy Index range is on a
polynomial scale between zero and five
Zero = No Privacy
All is open
Five = Full
Privacy Protection
0
.
.
.
.
5
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

20. The Privacy Index
▪ For contract purposes,
use a graphical
representation
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
4
3
2
1
0
5
Privacy Protection
ImpactonPrivacy
Data Movement
Low Privacy Index
Medium Privacy Index
High Privacy Index
Major Privacy Index
No Privacy Needed
1235 4 0
0.9
Index 0.9: W004Terminate Employment
Index 1.2: X009 Confirm Report
Index 1.5: R001 Get Employee ID, +8
Index 1.8: X011 Confirm Termination, +1
Index 2.0: X007 Confirm Data Retrieval
1.5
1.21.5
4.4
2.0
Index 2.4: E001 Add Employee , +11
Index 3.2: X005 Confirm Merge
Index 4.4: R002 Read Employee Data, +12.43.2

21. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

22. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
Safety Metrics
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

23. Safety for Software
▪ Software impacts
physical world
▪ Level 1: Low – break
▪ Level 3: Medium
▪ Level 5: High – crash
▪ Probability Levels
▪ Less likely on motorways
▪ Highly probably in villages
and urban areas
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

24. Measure Safety Risk
▪ Impact through actuators
▪ Steering Wheel in Autonomous Cars
▪ Closed Door at Smart Homes
▪ Indicated by an index 1 ⋯ 5
▪ Other classical software risks remain:
▪ Causing losses for business
The principle for Each Model Element is
Safety = Probability * Impact
0
.
.
.
.
5
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
Zero = All Safe!
Highly unlikely
Five = Danger!
Safe Our Souls!

25. What means Safety for Software?
▪ Impact Levels
▪ Level 1: Low
▪ Level 3: Medium
▪ Level 5: High
▪ Probability Levels
▪ By percentages
1
2
3
4
5
Probability
ImpactonSafety
Data Movement
Major Safety Risk
High Safety Risk
Medium Safety Risk
Low Safety Risk
Risk Incurred/ Feature
100%80%60%20% 40%0%
3.2
Index 3.2: E010 Terminate Employee
2.0
Index.2.0: X010 Block IT Access
Index 1.2: W004Terminate Employment
0.6
Index 0.8: X011 Confirm Termination
Index 0.6: X008 Weekly Report, +24
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
0.8
1.2

26. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

27. Agenda
Test Metrics
Privacy Metrics
Safety Metrics
Metrics in Contracts
Metrics in Contracts
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

28. Privacy of Data Movements
▪ How private is a
Data Movement?
▪ Can it be intercepted?
▪ Not really if machine-internal
▪ Quite easy if HTTPS
▪ Less easy with two-way
authentication
▪ Most difficult with
private key encryption
▪ ISO/IEC 19761 COSMIC provides
a model for measuring privacy
that is well suited for distributed
systems such as an IoT Concert,
Embedded Systems, or Mobile
Apps
Software Boundary
Functional
Process
Persistent Data
Trigger
Entry
Entry
eXit
eXit
Write Read
Device User Application User
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

29. Name Label Data Movement Sub-Process Description
1) E001 Add Employee Enter new employee data
2) R001 Get Employee ID Unique ID for employee identification
3) W001 Write Employee Data Store in database
4) X001 Show Employee ID Confirm new employee added
5) X002 Confirm Transaction Confirmation or error message
Data Movements
23) E008 End of Week Timing Signal
24) R006 Get Data for Weekly Report Select employee data that changed during past week
25) E009 Read Access Rights Get external IT access rights
26) X008 Weekly Report All data including termination records
27) X009 Confirm Report Weekly report done
28) E010 Terminate Employee Enter termination control flag
29) X010 Block IT Access Block access to IT
30) W004 Terminate Employment Update an employee record
31) X011 Confirm Termination Confirmation or error message
Add Row Ins Row Del Row Validate Extract SNAP
1.22
Effectwhen
PrivateDatais
Disclosed
Privacy
Protection
Level
Privacy
Probability
Impacton
Safety
Safety
4: Makes guilty 2: Strong 2.4 10% 2: Little 0.2
4: Makes guilty 1: Weak 1.5 5% 2: Little 0.1
4: Makes guilty 2: Strong 2.4 10% 2: Little 0.2
4: Makes guilty 1: Weak 1.5 5% 2: Little 0.1
4: Makes guilty 2: Strong 2.4 10% 2: Little 0.2
4: Makes guilty 1: Weak 1.5 10% 2: Little 0.2
4: Makes guilty 1: Weak 1.5 20% 2: Little 0.4
4: Makes guilty 1: Weak 1.5 5% 4: Quite 0.2
2: Harmful 3: Two-way 2.0 20% 3: Medium 0.6
1: Inconvenient 3: Two-way 1.2 10% 3: Medium 0.3
1: Inconvenient 4: Enclosed 1.5 80% 4: Quite 3.2
1: Inconvenient 5: Internal 1.8 50% 4: Quite 2.0
1: Inconvenient 2: Strong 0.9 30% 4: Quite 1.2
1: Inconvenient 5: Internal 1.8 40% 2: Little 0.8
2.4 0.2
0.9 3.2
Safety Index:
Minimum Privacy: Maximum Risk:
Privacy Index:
Assessing Privacy & Safety
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
Assessing
Implementation
Assessing
the Model

30. Privacy of Data Functions & Transactions
▪ How private is Data / are
Transactions?
▪ Can it be accessed?
▪ Not really if machine-internal
▪ Quite easy with password
▪ Less easy with two-way
authentication
▪ Most difficult with
private key encryption
▪ ISO/IEC 20926 IFPUG provides a
model for measuring privacy
that is well suited for web pages
and transactional systems
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
Software Boundary
ILF
User
EIF
EI EO EQ

31. Assessing Privacy & Safety
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS
Name Label Description of Elementary Data Function ILF EIF Type
1) D001 Employee Data A persistent logical entity maintained by the application 1) ILF
2) D002 Active Directory The active directory provides the unique username and the account list where he or she has access1) EIF
1
EIF: 1
Elementary Data Functions
ILF:Add Row Ins Row Del Row Validate Extract SNAP
1.22
Effectwhen
PrivateDatais
Disclosed
Exposureto
Privacy
Violation
Privacy
Probability
Impacton
Safety
Safety
4: Makes guilty 2: Strong 2.4
3: Costs money 3: Two-way 2.6 5% 1: Low 0.1
SNAP 2.4 0.2
0.9 3.2
Privacy Index: Safety Index:
Minimum Privacy: Maximum Risk:
Name Label Description of Elementary Transaction Type
1) T001 Add Employee Primary intent is to maintain data in the ILF EI
2) T002 View Employees Primary intent is to present data to the user, data is retrieved both from an ILF and an EIF, and there are none of the following involved: calculations, derived data, ILF update or aEQ
3) T003 Merge Employees Primary intent is to maintain data in the ILF EI
4) T004 Weekly Report Show total employment status with liabilities by the salaries EO
5) T005 Update Employee Primary intent is to maintain data in the ILF EI
6) T006 Terminate Employee Stops access to the IT and terminates employment EI
EI: 4
EO: 1
EQ: 1
Elementary Transactions
Add Row Ins Row Del Row More FTR Validate Extract SNAP
1.22
Effectwhen
PrivateDatais
Disclosed
Exposureto
Privacy
Violation
Privacy
Probability
Impacton
Safety
Safety
4: Makes guilty 2: Strong 2.4 10% 2: Little 0.2
4: Makes guilty 4: Enclosed 3.8 10% 3: Medium 0.3
4: Makes guilty 2: Strong 2.4 20% 1: Low 0.2
4: Makes guilty 3: Two-way 3.2 10% 4: Quite 0.4
4: Makes guilty 2: Strong 2.4 10% 2: Little 0.2
1: Inconvenient 2: Strong 0.9 80% 4: Quite 3.2
2.4 0.2
0.9 3.2
21
Maximum Risk:
Privacy Index:
Minimum Privacy:
Safety Index:

32. Metrics for Software Contracts
▪ Functional Size
▪ COSMIC
▪ IFPUG, NESMA
▪ …
▪ Test Size
▪ Test Intensity
▪ Acceptable Defect Density
▪ Privacy Index & Minimum Privacy
▪ For Data Movements
▪ For Data Functions &
Transactions
▪ Security Index & Maximum Risk
▪ For Data Movements
▪ For Data Functions &
Transactions
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

33. Metrics for System Contracts
▪ Hardware Vulnerability
▪ Software might destroy itself
when Hardware is
compromised
▪ Functional Size
▪ COSMIC
▪ IFPUG, NESMA
▪ Test Size
▪ Test Intensity
▪ Acceptable Defect Density
▪ Privacy Index & Minimum Privacy
▪ For Data Movements
▪ For Data Functions &
Transactions
▪ Security Index & Maximum Risk
▪ For Data Movements
▪ For Data Functions &
Transactions
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

34. Conclusions
▪ Contracts for safety-critical software must specify how well we protect
privacy and how safe the software behaves
▪ Based on a functional size model
▪ Contracts without specifying test metrics are dangerous and bound to
later arbitration
▪ Currently, test metrics work with ISO/IEC 19761 COSMIC only
▪ Make ISO/IEC 20926 & 24570 compliant with the VIM and the GUM
▪ Consumers need such test metrics, privacy and safety metrics when
purchasing software or autonomous systems
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

35. Logos Press
Berlin 2016
Questions?
| METRICS FOR PRIVACY & SAFETY IN CONTRACTS

36. | METRICS FOR PRIVACY & SAFETY IN CONTRACTS
New Book on Six Sigma Transfer Functions