The document describes the CRISP methodology for evaluating video surveillance systems. It involves four main steps: 1) an initial configuration process where details of the system and use cases are specified, 2) an assessment where experts answer questions regarding the security, trustworthiness, efficiency, and freedom dimensions, 3) identifying any conflicts between dimensions, and 4) resolving conflicts in order to achieve certification. The goal is to systematically assess systems, involve stakeholders early, and transparently address any issues found between criteria to help ensure privacy and ethics are protected.

1. CRISP Evaluation
16th of March 2017, CoU Meeting, Brussels
Dr Nathalie Hirschmann
(Center for Technology and Society – Technische Universität Berlin)

2. EVALUATION
CRISP Methodology
Source: Deliverable 5.2. February 2016
Information provider
Audit and
Inspection
review &
decision
Auditor (third party)
EVALUATION CERTIFICATION
SurveillanceAttestation
S-T-E-Fi
Assessment
Configuration
selection and
determination
R3R2
R1
2

3. Systematisation of
criteria
Video surveillance
3
performance
accuracy
robustness
fair distribution
accountability
withdrawal mechanisms
end user safety
system interference
risk
user error protection
environment
awareness
transparent use
feeling of uneasiness
physiological invasiveness
reliability
maintenance resources
usability
ethical codes
good practice/safety codes
documentation protective measures to ensure efficiency
usability
user manual
customisation
interoperability
lifecycle costs
energy efficiency
personal data
prohibition of discrimination
due process

4. Systematisation of
criteria
Video surveillance
4
performance
accuracyrobustness
SECURITYfair distribution
accountability
withdrawal mechanismsend user safety
system interference
risk
user error protection
environment
awareness
transparent use
TRUST
feeling of uneasiness
physiological invasiveness
reliability
maintenance resources
usability
ethical codesgood practice/safety codes
documentation
protective measures to ensure efficiency
usability
user manual
customisation
EFFICIENCYinteroperability
lifecycle costs
energy efficiency personal data
prohibition of discrimination
due process
FREEDOM
INFRINGEMENT

5. Video surveillance
5
SECURITY TRUST
EFFICIENCY
INFRINGEMENT
FREEDO
M
S-T-E-Fi experts

6. ‘Information provider’
S-T-E-FI ASSESSMENTCONFIGURATION
R1 R2
C
E
R
T
I
F
I
C
A
T
I
O
N
10
Evaluation process
Video surveillance

7. ‘Information provider’
S-T-E-FI -ASSESSMENTCONFIGURATION
R2
1. Specification of the
security area
Client informs about:
2. Detailed technical
specifications of
the system
3. Basic information
on at least three
application
scenarios
11
Evaluation process –
configuration
Video surveillance
1. Reliable answering
of all assessment
questions by the
project leader and
‘appointed experts’
2. Identifying conflicts
between and within
S-T-E-Fi
dimensions
3. Conflict resolution
R 1

8. ‘Information provider’
S-T-E-FI -ASSESSMENT
12
Evaluation process – link between
configuration and assessment
Video surveillance
R 1

9. 13
Evaluation process –
assessment
Video surveillance
‘Information provider’
S-T-E-FI -ASSESSMENT
R1
1. Reliable answering
of all assessment
questions by the
project leader and
‘appointed experts’
2. Identifying conflicts
between and within
S-T-E-Fi
dimensions
3. Conflict resolution
R 2

10. 14
Assessment: report R2
R 2
 Summary of how the process was conducted
 Range of stakeholders involved
 Information gathered during configuration
 Selected assessment questions
 Results of the assessment questions
 Type and number of identified conflicts
 Conflict resolution
 Potentially remaining, unsolved conflicts

11. ‘Information provider’
S-T-E-FI -ASSESSMENTCONFIGURATION
R1 R2
1. Specification of the
security area
Client informs about:
2. Detailed technical
specifications of
the system
3. Basic information
on at least three
application
scenarios
C
E
R
T
I
F
I
C
A
T
I
O
N
15
From evaluation to certification
Video surveillance
1. Reliable answering
of all assessment
questions by the
project leader and
‘appointed experts’
2. Identifying conflicts
between and within
S-T-E-Fi
dimensions
3. Conflict resolution

12. 18
Assessment criteria –
example
“Yes” or “No”
Is the personnel trained
to increase public
awareness on the
possible impacts of the
system?
What kind of training
does the personnel
receive and how often?
QUESTIONS:
T R U S T
Awareness, Training
e.g. “Our personnel
is trained on the
system in the field
every year. This is
recorded in the
personnel files.”

13. 19
Conflict identification –
example
RESPONSE
OPTIONS
NO YES
NO C- (0) C+ (1)
YES C- (0) C- (0)
Criterion A: ‘Observability’ (T)
‘Are people constantly observed by the system?’
Criterion B:
‘Transparency’ (T)
‘Is the system clear
on what it offers?’
C – (0): no conflict; C+ (1): conflict; needs to be
addressed.

14. 24
Identification of conflicts

15. 25
Conflict resolution
1. Ensuring compliance with applicable legal
requirements
2. Implementation of technical changes to the
security system and/or implementation of
changes to the applied operating procedures
3. Negotiating a solution by following a practical
and legitimate decision-making process

16. participation & systematisation:
enabling/encouraging early
stakeholder participation;
assessment criteria are
systematised according to four
dimensions
systemic view: transferring
assessment criteria into a
matrix structure; identifying
conflicts between/within
dimensions and making them
visible
conflict resolution: indicating
identified conflicts which need
to be solved to involved parties
in order to move on with
certification
S
Fi
T
E
27
CRISP’s approach

17. 28
For further questions, please contact:
Dr. Leon Hempel
Zentrum Technik und Gesellschaft – TU Berlin
Hardenbergstr. 16-18
10623 Berlin – Germany
hempel@ztg.tu-berlin.de