How to critically evaluate performance when the primary purpose of intelligence is to inform decision-making to improve its outcomes, and not to justify accuracy or productivity of intelligence deliverables.

1. MEASURING THE VALUE
OF CI
Avner Barnea
Lecturer on Competitive Intelligence
Western Galilee Academic College, Israel
Brig. Gen. ISA (Ret.), Israel
1
avnerpro@netvision.net.il8-Nov-14
Barnea on Measurement
19th Annual SCIP European Summit SCIP Conference
3-5 November 2014, Amsterdam

2. Evaluate intelligence performance
How to critically evaluate performance when the
primary purpose of intelligence is to inform
decision-making to improve its outcomes, and not to
justify accuracy or productivity of intelligence
deliverables.
2
8-Nov-14Barnea on Measurement

3. Why measuring is needed?
 Benefit of measuring intelligence is twofold:
1. To improve practice
2. Investment in intelligence
 Initiated by senior management. Not CI professionals!
 In national intelligence measurement tools are not
developed but it is measured heavily in cases of
intelligence failures mainly by investigation
commissions.
 In national intelligence success stories are rarely
available to the public, contrary to intelligence success
in the business sector.
3
8-Nov-14Barnea on Measurement

4. Methodology 1 - Major challenges
 Access to decision-makers;
 Subjectivity of data/ information;
 Issues of quantification, representation, and
accuracy of information;
 Time delay needed for outcomes of improved
decision-making to manifest;
 Accounting for other factors at play, to link
intelligence to outcomes and effect relationship
between CI and decision makers.
Must intelligence measurement be accurate?
4
8-Nov-14Barnea on Measurement

5. Methodology 2: Measurement….
 Measurement has been traditionally defined as “to
count”.
 Trying to measure “intangible” – i.e. reputation,
trust of intelligence outputs….
 Also indirectly evident to the observer….
 Scales that can estimate error?....
Subjective vs. Objective
Qualitative vs. Quantitative
5
8-Nov-14Barnea on Measurement

6. Methodology 3: Obtaining from
national intelligence to business
6
 System metrics – measure technology (software
systems) in use
 Output metrics – measure process or
product(customer satisfaction of deliverables:
accuracy, timeliness, usefulness/ value)
 Outcome metrics – measure effects on the
organization
In business – quality of information and the quality of information sources are more
important than timing while it depends on OSINT.
Subjective vs. objective….Qualitative vs. quantitative
8-Nov-14Barnea on Measurement

7. Methodology 4: Considerations
7
 What is to be measured?
 For whom?
 For what purpose?
Return on Investment ROI = usually financial figures
Balanced Score Card BSC = account for tangible
Measures of Effectiveness MoE = account for intangible
ROCI=return on CI
What is to be measured – added value of intelligence
For whom – for CI, and others including decision makers
For what purpose – to improve CI contribution
8-Nov-14Barnea on Measurement

8. What are we measuring?
9
 Scale 1 –Standard of demand for intelligence
 Scale 2 – Standard of supply of intelligence
 Scale 3 – Level of process
 Scale 4 – Organizational culture
Better added value?
8-Nov-14Barnea on Measurement

9. Scale 1: Demand for intelligence
Standard of demand by internal
customers – how much is demanded,
awareness to the value of intelligence in
decision making process, demand for
continuous activities and for special
projects, demand in the annual work plan.
10
8-Nov-14Barnea on Measurement

10. Scale 2: Supply of intelligence
Standard of supply - service
granted to internal customer -
satisfaction of customers – quality,
timely, range of coverage, quantity
of deliverables, feed back received,
cost of production.
11
8-Nov-14Barnea on Measurement

11. Scale 3: Internal intelligence process
How much CI became a tool used by
the management team? How much it
has been established in the decision-
making process?
12
8-Nov-14Barnea on Measurement

12. Scale 4: Organizational culture of intelligence
The strength of sharing of
information, expert tools used for
this purpose, the use of internal
experts for analysis, the use of CI
software.
13
8-Nov-14Barnea on Measurement

13. Parameters & Issues for measuring
Quantitative/ HardQualitative/ Soft
On going
Periodical
14
Organizational cultureProcessSupplyDemand
1. Using internal
experts
2. Sharing of info.
3. Internal discourse
4. Using reports
5. Embedded in
training programs
1. Involve in
decision-
making
2. Effectiveness
3. By procedures
4. Cost of
production
1. Customer
satisfaction
2. No. reports
3. Quality of
reports
4. Speed of
response
5. Access to info.
6. Update of info.
1. Reply to need
2. Present
intelligence
needs (KITs)
3. No. of
requests
4. Volume of use
of dedicated
system
5. No. of CI users
Parameters
Issues
8-Nov-14Barnea on Measurement

14. Questions and parameters/issues -example
15
Quantitative/ HardQualitative/ Soft
On goingQ 1: How timely are CI
reports?
Periodical
On goingQ 2: How well you get
answers to your KIT’s?
Periodical
On goingQ 3: How many significant
alerts have you received?
Periodical
On goingQ 4: How much CI is
involved in strategic
decisions?
Periodical
8-Nov-14Barnea on Measurement

15. Questions by parameters/ issues
Demand Supply Process Culture
1 2 3 4
12 5 8 7
6 9 10 11
16
8-Nov-14Barnea on Measurement

16. Questions and replies
17
0
50
100
150
200
250
300
350
400
450
seldomoftenas neededno replytotal
1234
0
50
100
150
200
250
300
350
400
450
do not
know
a littlemediumwellvery wellno replytotal
12345
Series1
Series2
Series3
How timely are CI reports?
How well you get answers
to your KIT’s?
8-Nov-14Barnea on Measurement
supply
demand

17. Questions and replies
18
0
50
100
150
200
250
300
350
400
450
01234more
than 5
‫סה‬"‫כ‬
12345
Series1
Series2
Series3
How many significant alerts
have you received?
0
50
100
150
200
250
300
350
400
450
Neveroftenalwaysno replytotal
123
Series1
Series2
Series3
How much CI is involved in
strategic decisions?
8-Nov-14Barnea on Measurement
supply
Process

18. Combined analysis
Based on the following information:
1. Output metrics – Questionnaire 12 questions, 4 issues
2. Output metrics - Accumulation of continuous feedback.
Analysis of feedback
3. Outcome metrics - Interviews with key people
15 interviews: management & focal users
4. System metrics- Information gathered from CI expert system
(how many read, how many comment, how many asking for
more info etc.) Gathering software info, statistical rate
of use.
19
8-Nov-14Barnea on Measurement

19. Final Comments
 Measuring CI value is a must internal process.
 To be initiated by CI director.
 Supported by CI Champion.
 Results to be distributed/ published.
20
8-Nov-14Barnea on Measurement

20. MEASURING THE VALUE OF CI
Avner Barnea
Lecturer on Competitive Intelligence
Western Galilee Academic College,
Brig. Gen. ISA (Ret.), Israel
avnerpro@netvision.net.il
21
8-Nov-14
Barnea on Measurement