Tesi di laurea triennale in Ingegneria Elettronica e Informatica

1. UNIVERSITÀ DEGLI STUDI DI TRIESTE
Dipartimento di Ingegneria e Architettura
Corso di Laurea in
Ingegneria Elettronica e Informatica
Extended summary of
“A different cup of TI?
The added value of commercial threat intelligence”
Anno Accademico 2020/2021
Candidata
Alessandra Amato
Relatore
Chiar.mo Prof.
Alberto Bartoli

2. 1
Table of Contents
1. Introduction......................................................................................................... 2
2. Methodology........................................................................................................ 2
3. Paid TI services and pricing................................................................................ 3
4. Comparison......................................................................................................... 3
5. Use and perceived value of TI............................................................................. 5
5.1 Application of TI........................................................................................... 5
5.2 Value perception............................................................................................ 5
6. Conclusions......................................................................................................... 6
Bibliography....................................................................................................... 6
Riassunto in lingua italiana................................................................................. 7

3. 2
1. Introduction
Threat intelligence, also known as TI, refers to the knowledge about attackers’ behavior and techniques,
that help identify the risks and improve the defenses from cyber-attacks.
Although today's organizations can implement their own TI systems, these are only able to capture a
limited fraction of the threat landscape, thus leading companies to rely on external sources of threat
intelligence.
There are three main sources of TI, accordingly open, shared and paid.
Open TI (OTI) consists of public lists of indicators, in the form of blacklists and abuse feeds, while shared
TI (STI) involves the sharing of threat data among members of formal and informal trusted communities.
The third source, paid TI (PTI), is provided by security firms that offer threat intelligence for sale.
However, the world of commercial TI is still an academically unexplored territory and this absence of
information affects security investments decisions of customers.
This paper aims at giving an insight into the market of TI and shedding light on:
• the services offered by paid TI and price levels involved.
• the differences in the data provided by PTI and OTI sources.
• the perceived value by customers and the use cases of TI into organizations.
2. Methodology
Figure 1. Data sources considered in the research
The first part of the research analyzed the data provided by PTI and OTI sources.
Initially, 3 to 5 subsets, regarding the same area of interest, were collected from two market leaders. The
authors mapped 30% of indicators included on the reports to a common threat actor and, to determine the
overlap between the vendors, compared the ones related to the same actor group.
Then, only the indicators regarding a period of detection from August to December 2018 were compared
to one month of data (October 2018) released by 4 open sources of TI, aiming again at finding the overlap
between the sources.
In the second part of the research, 14 security professionals working with PTI were interviewed about
their definition of cybersecurity threat intelligence, their experience with commercial and non-commercial

4. 3
sources, respectively in terms of costs, use cases and performance of providers, and the motivations for
abandoning certain sources.
Finally, the data collected from the responses was labeled into analytic codes, identifying the most
common answers regarding TI services, the use cases in organizations and clients’ value perception.
3. Paid TI services and pricing
The following paragraph analyzes the services offered by PTI vendors, based on the responses provided.
In addition, the costs of annual subscriptions to paid TI sources are briefly examined.
The findings of the research identified the most popular intelligence products provided by PTI vendors:
• Indicators, also known as indicators of compromise (IOCs), are the proof of presence of an
attacker: pieces of data, such as IP addresses, domains or file hashes, indicating malicious activities
in the organization’s network.
• Reports provide information regarding the attackers’ purposes and the techniques implemented
when attempting to sabotage customers’ informatic systems. This service was described to be
extremely useful, as it allows analysts to give context on events and alerts to their clients.
• Requests for information (RFIs) lead to a deeper analysis of the vendors’ own threat data, to
supply additional information concerning quality or context requirements previously inquired by
customers.
• Portals bring together all the knowledge and the insights that a PTI provider has acquired and
distributed over time into a website.
• Data mining platforms and aggregators collect and organize threat data from multiple sources.
These platforms allow subscribed customers to run queries and share their own PTI information.
• Custom alerts inform clients in the case of a new malware threatening their organization or
whether their software vulnerabilities are being targeted by suspicious actors. However, none of
the two leading vendors contacted provided this particular service.
The reported costs charged for these services ranged from $30,000, for smaller providers, only capable of
aggregating the data drawn from external sources, up to $650,000 for vendors of their own TI. Although
price levels are negotiable and based on the customer’s organization proportions, it was observed that only
businesses with voluminous funds for information security are able to afford PTI.
4. Comparison
In the next paragraph, indicators of PTI and OTI sources are compared to each other, to assess the
differences between TI suppliers. The study was based on the number of overlapping indicators and on
the timeliness in detecting and distributing threat intelligence.
• Overlap
The overlap of indicators from different feeds represents a fundamental aspect, determining the ability of
the sources to capture the threats that specific actors exert.

5. 4
However, as portrayed in Figure 2, by comparing the indicators that the two PTI vendors recorded on
their feeds, a minimal overlap was detected, equal to 1,3% and 13% for Vendor 2 and Vendor 1. Indeed,
even when considering the same 22 actors recorded from 2013 to 2018, no more than 4% of overlap was
identified for each indicator type.
Likewise, only 1% of the indicators present on the PTI sets was also recognized on the OTI ones. Vice
versa, the percentage of sharing indicators of the open sources with the paid sources dropped to 0,0%,
considering the greater proportions of their lists.
Figure 2. Overlap between OTI indicators reported in October 2018 and
PTI indicators published from August to December 2018.
• Timeliness
By measuring the delays in which common indicators appear on the sets of each TI provider, the authors
obtained information regarding the timeliness of each source.
When considering the two PTI vendors, an average delay of more than one month was estimated. Indeed,
it was extremely rare for the two paid sources to report the same data within the same week.
For the comparison between PTI and OTI, no substantial conclusions could be obtained, since overlap
was based on a small fraction of indicators.
Figure 3. Timeliness comparison between the different sources,

6. 5
where (n) represents the instances in which the former source was faster than the latter one.
5. Use cases and perceived Value of TI
5.1 Application of TI
From the answers provided by the professional users of TI, several use cases were identified, among which
three are of particular interest:
Network detection, described as the main use of TI among respondents, concerns all the instances in which
TI is employed automatically to mitigate and avoid cyber-attacks, including loading it into security
systems. Individuals also highlighted the use of threat intelligence in order to develop situational
awareness, aiming at improving the analysts’ perception of the enterprise’s risk situation and threat
environment. Moreover, customers also declared to use TI to direct the investigation of SOC teams, in
charge of responding and detecting cyber threats around the clock, towards the most relevant ones and to
upgrade the quality of the information they gather.
Moreover, further use cases of TI have been identified, such as reinforcement of organizations’ own threat
intelligence and business decision support.
5.2 Value perception
The analysis on the value aspects of TI was achieved by examining the analytic codes drawn during the
interviews.
It emerged that the majority of the customers tend to prioritize the threat intelligence that reduces the
inputs into the workload of analysts. Indeed, a great number of respondents perceived as more valuable
the TI that, integrated into the organizations’ systems, has a limited operational impact, as well as the
sources that provide smaller and more accurate sets. Although low-volume leads to an increase of false
negatives in the feeds, it seemed that customers were actually interested in minimizing the number of false
positives instead. The initial value perception was also reflected in terms of coverage of the relevant
threats, regarding specifically the organizations’ threat landscape, industrial sector and geographical
position. However, customers do not seem to evaluate coverage as much, since only 50% of the responses
described it as a valuable feature.
On the other hand, users mentioned the ability to provide context as an important aspect, as it improves
the general understanding of alerts.
In respect of these criteria, it was observed that PTI is defined as higher quality compared to other TI
providers, offering better curated and more polished products, although not in terms of timeliness.
Still, the authors revealed that 8/14 of the professionals interviewed are actually unable to implement
formal evaluation processes to compare the different sources. The approach used is, in fact, based on
personal perception developed during their experience with different TI sources.

7. 6
6. Conclusions
Ultimately, the main findings allowed the authors to highlight important aspects of the study.
When comparing two PTI vendors, a slight overlap and a significant delay in the reported indicators were
detected, raising doubts regarding the actual coverage that PTI provides.
As for the comparison between PTI and OTI, the minimal number of common indicators suggested that
the two sources represent different kinds of intelligence.
However, these issues seem to be less of a concern for customers, as they value TI based on the impact on
analysts’ time and false positives ratio.
Finally, it was observed that, due to the lack of transparency of PTI providers, customers are not capable
of comparing the different services and understanding the worth of their purchases. As a consequence,
they tend to rely on expensive market leaders, deemed as “the safe choice”.
Nevertheless, the authors believe that, eventually, the asymmetry in the information shared by PTI will
increasingly reduce the willingness of clients to consume, leading paid sources to no longer be able to
benefit in the long run.
Bibliography
Xander Bouwman, Harm Griffioen, Hasso, Jelle Egbers, Christian Doerr, Bram Klievink, Michel van
Eeten. A different cup of TI? The added value of commercial threat intelligence.
August 12–14, 2020

8. 7
Riassunto in lingua italiana
La threat intelligence, conosciuta anche come TI, è la conoscenza del comportamento degli attaccanti che
permette di potenziare le difese dagli attacchi informatici. Le organizzazioni tendono ad affidarsi
principalmente a tre fonti esterne di TI, le open, paid e shared sources. L’articolo mira a fare luce sul
mercato della TI andando ad analizzare i seguenti aspetti: i servizi offerti dalle paid sources e i relativi
costi, le differenze nei dati forniti dalle open e paid sources, il valore percepito dai clienti e l’utilizzo della
TI all’interno delle organizzazioni.
Per condurre lo studio sono stati prelevati da 3 a 5 subsets appartenenti a due leader di mercato ciascuno
e, al fine di determinarne la sovrapposizione, sono stati confrontati tra di loro i dati presenti.
Successivamente, gli stessi dati sono stati messi nuovamente a confronto con gli indicatori forniti da
quattro diverse open sources. Infine, sono stati intervistati 14 professionisti della sicurezza, i quali
lavorano a stretto contatto con le paid sources.
I risultati della ricerca hanno permesso di identificare i prodotti di intelligence più utilizzati dai clienti
delle paid sources. Tra questi i più significativi sono: gli indicatori di compromissione, ovvero segnali
dell’ipotetica presenza di un attaccante, i reports, i quali forniscono delle analisi riguardanti le tecniche
utilizzate e gli obiettivi degli attaccanti, le richieste di determinate informazioni da parte dei clienti e,
infine, i portali, che collezionano tutti i dati e approfondimenti acquisiti dalle paid sources nel tempo.
I costi relativi a tali servizi variano da 30,000$ fino a 650,000$ all’anno.
Confrontando gli indicatori forniti dai due leader è stata rilevata una minima sovrapposizione. Di fatti,
considerando gli stessi 22 attori malevoli che entrambi i venditori ritengono di tracciare, non è stato
individuato più del 4% di sovrapposizione per ogni tipo di indicatore. Similmente, solo l’1% degli
indicatori presenti sui set dei due fornitori è stato riconosciuto su quelli rilasciati dalle open sources.
Per determinare le differenze tra le open e le diverse paid sources, inoltre, è stata studiata la tempestività
delle fonti nel distribuire i propri dati, analizzando il ritardo con cui un indicatore, individuato da una
determinata fonte, viene osservato anche dall’altra fonte considerata. Tra i due venditori è stato misurato
un ritardo medio pari ad un mese, mentre per il confronto tra le paid e le open sources non è stato possibile
ottenere conclusioni sostanziali.
Le risposte fornite dai professionisti intervistati hanno permesso di identificare diversi casi d'uso. Il più
comune è quello relativo alla network detection, la quale comprende tutte le istanze in cui la TI viene
impiegata automaticamente per mitigare gli attacchi informatici, ma risultano essere significativi anche
gli utilizzi della TI atti a migliorare la situational awareness degli analisti e indirizzare le indagini dei SOC
teams.
Dall’analisi dei criteri di qualità è emerso che i clienti tendono a valorizzare la TI che riduce gli input nel
carico di lavoro degli analisti, prediligendo le fonti che forniscono set più ridotti e accurati. Nonostante
un volume più modesto comporti ad un aumento dei falsi negativi, risulta che i clienti, invece, siano
interessati a minimizzare il numero di falsi positivi presenti nei feed. Appare, perciò, che la copertura dalle
eventuali minacce non sia un aspetto estremamente rilevante.
Tuttavia, gli autori hanno rivelato che 8/14 dei professionisti intervistati non sono in realtà in grado di
attuare processi di valutazione formali per confrontare le diverse fonti.

9. 8
In definitiva, i risultati principali hanno permesso agli autori di evidenziare aspetti importanti dello studio.
Confrontando i due leader tra di loro e con le quattro open sources è stata rilevata una minima
sovrapposizione e un significativo ritardo negli indicatori riportati, sollevando dubbi sulla copertura
offerta da queste fonti. Ciò nonostante, è stato osservato che i clienti ambiscono principalmente a
minimizzare il flusso di lavoro dei propri analisti e il numero di falsi positivi, piuttosto che ottimizzare il
rilevamento delle effettive minacce.
Infine, gli autori hanno constatato che, a causa anche della mancata trasparenza da parte dei venditori, i
clienti non sono capaci di attuare veri processi di valutazione e tendono ad affidarsi ai leader di mercato
più costosi, identificati come “la scelta sicura”.