This document summarizes research on classifying malicious URLs on Twitter in real-time using machine activity data. The researchers collected data on URLs shared on Twitter during sporting events and used a honeypot to identify malicious ones. They built machine learning models to predict maliciousness based on metrics like CPU usage, network traffic, and processes when a URL was clicked. The best model was a multi-layer perceptron that achieved up to 72% accuracy. It showed network activity, CPU usage, and processes were predictive. Testing on a new dataset showed some independence between events. Using only 1% of training data caused a small 5% drop in performance, alleviating concerns over data requirements.

1. Real-time Classification of Malicious URLs on
Twitter using Machine Activity Data
Pete Burnap, Amir Javed, Omer Rana & Shahzad Awan
Social Data Science Lab
School of Computer Science and Informatics & School of
Social Sciences
Cardiff University
@pbFeed @omerfrana @socdatalab

2. Social Data Science Lab - @socdatalab
• Formed in 2015 out of the Collaborative Online Social Media
Observatory (COSMOS) programme of work (cosmosproject.net)
• Mission is to continue the work of COSMOS in democratising access
to big social data (e.g. Twitter, Foursquare, Instagram) amongst the
academic, private, public and third sectors.
• Recent grant capture of over £2.7 million through 20 grants
• A significant proportion of these funds have been awarded to collect
and analyse social media data in the contexts of Societal Safety and
Security e.g. social tension, hate speech, crime reporting and fear
of crime, suicidal ideation
• Working with Metropolitan Police, Department of Health,
Food Standards Agency

3. The Problem
• Our previous research has studied online social networks as “social
machines” that enable spread of malicious or potentially dangerous
information (e.g. rumour, hate speech, suicidal ideation)
• OSNs widely used around large events where information seeking
occurs (e.g. natural disaster, sporting events) – particularly
susceptible to spread of drive-by-downloads - URLs pointing to
malicious servers but hidden in attractive content
• Most research to date focuses on social network properties of
users to identify malicious accounts
• We explore behavioural analysis of malware with “real time”
machine activity logs and honeypots

4. Example
“Check out these awesome pics from #superbowl http://bit.ly/3s5dh3”
Modify registry
Download exploit
Open backdoor

5. Research Aims
• Develop a ‘real-time’ machine classification system to
distinguish between malicious and benign URLs within
seconds of the URL being clicked
• Examine and interpret the learned model to explicate
the relationship between machine activity and malicious
behaviour exhibited by drive-by-downloads on Twitter.
• Identify features/signals that generalize across events
• Understand data volume requirements

6. Method
• Data collected during two large sporting events – the
Superbowl and the Cricket World Cup
• Identify malicious URLs using client honeypot
• Build machine classification models using machine activity logs
generated while interacting with URLs extracted from Twitter
• Refine measurable activity metrics to identify most predictive
features by examining how they are used to make decisions
• Build a learning-curve chart that demonstrates using 1% of
the training data only has a small detrimental effect on
performance

7. Data
• Data collected during two large sporting events – the
Superbowl and the Cricket World Cup
• Search using event hashtags and restrict to tweets with URLS
• Superbowl – 122,542 unique URLs
• Cricket (semi and final) – 7,961 unique URLs

8. Identifying Malicious URLs
• Utilised a high interaction client honeypot (Capture HPC) to
visit each URL (for 5 mins) and log changes to system state
• Behaviour analysis avoids issues with static analysis where
‘signatures’ in code of new/dynamic codebases are unseen
• Use an exclusion list of malicious behaviours – changes to
registry, file system, running processes – to identify
malicious behaviour and tage URLs as malware
• Lists need updating and assumes you are able to capture all
potential malicious behaviour (difficult) – but methods
exist to support this

9. Predicting malicious behaviour
Waiting until malicious behaviour occurs is not ideal – it would be
useful to have some insight into measurable machine activity metrics
that provide a ‘signal’ before the attack occurs
Metrics:
1. CPU usage
2. Connection established/listening (yes/no)
3. Port Number
4. Process ID (number)
5. Remote IP (established or not)
6. Network Interface (type e.g. Wifi, Eth0)
7. Bytes Sent
8. Bytes receive
9. Packets Sent
10. Packets Received
11. Time since start of interaction

10. Model selection
Sampled 2k tweets for training (Superbowl) and 2k for testing (Cricket)
– 1k malicious and 1k benign. 5.5 million obs. Each obs. Is a feature
vector containing metrics & malicious/benign annotation
Methodological considerations:
• High variance in the mean recorded CPU usage, bytes/packets
sent/received, and ports used between the two datasets suggests
identifying similar measurements between events would be
challenging
• Standard deviation in both datasets very similar, which suggests the
variance is common to both datasets, but the deviation is high,
which suggests a large amount of ’noise’ in the data
• Behaviours in both logs are largely benign, creating a large skew in
log activity towards the benign type
• Generative or discriminative model?

11. Phase I Modelling
Baseline experiments - which model would be most
appropriate based on prediction accuracy?
Generative models that consider conditional dependencies in
the dataset (BayesNet) or assume conditional
independence (Naive Bayes)
Discriminative models that aim to maximise information gain
(J48 Decision Tree) and build multiple models to map input
to output via a number of connected nodes, even if the
feature space is hard to linearly separate (Multi-layer
Perceptron).

12. Phase I Results

13. Model performance over time

14. Best performing model analysis
We examine the MLP model during the training phase (t=60) to investigate how the model is
representing a learned weighting between features.
Model produced 9 hidden nodes with weighting given to each node for each class (malicious or
benign)
Nodes 3, 4 and 9 have high weightings towards a particular class. Node 9 stands out as the
most discriminative positive weighted node for malicious URLs.

15. Best performing model analysis

16. Sampled Learning
Storing Twitter data around ongoing real-world events is an issue given
that events can last several weeks = massive data
Less data = less storage space = less computational time required to
extract model features and run models
Questions could be asked around whether the training set is missing a
significant proportion of malicious activity given that not all URLs
can be visited in real-time given “take downs”
Demonstrating small training sample achieves similar performance to
full samples alleviates these issues to some degree as it
demonstrates the most explanatory features present in the smaller
sample.
We retained the full test dataset and sampled (using no
replacement) from the training data at 1%, 5%, 10% and
increments of 10% up to 100.

17. Sampled Learning
• Sample of 20%, 30% and 40% yields a performance of 68%, only
1% lower than the optimal performance of 69%.
• Performance using a 1% sample is 63% - a drop of only 5% on the
optimal performance with a complete sample.
• Based on the mean of two
runs for each sample

18. Conclusions
Multi Layer Perceptron (MLP) approach all worked extremely well
during training, achieving over 90% accuracy, up to 97% - machine
logs work well for prediction
Discriminative models performed better than generative models in
testing - MLP model performed best overall up to 72% - suggests
linearly separable features
Bayesian approach performed best in the early stages of interaction
(within 5 seconds of clicking the URL), achieving 66% accuracy when
the model had the least information available (Real time)
Drop in performance on a new event suggests attack vectors
slightly different across events, but with reasonably high degree
of accuracy we can claim some independence between
predictive features and events in Twitter URLs

19. Conclusions
MLP model shows key predictive machine activity metric was network
activity - particularly packets sent and received
CPU use and process IDs also had clear raised and correlated
weighting in the model, as did the bytes sent from the network when
correlating with new connections to remote endpoints, suggesting
data exfiltration exercises can be distinguished from general data
transfer.
Learning curve using small samples revealed only a small drop in
classification performance when compared to using the full training
sample, alleviating some concerns over appropriate sampling
mechanisms, lack of a complete log of all Twitter URL activity, and
the requirement for large amounts of data storage.

20. Thanks
Questions?
@pbFeed