Malicious domains are one of the main resources used to mount attacks over the Internet. It is important to detect such activities by mining the large-scale network traffic data and identifying malicious URLs, domains or IPs. The attackers often take advantages of vulnerabilities in DNS and commit activities such as stealing private information, spamming, phishing, and DDoS attacks, and tend to by-pass botnet detection by generating domain clusters from Domain Generation Algorithms (DGA). We have billions of DNS records per day. Spark AI platform hence serves as an efficient distributed platform for the processing and mining of this huge amount of data. We work on the following two cybersecurity use cases. 1. Detect DGA, Porn, and Gambling domains Each malware-compromised host machine will have a large amount of DNS request in sequential order. The domain names are either generated by DGAs or preserve particular string patterns by design. We use spark to generate DNS request domain sequences and use Word2Vec to estimate the embedding of the domains. We then estimate the similarity and the most similar domains in the embedding space are discovered as the potential malicious domains. 2. Detect cryptocurrency mining pool domains The attackers are interested in accessing computing resources to mine cryptocurrency. The malware infected computers will be directed to attacker-controlled mining pool domains. This type of DNS request does not preserve sequential order and is relatively random. Since each mining pool domain cluster is visited by a wide range of different host machines, we used LSH to evaluate the similarity among sets of hosts. As a result, LSH generates domain-bucket bipartite graph and FastUnfolding algorithm is used to discover the domain clusters. We leveraged spark AI for large scale DNS data analysis and discovered hundreds of thousands of malicious domains each day at high precision. Authors: Ting Chen, Hao Guo

1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics

2. Hao Guo, Tencent
Ting Chen, Tencent
Large-scale Malicious
Domain Detection with
Spark
#UnifiedAnalytics #SparkAISummit

3. About The Speakers
Hao Guo
• Applied Research Scientist @ Tencent Security
• Master degree in Computer Science from HIT with research
interest in NLP, deep learning and large-scaled machine
learning
Ting Chen
• Director, Applied Machine Learning @ Tencent Jarvis Lab
• PhD degree in Computer Science from UFL with research
interest in computer vision and machine learning
• Previously, Senior ML engineer and DS manager at Uber
3#UnifiedAnalytics #SparkAISummit

4. Agenda
• DDoS Attack & Advance Persistent Threat
• Sequence based detection
• Crypto Mining Malware
• Locality Sensitivity Hashing based detection
• Conclusion
4#UnifiedAnalytics #SparkAISummit

5. What is DDoS Attack
https://medium.com/@kapil.sharma91812/understanding-ddos-attack
5#UnifiedAnalytics #SparkAISummit

6. DDoS Attack Trend
2018 https://securelist.com/ddos-attacks-in-q4-2018/89565/
6#UnifiedAnalytics #SparkAISummit

7. DDoS Attack Trend
http://francescomolfese.it/en/2018/12/la-protezione-da-attacchi-ddos-in-azure/
7#UnifiedAnalytics #SparkAISummit

8. What is APT
Symantec APT white paper
8#UnifiedAnalytics #SparkAISummit

9. APT Activities
https://www.fireeye.com/current-threats/annual-threat-report.html
9#UnifiedAnalytics #SparkAISummit

10. Behind The Attacks: C&C
10#UnifiedAnalytics #SparkAISummit

11. Behind The Attacks: DGA
11#UnifiedAnalytics #SparkAISummit

12. Malicious Domain Detection
Scenario 1: DGA
uuybcc.com
igmgdc.com
lpppxa.com
swdosv.com
grevun.com
djiyei.com
cvevrm.com
vyjyui.com
Victim
hosts Sequences of domains accessed for each host
12#UnifiedAnalytics #SparkAISummit

13. Crypto Mining Malware
13#UnifiedAnalytics #SparkAISummit

14. Malicious Domain Detection
cab217f6.space
6850c644.space
cbb21989.space
c8b214d0.space
ceb21e42.space
cfb21fd5.space
c9b21663.space
d4b227b4.space
Scenario 2: Crypto mining domains
Victim hosts
14#UnifiedAnalytics #SparkAISummit

15. Data Scale @ Tencent
Billions
(day)
Billions
(day)
Tens
Millions
(day)
TB
DNS Records Domains IPs/Hosts Storage
Spark enables large-scale data analysis.
#UnifiedAnalytics #SparkAISummit
15#UnifiedAnalytics #SparkAISummit

16. Sequence Based Detection
16#UnifiedAnalytics #SparkAISummit

17. Sequence Based Detection
sentence
word
Victim
hosts
17#UnifiedAnalytics #SparkAISummit

18. Sequence Based Detection
document
Victim
hosts
18#UnifiedAnalytics #SparkAISummit

19. Domain2Vec Representation
ii-1i-2 i+1 i+2
Key Idea
• Estimate the domain2vec (work2vec)
representation of each domain with CBOW
Example domain2vec representation
• uymbkc.com
CBOW framework
-0.10 0.58 -0.04 … 0.29 0.26 -0.17
Victim
hosts
19#UnifiedAnalytics #SparkAISummit

20. Domain Clustering
• Start with seed domains (known malicious
domains)
• Find most similar domains
𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦 = cos θ =
𝑑𝑜𝑚𝑎𝑖𝑛𝑉𝑒𝑐4 5 𝑑𝑜𝑚𝑎𝑖𝑛𝑉𝑒𝑐6
||𝑑𝑜𝑚𝑎𝑖𝑛𝑉𝑒𝑐4||||𝑑𝑜𝑚𝑎𝑖𝑛𝑉𝑒𝑐6||
20#UnifiedAnalytics #SparkAISummit

21. Domain Clustering
21#UnifiedAnalytics #SparkAISummit

22. Example Clustering Results
Virus Domains
uuybcc.com
igmgdc.com
lpppxa.com
swdosv.com
grevun.com
djiyei.com
cvevrm.com
vyjyui.com
... ...
Nymaim Domains
pjbgwwt.ru
mphsnkjgnfh.biz
nkposroyfkr.net
uusuux.ru
dsvlvlnkcj.ru
jtduakh.ru
... ...
Conficker Domains
kicuxexj.org
xiaxyvyn.net
jhpruj.biz
cvlyfcz.org
dsvevamq.biz
blqdisrp.cc
eujyvcvj.org
... ....
22#UnifiedAnalytics #SparkAISummit

23. Implementation Framework
billions/day
23#UnifiedAnalytics #SparkAISummit

24. Key Functions
Domain Sequence Generation
Domain2Vec Calculation
Similarity Domain Clustering
domain_sequence_rdd =
input_rdd.combineByKey(to_list, append, extend, numPartitions = N).mapValues(sortbytime)
domain_sequence_rdd = domain_sequence _rdd.map( lambda r:Row(r) )
domain_sequence = spark.createDataFrame(domain_sequence_rdd,[”domainSequence”])
domain2vec = Word2Vec(vectorsize=100, minCount=3, numPartitions =N, seed=42,
inputCol=”domainSequce”, outputCol=”model”, windowSize=8, maxSentenceLength=1000)
model = domain2vec.fit(domain_sequence)
from pyspark.sql import SparkSession, Row
from pyspark.ml.feature import Word2Vec, Word2VecModel
model.findSynonyms(domain_seed, M).select("word", fmt("similarity", m).alias("similarity"))
24#UnifiedAnalytics #SparkAISummit

25. LSH Based Detection
25#UnifiedAnalytics #SparkAISummit

26. LSH based detection
host IP sets for accessing domain1:
host IP sets for accessing domain2:
Jaccard similarityVictim hosts
𝑆9 = {𝐼𝑃0, 𝐼𝑃1, 𝐼𝑃2, 𝐼𝑃3}
𝑆C = {𝐼𝑃1, 𝐼𝑃2, 𝐼𝑃3, 𝐼𝑃4}
Similarity of domain1 and domain2 is:
𝑠𝑖𝑚 𝑑𝑜𝑚𝑎𝑖𝑛1, 𝑑𝑜𝑚𝑎𝑖𝑛2 =
| EF9,EFC,EFG |
| EFH,EF9,EFC,EFG,EFI |
= 3/5
26#UnifiedAnalytics #SparkAISummit

27. • High dimensional and sparse
– tens of millions hosts
• O(N*N) comparisons
– million unique domains
• Spark provides Locality Sensitivity Hashing for fast near-duplicate detection
Why LSH
27#UnifiedAnalytics #SparkAISummit

28. LSH
With hight probablity domain1 and doman2 are
hashed into the same buckets .
With high probablity domain1 and domain2 are
hashed into the different buckets.
28#UnifiedAnalytics #SparkAISummit

29. Minhash and Jaccard Similarity
• There is a suitable hash function for
the Jaccard similarity : minhash
• The probability that
minhash(domain1) = minhash(domain2)
is equal to the similarity of
Jaccard(domain1, domain2)
29#UnifiedAnalytics #SparkAISummit

30. FastUnfolding
6850c644
.space
abc.comcbb21989
.space
ceb21e42
.space
defag.ur gha.com d4b227b4
.space
30#UnifiedAnalytics #SparkAISummit

31. Modularity:
FastUnfolding
Fast unfolding of communities in large networks, VD Blondel et.al. J. Stat. Mech. (2008)
31#UnifiedAnalytics #SparkAISummit

32. Implementation Framework
billions /day
32#UnifiedAnalytics #SparkAISummit

33. Key Functions
Domain-host Mapping Generation
Input RDD[(IP,domain)] // both IP and domain of DNS query
zipwithIndex,join input into RDD[(IP_id,domain_id)]
map,combineByKey into RDD[(domain_id,List[IP_ids])] // both domain and IP sets
map into RDD[(domain_id,sparseVector(IP_ids))] // dense vector map into sparse vector
Output RDD[(domain,sparseVectors(IPs))]
33#UnifiedAnalytics #SparkAISummit

34. Key Functions
LSH
Input RDD[(domain_id,sparseVector(IP_ids))] //domain_id and high dimensional sparse vector
MinHashLSH into RDD[(domain_id,List(hashvalues))] // reduce dimension into hundreds
flatMap into bucket RDD[(domain_id,List(bucket_id))] // map similar domains into the same bucket
fastunfolding // self implemented function
Output RDD[(domain_id,bucket_id)]
34#UnifiedAnalytics #SparkAISummit

35. Clustering Result
• Crypto mining domains
cab217f6.space
6850c644.space
cbb21989.space
c8b214d0.space
ceb21e42.space
cfb21fd5.space
c9b21663.space
d4b227b4.space
... ...
35#UnifiedAnalytics #SparkAISummit

36. Conclusion
• Sequence-based mining to detect DGA domains
– Sequence and co-occurrence
– Find near-neighbors/most similar domains
• LSH-based detect cryptocurrency mining domains
– Jaccard similarity
– Clustering
36#UnifiedAnalytics #SparkAISummit

37. Tencent Security
37#UnifiedAnalytics #SparkAISummit
Tencent Security

38. Thank You
38#UnifiedAnalytics #SparkAISummit
Acknowledgement
Shubing Long Chunhua Hong
Yu Liang Yong Deng
Mengling Han Xiangqian Wei
Na Yi Tingwei Mao
Tencent Security

39. DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT