Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to INtroduction to Zagros!
Similar to INtroduction to Zagros! (20)
Recently uploaded
Recently uploaded (20)
INtroduction to Zagros!
- 1. Mining VirusTotal for Operational Data and Applying Quality Control on the Obtained Results. Wall of Sheep, 2016 Gita Ziabari
- 2. © Fidelis Cybersecurity Agenda When Automation is needed? Techniques to automate processing of data mining malware. Introducing Yalda Overview of Algorithm. Stats of obtained data by Yalda. Demo. GitHub link to get the tool for free! How to use the tool.
- 4. © Fidelis Cybersecurity Zagros Makes it possible to get the latest malicious hashes and URLs seen in wild with minimum false positives. Could lead to an organized research analysis based on malware type, malware name and malware family. Great source for feeding cuckoo sandbox. Determine strength of your device in detecting malicious data. Performance testing tool. Zagros is not an Antivirus tool!
- 5. © Fidelis Cybersecurity VT Data Mining VT Data Mining Mining Hashes Live File Feed VTI Search VT Cluster VT Report Download Files Mining URLs Live URL Feed
- 7. © Fidelis Cybersecurity What to look for?VTFileFeedReport Hashes Positives Scans
- 9. © Fidelis Cybersecurity Positives Indicates total number of AntiVirus Engines detected the hash malicious.
- 10. © Fidelis Cybersecurity Scans AV Engine Score Malware Type Score
- 11. © Fidelis Cybersecurity AV Engine Score
- 12. © Fidelis Cybersecurity AV Engine Score Mapping Score AV engines based on your priorities and level of interest.
- 13. © Fidelis Cybersecurity Total AV Engine Score Microsoft 3 AVG 1 Av Engine Score 4
- 14. © Fidelis Cybersecurity Malware Type Severity
- 15. © Fidelis Cybersecurity Malware Type Severity Malware Type Severity Score WORM 3 VIRUS 3 TROJWARE 3 EXPLOIT 3 ADWARE 1 BOT 1 RANSOMWARE 5 ROOTKIT 4 RISKWARE 2 MALWARE 2 HOAX 1 POS 4 SPYWARE 2
- 16. © Fidelis Cybersecurity Hash Threat Score Total AV Score Malware Type Score Hash Threat Score
- 18. © Fidelis Cybersecurity Storing Obtained Hashes CSV files High scored hashes Mid scored hashes MongoDB Database: zagros Collection: zagros_hashes
- 19. © Fidelis Cybersecurity Example of Obtained Hashes in mongoDB "_id" : ObjectId("57a124d89bf13c5985c493cd"), "Indicator" : "b772a1a03984c18b00eae4da49d6e7b8”, "IngestTime" : "2016-08-02T18:55:20.177502", Attributes" : "malware_type" : "ADWARE", "malware_name" : <malware_name> "hash_type" : "high_scored", "severity" : 1 "sha256" :"3312f7c6cacae837647ca68247f98c0b19a3ff7c93063ead77e7a0390f73c574” "Type" : "hash”
- 20. © Fidelis Cybersecurity Live File Feed Get the latest hashes being submitted to VT in real time. More than 2 million hashes get evaluated on daily basis by Zagros and only ~300k get selected. 20% get mined by Zagros. Execute the script once and it will stay in synch with VT every minute! virustotal_data_mining_file.py
- 21. © Fidelis Cybersecurity VTI Search Search on file type, file size, positives, malware type, behavior,… 60% of given hashes are being added to database. Execute the script on hourly basis. virustotal_data_mining_vti_search.py
- 22. © Fidelis Cybersecurity VT Clustering Get clustered data from VT. PE, PDF, DOC and RTF files. 35% of given hashes are being added to database. Run the script on daily basis to get the hashes clustered the day before. virustotal_data_mining_clustered_hashes.py
- 23. © Fidelis Cybersecurity Expand the Search Get the list of latest hashes mined by Zagros. Search for similar-to hashes. Execute the script on hourly basis. 24% of given hashes are being added to database. virustotal_data_mining_file_similar_to.py
- 24. © Fidelis Cybersecurity Stats 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 Live File Feed VT search Clustering Similar-to Total VT Zagros
- 25. © Fidelis Cybersecurity Download and Extract Embedded Files Get the list of malicious hashes from Zagros database. Download the files from VT. Extract embedded files from it. virustotal_data_mining_download_files.py
- 28. © Fidelis Cybersecurity Sandbox Analysis Copies itself to AppdataLocalTemp. Begins using FindFirstFile and opening files. Reads the file before overwriting it with the virus. Opens the file to infect for writing. Writes the virus code to the file along with an string 'visua’. Writes the original file. Appears that it attempts to infect all files and not just exe files.
- 30. © Fidelis Cybersecurity What to look for?VTURLFeedReport URL Positives AV Engine Additional Info
- 32. © Fidelis Cybersecurity Scoring HTTP Response HTTP Response Score 200 2 100 1 403 1 404 1 Rest 0
- 33. © Fidelis Cybersecurity Scoring Categories blogs 1 uncategorized 0 malicious web sites 4 suspicious content 1 business 0 parked 0 phishing and other frauds 5 business and economy 0 travel 0 Bot networks 4 Parked domain 0 computersandsof tware 0 health 0 hacking 5 Not recommanded site 3 Elevated exposure 2 Proxy evoidance 2 potentially unwanted software 2
- 34. © Fidelis Cybersecurity URL Threat Score HTTP Response Score URL Categories Score URL Threat Score
- 35. © Fidelis Cybersecurity Whitelist Trusted URLs 7-zip.org acer.com adobe.com ahnlab.com microsoft.com blackberry.com ebay.com oogle.com hp.com mozilla.net oracle.com opera.com skype.com sophosupd.com samsung.co symantec.com toshiba.com dellbackupandrecovery.com,
- 36. © Fidelis Cybersecurity Malicious URL Positives AV Score Whitelisted URLs URL Categories Score HTTP Score
- 37. © Fidelis Cybersecurity Trim URLs Include up to two path of the URL. Expand possibility of getting malicious URLs.
- 38. © Fidelis Cybersecurity Obtained URL URL Short URL Positives AV Engine Score URL Threat Score http://www.i gmarealty.r u/index.php /?option=co m_content &task=view &ida www.igmar ealty.ru 10 10 12
- 39. © Fidelis Cybersecurity Live URL Feed Get the latest URLs being submitted to VT in real time. More than 3.7 million URLs get evaluated on daily basis by Zagros and only ~350K get selected. 9% get mined by Zagros. Execute the script once and it will stay in synch with VT every minute! virustotal_data_mining_url.py
- 40. © Fidelis Cybersecurity Malicious Hashes from URLs virustotal_data_mining_url_itw.py
- 41. © Fidelis Cybersecurity Storing Obtained URLs CSV files Stores URLs MongoDB Database: zagros Collection: zagros_urls
- 42. © Fidelis Cybersecurity Quality Control Data Hashes URLs
- 43. © Fidelis Cybersecurity Aging Hashes Removing the low scored hashes from mongoDB. virustotal_aging_mined_data.py
- 44. © Fidelis Cybersecurity Aging URLs Removing low scored URLs from database from mongoDB. Script:virustotal_aging_mined_urls.py
- 45. © Fidelis Cybersecurity Conclusion %100 accuracy in obtained results in terms of being malicious. Makes it possible to get the latest malicious hashes and URLs seen in wild. Could lead to an organized research analysis based on malware type, malware name and malware family. Great source for feeding cuckoo sandbox. Functional and performance testing tool.
- 46. © Fidelis Cybersecurity Code is available at Fidelis gitHub https://github.com/fideliscyber/data_mini ng
- 47. © Fidelis Cybersecurity What you would need to use Zagros VT key Python 2.7.6 or later Linux OS Python modules to download: urllib2, urllib, json, requests, pymongo MongoDB(optional)
- 48. © Fidelis Cybersecurity How to use Zagros
- 50. © Fidelis Cybersecurity Fidelis Cybersecurity Gita Ziabari Senior Threat Research Engineer Email: gita.ziabari@fidelissecurity.com Twitter: @gitaziabari
Editor's Notes
- hex editor and see multiple occurrences of the split 'visua' followed by an MZ header possibly proving that the file infector contains many other files in it that it has attempted to infect over time as it’s spread.
- 200 ok -> The request has succeeded. 100 Continue The client SHOULD continue with its request. This interim response is used to inform the client that the initial part of the request has been received and has not yet been rejected by the server. The client SHOULD continue by sending the remainder of the request or, if the request has already been completed, ignore this response. 403: Forbidden The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. 404 Not Found The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent.
- https://www.forcepoint.com/master-database-url-categories categories_score = {"blogs" : 1, "uncategorized" : 0, "malicious web sites": 4, # sites containing code intentionally modify users "suspicious content" : 1, # sites with suspicious content "business" : 0, "known infection source" : 5, "parked" : 0, "phishing and other frauds" : 5, # counterfeit legitimate sites "business and economy": 0, # Sites sponsored by or devoted to business firms "travel" : 0, "bot networks": 4, # Command and control centers "parked domain": 0, # Sites that are expired, offered for sale, .. "computersandsoftware" : 0, "health" : 0, "real estate" : 0, # Sites that provide information about renting, buying, selling "information technology" : 0, # Computers, software, the Internet and related business firms "entertainment" : 0, "compromised websites" : 5, # Sites that are vulnerable and known to host an injected malicious "dynamic content": 2, # URLs dynamically being generated "not recommended site" : 3, "potentially unwanted software" : 2, # Sites altering operation of a user's hardware, software, ... "web and email spam" : 2, "application and software download" : 1, "personal network storage and backup" : 1, #store personal files on web servers for backup or exchange "hacking" : 5, "elevated exposure" : 2, "education" : 0, "web hosting" : 0, "marketing" : 0, "radiomusic" : 0, "internet radio and tv" : 0, "videos" : 0, "proxy avoidance" : 2,