## Talk delivered at artintoscience.com ## One of the primary data sources we use on the Splunk Security Research Team is attack data collected from various corners of the globe. We often obtain this data in the wild using honeypots, with the goal of uncovering new or unusual attack techniques and other malicious activities for research purposes. The nirvana state is a honeypot tailored to mimic the kind of attack/attacker you are hoping to study. To do this effectively, the honeypot must very closely resemble a legitimate system. As a principal security research at Splunk, co-founder of Zenedge (Now part of Oracle), and Security Architect at Akamai I have spent many years protecting organizations from targeted as well as internet-wide attacks, and honeypots has been extremely useful (at times better than threat intel) tool at capturing and studying active malicious actors. In this talk, I aim to provide an introduction to honeypots, explain some of the experiences and lessons learned we have had running Cowrie a medium interaction SSH honeypot base on Kippo. How we modified cowrie to make it more realistic and mimic the systems and attack we are trying to capture as well as our approach for the next generation of honeypots we plan to use in our research work. The audience in this talk will learn how to deploy and use cowrie honeypot as a defense mechanism in their organization. Also, we will share techniques on how to modify cowrie in order to masquerade different systems and vulnerabilities mimicking the asset(s) being defended. Finally, share example data produced by the honeypot and analytic techniques that can be used as feedback to improve the deployed honeypot. We will close off the talk by sharing thoughts on how we are evolving our approach for capturing attack data using honeypots and why.

1. How to Make a Honeypot
Stickier (SSH*)
lessons learned by Splunk Security Research

2. Honeypots Work

3. Agenda
● Introduction
● The data challenge
● Our first attempt
● Analysis methodology
● What we learned
● Our next steps
● How to get started

4. whoami
● Former Prolexic/Akamai Architect
● Co-founded Zenedge, which was acquired by Oracle
● Long time Splunker, recently returned to do research

5. The Data Challenge
● Discover and characterize techniques used in the
exploitation of “vulnerability X” in the wild.
● Determine what’s *actually* targeting our environment.

7. Why Splunk Security Research Uses Honeypots
We lure would-be attackers to a faux system and then capture data regarding their
movements and attack techniques
We want to programmatically produce Splunk Enterprise Security Content
(ESCU)
Our goal is to cover relevant attacks happening in the wild (especially those
without POC exploit code)

8. Our Goals
● Collect downloads, payloads, connections, and behaviors
● Emulate and manipulate common system parameters
● Ensure that our system was easy to deploy/distribute/build
● Include (plus) sane logging, ideally populating Splunk
We selected Cowrie, a fork of Kippo.

9. Before and After

10. Just Connections vs Interaction

11. Files Dropped

12. Our Response

13. GIF of Attacker Here

14. Our Response
Change Cowrie to emulate a Ubuntu 14.04 instance running on AWS by:
Changing in /home/cowrie/cowrie/etc/cowrie.cfg
● Hostname: defaults to svr04, which is a dead giveaway of the fact that this is a Cowrie instance. You
will want to change this.
● Interactive_timeout: defaults to 180, increase it to 300 to make sure we do not disconnect potential
attackers from a bad connection early.
● kernel_version: critical that this is an update reflecting the kernel you want to emulate. In our case,
the default kernel installed with Ubuntu 14.04 is 3.13.0-158-generic
● Kernel_build_string: same as above. Each OS is slightly different. In our case, it was ##208-Ubuntu
SMP Fri Aug 24 17:07:38 UTC 2018
● version - SSH banner version to display for a connecting client. Make sure this matches your OS’s. In
our case, for a default install this is is: SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.10

15. Our Response
Change Cowrie to emulate a Ubuntu 14.04 instance running on AWS by:
Changing in /home/cowrie/cowrie/etc/userdb.txt
● Add new user names that failed authentication
● Exclude admin user, as it was creating lots of noise

16. Our Response
Change Cowrie to emulate a Ubuntu 14.04 Instance running on AWS by:
Changing in /home/cowrie/cowrie/share/cowrie/fs.pickle
● Update file system to match whom you want to emulate
● Cowrie Ships with a great tool for this
~/cowrie/bin/createfs -l /. -o
~/cowrie/share/cowrie/ubuntu14.04.pickle -p

17. Our Response
Change Cowrie to emulate a Ubuntu 14.04 Instance running on AWS by:
Changing in /home/cowrie/cowrie/share/cowrie/txtcmds
● Update common commands prebuilt outputs
● We saw attackers commonly use: b
○ bin/dmesg
○ bin/mount
○ bin/lscpu
○ bin/df
○ usr/bin/lscpu

19. Greynoise Tags
The most common garbage that hits
your AWS instances besides Mirai

20. Analysis Methodology

21. Analysis methodology
1. What rare files were dropped?

22. Analysis methodology
1. What rare files were dropped?
2. Does VirusTotal flag them as bad? If not, can we find it in the public domain?
Bash One Liner
~/virustotal$ hashes=$(ls /home/Cowrie/Cowrie/var/lib/Cowrie/downloads/ |
grep -v tmp | grep -v .sh | grep -v Evlon);
for h in $hashes; do python vt_driver.py file-report $h | jq;
sleep 25;
done
Use VT CLI tool (from Github)
List downloaded hashes
Filter out the crud
Slow down VT rate limits 4/rpm

23. Analysis methodology
1. What rare files were dropped?
2. Does VirusTotal flag them as bad? If not, can we find it in the public domain?
3. Is it known by GreyNoise?

24. Analysis methodology
1. What rare files were dropped?
2. Does VirusTotal flag them as bad? If not, can we find it in the public domain?
3. Is it known by GreyNoise?
4. Is there POC code out there exploit-db, metasploit modules?

25. Analysis methodology
1. What rare files were dropped?
2. Does VirusTotal flag them as bad? If not can we find in public domain?
3. Is it known by GreyNoise?
4. Is there POC code out there exploit-db, metasploit modules?
5. Get IOCs -> Yara -> Set up a hunting rule in VirusTotal

26. What We Learned

27. What we learned
1. Some actors knew they were in Cowrie (or a Honeypot)
2. Spending lots of time in analysis and hard-to-piece-together searches (ESCU)
from Cowrie data
3. Not application-specific, which provided us with a limited view of what we
cared about
4. Analysis timing is key

28. Next Steps
Kernel Log APPlication (KLAPP)
● Sysdig: used to capture level kernel information from the operating system
● Falco: used as an early alert system when a honeypot has been tampered with
● Application: collected logs from the vulnerable application being monitored
● S3 bucket Sync: tool to offload sysdig binary files, as well as application and
system logs to S3

29. Diagram here

30. Sysdig chisel’s FTW
Give me all system logs $> sysdig -c spy_logs -r <sysdig capture file>.gz2
Show me TCP connections sorted $> sysdig -c topconns -r <sysdig capture file>.gz2
Show me HTTP events $> sysdig -c httplog -r <sysdig capture file>.gz2
Show opened shells $> sysdig -c list_login_shells -r <sysdig capture file>.gz2
Give me all traffic for port $> sysdig -c spy_port 22 -r <sysdig capture file>.gz2

31. Falco’s early warning

32. How to get started
1. Download install Cowrie
2. Start an AWS EC2 Ubuntu 14.04 instance
3. Run easy_button.sh
wget -q
https://raw.githubusercontent.com/d1vious/splunk_cowrie/master/easy_button.sh
sudo ./easy_button.sh -s <splunk server url> -t <splunk HEC auth token>

33. Thank you
Questions
@d1vious