NGC17 Talk @ Pinterest - June 5, 2017

1. Learning by s/doing/h4ck1ng/
Our experience learning application security through hacking competitions
Marcello Pogliani
NECSTLab @ Pinterest
San Francisco, 5 June 2017

2. 2

3. 3

4. 4

5. Write secure code
Think like an attacker
5

6. Security competitions (CTFs) - what this talk is about
Information security-oriented “game”
Try to break into (toy) applications “for fun”
… get flags: flag{this_is_a_flag}
6

7. Why?
Fun (above all)
Skillz (and motivation)
Awareness
Creds
7

8. How?
8
Online or live

9. How?
Jeopardy Attack-Defense
9

10. Which skills?
reverse engineering
binary exploitation
web application security
cryptography
forensics
… and others
Good teams
generally have
strong skills and
experience in all
these areas
10

11. Is this just academic?
● Stripe CTF https://github.com/stripe-ctf
○ CTF 2.0 (2012): web oriented
○ CTF 3: same concept, applied to distributed systems
● Google
○ CTF https://capturetheflag.withgoogle.com/
● Facebook
○ Open CTF platform http://github.com/facebook/fbctf
● many others (internal or external)
11

12. What we do (as a team && as a research group)
We play CTFs
We “hack” together
We organize CTFs
12
Tower of Hanoi

13. Some history
● UCSB iCTF, back in 2004.
● ToH won the iCTF edition of 2004 and 2005.
● Playing iCTF every year since then.
● Since 2010, several other CTFs: DEFCON
Quals, CSAW, Plaid CTF, RuCTFe, codegate, …
13

14. Playing CTFs
● Team Organization
○ Tasks / specialization
○ Tools
● Strategy (before being awake for 48h)
○ “observe” other teams
○ “play” the organizers
14

15. 15
PoliCTF 2015
WIP: PoliCTF 2017
7 - 9 July 2017

16. PoliCTF 2015 in numbers
● 48 hours no stop
● > 1k registered teams
● 388 participating teams
○ solved at least one challenge
● 25 challenges
● 500$ infrastructure
16

17. Easy?
Designing effective challenges
Have a solid infrastructure
lasts 48 hours and (mostly) can’t be reused
17

18. Design effective challenges

19. ● No guessing
○ we all know how to use dirbuster
● No bruteforcing
○ challenge people’s brains, not Intel’s CPUs
● No standard challenges
○ google can solve them for you
○ CTF != pentest
● A flag is a flag is a flag
Good challenge = fun
19

20. ● Newbies & experts
● Different skills
○ Forensics, crypto nerds, networking experts, reverse
engineers, …
○ Unusual topics / language / … are ok
Good challenge = fun for the whole team
20

21. ● Don’t rely on an obscure bug of openssl 2.0.54b you can
find on an obscure bug tracker you can find only by
chance because the subject is completely wrong…
○ unless testing people’s google fu (recon)
● Beware of multi-step challenges
Good challenge != impossible challenge
21

22. Idea → challenge (what works for us)
1. Call for challenge
2. Challenge submission
3. Challenge peer-review (without knowing the solutions)
○ Test, test, test
○ Get feedback (and spot loads of bugs)
○ Assign points (alternative: “dynamic” challenge rating)
4. Select, fix & deploy
22

23. Have a solid infrastructure

24. Infrastructure
Amazon AWS
DNS, Storage, Hardened EC2 VMs,
Load Balancers, Security groups.
1 challenge = N VMs (w/scaling)
24

25. Monitoring
Know if a challenge is down
before people starts complaining
Icinga + scripts trying to actually
solve the challenge (+ ELK for logs)
25

26. 26

27. What didn’t work
● Crypto
○ guessing
● Challenges with more than a unique solution
○ pretty simple “flag based” verifier → didn’t work out → support
effort, bad player experience
● Bad hardening
○ screwed up (badly) the permissions in an NFS share with the
.pcap of all the traffic…
○ A team reported this to us!
27

28. References
● https://github.com/pwning/docs/blob/master/suggestions
-for-running-a-ctf.markdown
● http://captf.com/maxims.html
● https://ictf.cs.ucsb.edu/pages/docs.html
28

29. marcello.pogliani@polimi.it
http://toh.necst.it
http://www.polictf.it
@towerofhanoi
29

30. NECSTLab Research in System Security | some examples
30
● Malware Analysis
○ A. Continella et al., ShieldFS: a self-healing, ransomware-aware
filesystem - ACSAC 2016 & BlackHat 2017
○ M. Polino et al., Jackdaw: Towards automatic reverse
engineering of large datasets of binaries - DIMVA 2015
● Cyber-Physical System Security
○ D. Quarta et al., An Experimental Security Analysis of an
Industrial Robot Controller - S&P 2017 & BlackHat 2017

31. NECSTLab Research in System Security | some examples
31
● Fraud Detection
○ M. Carminati et al., BankSealer: an online banking fraud analysis
and decision support system - IFIP 2014
● Mobile Security
○ C. Zheng et al., On-chip system call tracing: A feasibility study
and open prototype - CNS 2016
○ N. Andronio et al., HelDroid: Dissecting and detecting mobile
ransomware - RAID 2015 and BlackHat EU 2016
○ L. Falsina et al., Grab'n Run: Secure and Practical Dynamic Code
Loading for Android Applications - ACSAC 2015

32. marcello.pogliani@polimi.it
http://toh.necst.it
http://www.polictf.it
@towerofhanoi
32