Designing Cybersecurity Policies with Field Experiments

1. Designing Cybersecurity
Policies with Field Experiments
Gene Moo Lee
University of Texas at Austin
Joint work with Shu He, John S. Quarterman, Andrew B. Whinston
Supported by NSF 1228990
February 25, 2015
KAIST

2. Gene Moo Lee, KAIST, Feb 2015
“Although the threats are serious and they
constantly evolve, I believe that if we address them
effectively, we can ensure that the Internet remains
an engine for economic growth and a platform for
the free exchange of ideas.”
—Barack Obama
2

3. Gene Moo Lee, KAIST, Feb 2015
Motivation
• Inadequate cybersecurity is a serious threat
• avg cost $3.5 million in 2013, 15% increase
• # of compromises increased by 25%
• data breaches of 2.6 million Target consumers
• U.S. government’s measures
• Cybersecurity Policy Review (2009)
• Executive Order 13636 (2013) “Improving Critical
Infrastructure Cybersecurity”
3

4. Gene Moo Lee, KAIST, Feb 2015
Approaches
• Technical approaches:
• spam filtering, intrusion detection systems (IDS), digital
forensics
• Sahami et al. (1998), Cormack and Lynam (2007), Denning
(1987), Lee and Stolfo (1998), Casey (2011), Taylor et al.
(2014)
• Economic approaches:
• underinvestment due to (1) information asymmetry, (2)
network externalities, (3) moral hazards
• van Eeten et al. (2011), Moore and Clayton (2011), Arora
et al. (2004), D’Arcy et al. (2009), Wood and Rowe (2011)
4

5. Gene Moo Lee, KAIST, Feb 2015
Our approach
• We found evidence that spam evaluation publication help improving
security levels in country level
• Quarterman et al. (2012), Qian et al. (2013)
• Use outbound spam to estimate latent security level
• 90% spam is from compromised computers controlled by botnets
(Rao and Reiley 2012, Moore and Clayton 2011)
• Ultimate goal:
• Evaluate the effectiveness in organizational level
• government sponsored institution to monitor and evaluate
organizational security levels (Moody’s, S&P for bonds)
• Counterfactual policy analysis with randomized field experiments
5

6. Gene Moo Lee, KAIST, Feb 2015
Research questions
1. Our goal is to set up an independent institution to evaluate
and monitor all organizations’ cybersecurity level
2. Does information disclosure change organizational
behaviour? In other words, spam reduce?
• Method: Randomized field experiment
• Two treatment groups with different info disclosure
• Two cycles of emails at January/March 2014
• A website built on Google cloud
6

7. Gene Moo Lee, KAIST, Feb 2015
Experimental design
• 7919 US organizations, three groups: control, private, public
• Private treatment: email with spam volume, rank, IP addr
• Public treatment: email + publication in public website
7

8. Gene Moo Lee, KAIST, Feb 2015
Randomization
• Stratification with industry sectors and IP counts
• Pair-wise matching with pre-experimental spam volume
• Re-randomization: 10,000 times and power calculation
8

9. Gene Moo Lee, KAIST, Feb 2015
Treatment channel: email
9

10. Gene Moo Lee, KAIST, Feb 2015
Website: search engine
10
• http://cloud.spamrankings.net

11. Gene Moo Lee, KAIST, Feb 2015
Website: overall stats
11

12. Gene Moo Lee, KAIST, Feb 2015
Website: detail charts
12

13. Gene Moo Lee, KAIST, Feb 2015
System implementation
• Back end: data collector, peer ranker, web generator, MySQL, JSON
• Front end: Google cloud, search engine, analytics
13

14. Gene Moo Lee, KAIST, Feb 2015
Data: CBL and PSBL
14
• A spam blocklist uses spamtraps to collect IP adresses
sending out spams:
• CBL: http://cbl.abuseat.org/
• PSBL: http://psbl.org/
• Spamtrap
• honeypot used to collect spam
• email addresses not for legit communications
• CBL daily avg data
• 8 million IP, 190K netblocks, 21K ASNs, 200 countries

15. Gene Moo Lee, KAIST, Feb 2015
Organizational spam data
15
• IP > netblock > ASN > organization
• IP > netblock: IP lookup
• netblock > ASN: Team Cymru
• ASN > org: algorithm + manual inspection
• Organization data from LexisNexis
• 7919 U.S. organizations identified
• Industry codes: SIC, NAICS
• Public/private, # employees

16. Gene Moo Lee, KAIST, Feb 2015
Org level spam volume and IP address
16

17. Gene Moo Lee, KAIST, Feb 2015
Industry sectors
17

18. Gene Moo Lee, KAIST, Feb 2015
Industry level spam volume/host
18

19. Gene Moo Lee, KAIST, Feb 2015
Hypothesis development
1. Information disclosure effect
2. Publicity effect
3. Pre-experimental security level
4. Industry competition level
19

20. Gene Moo Lee, KAIST, Feb 2015
Info sharing and publicity effects (H1, 2)
20

21. Gene Moo Lee, KAIST, Feb 2015
Large spammers (H3)
21

22. Gene Moo Lee, KAIST, Feb 2015
Competition (H4)
22

23. Gene Moo Lee, KAIST, Feb 2015
Empirical analysis summary
1. Private info sharing doesn’t work
2. Publicity matters
3. Organizations with (1) large spam, (2)
less competition reacted
4. Peer effect exists after the treatments.
Stronger with treatment groups.
23

24. Gene Moo Lee, KAIST, Feb 2015
Robustness check
1. Placebo test: change experiment time
2. Subsample analysis: only include
moderate spammers
3. Alternative pre-experimental spam
measure: 6, 4, 2, months
4. Control variables
24

25. Gene Moo Lee, KAIST, Feb 2015
Directions
1. Robust security evaluation: spam,
phishing, DDoS, etc.
2. Different environment: China, Korea
3. Treatment channel: social media
4. Cybersecurity insurance
5. Cloud security
25

26. Gene Moo Lee, KAIST, Feb 2015
Thank you!
Contact: gene@cs.utexas.edu
26

27. Gene Moo Lee, KAIST, Feb 2015
References (1)
[1] Adelsman, Rony M., and Andrew B. Whinston (1977). "Sophisticated voting with information
for two voting functions." Journal of Economic Theory 15, no. 1: pp. 145-159.
[2] Anderson, Axel, and Lones Smith. "Dynamic Deception." American Economic Review 103, no.
7 (2013): 2811-47.
[3] Anderson, Ross (2001). "Why information security is hard: An economic perspective." IEEE
Computer Security Applications Conference, pp. 358-365.
[4] Aral, Sinan, and Dylan Walker. "Identifying influential and susceptible members of social
networks." Science 337, no. 6092 (2012): pp. 337-341.
[5] Arora, Ashish, Ramayya Krishnan, Anand Nandkumar, Rahul Telang, and Yubao Yang (2004).
"Impact of vulnerability disclosure and patch availability-an empirical analysis." Workshop on
Economics of Information Security, vol. 24, pp. 1268-1287.
[6] Bauer, Johannes, and Michael van Eeten (2009). “Cybersecurity: Stakeholder incentives, externalities,
and policy options.” Telecommunications Policy, Vol. 33, pp. 706-719.
[7] Blei, David M., Andrew Y. Ng, and Michael I. Jordan (2003). "Latent dirichlet allocation."
Journal of Machine Learning Research 3: pp. 993-1022.
[8] Bratko, Andrej, Gordon V. Cormack, Bogdan Filipic, Thomas R. Lynam, and Blaz Zupan
(2006). Journal of Machine Learning Research 6: pp. 2673-2698.
[9] Bruhn, Miriam, and David McKenzie (2008). "In pursuit of balance: Randomization in practice
in development field experiments." World Bank Policy Research Working Paper Series.
[10] Casey, Eoghan (2011). Digital evidence and computer crime: Forensic science, computers and
the Internet. Academic Press.
[11] Cormack, Gordon V., and Thomas R. Lynam (2007). “Online supervised spam filter evaluation.”
ACM Transaction on Information Systems, Vol. 25(3)
27

28. Gene Moo Lee, KAIST, Feb 2015
References (2)
[12] D’Arcy, John, Anat Hovav, and Dennis Galletta (2009). "User awareness of security countermeasures
and its impact on information systems misuse: A deterrence approach." Information
Systems Research 20, no. 1: pp. 79-98.
[13] Denning, Dorothy E. (1987). “An intrusion-detection model.” IEEE Transactions on Software
Engineering, Vol. 13(2): pp. 222-232.
[14] Dharmapurikar, Sarang, Praveen Krishnamurthy, and David E. Taylor (2003). “Longest prefix
matching using bloom filters.” Proceedings of the ACM SIGCOMM Conference: pp. 201-212.
[15] Dice, Lee R. (1945). “Measures of the amount of ecologic association between species.” Ecology
26(3): pp. 297-302.
[16] Duflo, Esther, Rachel Glennerster, and Michael Kremer. "Using randomization in development
economics research: A toolkit." Handbook of development economics 4 (2007): 3895-3962.
[17] Fracassi, Cesare (2014). "Corporate finance policies and social networks." In AFA 2011 Denver
Meetings Paper.
[18] Festinger, Leon. "A theory of social comparison processes." Human relations 7, no. 2 (1954):
117-140.
[19] Gal-Or, Esther, and Anindya Ghose (2005). "The economic incentives for sharing security
information." Information Systems Research 16, no. 2: pp. 186-208.
[20] Graham, Bryan S. (2008). "Identifying social interactions through conditional variance restrictions."
Econometrica 76, no. 3: pp. 643-660.
[21] Harper, Yan Chen, F. Maxwell, Joseph Konstan, and Sherry Xin Li. "Social comparisons and
contributions to online communities: A field experiment on movielens." The American economic
review (2010): 1358-1398.
[22] Harrison, Glenn W., and John A. List (2004). "Field experiments." Journal of Economic Literature:
pp. 1009-1055.
[23] Kugler, Logan (2014). “Online Privacy: Regional Differences.” Communications of the ACM,
Vol. 58 No. 2, pp. 18-20.
28

29. Gene Moo Lee, KAIST, Feb 2015
References (3)
[24] Krebs, Brian (2014). Spam Nation: The Inside Story of Organized Cybercrime - from Global
Epidemic to Your Front Door. Sourcebooks, Inc.
[25] Lee, Wenke, and Salvatore J. Stolfo (1998). “Data mining approaches for intrusion detection.”
Proceedings of 7th USENIX Security Symposium.
[26] Levchenko, Kirill, Andreas Pitsillidis, Neha Chachra, Brandon Enright, Márk Félegyházi, Chris
Grier, Tristan Halvorson, Chris Kanich, Christian Kreibich, He Liu, Damon McCoy, Nicholas
Weaver, Vern Paxson, Geoffrey M. Voelker, and Stefan Savage (2011). "Click Trajectories:
End-to-End Analysis of the Spam Value Chain." IEEE Symposium on Security and Privacy.
[27] Moore, Tyler and Richard Clayton (2011). "The Impact of Public Information on Phishing
Attack and Defense." Communications & Strategies 81.
[28] Morgan, Kari Lock, and Donald B. Rubin (2012). "Rerandomization to improve covariate
balance in experiments." Annals of Statistics 40, no. 2: pp. 1263-1282.
[29] Popadak, Jillian A. (2012). "Dividend Payments as a Response to Peer Influence." Available
at SSRN 2170561, http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2170561.
[30] Pitsillidis, Andreas, Chris Kanich, Geoffrey M Voelker, Kirill Levchenko, Stefan Savage (2012).
“Taster’s choice: A comparative analysis of spam feeds.” Proceedings of the 2012 ACM Internet
Meassure Conference: pp. 427-440.
[31] Rao, Justin M., and David H. Reiley (2012). "The economics of spam." Journal of Economic
Perspectives 26, no. 3: pp. 87-110.
[32] Roesch, Martin (1999). “SNORT: Lightweight intrusion detection for networks.” Proceedings
of 13th Large Installation System Administration Conference, pp. 229-238.
[33] Rothschild, Michael, and Joseph Stiglitz (1992). “Equilibrium in competitive insurance markets:
An essay on the economics of imperfect information.” Springer Netherlands.
[34] Sahami, Mehran, Susan Dumais, David Heckerman, and Eric Horvitz (1998). “A Bayesian
approach to filtering junk e-mail.” Learning for Text Categorization 62: pp. 98-105.
29

30. Gene Moo Lee, KAIST, Feb 2015
References (4)
[35] Shue, Kelly (2013). "Executive networks and firm policies: Evidence from the random assignment
of MBA peers." Review of Financial Studies 26, no. 6: pp. 1401-1442.
[36] Tang, Qian, Leigh Linden, John S. Quarterman, and Andrew B. Whinston (2013). “Improving
Internet security through social information and social comparison: A field quasi-experiment.”
In Workshop on the Economics of Information Security.
[37] Taylor, Robert W., Eric J. Fritsch, and John Liederbach (2014). Digital crime and digital
terrorism. Prentice Hall Press.
[38] Taylor, Shelley E., and Marci Lobel (1989). "Social comparison activity under threat: downward
evaluation and upward contacts." Psychological review 96, no. 4: p. 569.
[39] van Eeten, M., H. Asghari, J. M. Bauer, and S. Tabatabaie (2011). "Internet service providers
and botnet mitigation: A fact-finding study on the Dutch market." Delft University of Technology.
[40] Wood, Dallas, and Brent Rowe (2011). "Assessing home Internet users’ demand for security:
Will they pay ISPs?" Workshop of Economics of Information Security.
30