Data Driven Security, from Gartner Security Summit 2012

AgendaWho am I? Who is Etsy? Splunk at Etsy?Web Application SecurityAccount TakeoverPayments and PCICredits, Data, Further Reading 2

@ngalbreathhttp://slidesha.re/KPvHYu 3

Whois Nick GalbreathDirector of Engineering at Etsy covering:– Fraud– Security 2– Support Engineering -06-1– (and other stuff outside of this talk) 2012 y is mSoftware Development background in two year e rsaryE-Commerce and Social Media anniv tsyBooks, Patents, Oh My… http://client9.com/ at E 4

$525,000,000 in community sales 875,000 active sellers 41MM unique visitors 15MM registered members 150 countries5

What Could Possibly Go Wrong? • Marketplace Risk like Big Auction Site • Payment Risk like Payments Company • Social Risk like that Big Social Network With a member base frequently: • New to Etsy • New to Running a Business • New to the InternetPhoto Credit: Rod Ramsey http://bit.ly/KnI8uB 6

To Make It More Interesting: Continuous DeploymentOn average, there are 50+ production code changes per day. So when wehave a problem:Is it an operations problem?Is it a development problem? Learn moreIs it a product problem causing http://bit.ly/KFYYlZcomplaints to come in?Or is it an attack? 7

Old Workflow: #notwinning Logging into production network (!) Finding the right file Unzipping the right file Grepping• Writing very clever scripts to extract data• Writing more clever scripts to merge data• Making a report – in plain text 34 minutes for• Alerting one day’s log for nothing! 8

Splunk installed at Etsy mid-2010"Hey. .. lets go try this NEW thing!" (door slamming shut) "Sorry.... were closed.” Steve Martin. Comedy is not pretty. 1979. Track 8 ~2:45 Serious New Technology Fatigue Why don’t we use a Real Database with SQL? Grep technology works &^#*&@^*#^%^ YAQL – Yet Another Query Language 9

L’OutrageThen a colleague: • Didn’t know Etsy’s stack (new) • Remote and out of office • Didn’t have production access • Didn’t know any of my very clever scripts • Not experienced with Splunk • In about 30 minuteswhips up a real-time email alert fora velocity check on a particular URL I only have one thing to say about this….. 10

OH, YEAAHH! 400+GB indexed per day 30+ TB total storage 60+ data sources from “hundreds of servers” (via central syslog aggregation) 11

Data-Driven Security Three examples of how we use data and Splunk to help make Etsy a safer place to conduct business. •Web Application Security •Account Takeover •Payments and PCI That said we are barely scratching the surface of Splunk!Data-Driven By Mat Edelson. John Hopkins Engineering Magazine, Fall 2011http://eng.jhu.edu/wse/magazine-fall-11/item/data-driven/Illustration by Mark McGinnis No association, just a great article & illustration 12

WebApp Security

Make Security Visible Your peers actually are interested in security. But are you letting them? Turn security from a binary event into a continuous event. 14

Detect the Steps A journey of a thousand miles begins with a single step. Lao-tzu, China 600BCA single breach begins with a journey of a thousand steps. Nick Galbreath, USA 2012AD 15

SQLi, XSS, CSRFsource=“info.log” log_name_space=“SECURITY” attacktype=“XSS” That was easy 16

SQLi, XSS, CRSFsource=“info.log” log_name_space=“SECURITY” attacktype=“SQL”| geoip ip Paints a different picture 17

The Dumbest Check Possible for SQLiWe have some snazzy technology for detecting SQLi in Splunk, but youdon’t need it to get started:source=access.log (uri="*UNION+ALL*" OR uri="*UNION%20ALL*”)Will wildly undercount but also low false positive rateWill detect scans from various toolsWill get you started in making security visible 18

SQLi and Database Errorssource="error.log" ( "syntax error" NOT "smarty" NOT "ClientLogger" ) | eval event=_raw | table event• We use Splunk to alert on any database syntax errors too.• SQLi attacks and probes will likely trigger a bust of syntax errors if code doesn’t properly sanitize data was That e clos Do the same with server 500 errors, core dumps 19

Investigating Rent-A-CPU Trafficsource=“access.log”| lookup datacenter-cidrs provider_cidr AS true_client_ip OUTPUTNEW provider_name| where isnotnull(provider_name)| top provider_name Publi c Dat a S ee Appe ndix 20

Attacker-Driven Testing“I thought I found something but then it stoppedworking…” Email to security-reports@etsy.com from ethical hacker Attacker-driver testing augments Etsy’s proactive security measures Splunk alerts us on potential attacks using a number of parameters What URLs are being targeted? Maybe they found something? Can it be reproduced? (sometimes completely automated validation) Fixes can be pushed out that day, if not within minutes. 22

Security Post-MortemsFor any security vulnerability, found either external or internal,exploited or not, we hold “blameless post-mortems”Use to teach about security issues e.g. review OWASP Top 10 http://bit.ly/fXsJg6Can we make it so this mistake doesn’t happen again or can beautomatically detected?A Key to post-mortem is know when something started and when itended. Logs “at your fingertips” via Splunk helps greatly(and absolutely essential for actual incidents) 23

Account Takeover

Account Takeover • Stolen credentials • Brute forcing of credentials • Using account takeover of email to further takeover other accountsHorrible for victim and really slow to clean up 25

Many Users Failing to Sign-in from One IP source=“info.log” log_namespace=“login” reason="wrong password” true_client_ip!=38.117.156.X X X | dedup etsy_username,true_client_ip | transaction true_client_ip | where eventcount > X X X X | table true_client_ip,etsy_username | geoip true_client_ip | table true_client_ip,true_client_ip_countryname,etsy_username 26

Brute Forcing Passwords?source=”info.log”log_namespace="login” Peop le wilreason="wrong password" try 10 ltrue_client_ip!=38.117.156.X X X passw 0 ords| transaction etsy_username manu ally| where eventcount > XXXX| table etsy_username,true_client_ip,eventcount| sort -eventcount Frequency Buckets set in Splunk Dashboard 27

I Forgot My Password x1000 source=“/web/access.log” request_uri=/forgot_password.php http_method=POST| transaction true_client_ip| where eventcount > X X X| table true_client_ip,eventcount o from| sort –eventcount Hell bia! Ser Not just fraud… has disclosed problems in email transport and product problems with our reset flow 28

Apply the same analysis to other things that should not change much– Payment cards– Email addresses– Passwords (successful change)– Regular physical addresses 29

CAPTCHA Splunk 2x2 dashboard keeps us in-the-know on how often CAPTCHAs are being shown, to whom, and how often they pass.reCAPTCHA http://www.google.com/recaptcha 30

Integrated into Support Tools Splunk is glued into our internal tools used by General Support and MITS (Marketplace Integrity / Trust & Safety) teams. 31

Payments and PCI

Payments @ EtsyRamping up on our own payments platformFull PCI EnvironmentWith separate Splunk installation This space intentionally left blank. 33

Alerting on Unusual Payment ActivityAll the WebApp security and account take-over rules apply, along with special checksfor payment activity Abnormally large payments Part of Payment velocity a larger paymen Very small payments (skimming?) t risk solution The usual IP address checks. 34

Compliance and ReportingInstead of building custom applications with fuzzy requirements“Log it, let Splunk figure it out later”Even the business guys can use it for ad-hoc queries.Unexpected side effect: removing and/or changing data is really hard.This is good. Compare to SQL. (Splunk also has a secure log system)Easy to make reportsPCI QSA so far says this meets PCI requirements. 35

Internal RiskAgain, instead of build out of newapplication (with fuzzy requirements)Log It, Splunk it later.Who, is what making what changesWho is looking at potentially sensitivedataAnd alert on it.Used in payments and main support Etsy Support and MITS 2012applications 100% Good Eggs Team Etsy 2012 36

Credits

AcknowledgementsThis presentation would not be possible without the hard work by: Marcus Barczak Jerry Soung Zane Lackey Operations Fraud and Risk Security Engineering EngineeringBig thanks to everyone at Etsy in Engineering, Payments, Operations,Support and MITSAnd of course, the fine folks at Splunk! 38

Data and ReferencesDatacenter IP List: https://github.com/client9/ipcatISC Top Troublemaker IPs: http://isc.sans.edu/ipsascii.html http://isc.sans.edu/sources.htmlOn Security and Continuous Deployment: http://bit.ly/KFYYlZOther presentations on Etsy and Security/Fraud/DevOps: http://slidesha.re/Kw5zdV http://slidesha.re/IMaavq http://slidesha.re/JGaU2s 39

Security Engineering and “Just Culture”Treating security mistakes as “accidents” (whether exploited or not)Based originally on health care initiativesPatient Safety and “Just Culture”, David Marx JD– http://psnet.ahrq.gov/resource.aspx?resourceID=1582– http://bit.ly/LhRHaT (presentation)John Allspaw on Blameless Post-Mortems:http://codeascraft.etsy.com/2012/05/22/blameless-postmortems/ 40

www.etsy.com It’s time for questions! Nick Galbreath @ngalbreatht t p : //s l i d e s h a . r e /K P v H Y u

http://tecnologiaeninformaticaparaelfuturo.blogspot.com	74
http://tecnologiaeninformaticaparaelfuturo.blogspot.mx	33
http://tecnologiaeninformaticaparaelfuturo.blogspot.com.es	3
http://tecnologiaeninformaticaparaelfuturo.blogspot.com.ar	2
http://www.linkedin.com	1
http://www.tecnologiaeninformaticaparaelfuturo.blogspot.com	1
http://www.twylah.com	1
http://tecnologiaeninformaticaparaelfuturo.blogspot.ch	1

Data Driven Security, from Gartner Security Summit 2012

by Nick Galbreath on Jun 12, 2012

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 116

Statistics

Data Driven Security, from Gartner Security Summit 2012 Presentation Transcript