This document discusses how Etsy uses data metrics and analytics to secure its ecommerce platform. It summarizes how Etsy collects application statistics and metrics, visualizes data using graphs, analyzes logs and databases to detect anomalies and security incidents, and shares case studies of how metrics helped address issues like full-site SSL adoption, fraud detection, and blocking bad bots. The document emphasizes that an application's security posture is directly related to how much is known about the application from collected data and metrics.

2. Securing eCommerce
with Data Metrics
Corey Benninger

3. Founded in 2005, Etsy is an e-commerce website focused on
handmade and vintage items, as well as art and craft supplies.

5. Continuous
Deployment
Average 35 deploys
to production a day
About 10,000 lines
of code a day

6. Corey Benninger
Senior Software Security Engineer
@0xb3nn

7. https://www.etsy.com/listing/92868829/the-oh-my-orange-elephant-designer-wall
Overview
Collecting Metrics
Viewing Metrics
Taking Action
Case Studies

8. First, a thesis

9. The security posture of your
application is directly
proportional to how much
you know about your
application.

10. https://www.etsy.com/listing/96459220/stick-your-head-in-the-sand-pen-ink
Looks fine from here

11. Data Collection

12. Application Stats

13. https://github.com/etsy/statsd

14. if (preg_match(self::PATTERNXSS, $this->url) == true)
{ $msg = “attacktype=XSS url=” . $this->url;
Logger::log_info($msg, ‘SECURITY’);
StatsD::increment(‘security.potential_xss’); if (!
$this->rate->checkIncrement(self::XSS_WEIGHT)) { $this-
>drop_request = true; }
}
StatsD

15. We <3 Graphs

16. https://github.com/etsy/dashboard

18. Is this normal?

19. Is this normal?

22. Smoothing Data

23. Get Historical

24. Log Analysis

25. https://www.etsy.com/listing/130330032/all-the-things-internet-meme-embroidered
Log all the things

26. Event logs
Visit logs
Error logs
Mail logs
API logs
Search logs
DNS logs...

27. Splunk

32. Databases

33. https://www.etsy.com/listing/128620213/vintage-happiness-is-a-humongousl
Databases
Relational (row) database
Columnar (column) database
MapReduce (clustered data
processing)

34. Awesome Data Team

35. 160 nodes
3840 cores
15 TB of RAM
960 TB storage

36. ! Ad-hoc analysis of a large dataset
! Needs to be fast
(or scalable)
! Might not do it more than once
(for a data set)

37. Why: Analytics

38. SuperBIT

40. Case Studies

41. Full Site SSL
resource cost

42. Goal
Full-site SSL for all
Etsy sellers

43. Opt In

44. analytics_cascade do
analytics_flow do
analytics_source 'event_logs'
tap_db_snapshot 'users_index'
assembly 'event_logs' do
group_by 'user_id', 'scheme' do
count 'value'
end
end
assembly 'users_index' do
project 'user_id', 'is_seller'
end
assembly 'ssl_traffic' do
project 'user_id', 'is_seller', 'scheme', 'value'
group_by 'is_seller', 'scheme' do
count 'value'
end
end
analytics_sink 'ssl_traffic'
end
end

46. Incident Response
for web attacks

47. https://www.etsy.com/listing/152084181/boba-fett-the-good-the-bad-and-the
Finding Vulns
Bug Bounty Program
Launched Sept 2012
Reward: $500 - $2000

48. Needle in a haystack

49. • URL Patterns
• IP Addresses
Simple Patterns

50. analytics_cascade do
analytics_flow do
analytics_source 'access_logs'
assembly 'incident_response' do
query_event 'timestamp', 'request_uri', 'useragent', 'ip'
where '"/bad_url.php'".equals(request_uri:string)
group_by ’url’ do
count 'value'
end
end
analytics_sink 'incident_response'
end
end

51. When to Alert
setting thresholds

56. • Per time period, count password resets
• Sort the amounts
• Discard outliers
• Average remaining
• Compare with past known attacks
Big Data Answer

57. Collusion Fraud

58. The Price is Wrong?

59. Overpaid

60. Analysis
Check for meta-data
Exact hash and fuzzy hashing
Analysis of key properties
(shadows, patterns, shading...)

61. Grow Stronger
Detection is timely (hashing ~1ms)
Each new data point helps for analysis

62. Phishing Attack
reactive to proactive

63. Not Etsy

64. Reactive

65. source=”access_logs” client_ip=10.163.2.3 | transaction request_uri
Incident Response

66. Normal

67. Proactive

68. Scanners
low hanging fruit

69. Bad Deploy?

71. https://www.etsy.com/listing/162962424/robot-dress-up-costume
Block Only Bad Bots
Allow legitimate users
(including API requests)
Allow search engines
Allow our own scans

72. Asimov?

73. Bad Bots

74. Bad Bots

75. False Positives

76. https://www.etsy.com/listing/159148839/robot-card-trust-no-one
Bad Bots
Disobey
404
Time
Announce

77. Detection
Nick Galbreath at DefCon 20
“LibInjection” for detecting SQLi
Does it parse as SQL? Yes, then it’s
SQL
Do you have “.aspx” files? No, then why
is someone requesting one?

78. if (preg_match(self::PATTERNXSS, $this->url) == true)
{ $msg = “attacktype=XSS url=” . $this->url;
Logger::log_info($msg, ‘SECURITY’);
StatsD::increment(‘security.potential_xss’); if (!
$this->rate->checkIncrement(self::XSS_WEIGHT)) { $this-
>drop_request = true; }
}
Log and Limit

79. 439 - Not Handmade

80. Check the Graphs

81. Conclusions

82. ! Instrument your application, log
everything
! Get familiar with data resources: people
and tools
! Use your data to help drive security
alerts, investigations, and actions

83. Thanks!http://codeascraft.com