1. The Snake Oil Cycle
Dave Cole, dave@openraven.com

3. OPEN RAVEN
Modern Attack Surface Management

10. Meet Clark Stanley
It’s not about Clark or even a type of person

11. Genuine problems lay beyond the

13. The obvious stuff
If you did a great job, many times nothing
bad will happen. Which is impossible to
prove. The really compelling “before &
after” stories are elusive.

14. I’m going to be unfair &
pick on another product
category & a single
company as an example.

15. Imagine this in security, esp
in the middle of an incident…
measure progress in months!

16. Standard expectation: real-
time or very close to it.

17. Our expected uninvited guests

18. I’m sure it’s hard to build an HRIS, but their
Alice & Bob don’t have an Eve. Or a Mallory.
Or a Trudy.
Or re-architect because a new attack type
changed their original assumptions.
Or crank a new remediation engine
because they can’t remove the latest
threat.
Or make sure that OS change didn’t just
bust your behavioral defenses.
Or properly identify every IoT device.

19. A short
story.

22. Building security products is hard.
Being a security professional is hard.
Natural empathy should drive us together.
It rarely does.

23. The U.S. employs
nearly 716,000 people
in cybersecurity
positions, with
approximately 314,000
current cybersecurity
openings
https://www.cyberseek.org/heatmap.html

25. Breach Level Index 2018

26. https://blog.rapid7.com/2018/04/30/cve-100k-by-the-numbers/

27. https://blog.rapid7.com/2018/04/30/cve-100k-by-the-numbers/

31. “7% of all S3 buckets have unrestricted
public access, and 35% are unencrypted”
SkyHigh Networks (now McAfee) September 2017… before Amazon changed policy settings to proactively warn users of public buckets

32. Who When What
Booz Allen Hamilton May 2017 Battlefield imagery & admin credentials to sensitive systems
Deep Root Analytics June 2017 Personal data about 198 million American voters
Dow Jones & Co July 2017 Personally identifiable information for 2.2 million people
WWE July 2017 Personally identifiable information of >3 million wrestling fans
Verizon Wireless July/Sept 2017 Personally identifiable information of ~6 million people &
sensitive corp. information about IT systems, incl. login creds
Time Warner Cable Sept 2017 Personally identifiable information about 4 million customers,
proprietary code & administrator creds
Pentagon Exposures Sept/Nov 2017 TBs of info from spying archive, resume for intelligence
positions—incl. security clearance & ops history, creds &
metadata from an intra-agency intelligence sharing platform
Accenture Sept 2017 Master access keys for Accenture's account w/ AWS Key Mgmt
System, plaintext customer password DBs & proprietary API data
Natl. Credit Federation Nov 2017 111GB financial info—incl. full credit reports-- ~47K people
Alteryx Dec 2017 Personal info of ~123 million American households
https://businessinsights.bitdefender.com/worst-amazon-breaches

34. On average, sec teams are understaffed,
undertrained, under intense pressure from
increasing threats & vulnerabilities with
regulatory mandates to make fast
decisions… & circumstances can change
w/o warning.
Let’s try to find them some help...

36. Security design can sometimes look like this

37. Which leads to this

38. Historically we have assumed the person
behind the console is an expert

39. Increasingly, we have products designed for
experts in the hands of novices.
The bar must be lower.

40. Before we move on, a word on how products
are evaluated prior to purchasing*
*This is hard to generalize, your mileage may vary

41. Proof of Concept

42. Normal Operations

43. OMG Operations

44. What if someone had a solid test lab that we
could all lean on for quality, unbiased
product reviews?

48. There’s no genuine equivalent for the
security community… why?

49. They are spread too thin, like every other
journalist on the 24x7 coverage beat.

51. They also have the same pressures as
political journalists– if you’re overly critical
you lose access.

52. They typically do not have the lab
environments they require. Most vendors
have someone dedicated to lab ops at scale.

53. Even if they had the labs they need, the skill
set required to test one product is not what
you need to test the next product.

54. And if they truly were this skilled, they would
be unlikely to remain a journalist for long.

55. So our security reviewers are unlikely to
have the focus, objectivity, labs & skills they
need to publish great reviews. And if they
did, we would hire them in a heartbeat.
What about the industry analysts?

56. Same
• Increasingly spread too thin; lack of focus
• Typically little or no experience as
practitioner
Worse
• Purely anecdotal-- no real tests
• $$$ of a subscription; not readily available
Better
• More direct customer feedback
• In-depth reports w/ longer horizon

61. AntiVirus Industry – An Exception

62. What Opinion
File scanning tests Worked fine until this was only a subset of defenses… early 2000s
On access tests About the same as file scanning
Retrospective tests Purgatory b/w file scanning & real world tests; misleading
Real world tests When done properly, effective; rarely done properly
Remediation tests Few and far in between…
Performance tests Largely a success, materially improved the industry

63. At their best, public tests push vendors to
make better products, but they are too often
designed poorly or easily misunderstood.
Usually, they simply don’t exist.

64. So what’s a security marketer supposed to
do?

66. Hypothetical example: new scan engine

67. Product Person
“Our year long effort to create an awesome network
scanner that we can improve more rapidly and
support better in the future just completed! In some
instances, you’re going to see some nice speed
improvements.”

68. Marketing
“Our year long effort to create a strong network
scanner that we can improve more rapidly and
support better in the future just completed! In some
instances, you’re going to see some nice speed
improvements.”
Customer value not
readily apparent to many
Vague

69. Post Marketing
“New scan engine frequently boosts scans by 30%,
reducing time needed to scan and easing network
impact in many instances.”

70. Customer
“New scan engine boosts scans by 30%,frequently
reducing time needed to scan and easing network
impact in many instances.”

71. Recap
Opaque product benefit
↓
Well-intentioned attempt at clear message
↓
Misunderstanding due to nuance, pressure
↓
Disappointment & loss of trust

72. Marketing security products often requires a
strong grasp of nuance, understanding not
only the tech, but also how the customer will
receive the message. These people are rare.

73. And even if they are really that good, things
change quickly. And they change security
markets… which changes everything.

74. Marketing is also affected by how dynamic
security companies are– more so than other
functions. And more so than other areas in
software.

77. Often distracted (2018 #s)
• 184 M&A transaction
• 406 Financing transactions

78. Even if you can avoid the distractions, any
company will change significantly as it
briskly moves from start-
up→growth→behemoth
This often alienates employees & users

79. “I have never seen such a fast-growing market
with so many companies on the losing side”
David Cowan, Partner, Bessemer Venture Partners
Under threat: Cyber security startups fall on harder times
Reuters, January 17, 2018

80. If you knew you had to show strong sales
performance to establish your company, but
were unsure of its longevity, how would you
build & sell your product?

81. Given the rough & tumble nature of the
industry, far too often companies optimize for
creating the “reach for your wallet” moment
over establishing a strong foundation

82. Closing Thoughts
Breaking the Cycle

83. It’s too easy to blame any one person or
type of person, the problem is pervasive
& self-reinforcing

84. Security Professionals
1. Do a process walk thru w/ vendor or 3rd party
2. Do whatever you can to best simulate genuine
production conditions before purchasing
3. Ask for a “reverse roadmap”
4. Insist on references, preferably long-term
ones
5. Take updates & new versions ASAP

85. Security Vendors
1. Definition of done: customer is successful
2. Hire & empower a real design team
3. Build out a complete customer feedback loop
4. If no one else will do it, produce your own
metrics & be ready to share & explain them
5. Invite Sales & Marketing into “the factory”

86. Thanks
Dave Cole, dave@openraven.com