Download to read offline
Uploaded byAdrian Sanabria
RSAC 2016: CISO's guide to Startups
The presentation outlines the challenges and processes involved in purchasing security products from startups, emphasizing the need for a tailored due diligence approach. It highlights the rapid emergence of new security startups and the importance of understanding their potential risks and benefits. Attendees are encouraged to apply a structured framework for evaluating startup offerings and to engage thoughtfully in vendor relationships to maximize ROI and avoid pitfalls.
More Related Content
Similar to RSAC 2016: CISO's guide to Startups
RSAC 2016: CISO's guide to Startups
- 1. SESSION ID: #RSAC Adrian Sanabria SecurityStartups - The CISO’s Guide to Flying High Without Getting Burned PDIL-W03 Senior Security Analyst 451 Research @sawaba
- 2. Enjoy the presentation,but there’s more! 2 Three ways to get a copy of this session’s supplemental handout: 1. Send an email to sawaba@zip.sh with rsa2016 as the subject 2. Go to http://zip.sh/z/sawaba/rsa2016 3. Scan the QR code to the right Note: I’ve been told QR scanning might not work well in this environment, so YMMV.
- 3. Mature security productshaven’t kept up Products from startups are unproven - an unknown risk Rock and a hard place? Why are we here? 3 The process of buying security products for the enterprise is broken
- 4. What are weup to? 4 Agenda What you need to know about startups before doing business with them This isn’t your CFO’s due diligence... Due diligence in a 6-stage process Advice and stories from the trenches Goals Learn tips and advice for fixing the process of buying security products Understand how doing business with startups is different Leave with a framework to put into practice and the resources necessary to be successful with it
- 5. #RSAC What you needto know about startups
- 6. The security industrymoves fast WE SEE… WE HAD… 6 9 new startups every month 5 new categories every six months 1238 enterprise security companies in our database 134 security M&A deals in 2015, worth… $9.98 billion, with an average of… $192m paid by acquirers
- 7. Greener grass 7 security start-up nounsi-ˈkyu̇r-ə-tē ˈstärt-ˌəp A new company you will pay to do a better job at something you already pay an older company for, though the new company has less experience doing it, there are no guarantees it will do a better job and you’re going to keep paying the older company.
- 8. Why do securitystartups exist? 8 • Displace existing vendors • Address (security) gaps • Solve technical challenges • Address new market segments or environments Security startup goals aren’t that different
- 9. Why do securitystartups exist? 9 Security is always a secondary or enabling layer
- 10. Understanding the startupcycle 10 Idea Founded Seed Funding GA/MVP Growth & funding Exit Founders leave Acquisition? Acquisition? Acquisition? Founders leave?
- 11. Cutting through themarketing 11
- 12. How do Ifind a startup? Security startup pool InfoSec Mgrs Industry Analysts Cons VCs Forums Email, LinkedIn, Cold Calls Partners Sales Pres, Demos
- 13. #RSAC Getting the mostout of a startup relationship through due diligence
- 14. What does ‘duediligence’ mean to you? 14 That’s where I send the vendor a checklist with items like ISO 27000, SSAE 16, HIPAA and PCI on it, right? List of references Financial stability Company history Compliance Customer Complaint history Insurance Audit results (SSAE 16, ISO 27001, PCI) Contracts Breach/IR plans
- 15. What does ‘duediligence’ mean to you? 15 Does the product work? Can vendor claims be validated? How could efficacy be measured and compared to other options?
- 16. How do youvalidate a security product actually works? 16
- 17. A startup-specific duediligence process 17 1 Get the big picture • Find gaps • Determine greatest needs 2 Create requirements • Based on needs and resources • Budget • Staff • Skills 3 Vendor research • Find targets • Research targets 4 Initiate Relationship • Start conversation • Test product 5 Make/Break • Does it make sense? • Feedback loop • Formal relationship 6 Manage relationship • Product/vendor monitoring • Product development feedback loop Search cycle Dating cycle Not quite ready… Try again!
- 18. Take a stepback 18
- 19. The process 19 Research thestartup (“Passive Recon”) Engage the startup Ensure a good product/environment fit (avoid Shelfware!) This is a startup: the roadmap IS the product Proper preparation makes the most of your PoC Contracts, agreements, liability – rubber, meet road Uh-oh, they got acquired!
- 20. When you engage… 20 Don’tshy from questions*: “We’re 62 minutes into this sales presentation and I don’t know what your product is.” “Plan to dump before you jump” (i.e. Have an exit plan before you start) You are a valuable asset to a startup; this gives you leverage Use this leverage! * - real story
- 21. Ensure a goodproduct/environment fit What is shelfware? Why does it occur? What ends up on the shelf?* 21 * See handout Top five reasons products become shelfware according to buyers: 1. Compliance-driven purchase 2. Internal Politics (tied for #1) 3. Lack of staffing/headcount 4. Lack of time/expertise 5. Features overpromised or missing
- 22. Roadmap fit 22 Be clear:what are you willing to wait on versus need now? Integration path – just APIs or deeper partnership? Platform-based architecture? What are the long-term goals? Are they feasible/reasonable? Better Best Unicorn Unimaginable wonders to behold The average roadmap
- 23. The value ofsecurity products Can you calculate the value you should get from it? What’s the Time-to- Implementation? What’s the Time-to-Value? What’s the True Cost? 23 Drawing and concept by Henrik Kniberg http://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp
- 24. Example: the valueof threat intelligence 24 …box of rocks threat intelligence! $10k
- 25. Example: the valueof a SIEM 25 $1.5M Per year
- 26. Advice from thetrenches 26 Q: What are some challenges to watch out for? A: Overly vague descriptions of their IP. Not being multi-platform ("oh, we'll support Macs in our *next* major release!"). “…figure out how to short circuit the purchasing system… the startup needs your money more than you do...” –Richard Stiennen
- 27. Advice from thetrenches 27
- 28. A story fromthe trenches 28
- 29. ***SEND PACKET*** FLIXMU WIFI-PRODUCT WIFI-PRODUCT 0007E897A65E5 172.23.1.6 1.245.10 ProductName 1.00 A71978AC4B00 2012-10-03-14.10.10.000000 Underestimatingthe difficulty of properly designing a cloud-managed architecture 29 + 0007E97A65E5
- 30. Lessons learned 30 Why didthis happen? Small company Three engineers No Security expertise No third-party security audit Conclusions Due diligence of technical products requires technical assessments Ask if a third-party audit has been performed Consider impact and liability to other customers before taking assessment too far Keep pressure on the vendor to fix the issue, even if you decide not to buy the product
- 31. Recommendations: brace forimpact 31 Not comfortable? Don’t do it, or do it through a trusted partner Don’t have the spare staff/skills/cycles? Don’t do it. Plan to lose most of one FTE’s productivity to testing, implementation and bug reporting activities, at least initially. Look for products with a high potential reward/effort ratio - threat prevention technologies, for example. Check workflow integration before purchasing!
- 32. Shoutout: Yu’s CyberDefense Matrix tools 32
- 33. Apply what youhave learned 33 Later today you should: Check out Sounil Yu’s Cyber Defense Matrix Follow-On talk at 4:30pm in West 2016 This week you should: Take the vendor marketing challenge in the expo: don’t be afraid to ask questions Within three months of this conference: Go through the first half (steps 1-3) of the due diligence cycle for at least one product Have a few trusted sources for gathering information/recommendations on startups Within six months: Go through the second half of the due diligence cycle (steps 4-6) Refine your due diligence process and share your results with others if comfortable
- 34. Thank you! 34 Please, continuethe conversation, chat or ask questions: Twitter: @sawaba Adrian.Sanabria@451Research.com Adrian.Sanabria@gmail.com Spiceworks (sawaba) Peerlyst
Editor's Notes
- #3 There was a lot more information I wanted get into this talk, but there just isn’t time to show it all to you. For that reason, I’ve created a handout and some other additional materials for you. Feel free to look through it during this presentation, or just go ahead and
- #4 The process of buying security products for the enterprise environment is broken. Throw startups into the mix with that last statement, and you’re in for a ride if you aren’t prepared. Traditional due diligence doesn’t work for security products, and often, it just doesn’t happen, period. Mature security products haven’t kept up Products from startups are an unknown risk Rock and a hard place? Two VERY important concepts with startups: Startups are effectively a reverse lottery – we don’t actually know if their product has value until AFTER they’ve received money for it. It is possible for a security startup to be a complete success in the BUSINESS sense, while being a complete FAILURE in terms of delivering value to the customer. It is inherently very, very difficult to prove or demonstrate efficacy or value with security products. THIS is why I say that the process of buying them is broken.
- #5 Emphasize the problem statement and goals of this talk: Problem statement: Startups may get pitched as the solution to all our problems, but there’s a lot I’d like to share with you before you jump into these waters. At the risk of spoiling my talk, I’m going to tell you up front that effective due diligence is going to be 90% of what you need to avoid getting burned when working with a startup. The other 10% is understanding startups and applying the former 90% to the latter 10. Goals/Agenda The title mentions CISOs, but this talk should be useful for anyone involved in the process of finding security products that are a good fit for their company. Some tips on how to evaluate both startups and their products; I’ve also got some information from CISOs and startup founders to share. Address the duality of cutting edge startup technology that may or may not be ready or suitable for your business’s production environment.
- #8 Is the grass always greener? Are we really getting any better with each startup iteration?
- #9 Or I could add a diagram showing how security is a layer on top of everything else
- #10 In order to have network security, we need to have a network. In order to have endpoint security, endpoints first need to exist. Ditto, cloud, mobile, etc.
- #11 It is important to understand this cycle prior to working with startups. We’ll reference this more later on. Startups can stay in stealth mode for years! This is often when many startups begin to look for companies to be beta testers or early adopters/partners The growth/funding cycles usually represent most of a startup’s life Spend a little time talking about how startup relationships are different from normal vendor relationships: though due diligence is pretty much the same, regardless of security product or vendor, the startup relationship is very different. Point out that VCs/investors are often on the startup’s board, so they’re a good PoC to start the relationship from You are not merely their customer - you are their beta tester, their partner, their guide, potentially their advocate. Don’t just ditch them if you don’t like things about the product, WORK with them. Be honest. Be prepared to spend a lot of time and effort giving feedback. Without brutally honest feedback and thorough reviews of issues and problems, the startup can’t get better – YOU have the perspective they lack – the perspectie they need. If you can’t do these things, don’t go after startup relationships. By nature, they require more patience and support. Don’t expect mature version 8 products. We’ve seen several acquisitions before some startups even come out of stealth. We don’t even know what we missed! It is also important to understand the cycle represented here is a very loose representation of the path the average startup takes, and can be disrupted at any stage by a number of events. We recently saw one startup get successfully sued for every dime of revenue it had ever made!
- #12 Can we tell what all thee products do? Not really – this is all hyperbole. The problem with a lot of this is that the marketing tends to intentionally divert your eyes away from the gaps each product has, making your job that much harder. I will say, some of these vendors are very up front, and while their marketing might look like everyone else on the surface, when I do a briefing with them, they’ll even have a slide titled “what we don’t do” or “our challenges”. THAT’s a good sign. Any vendor that will point out their own products’ weak points is a vendor making your job easier. The less work you have to do to cut through the marketing, the more time you have to get the job done. CHALLENGE: Visit the expo and find a company you aren’t familiar with. See how long it takes you to figure out what this company and their products do. If you have someone with you, make a game out of it, even – time it. I’d love to hear your experiences and results – this is what I do as part of my job almost every day! <-- Tweet this.
- #13 How do I find a good startup? They’re almost certainly already spamming you to death in your inbox, at conferences, on websites, in airports… How do I cut through marketing and hype? I can’t figure out what the hell they do or whom they’re actually competing with. Put together a trustworthy network of other CISOs, VCs and industry analysts. You need informants! Places to discover startups and discuss experiences: PeerLyst, Spiceworks forums, conferences Put emphasis on sources that have experience with either the startup’s people or products. You need a good relationship with both. VCs and Analysts are the most likely to know the founders, their vision and will likely have an opinion on whether they’d recommend doing business with them Other InfoSec managers and forums are good places to find people familiar with using the product Instead of considering it a bonus when new products can integrate with your other core security products, why not make integration a priority? Open a dialogue with existing vendors you’re happy with. Ask them who their technology and integration partners are, and start your search there. Most vendors will go through their own due diligence and vetting process before entering into a partnership agreement with a startup. It’s not a guarantee, but it is another layer of assurance that costs you nothing and you wouldn’t have otherwise. Webinar-style sales presentations and Demos – forget it, at least as an introduction to the startup. If you do this, leave it for later, after you already have a good feeling about the company, a recommendation, etc. It is just too much time to spend on something for far too little value. It generally takes me about 5 minutes on a company’s website to figure out: what a product does what it’s core features and capabilities are Other products it can integrate with Whether or not it roughly fits what I’m looking for. Often, you can even find short videos or pre-recorded content to help explain how the product works So… why the hell would I waste an hour watching a webinar or demo to get the same amount of information? First, they’re going to spend at least 10-15 minutes explaining to you why you need the product. We call this the “problem statement”. You already understand the problem – that’s why you’re actively searching for a product in the first place. Tell them you understand the problem already, and they can skip that part. Better yet, if it is a scheduled meeting, set the agenda yourself and list your expectations for the meeting. Ensure that there is someone technical on both sides to understand any architectural or compatibility showstoppers answer technical questions spot (from your side) egregious BS Personally, I’m thrilled when I can try out the product myself without having to talk to anyone in sales, or even a video of someone using the product can give you a good feel for what it can do without having to make a time commitment and blocking off time on your schedule. Finally, there are just too many vendors out there to allow each of them to take an hour of your time individually. You also can’t PoC everything. It just isn’t feasible for anyone except dedicated labs.
- #14 Awww, patience? Really?
- #15 Ask the crowd how many already have some sort of formal due diligence process they use? Chris Tarnovsky’s idea of due diligence might involve dipping a TPM chip into a hydrochloric acid bath and polishing away layers of ceramic and silicon so he can explore microprocessor secrets with an electron microscope. That’s one extreme. The other is almost entirely business-oriented. Yeah, you should probably do all the business stuff. In fact, someone has probably already automated that process for you. HOWEVER...
- #16 THIS is what bothers me about due diligence. I’ve run into situations where security products actually lowered our security posture and had negative value for at least the first year.
- #17 The process of buying security products for the enterprise environment is broken. Throw startups into the mix with that last statement, and you’re in for a ride if you aren’t prepared. Traditional due diligence doesn’t work for security products, and often, it just doesn’t happen, period. Mature security products haven’t kept up Products from startups are an unknown risk Rock and a hard place?
- #18 Due diligence is a process, rather than a cycle, but it contains several cycles. The big picture – what are you missing? Requirements – you know what you’re missing, but what do you have the staff, budget and skills to take on? Vendor research – find startups that meet your needs and begin researching them. Approach the vendor to begin the relationship – begin testing the product Make/Break – does it work or not? If yes, go to #6 If no, go back to #4 Manage relationship maintain relevance keep product up-to-date and running smoothly Provide feedback for the startup on problems, successes and requests
- #19 What do we need most urgently? What are our requirements? What resources do we have to spare? Architectural/deployment options? Where do we need it to work? How well does it need to scale? What are the architectural/deployment options Do I have a hard requirement for the technology to live on premises? Many do for a variety of reasons. Do I prefer SaaS (not having to devote hardware, rack space or fire up a VM on-site)? Do I prefer a managed offering (this is a labor-intensive product, and I know I can’t run it properly myself – we need help)? Are there hybrid options? What would that look like? What’s the environment coverage/efficacy? Where does it work? Does the product work for my employees and assets that leave the corporate network? Examples: There’s a few “agentless” endpoint security products out there – how does that protect my laptop while I’m here at RSAC? Perimeter network security products (use my diagram?) usually does nothing for mobile devices, assets that are off-prem or assets that regularly travel on and off prem The vast majority of security budgets are concentrated here! Does the product cover all my endpoints? Or just 0.1% of them (my crown jewels???) AND only when they’re on the corporate network? “How much of your security budget is still protecting endpoints when they are at employee’s homes? Coffee shops? Hotels? Airports? Can I deploy it to servers? What about auto-scale environments? What does creating and destroying 8000 servers every day do to the reporting and management console? Can it scale? What’s the asset coverage like? Ask for a list of product features and capabilities Ask to see the most recent roadmap Who is going to be responsible for the product? Who will work with support? Who will patch it and upgrade it? Who will go to training and become the SME? Does it make sense to have an SME, or is a SPOF a bad idea?
- #20 You want to know as much as possible about them and their product before you ever talk to them. It will save you a TON of time, honestly.
- #21 Working with a startup is more like a partnership than typical customer/vendor relationship. Don’t put all your eggs in this basket! Always have a plan to replace this vendor, should something happen The more critical this vendor is to your company’s security, the more important this plan becomes. If you can’t easily replace this vendor, consider other options. Don’t get into a vendor relationship lightly if you know it will be tough to extract or replace their product. You can potentially help them as much as they can help you! You can spread recommendations through word of mouth You can lend credibility to their product claims You can help them land other customers, additional investors, potential acquirers... You have leverage offering to talk to other potential customers allowing them to publicly state you as a customer (Be willing to trade public endorsement for promise to build for your needs. Startups need customer wins they can share publicly. Most companies don't like to publicly share what they're using internally, like that somehow makes them safer.) allowing them to do a case study Use this leverage! Don’t settle for a product that still requires you to do significant amounts of custom development to get it working properly Plan to spend considerable time reporting bugs, errors, issues - worst-case, be willing to devote at least one person full-time to this product. That said, define your limits before you begin and be willing to cut your losses if you need to.
- #22 Do you really need it? Platform or ‘feature as a product’? Workflow fit Staff Managed options Integration, APIs Do you really need this product? How would it improve your company’s security posture? Be careful not to get sucked into “cool new toy” syndrome Is it really a standalone product or something that will soon become a feature of a larger product? Does it fit into your workflow? Does it fit with how you and your team run security in the organization, or does it require significant new training, new consoles, new operational responsibilities, a new source of logs... Email is not an API! Does this product integrate with your existing IT and security products? Are they planning on an MSSP offering of the product, or partnering with an MSSP? Staff - Is your staff skilled enough to use the product effectively? Do you have enough staff to use the product effectively? It doesn’t matter if their product is the greatest thing in the world if you can’t effectively use it. As boring as it sounds, things like active directory and ticketing system integration can be as critical as whether or not the product can actually make good on its claims
- #23 Ask to see their roadmap. What’s the long-term vision for the company? Does it jive with what your plans are? Is the product architecture designed as an extensible/modular platform? Are they integrating with other vendors now, or do they plan on formal product/technology integrations? Do they integrate with your current vendors and products you already use? Are they willing to? Do they have an open API?
- #25 Fun Scenario/Analogy - the point here is to think about what the product does, and determine whether or not it even provides any value to you. It is shocking the number of companies that don’t consider this. You own a shop, and someone recently threw a rock through the window. To prevent this from happening again, you replaced the previous glass with shatterproof glass. (prevention technology) A salesman comes into your shop the next day and says, “hey, I saw you had a rock thrown through your window the other day. I’ve got a great deal for you.” He tries to sell you a “threat notification” service. “What’s that?” you say. The salesman lays down a photo (box of rocks) “That’s a box of rocks”, you say. “No it isn’t!” the salesman protests - “this is threat intelligence! Someone could throw these at your nice, new windows! You need to know about them!” “Okay”, you say, “how much are you charging for this… service?” “I’ll send you these pictures every day, and you pay me $10k a year for this service.” You point out that the shatterproof glass you installed isn’t perfect, but it should prevent against most rocks that the average person could even lift and throw.
- #26 Ditto for SIEM. Is it working? I don’t know, you have to do a bunch of custom development work to even figure out if the product is working. Photo of construction supplies “Here’s your house!" $1.5m Per year And you’ll need to hire some people to build it If you want a garage, that will be extra No shingles, eh? The roof is extra. Back to reality from analogy: SIEM is a development platform, not a product. Learn to recognize the difference! You need skilled people to build it and put everything together Support for logs from unsupported products or custom apps: build it yourself Threat intel: “we support STIX and TAXII” <— in other words, go buy it separately yourself. Good luck integrating it. Triggers/Automation: build it yourself Also, you’ll randomly notice you’re not getting logs from 5% of your log sources, and you’ll have to chase down those issues. This will happen once a day.
- #27 Is due diligence different from validation? I would argue NO. It isn’t. After all, if the product doesn’t do what it is supposed to, what good is the rest of your due diligence process? This advice is specific to products with threat detection/prevention capabilities, but I felt it was important data to share, given that more than half the security industry is threat detection/prevention-focused.
- #28 Is due diligence different from validation? I would argue NO. It isn’t. After all, if the product doesn’t do what it is supposed to, what good is the rest of your due diligence process? This advice is specific to products with threat detection/prevention capabilities, but I felt it was important data to share, given that more than half the security industry is threat detection/prevention-focused.
- #29 So, a ways back, I joined an enterprise for a short time to help them get their security program going. The CTO there brought me in, and I worked directly for him. One day, he comes to me and says, ”We’re considering putting these cloud-managed wireless access points in all the manufacturing centers. Could you take a look at them and make sure they’re secure?” There’s nothing more exciting as a security geek than when someone hands you something brand new and says, “I need you to try to break this”. The hardware was pretty off the shelf, nothing remarkable. What was supposed to be really cool about these devices is that they were cloud-managed, like Cisco Meraki stuff, but at a fraction of the price. I had experience with cloud-managed products before. In a previous life, we had some Qualys scanner appliances, which were the first cloud-managed devices I could remember. You just logged into Qualys.com, and controlled these scanners from there. You never had to worry about having poke holes in the firewall, set up VPNs or enable any sort of inbound connections. Qualys accomplished this by having the scanner reach out to Qualys.com every 60 seconds or so to check for new instructions. So, while it could take as long as a minute for your commands to make it to the scanner appliance, it was still a pretty slick and impressive setup back in 2004. Well, these guys were trying to do the same thing. I must admit, I never dug into the details of how Qualys did it back in the day, but being 2012, I figured the process had been refined over the last 8 years. Meraki was well known for it, and I started this testing about a month before Cisco picked them up for $1.2bn. There was nothing refined about how these products were designed. Due diligence is tough to define for any given scenario. Simply put, it is the result of the time, money and skill you feel like you have available to make sure you’re not making a big mistake. In this case, I decided if there was nothing else I had time for, I wanted to understand how these devices talked to the cloud management portal. I prioritized that task first. While myself and another colleague ended up spending many hours validating this vulnerability, we knew within the first few hours that this product wasn’t secure, and wasn’t going to get our approval.
- #30 Due diligence time. Our first question: how does this vendor’s “cloud managed” architecture work? Let’s see Start a packet capture and turn it on Check out the packet capture Uh oh. Entire config pulled over HTTP. Not HTTPS. And then gets sent back via syslog I wonder, how does the server know to give it this config? How does it authenticate? All I see is a mac address. It couldn’t be that simple… Yep, it is. If these were all manufactured in the same run... <wrote some python> Ran the python – yes, we got the config for every other device by sequentially guessing the MAC. Most configs included the name of the customer and the physical ADDRESS where these devices were installed.
- #31 A full year later, it was still serving up config files for whomever tossed a MAC address at it. One year to get it fixed. We never bought it. I was even working for a different company when they finally fully put these vulnerabilities to bed. Benefit: We didn’t spend money on OR deploy devices that had critically flawed security Drawback: We got sucked down a year-long rabbit hole. Though we didn’t have to, we felt morally obligated and even wrote code to determine the full extent of the vulnerabilities and to ensure the vendor got them fixed. Also, it was really, really exciting.
- #32 Preventing shelfware! We hate shelfware, right? How many of you have shelfware? (ask the audience – explain shelfware if they appear confused) Please, please, please don’t just go buy stuff!
- #33 One of my peer reviewers suggested I check out his slide deck. His Cyber Defense Matrix concept is a great tool for implementing several of my suggestions in the due diligence process. His 8th use case here is great for gap analysis exercises and visualizing the ‘big picture’ in your organization.