BlueVoyant: 7 Experts Share Key Questions To Ask When Evaluating Providers

Sponsored by
Experts Share Key
Questions To Ask
When Evaluating
Providers
How Resource Constrained Security Teams Can Achieve the
Capabilities of the Most Well-Defended Organizations

2
INTRODUCTION
As cyber threats and the solutions employed to address them become more challenging, businesses
are increasingly turning to managed security service providers (MSSPs) for help. Doing so makes a
lot of sense, because cyber security is not the core competency of most businesses. However, it is
the business of MSSPs, and good MSSPs maintain the staff, expertise, and technologies needed to
always stay on top of the latest cyber threats.
But what is a good MSSP? Selecting the right service provider can be a lot more difficult than
deciding you need one. With BlueVoyant’s generous support, we took a closer look at how security
professionals make managed security services decisions by asking seven security experts these key
questions:
l When should you consider partnering with an MSSP?
l How do you know if an MSSP has the technology resources to meet your security needs?
l How do you decide the service level and scope of coverage you need from an MSSP?
l How do you evaluate an MSSP’s threat hunting capabilities and their ability to proactively
detect new threats?
l How do you evaluate a service provider’s ability to end, prevent, and respond to breaches?
The answers they provide are interesting and varied, but they point to the importance of an MSSP
having an appropriate technology stack and an open relationship that relies on visibility and
responsiveness. Anyone who has engaged with an MSSP at any level will find insight and a lot of
good advice in these essays.
Mighty Guides make you stronger.
These authoritative and diverse
guides provide a full view of a topic.
They help you explore, compare, and
contrast a variety of viewpoints so
that you can determine what will work
best for you. Reading a Mighty Guide
is kind of like having your own team
of experts. Each heartfelt and sincere
piece of advice in this guide sits
right next to the contributor’s name,
biography, and links so that you can
learn more about their work. This
background information gives you
the proper context for each expert’s
independent perspective.
Credible advice from top experts
helps you make strong decisions.
Strong decisions make you mighty.
© 2019 Mighty Guides, Inc. I 9920 Moorings Drive I Jacksonville, Florida 32257 I 516-360-2622 I www.mightyguides.com
All the Best,
David Rogelberg
Publisher, Mighty Guides, Inc.

3
Resource-Constrained Security Teams Can Achieve the Capabilities of the Most Well-Defended Organizations
Most world-class security technologies are available only to the “security 1%”: banks, national governments, and
the largest enterprises. These organizations have sizeable budgets to hire and retain significant Expertise and
purchase or develop premier security solutions.
These large enterprises drive innovation, but their solutions don’t map well to small-to-mid-sized organizations
the other 99%. Smaller enterprises are typically constrained by budget and resources and are forced to
compromise when it comes to security.
BlueVoyant provides a new approach for resource-constrained teams. We democratize cybersecurity by
protecting organizations of all sizes against agile and well-financed cyber attackers through highly-scalable
service offerings tailored to meet the needs of our clients. We partner with our clients to achieve a level of
security that they couldn’t reach on their own. We provide technology and integration they couldn’t otherwise
afford. We offer threat intelligence that they wouldn’t have access to. We staff our Security Operations Centers
with experts they would have difficulty hiring and retaining. As a result, we trim high costs and help IT teams
achieve a level of security previously only available to the largest and most well defended organizations.
Founded and led by experts in the cybersecurity and government security sectors, BlueVoyant makes superior
technology, proprietary threat intelligence, 24x7 Security Operations Centers (SOCs), and deep cybersecurity
expertise available to enterprises of all sizes. We provide mutually reinforced solutions that allow clients to right
-size services to meet their unique needs.
The first step in determining the proper security for your organization is to arm yourself with the right questions.
The experts that have contributed to this Mighty Guide will help prepare you to move forward on your quest for
improved cybersecurity. Enjoy the book.
Regards,
David Etue
Global Head of Managed Security Services
BlueVoyant
BlueVoyant is an analytic-driven
cybersecurity company whose
mission is to protect organizations
of all sizes against agile and well-
financed cyber attackers. Founded
and led by experts in the cybersecurity
and government security sectors,
BlueVoyant’s offerings are built with
real-world insight and applicability.
Through our Advanced Threat
Intelligence, Managed Security
Services, and Incident Response
Services, we excel in intelligence
gathering, cybersecurity defense,
detection of attacks, and response
coupled with remediation.
Our 24/7 SOCs, offices around the
world, and our security analytics
platform positions us to best help our
customers defend against emerging
cyber threats. For more information,
visit bluevoyant.com
FOREWORD

TABLE OF CONTENTS
4
CHAPTER 1
When to Leverage a Managed Security Services Provider (MSSP) 7
CHAPTER 2
MSSP Tech Stack 22
CHAPTER 3
Service Levels and Scope 35
CHAPTER 4
Threat Hunting and Incident Response 49
CHAPTER 5
Ending and Preventing Breaches 63

MEET OUR EXPERTS
BRIAN SHEA
Chief Information Officer /
Chief Compliance Officer, MBX
Medical Billing Experts, LLC
and VSTRATEGY, LLC
RACHEL GUINTO
CISO, Ontario Pension Board (OPB)
DANIEL PAULA
SVP, Information Security Risk
Management, Charles Schwab
RAFAEL NAREZZI
CISO - Chief Cyber Security
Strategist, WiseEnergy - Smart
Renewables Services
GREG FITTINGHOFF
Former Vice President
and Chief Information
Officer, Fashion Institute of
Technology
TAD DICKIE
VP & CSO, Colonial
Companies
PATRIC J.M. VERSTEEG
Leading Security Change
at Enterprise Organisations
(CISO)

7
One of the big challenges in deciding to partner with
a managed security services provider (MSSP) is
deciding whether it is the right thing to do for your
business. It is an important decision that goes beyond
the simple cost-benefit analysis. We looked at how
companies make the decision by asking our experts
the following question:
When should you consider partnering with a
managed security services provider?
WHEN TO LEVERAGE
A MANAGED SECURITY
SERVICES PROVIDER (MSSP)
CHAPTER 1
7

8
Rachel Guinto is a 17-year veteran
of Information Security in financial
services. Her operational and leadership
roles include Risk Assurance,
Governance, Regulatory Compliance,
Cryptography and Customer Online
Protection, Intelligence and Vulnerability
Management, and CISO. Rachel is a
CISSP and CISM designate, with a B.A.
in political science and a Diploma in
computer programming. She volunteers
as a cyber safety and education
advocate and career mentor.
Rachel Guinto, CISO, Ontario
Pension Board (OPB)
“However, even when making a managed
security services decision, there’s always a part
of security governance you will need to keep
in-house so you can effectively manage the
service provider relationship.”
Building and maintaining an internal cybersecurity program is a
significant proposition. Cybersecurity is a data-intensive operation that
requires processing log data that was generated by activity throughout
the IT environment. This requires skilled staff, and it requires technology
such as security incident and event management (SIEM). The ability
to attract and maintain talented staff is a key consideration. There is a
shortage of qualified security people, which makes finding and attracting
them difficult. If you succeed in hiring the right people, keeping them is
even more challenging. The largest enterprises with the biggest budgets
and most expansive security programs have the most to offer to this rare
talent, which puts many midsized companies at a disadvantage.
For many businesses, deciding to leverage managed security services
is a purely practical decision to achieve economies of scale and have
access to the talent they need, without paying directly to build and
maintain it themselves. However, even when making a managed security
services decision, there’s always a part of security governance you will
need to keep in-house so you can effectively manage the service provider
relationship. Your focus shifts internally toward making sure that you

9
have the right governance model to provide an appropriate level
of oversight. You should partner with an MSSP that believes in
a high level of transparency and offers a portal and dashboards
that provide a full view of all analysis, activities, responses, and
remediation affecting your environment.
Key Questions You Should Ask:
Do you have the skills and resources needed to monitor and
correlate the large volumes of activity data in your environment?

10
Brian is one of the most dynamic IT
professionals working in Columbus
today. He brings 20+ years of IT
related experience to the table,
focusing primarily on enterprise
infrastructure, operations and
security.
Brian Shea, Chief Information Officer /
Medical Billing Experts, LLC and
VSTRATEGY, LLC
“The size of a company often does not
equate to the size of its security needs.”
If and when to choose managed services depends on a number of factors
specific to the business. These include company size, industry, how
the company accounts for its security resources (for example, whether
security is considered an operating expense, a capital expense, or both),
and internal resources available to address the company’s security needs.
These internal resources include technology, staffing, and security skills.
The answers are not always simple. For example, the size of a company
often does not equate to the size of its security needs. A small medical
practice with a simple IT infrastructure can have the same security
requirements as a large hospital. They both deal with personal health
information (PHI), personal identity information (PII), and payment card
information (PCI), and both are highly regulated. The small medical
practice certainly won’t have the same level of in-house resources to
apply to its IT security. Such a business could be an ideal candidate for
managed security services. By the same token, a large company might
have extensive security resources but suffer from the inability to recruit
the talent or implement the technology it needs in-house. This large
company might also benefit from leveraging managed security services.

11
In considering whether you need an MSSP, the first question you
should ask yourself is this: do you have the internal resources,
staffing, and expertise you need to implement and operate
the security technologies necessary to meet your compliance
requirements and keep cyber-risk at an acceptable level? Note
that this is a tricky question because the regulatory and threat
environment is constantly changing, as is your business’s IT
infrastructure. It is a question you need to ask yourself every day.
Do you have the internal resources, staffing, and expertise
you need to implement and operate the security technologies
necessary to meet your compliance requirements and keep cyber-
risk at an acceptable level?

1212
Patric has built and led information-
and cybersecurity teams around the
globe, leading strategic information and
cybersecurity change. With 20+ years
of experience in delivering strategic
planning, engaging leadership, sharp
analysis, and custom solutions, he
ensures that businesses stay secure
in the ever-changing (cyber) security
landscape.
Patric J.M. Versteeg, MSc. C|CISO
CISSP CISM, Leading Security Change
at Enterprise Organisations (CISO)
“If cybersecurity is not your core business, if you are not
an MSSP yourself, you should not think about struggling
to uphold a security posture for your company.”
The decision to outsource security operations to an MSSP is not a difficult one.
If cybersecurity is not your core business, if you are not an MSSP yourself, you
should not think about struggling to uphold a security posture for your company,
especially with all that is happening in the security world today.
Keeping a security practice up to par takes a lot of time and effort. It requires
maintaining staff when there is currently a shortage of security workers
worldwide. Even if you succeed in building staff, some of them would be
underutilized, whereas others would not have the skills you needed, and
depending on what you are doing, you might find it difficult to pay them enough or
make the job interesting enough to keep them engaged.
There are many managed security options available, including providers and
service plans that are affordable for midsized companies, and it’s reasonable to
expect you can find comparable coverage for the amount to spend in-house. Even
if you think that you will get less bang for the buck going to a service provider than
you would if you did it yourself, you should consider that regardless of whether
you buy less service than you have now because of the pricing, the quality of the

13
services that you buy will be better. That’s because the service
provider is dedicated to that security function and is quickly
scalable. Of course, this assumes the MSSP you choose is a
company that knows what it’s doing and is an expert in its field.
Is cybersecurity a core business capability that you possess?

14
Tad oversees all aspects of logical
and physical security for Colonial
Companies. He is an accomplished
information security principal with
more than 20 years IT and information
security experience including a decade
of executive information security
consulting with several leading
professional services firms. Tad has
extensive experience in providing
strategy and operational leadership for
all aspects of security management
and governance to diverse global
organizations.
Tad Dickie, VP & CSO, Colonial
Companies
t
“The primary driver for utilizing an MSSP should
be the opportunity to quickly scale a team of
experienced cyber security and incident response
specialists to support continuous security
operations”
When deciding whether or not to utilize an MSSP for some or all security
operations, an organization’s primary consideration should be whether
or not there’s justification for a full complement of security staff and
tools with the technical expertise to support a 24/7 security operation.
If your current security practice is not able to maintain this level of
coverage, you should be considering how to utilize managed security
services to fill gaps in your security program.
Although you can experience savings utilizing an MSSP, cost reduction
shouldn’t be the leading driver in the decision process. Instead of
organizations making the significant investments required to assemble,
maintain and staff a dedicated security operations center, a more cost-
effective solution may be to utilize managed services for 24/7 managed
detect and response. However, bigger gains can be experienced when
utilizing an MSSP’s experienced staff who are continuously responding
to a broad set of threats with a full complement of supporting technology
and senior leadership. MSSP staff are sure to grow and maintain their
skills to be effective.

15
Additionally a team may periodically need a forensic specialist,
even though the organization doesn’t experience a high volume
of activity requiring forensics. Not only will keeping a forensics
analyst on staff likely under utilize their costly skill set, their
skills may degrade or the organization experience high turnover
keeping the position filled. Conversely, a forensic specialist in a
busy managed security services operation is continuously honing
their skills while receiving extended training and peer exposure.
This situation also extends to other coveted skill sets such as
vulnerability management and threat hunting. The primary driver
for utilizing an MSSP should be the opportunity to quickly scale
a team of experienced cyber security and incident response
specialists to support continuous security operations without
a costly and frequently painful onboarding and implementation
period.
Do you have a full complement of security staff that is large
enough—and has the technical expertise—to support a 24/7
security operation?

16
Daniel Paula is the SVP, Head of
Information Security Risk Management
(ISRM) at Charles Schwab. He started
his career with KPMG in 1999. After 10
years with KPMG, the Federal Reserve
recruited him as a Senior IT Examiner in
the wake of the Great Recession in 2009,
where he developed extensive know-how
of the global regulatory environment. In
2016 he was recruited by Charles Schwab
to support efforts to strengthen the
cybersecurity risk program. He has built
and operated core IT Risk/Cybersecurity
programs for three different Fortune 500
companies.
Daniel Paula, SVP, Information Security
Risk Management, Charles Schwab
“You may benefit greatly by partnering with a
provider that offers compliance assessment tools
and helps to automate reporting and attestation.”
Two key considerations will influence the decision to outsource security functions
to an MSSP:
1. Skills requirements—This is basically whether or not you have the
skills and resources in-house to run the operation. This includes the
ability to keep up with a rapidly changing threat landscape and with new
technologies being developed and deployed that are needed to secure your
environment.
2. Ramp-up speed—This is the time it takes you to meet a new security
requirement. If it would take you a year to build out a team to meet an
urgent security requirement, but you could get an MSSP up and running in
a shorter period of time, you need to consider the MSSP option.
Of course, cost is always a consideration, and that will influence the kind of
MSSP vendor you select. Another factor that can influence the decision for some
businesses is the nature of the regulatory environment. For instance, if you are
in an industry with complex regulatory requirements, such as financial services,

17
you need to carefully consider not only the provider you choose,
but also the functions you outsource. You may benefit greatly by
partnering with a provider that offers compliance assessment
tools and helps to automate reporting and attestation.
Do you have the ability to ramp up new security capabilities fast
enough to fulfill a new security requirement?

18
Greg Fittinghoff has held CIO/senior level
technology positions across a broad
array of industries: Higher-Ed (Fashion
Institute of Technology), Media (HBO,
Time Warner), Medical Devices (Becton
Dickinson), and Consumer Products
(PepsiCo). He received a CIO Ones to
Watch Award, given to rising stars in IT for
theirleadership, innovation, and value to
their organization. He holds an MS and a
BS from Iona College.
Greg Fittinghoff, Enterprise
Solution Architect, Nintex
“Always maintain a level of security expertise
in-house, and seek out an MSSP to provide the
arms and legs that can cost effectively perform
functions you cannot do internally.”
Cybersecurity is a critical need for all organizations. Deciding if you
are going to outsource a security function or process to an MSSP
really comes down to asking yourself a series of questions. You need
to determine if you can better protect the organization by partnering
with a security services provider that can offer experts (and potentially
technology) not available internally.
If this is something that you need to do to protect the organization—
something customers require, a response to a regulatory issue, or
addressing a new threat—then the next issue is how to go about meeting
that need. Answering that question involves determining if you have the
ability to do it internally with the people and skills you have. Will it incur
significant cost because of its specialized nature, requiring people to have
certain certifications? Finding those people and maintaining them on a
24/7 basis may be essential because information security is a 24/7/365
proposition. There is never a day or a time when something is not
happening. It might involve collecting log files from servers. Depending
on how many you have and if they are virtualized, that in itself becomes a
burden. You need to maintain the tooling and the expertise to use it.

19
With this understanding of what’s needed, you have to drill into
exactly how many of those resources you require to support the
kind of security response and risk management the organization
expects. And at that point you can begin to make a realistic
evaluation as to whether this is something you can build internally
or if you need an external partner to implement it.
Every organization and situation is different, but regardless of
the path an organization takes, it should never outsource all
its information security capabilities. Too many security issues
arise that have business implications and require involvement of
business managers. Always maintain a level of security expertise
in-house, and seek out an MSSP to provide the arms and legs that
can cost effectively perform functions you cannot do internally.
What types of human and technology resources are needed
to support the information security and risk management
the organization expects? What is the most effective and
sustainable approach to get those resources?

2020
The British naturalized Brazilian holds a
master’s degree in computer forensics,
cyber-security and counter-terrorism from
Northumbria University in the United Kingdom.
Rafael also participated in the book “Strokes and
Frauds, Prevent Against the Stars”, by Leonel
Baldasso Pires, with a chapter on crimes in the
virtual world. Today the expert acts as a CTO.
In the last year, Rafael has been invited to
lecture in several countries and institutions of
Europe, such as: British Chartered Institute of IT,
University of West London, Amsterdam, Prague
Cybercentral, Bedfordshire University and
Singapore.
Rafael Narezzi, CISO - Chief Cyber
Security Strategist, WiseEnergy - Smart
Renewables Services
“An outside source can also provide key security
metrics and performance indicators that help you
make decisions about risk and where to prioritize
your security investments.”
There are several reasons to consider leveraging managed security services,
especially if you are a small or midsized company. There is no simple answer
for every organization, and although outsourcing can sometimes cost more
than actually having your security operation on premises, you may also find
that outsourcing is actually more cost effective and delivers a more secure
environment than trying to do it internally. One of the biggest challenges is that
cybersecurity is not something that you set up once and then forget about. It is
constantly changing, and keeping up with those changes can be difficult. This is
a problem for many companies, especially for smaller businesses. Outsourcing
security is a way of transferring some security operations activities, like threat
intelligence, continuous monitoring, detection, containment, response to threats
and security incidents responsibility to someone else.
Numerous benefits can be gained by outsourcing. For instance, operating in
today’s digital economy generates lots of traffic and enormous amounts of log
data that must be monitored and analyzed. Managing all that log data may require
setting up a security operations center (SOC) and SIEM solutions, which can be
costly—capabilities the MSSP already should have. An MSSP can also provide key
security metrics and performance indicators that help you make decisions about
risk and where to prioritize your security investments.

21
What are the key security metrics I need to make decisions about
cyber-risk to the business, and what is the best way to get those
metrics?

22
Once you have made the decision to outsource some
aspects of your security program to a managed
security services provider (MSSP), then begins the
process of finding the right vendor. That often starts
by finding a vendor with technical capabilities that
match your needs. We explored how organizations do
this by asking our experts the following question:
How do you know if an MSSP has the
technology resources to meet your security
needs?
MSSP TECH STACK
CHAPTER 2

23
Pension Board (OPB)
“With any security program, you need
to be sure that a security vendor can
cover all the basics.”
One way to evaluate security vendors is to survey the landscape and
determine the ones that are more suitable to your organization. Deciding
which provider is best for you requires that you first know your own
needs.
With any security program, you need to be sure that a security vendor can
cover all the basics. This includes the ability to aggregate and analyze log
data in a security incident event monitoring (SIEM)-type solution and to
deliver managed detection and response capabilities. Their technology
needs to fit your current program and where you might grow your
practice. For example, right now you may not need advanced artificial
intelligence (AI) and machine learning capabilities, but as you move more
operations into the cloud, these capabilities will become important to
you. You should anticipate those needs as you evaluate service provider
capabilities.
One approach to outsourcing security is to begin by assigning operational
functions to the vendor. To do that, you need to know the solutions they
work with. Do they use technology and solutions that are best-in-class

24
and trusted by the industry, as opposed to their own proprietary
solutions? Outsourcing operational functions gives the vendor a
line of sight into your environment and your security needs, but it
also gives you a view of their technologies and processes.
Also, when thinking about vendor qualifications, it is not just
about the tool sets and skills the vendor brings, although those
are important. You also have to think about the relationship you
expect to have with the vendor. Ideally, you want to have a long-
term relationship so that the vendor comes to know what is
important to you as an organization. To develop a valuable security
partnership, you have to build the relationship and make the vendor
part of the team.
What exactly are you looking for from a security provider, and is
it able to deliver those things?

25
security.
VSTRATEGY, LLC
“Many do a good job of providing the basic security
stack…however, they fall down when it comes to
proactive work that requires…deeper analysis.”
Evaluating vendor capabilities is challenging, partly because there are so
many players in the security space. Some focus entirely on security, and
some are managed services generalist that also offer security services.
Many do a very good job of providing the basic security stack that
includes antivirus and antimalware, firewalls, and patching. However,
they fall down when it comes to the proactive work that requires pulling
together to log data from many sources and doing the deeper analysis
of everything occurring in the environment. Finding the right vendor
for your situation requires a vetting process. Does the vendor use and
support a range of security technologies? Where does it hire its analysts
and experts? Does it offer automation and orchestration? You should
really check out a vendor’s background and references. You need to give a
vendor some true examples and have it walk through its methodology.
Another challenge is that if you are a smaller company looking to
outsource security functions—maybe because you don’t have the
resources to do it well yourself—you might not even know the right
questions to ask. A large enterprise may be outsourcing a well-defined

26
component of its security operation to address a resource issue
and have the internal expertise to vet potential service providers.
A smaller company may not have a CISO or the expertise to
really know what technologies it needs and if it is right-sized for
its organization. In that case, it would be a good idea to hire a
security consultant who can help the company define the services
it requires, evaluate service provider capabilities, and generally
represent an organization’s best interest.
Do you have the internal expertise to know what questions
specific to your needs you should be asking of MSSP
candidates?

2727
landscape.
“When you have identified candidates that fit culturally and
from a maturity level perspective, drill into their technological
capabilities, such as the tools they use, and whether they use
advanced technologies and the newest approaches.”
It begins with knowing yourself, and that involves evaluating your organization’s
culture and the maturity level of your current security practice. For example, if you
think in terms of capability maturity model integration (CMMI) levels, a company
that has no consistent approach to security might be at CMMI level 0, and one
that has implemented a SIEM solution and has an operational security operations
center (SOC) might have a security practice that operates at CMMI level 3 or 4.
Once you have an objective understanding of your own capabilities, seek an
MSSP that can meet you at your maturity level, one that fits your ambition for
further maturity, and one that understands what you are trying to do. If you want
to improve, so you should align with a vendor having a more mature security
practice. But you don’t want to align with one that is way beyond your current
level if that vendor can’t meet you at your current level because you need to
interact on all levels—and your developers and administrators will not be ready for
that. The vendor should be able to talk to people in your company and be able to
provide training, coaching, functional support, and personnel to help you grow and
expand. But if the vendor is unable to help you grow, or is too far beyond you in
service maturity, it will not be a good fit.

28
When you have identified candidates that fit culturally and from a
maturity level perspective, drill into their technological capabilities,
such as the tools they use, and whether they use advanced
technologies and the newest approaches. Are they able to provide
24/7/365 support? Do they augment the technology they use to
make it more effective? A good MSSP combines great technology,
a well-defined and transparent process, and exceptional security
talent.
Can the MSSP meet you at your maturity level, communicate
with your organization effectively, and help you improve your
practice?

29
organizations.
Companies
t
“If your vendor is strictly proprietary, keeping the
technology inaccessible, you may miss some
benefits of integration with other established
onsite analysis and reporting tools.”
Any organization considering outsourcing to an MSSP must have a base
understanding of its requirements. An organization needs the leadership
and expertise to know what capability gaps exist and be able to evaluate
the quality and alignment each service a provider has to offer. This
ensures you can obtain the services needed without over procuring.
MSSPs should have a full, scalable set of offerings available, including a
staffed 24/7 SOC, logging, threat detection / analysis, advanced reporting
and workflow management capabilities. They should also be able to
offer security orchestration, automation, and response (SOAR) services.
A provider’s technical security stack should be a highly tuned offering
comprised of all the components you might want to consume via à la
carte procurement. For instance, you might want to start with next-gen
anti-malware or endpoint detection and response, then add managed
IPS, SIEM and log analysis during a subsequent phase. Perhaps you
haven’t been able to secure funding internally for SOAR services or threat
intelligence, but you still want to partner with an MSSP that can provide
those services in the future. You’re going to develop a deep relationship

30
with an MSSP that ideally lasts over time. If you can’t justify some
services today, you want to be able to add them as needed later.
There are typically two types of technical security stacks an
MSSP may utilize. A provider may utilize a proprietary set of
tools they have developed over time, or an MSSP will utilize
more industry-recognizable solutions that provide established
APIs for integration. If your vendor is strictly proprietary, keeping
the technology inaccessible, you may miss some benefits of
integration with other established onsite analysis and reporting
tools. As your security program matures, you may want to dive into
the latest threat-hunting techniques or pull threat intelligence into
another tool. For instance, if your MSSP significantly reduces the
workload for the organizational team, they may be able to focus
on maturing other areas for increased program maturity. If the
MSSP utilizes a proprietary security stack, there will be challenges
leveraging data from daily security operations without engaging
the provider to extend services.
Does the MSSP use proprietary software in its technology stack,
or does it use recognizable industry solutions with open APIs?

31
companies.
“It’s important to drill into the technologies they use, not only
to see if they are using the latest detection, response, and
orchestration technologies, but how they use them and what
their level of competency is.”
Determining if a service provider has a technology stack to meet your security needs
is a crucial area that requires detailed due diligence. This includes going to the vendor
sites and seeing their SOC teams in operation, and seeing how they perform their
processes. It’s important to drill into the technologies they use, not only to see if they
are using the latest detection, response, and orchestration technologies, but how
they use them, the service levels and what their level of competency is. It’s a good
idea to have an independent firm or consultant to evaluate the vendor choices and
report on their capabilities. It’s also important to speak to existing clients to get their
perspective about how the service provider operates.
There is also an emotional intelligence aspect of the evaluation that is difficult to
quantify. Either during the contract discussions or during sales discussions, you
have to develop a sense of whether they are really going to be the business partner
you want them to be. This is the business partner that is going to receive that call in
the middle of the night when there’s a data breach. How much of a premium do you
want to put on trust? It’s a critical relationship.
Can you visit the MSSP’s security operations center? What are other clients
saying?

32
“At the very least, a qualified vendor must be capable
of providing services 24/7/365 and have systems
capable of real-time/near real-time monitoring of your
environment to detect and report on anomalies.”
The best way to begin qualifying an MSSP is to turn to industry resources
and do your own initial research. This can begin with industry analyst
reporting, but you should also speak to peers in the industry. The
traditional approach of sending out a request for information (RFI) is
something you can do, but it is often faster and easier to talk to research
organizations and peers to find out who’s using what technologies and
what these people in the industry think of the service providers. If you can
contact current customers and ask if they would recommend a particular
vendor, that can tell you a lot and help you eliminate potential vendors
early in the process. With that initial research, you can then dig into the
vendor’s capabilities.
This involves looking at a vendor’s technologies and how they use them.
At the very least, a qualified vendor should have a SIEM platform that is
capable of collecting log files from physical and virtualized servers in your
environment, analyzing that data, reporting on anomalies, and triggering
alerts. A good MSSP needs to have the skills and resources to maintain
that platform and optimize it for your environment. You absolutely want
a vendor to have detection and response tools and the ability to apply
behavioral analytics to the large amounts of data it will be collecting,

33
which implies a level of machine learning in the technology stack.
You want to see that the vendor has an innovation program where
it is evaluating its own technology stack to see how it can deliver
a higher level of capability. And ideally, a partner will offer visibility
into its process and its ongoing activities. Transparency breeds
trust and ensures you see all the efforts the MSSP partner is
undertaking on your behalf.
Sometimes a third-party consultant can help you define your
needs and evaluate a service provider. But you still need to have
that internal person who can take those findings and act upon
them. Outsourcing security is never about one thing. There are
technology considerations, internal resources considerations, and
business considerations. Evaluating a service provider usually
happens in the context of multiple things that are going on at once.
You need someone internally who can focus on those issues—
someone who has an appreciation for the art and science of
information security and finds it engaging and interesting. Relying
on a consultant to provide that for you is costly and ineffective.
What do industry watchers say about a vendor? When asked,
does a customer recommend that vendor? Why or why not?

3434
Singapore.
Renewables Services
“Determining which MSSP to use comes back
to the security roadmap for the company
and understanding what you’re trying to
accomplish.”
Determining which MSSP to use comes back to the security roadmap for the
company and understanding what you’re trying to accomplish. You will have
outsourcing criteria that may include a need for certain kinds of reports or
providing certain kinds of security analytics. You need to evaluate MSSPs on their
ability to deliver on these criteria.
For some services, such as a general service provider who is delivering SOC and
SIEM capabilities and basic security management services, you want to find
a vendor that can deliver on the criteria important to your business. In other,
more specialized functions, such as pen testing and security assessments,
you may want to rotate these vendors. That’s because good security requires
thinking out of the box. If you use the same pen testing or security assessment
companies over and over again, they become familiar with the network and their
work becomes routine, which sets them up to miss things. If you rotate these
specialized service providers, they are working hard to win you as a client, so they
will go the extra mile to prove their value. They are more likely to find new things
that may be issues you need to address.
Can the MSSP deliver key capabilities that are most important to your
organization’s cybersecurity?

35
Deciding which managed security services
provider (MSSP) is best for you requires
considering supplier capabilities in the context
of your security needs. Another important
determination you need to make is the scope
of the relationship you will have with the MSSP
and the level of service you will ask it to perform
on your behalf. We looked at how companies
think through relationships and service levels by
asking our experts the following question:
How do you decide the service level and
scope of coverage you need from an
MSSP?
SERVICE LEVELS AND
SCOPE
CHAPTER 3

36
Pension Board (OPB)
“You need your own standards, priorities,
security principles, and playbook. The
relationship needs to align to your
playbook.”
When deciding on the scope of an MSSP relationship, the service level
you agree to goes back to knowing what you expect to gain by engaging
with a service provider in the first place. It’s important to align the MSSP
services with your own playbook. Some things in your practice will not
change. What’s changing is who is doing them. For example, if it’s your
practice to investigate and resolve certain incidents within 24 hours, that
becomes part of the agreement. Those are things you have to decide for
yourself.
Some organizations treat the managed security service as a black box, or
they try to align their playbook to the MSSP contract. This is a mistake.
You need your own standards, priorities, security principles, and playbook.
The relationship needs to align to your playbook. These standards,
and how the MSSP reports on their work to support them, need to be
clearly laid out in the service-level agreement (SLA), and it becomes
your responsibility to track and verify that they are being followed. This
involves monitoring reports, monitoring key milestones, and having
regular reviews with the provider. The more visibility the MSSP provides
into all the activity it is undertaking, the better you will be able to monitor
and understand its value.

37
Typically, there is an overall master service agreement, and then
there are specific statements of work dealing with different types
of activity and functions they will be performing. You need to
decide if you want your MSSP to only alert your team, or if you
want it to be able to respond and remediate on your behalf, and in
what context. It’s important that all these service levels combine
to deliver the service you need. For example, an MSSP SLA may
specify requirements for incident response, but the MSSP may
not be the actual incident management team. That could be
fulfilled internally or by an infrastructure service provider. The
MSSP must understand the need to cooperate with other parties,
whether those are your internal people or other third parties
supporting your environment. Now it becomes as much about
managing partners and making sure everybody is living up to their
expectations as it is about the actual operations.
Can you write a service-level agreement that conforms to our
standards and our security playbook?

38
security.
VSTRATEGY, LLC
“To properly align services to security needs,
companies should be more methodical about
performing a security assessment.”
Before organizations can know exactly what they need from an MSSP
engagement, they need to perform an IT security assessment that lays
out their current security practice and gaps to be addressed. Every
security practice will have security gaps. To prioritize and make risk-
based judgments about which gaps to address first, organizations need
to perform this assessment in the context of critical business processes.
Many companies consider outsourcing after an incident wakes them
up to their vulnerability. They feel they need to take immediate action to
protect themselves, and they go directly to the endgame of looking for
a managed services partner. This can cause them to purchase services
they do not need without addressing important gaps in their security
program. To properly align services to security needs, companies should
be more methodical about performing a security assessment to really
determine where they need to spend their money, what they should
spend it on to address genuine risks they face, and how to prioritize that
investment. If you go to an MSSP seeking guidance in how to build a
service stack to meet your needs, keep in mind that it is in the business
of selling services, and it will be in the MSSP’s interest to sell you the

39
fullest complement of services possible. Understand the value it
is providing in terms of technology, resources, hours of coverage,
response times, and other key capabilities, and compare this
closely to what a similar level of cybersecurity would cost your
organization if you did it yourself.
Where are the gaps in our current security practice, and which
ones pose the greatest risk to the business?

4040
landscape.
“Good initial candidates for outsourcing are routine
tasks such as managing firewalls, antivirus, data
loss prevention, and vulnerability management.”
You need to tailor security coverage to your business needs, so first and foremost,
you need to turn back to company strategy, mission, and vision and see how
those align with risk and compliance requirements. The services you need are
determined first by your risk management and risk appetite, and second by the
maturity of your current security practice. For some businesses, compliance is an
important driver as well. You need to know your own needs and have a team that
can help you determine that.
Good initial candidates for outsourcing are routine tasks such as managing
firewalls, antivirus, data loss prevention, and vulnerability management. As you
become more sophisticated, you may consider outsourcing advanced endpoint
security like managed detection and response. There are always core security
functions you do not want to give up. You would not outsource management of
key high-privileged accounts, and you would not outsource your Chief Information
Security Officer (CISO). Also, you should not outsource your security architect;
that is the person who has security business knowledge about the service levels
and the scope of coverage you should be receiving. A good MSSP will be a partner
that works closely with your own organization to optimize results.

41
Of course, you should expect an MSSP to be able to deliver what
you need and have the flexibility to meet most of your special
demands. This might be special requirements around reporting,
24/7 coverage, security incident event monitoring (SIEM) and
security operations center (SOC) capabilities, and special
service-level capabilities. You must go to your MSSP with your
requirements to ensure compatibility.
Which security functions can we outsource, and which ones must
we always keep in-house?

42
organizations.
Companies
t
“When contracting for managed security
services, it’s often best to use a phased
approach rather than enabling all the
services at once.”
Service level and scope requirements should be the result of an
assessment. If the organization is in a highly regulated industry,
compliance and reporting artifacts could identify gaps in the security
program. Unregulated industry participants, should conduct a current
assessment using a leading industry framework. Identified gaps should
be risk based prioritized into a remediation roadmap for MSSP service
procurement. The assessment can be executed internally or via external
professional services. It may be possible to leverage potential MSSPs to
conduct independent pre-sales assessments to produce an approach and
proposal for implementation of their services.
When contracting for managed security services, it’s often best to use a
phased approach to ensure the service provider integration is functional.
A big bang approach to turning on services may lead to oversubscription,
confusion and lack of meaningful results initially. Additionally even
utilizing an MSSP with implementation services, there will still be
necessary tuning. Dependent on the provider and procured services, you
will need to identify false positives, whitelist and work out escalation
paths. When you are building a solution internally, it grows organically,

43
but when you bring in an MSSP, their business goals may not sync
with your implementation priorities. By using a phased approach,
capabilities are rolled out according to risk based priority as
outlined in your security remediation roadmap. Understand how
much of the technology, resources, and process the MSSP can
provide, and ensure the managed detect and respond tasks the
MSSP is performing are clearly defined.
Which services do we need most, and are we ready to work with
the MSSP to sort out the operational details of those services?

44
companies.
“Outsourcing security mitigates risks, but it also
involves accepting new risks. Companies have
to look at both sides of that equation.”
Deciding what to outsource depends on the nature of the organization and a
number of factors, including its financial resources, its internal skills, what the
business needs, the regulatory expectation for that business, and its risk appetite.
Outsourcing security to a service provider mitigates risks, but it also involves
accepting new risks. Companies have to look at both sides of that equation.
When it comes to contracting security services, there’s always a trade-off between
the convenience and the risk of outsourcing. This causes many organizations to
take a hybrid approach. For example, some organizations, such as government
agencies and large banks, will intentionally keep certain things in-house while
outsourcing others, even if it is more costly and painful to do this. They might
take a hybrid approach in which they contract with an MSSP to provide coverage
evenings and weekends while the in-house team covers security during business
hours.
A large bank under intense regulatory scrutiny and less regulated businesses
have more flexibility. It comes back to weighing what’s gained by outsourcing
versus the new operational risks you take on. A key benefit of outsourcing is that

45
you don’t have to build a new capability from scratch. That ability
to quickly ramp up a solution through a service provider can be
critical to a security strategy.
What risks are we mitigating by contracting with an MSSP, and
what new risks are we taking on?

46
“It’s not only the technical capabilities and the
services that they offer. It’s also about whether
a vendor meshes well with your organization.”
If you have a chief information security officer or a security architect
in-house, this person will be able to understand the types of skills you’re
going to need and services that go along with that. If you don’t have
that basic security expertise in-house—maybe you’ve finally decided to
address this area and you need to hire resources—you have to rely on
outside expertise. A security consultant can look at your operation and
help you design an information security program with all its different
components, including which ones you need to focus on first. This
strategic plan aligns with your business needs, and it addresses key
questions, such as areas of primary focus and the timeline for gaining
traction in those areas.
This needs to be done before you go to a security vendor. When you go
to an MSSP, you need to have your plan in hand. You are now looking
for resources and vendors who will work with you to deliver those kinds
of services in the timelines that are most appropriate. This usually
involves numerous discussions and meetings to find the right vendors
and resources. It’s not only the technical capabilities and the services

47
that they offer. It’s also about whether a vendor meshes well
with your organization. Can the vendor talk with you in ways that
you understand? Does it appreciate your strategic plan, your
motivations, what’s happening operationally, and why you need
to move in the direction you are? When vendors are true partners,
those are things that they would want to know, and if they don’t
ask those questions or don’t seem interested, that relationship
likely will not work over the long term.
You need to map a vendor’s service offerings to your strategic
plan to make sure there is a good match and that it can add value.
There’s a difference between a vendor that’s purely operational and
simply does exactly what you ask and a vendor that understands
your goals and is willing to leverage its experience with other
companies to deliver services that help you be more successful.
The latter is the type of vendor relationship you want.
Can a prospective vendor talk with you in ways that you
understand? Does it appreciate your strategic plan, your
motivations, what’s happening operationally, and why you need
to move in the direction you are?

4848
Singapore.
Renewables Services
“One of the fundamental things in
security is to have good visibility into your
environment.”
When deciding on service levels appropriate for your business, do not think about
security purely in terms of cost. Security is about reducing the risk of any potential attack
that might happen or is already happening and could put your company out of business.
One of the fundamental things in security is to have good visibility into your
environment. Also, you don’t want to be bombarded with thousands of logs of
everything happening through the day and many false positives that ultimately prevent
you from acting when something important happens. This means at a minimum that
the MSSP engagement needs to include a SOC and SIEM solution. The SOC should be
staffed with experts who can use automation and threat intelligence to quickly identify
threats and take action to secure your business.
It makes sense to work with managed security services for SOC and SIEM operations
because these capabilities are expensive to build and maintain yourself. Expert staff
is also in high demand and difficult to hire. It’s best to outsource specialized security
functions like pen testing and security assessments; it is not practical to maintain in-
house pen testing and security assessmentsexpertise for something that you may do
only two, three or four times a year. Both the skills and the technology become stale if
they go unused.
What is the value of risk reduction provided by a particular security investment?

49
Proactive security strategies have become a
necessity in today’s distributed IT environments,
and for many who partner with an MSSP, the
vendor’s threat-hunting capabilities are an
important consideration. Threat hunting is more
than just looking at anomalies and behavior. A
good threat hunter thinks like a hacker and draws
on multiple sources of proprietary and open
source threat intelligence to inform his process
and to identify and respond to threats. We dug
into the evaluation of a provider’s threat-hunting
capabilities by asking our experts the following
question:
How do you evaluate an MSSP’s threat-
hunting capabilities to proactively detect
new threats?
THREAT HUNTING AND
INCIDENT RESPONSE
CHAPTER 4

50
Pension Board (OPB)
“You have to know that as you move
forward, your MSSP can move forward
with you.”
Evaluating a vendor’s threat-hunting capabilities is not so easy until a
threat really happens. One way is to have the vendor share its plans and
process so that you can at least know what processes it is following.
Putting technical solutions aside, knowing that a vendor has a mature,
repeatable process is important. If you have any hint that a vendor is new
at this, it might not be the right provider for you. Also, check references
who give feedback on how the relationship has worked.
Another way to judge a vendor’s skill sets and capabilities is to look at
whether the vendor is doing things to move forward. Is the vendor stuck
in an early 2000s kind of security mind-set? Or has it evolved and looking
at more forward-thinking strategies and technologies to prepare for the
future? You have to know that as you move forward, your MSSP can move
forward with you. You should also ask if the MSSP’s process includes
a combination of automated procedures and human decision-making.
Automation can rapidly weed out false positives, but human experience is
the key to effective threat identification.
Some of the capabilities and service levels will be built into the contract,
whether it’s in the larger contractual agreement or the individual

51
statements of work. These service-level agreements (SLAs) outline
performance and process expectations, reporting requirements,
and other aspects of the relationship. If continuous improvement
is an important part of the value you expect from a vendor, you can
put continuous improvement into the contract.
Is the MSSP stuck in an early 2000s kind of security mind-set, or
has it evolved and looking at more forward-thinking strategies
and technologies that are preparing it for the future?

52
security.
VSTRATEGY, LLC
“Listen closely to how the vendor talks about
what it does. People use certain words when
they are being proactive or reactive.”
Any service provider can say it proactively looks for new threats and is
an aggressive threat hunter, but there are several ways you can dig into
that a little further. It involves understanding the service provider’s toolset
and its processes for bringing all that data together to learn what really
is the problem. A service provider needs to have the technology, but it
also needs automation and defined processes that make its practice
proactive. In vetting a service provider, you may need to get into the
weeds, but the provider should be able to show you how it does what it
says it can do.
One approach is to ask the provider to walk you through an actual
example of a scenario in which you received multiple data inputs, you
identified something and reacted to it, and a positive outcome resulted
from that. The vendor should be able to articulate this using a real-
life example without revealing confidential information. Listen closely
to how the vendor talks about what it does. People use certain words
when they are being proactive or reactive. For instance, if the vendor
describes receiving an alert and then going through a process of waiting
for outcomes at various stages and doing weekly or monthly threat

53
reviews, that would be a red flag. On the other hand, if the vendor
is doing a lot of things at once when an alert occurs, searching
the dark web and correlating with other threat intelligence and
other network activity while isolating the event, that’s the kind of
thing you want to hear. The goal in being proactive is not waiting
until the end of the month to decide something is high risk. The
language the vendor uses to describe the process can be an
indicator. You should also review the level of reporting the vendor
provides regarding its ongoing threat-hunting and response
activities. Ideally, you should be able to view reports or dashboards
that highlight exactly what the vendor is doing to defend your
organization.
Can you walk me through an actual case that illustrates your
threat detection and response process? How did that work, and
what was the outcome?

5454
landscape.
“You need to look at their internal and
external threat-hunting capabilities based on
the security architecture they lay down.”
There are two ey things to look for when evaluating an MSSP’s threat-hunting
capabilities:
l Many threat intelligence feeds are available to MSSPs. What threat
intelligence do they use that is specific to your business segment? Maybe
they have built their own feed. How much does their threat intelligence
cover your business segment?
l What capabilities do they have to tailor their threat intelligence to your
needs? For instance, there are services that allow you to upload your
300, 500, 1,000 or whatever number of key indicators and perform threat
hunting that focuses on those specific indicators. Those indicators are
important to you, and focusing on them gives you a more tailored threat
intelligence overview. MSSPs should utilize at least several generic types
of threat intelligence and then build their own set based on triggers you
give them.

55
Additionally, you need to look at how MSSPs are actually doing the
threat hunting. Are they using security incident event monitoring
(SIEM) and security operations center (SOC) technology, and
possibly artificial intelligence (AI), to perform internal threat
hunting on your own network? How do they correlate this with
external threat hunting, which involves threat intelligence?
You need to look at their internal and external threat-hunting
capabilities based on the security architecture they lay down.
If an MSSP says that they need to use particular tools but fails
to tie them together with an overall security architecture that
demonstrates it is fully aware of your internal network and what
kind of systems you have, that should be considered a red flag.
What threat intelligence feeds does the MSSP use that are
relevant to my business, and can it tailor its threat intelligence to
cover specific indicators you provide?

56
organizations.
Companies
t
“It’s possible to do threat-hunting proofs of
concept with MSSPs that demonstrate their
capabilities detail what you need to fulfill
your threat-hunting requirements.”
Evaluating threat-hunting capabilities is challenging because that’s
one of those things that improves as you work with the provider to tune
those processes. A key part of the provider’s service is threat intelligence.
Everybody advertises threat intelligence because they have some form of
it, but you should evaluate its quality and relevance to your organization,
targets, and stakeholders. If an MSSP can’t provide that, its threat
intelligence likely will not be leverageable for your threat-hunting needs.
It’s possible to do threat-hunting proofs of concept with MSSPs that
demonstrate their capabilities to fulfill your threat-hunting requirements.
A vendor might identify it’s threat intelligence may not meet expectations,
that situation doesn’t fulfill your threat intelligence requirements, but
it’s a sign you are dealing with a trustworthy vendor. Keep in mind
threat hunting is evolutionary. Leading-edge vendors have automated
components of their threat-hunting capabilities to speed response. The
more a team works with an MSSP, the more noise will be reduced and the
finer tuned your security operations become. As your tools and threat-
hunting process matures, it’s likely to raise expectations for an even
higher level of quality in threat intelligence.

57
Can the MSSP detail how it would work with your threat-hunting
program based on your specific parameters?

58
companies.
“You don’t want a vendor telling you
every quarter that 100 more people
are needed.”
There is no simple answer to how you evaluate an MSSP’s threat-hunting
capabilities, but there are several approaches that will provide insight into what an
MSSP can do.
First of all, look into the vendor’s threat-hunting processes, how they apply
technology and people to those processes, and what their plan is for dealing
with large numbers of alerts. It’s not practical to just throw more people into the
mix. You don’t want a vendor telling you every quarter that 100 more people are
needed. Is the vendor applying tools to better filter and process alerts, and is it
having success with those tools?
It’s also important to see how a vendor has performed in the past. Look at big
cases like WannaCry and SpectreMeltdown to see if a vendor experienced those
when it spotted them, and how it dealt with them. Walk through a case study with
the vendor.
Another way to evaluate MSSPs is to see how well plugged in they are to the
threat-hunting ecosystem. Threat hunting involves a number of players. There
are numerous threats and copious threat-intelligence sharing. Threat-hunting

59
work often gets done in collaborative forums. Look for references,
and find out how others in the ecosystem perceive that particular
MSSP. Do they work well together? Are they good players or not?
Do they have a reputation for being top-notch or just average? Are
they drawing talent from well-defended private institutions like
major banks or healthcare organizations? Are they hiring from
national defense agencies and the military?
Also, the MSSP should be open to a phased implementation over
a period of time as opposed to doing it all at once. That allows for
transition and provides an opportunity to see if the process can be
tuned to meet your threat-hunting needs.
Is the MSSP applying tools to better filter and process alerts, and
is it having success with those tools?

60
“All other things being equal, MSSPs
serving larger numbers of clients will be
tapped into larger amounts of threat data.”
One of the most important aspects of threat hunting is the size of the
vendor’s threat intelligence database and its level information sharing.
All other things being equal, MSSPs serving larger numbers of clients
will be tapped into larger amounts of threat data. For that reason, one
factor in evaluating an MSSP’s threat-hunting capabilities is the size of
its client list and the types of clients and threat intelligence it leverages.
Larger vendors with more clients will be in a better position to track more
threats—even threats that may not be relevant to you yet. Additionally, a
larger client list may help an MSSP identify a threat sooner. An individual
event could look like an anomaly. When seen in a narrow field, it’s difficult
to detect a pattern. But an MSSP looking across a large field of clients
can view it in the aggregate and perhaps see it happening in many places.
It’s no longer an anomaly. It’s an attack.
Another part of your evaluation should include checking with
organizations the vendor supports who are also your peers. You need
the perspective of someone in your industry—maybe even one who has
a higher level need than yours. For example, you may want to check with

61
a vendor client who is in a more highly regulated aspect of your
business area. The MSSP should have capabilities and experience
relevant to your industry.
Determine if the vendor has a program of innovation in which
it is bringing in newer sets of technologies to help expand its
capabilities, speed, identification, and response; reduce the cost of
the services it’s delivering; and improve its services.
How extensive is the MSSP’s client list, and is your industry and
geographic location well represented?

6262
Singapore.
Renewables Services
“You want to understand the tools the
MSSP uses and its predictive capabilities,
and you want to see thorough incident
reporting.”
The best way to evaluate an MSSP’s threat-hunting capabilities is to know the
company and the people who are working there. You want to understand how
the MSSP manages its SOC and how it does its work. You want to understand
the skill sets and certifications of the MSSP’s SOC personnel and the mind-set in
approaching a problem. You want to understand the tools the MSSP uses and its
predictive capabilities, and you want to see thorough incident reporting.
Another critical indicator is how the MSSP shares threat intelligence. One of the
reasons attackers are so successful is that they share knowledge on the dark
web about how to attack. Skilled and advanced threat hunters should also be
monitoring the dark web. To be effective, an MSSP must also share information
with other organizations so it can see threats beyond just the activity it monitors
in your network. The larger the number of clients, the more threat intelligence is
available to them.
Does the MSSP have predictive capabilities? What tools does it use for that?

63
The worst-case cybersecurity scenario
is having a damaging data breach. For
companies that have data worth stealing, it is
just a matter of time before they experience a
breach, if they have not already. Having a solid
breach response protocol is essential, but how
do you evaluate an MSSP’s breach response
and containment processes? We looked
into this by asking our experts the following
question:
How do you evaluate a service provider’s ability
to end, prevent, and respond to breaches?
ENDING AND PREVENTING
BREACHES
CHAPTER 5

64
Pension Board (OPB)
“You need to look at your own business
requirements and determine if the
MSSP’s processes and protocols can
support them.”
To evaluate an MSSP’s breach response practices and capabilities, you
need to look at its breach response playbook to determine if it meets your
requirements for ending and preventing breaches. Does the playbook
live up to what’s required by your various regulators? Does it meet what
you would expect for the type of business you run? If the breach is going
to cause an outage to operations that will tie into your own incident
response management, can the MSSP support that?
In cases of operational disruption, you probably have existing
measurements and objectives you have to meet that relate to how critical
certain systems are to your business. So if a business can never be down,
can the MSSP meet that kind of response time? You need to look at your
own business requirements and determine if the MSSP’s processes and
protocols can support them.
Then you want to see how an MSSP is going to deliver on that. Is it
using detection and response automation tools, and how do those work
into the execution of processes laid out in the MSSP’s playbook? Has it

65
automated procedures to speed identification and remediation
of threats? Many of the answers will come from sharing with the
MSSP enough information for it to understand your environment,
and listening to how it explains its processes. Those foundational
pieces of your existing security program guide you in what to tell
the MSSP about what is important in your environment.
Does the MSSP’s playbook live up to what’s required by your
various regulators, and does it meet what you would expect for
the type of business that you run?

66
security.
VSTRATEGY, LLC
“You want the MSSP to show you its processes
and explain the technologies it is leveraging to
automate detection, analysis, and response.”
The goal is to prevent a breach from happening and to limit the impact
and breadth of any threats. However, security isn’t a perfect science,
and organizations will be breached. In those scenarios, any organization,
especially an MSSP, should have a documented incident response
protocol. It will be a runbook that the MSSP uses for its organization as a
whole, and it should have incident response processes tailored for each
of its clients. If the MSSP is unable to demonstrate that it has that, steer
clear. Without those kinds of documented protocols, an MSSP will be too
slow in proactively fulfilling an incident response plan.
The incident response plan needs to include all the actions to be taken
and communications that will happen, and you need to walk through the
plan step by step to determine if the MSSP has solid knowledge around
chain of custody, what it is doing with the data, how it is preserving logs,
and how it is reporting incidents to business management and regulators.
If it is not mature in that space, it will not be talking about those things.
Then you can get into the weeds a bit to determine what technologies an
MSSP is using and what processes are driving the technology. You want

67
the MSSP to show you its processes and explain the technologies
it is leveraging to automate detection, analysis, and response.
You need to have the internal expertise to ask these questions
and understand what the MSSP is telling you, or you need to work
with a consultant who can help with the evaluation. Ultimately,
you must validate that the MSSP is doing what it says it can do.
The only way to do this is to actually see the MSSP in action,
performing these breach management functions.
What is the MSSP’s process for managing chain of custody of
digital forensics evidence in the event of a breach?

6868
landscape.
“If the MSSP’s breach response is to send you a
tool so you can make a server image, that’s not
enough.”
The first question to ask is this: does the MSSP have boots on the ground? Are
there teams available 24/7/365? Can the MSSP send in a remediation team in
an emergency situation? How much of the process can be initiated, managed,
or accelerated through the use of automation? If you have a breach, it needs to
be fixed immediately. You have people doing day-to-day operations who can’t
drop everything to do major incident response. They are probably not trained
for it because that kind of security is not your core business, which is why you
are going with an MSSP. If the MSSP’s breach response is to send you a tool so
you can make a server image, so they can investigate what happened, that’s not
enough; you’re dealing with damage control. You need an MSSP that can respond
immediately in a hands-on way. This may be an immediate remote response
followed by a person or a team or whatever it takes to fix the problem.
Another way to evaluate the MSSP in this area is to do a tabletop exercise
involving a breach scenario. If somewhere in performing that exercise you notice
that you need to log onto a website or you need to issue a ticket, forget that
vendor. First and foremost, you need to evaluate what the MSSP’s response looks
like. If there’s something in the process that smells like a problem, that MSSP
shouldn’t get the job.

69
Also look at how the MSSP balances its use of technology tools and
people. You need to evaluate the MSSP’s underlying toolset that can
range from a self-built scripted system in conjunction with other
tools, to fully developed commercial solutions. You would expect
the MSSP to have some kind of security incident event monitoring
(SIEM), but managed security services is broader than just a
managed SIEM. It includes threat hunting, automated detection,
and manual searches, all of which go to the sub capabilities of the
service provider. This indicates how good an MSSP will be in breach
containment as well. If you look at an enterprise-grade MSSP that
relies on its own proprietary tools it has developed in-house, that’s a
huge risk. It’s worth asking why the MSSP isn’t sticking with its core
business. Why isn’t it leaving the development of a SIEM solution to
the professionals who do that?
What are the MSSP’s immediate response capabilities in the event
of a breach, and can it send in a remediation team in an emergency
situation?

70
organizations.
Companies
t
“An MSSP should be able to quantify
performance through response time,
remediation time, and dwell time type
metrics.”
Breaches start off as events and escalate through an organization’s
incident response process. An MSSP needs a defined response protocol
and service-level agreements (SLAs) for how it manages incidents.
The providers response protocol should be evaluated for fit with an
organization’s incident response plan. If there are significant differences
in response SLAs to the organization incident response plan, that may be
an indicator a provider might not meet selection needs or criteria.
In addition to evaluating the MSSPs incident response escalation
procedures, you should review how many breaches the provider has
responded to in the past year. Does the MSSP have the capability to
respond in a timely fashion, minimizing dwell time? Dwell time is a huge
factor in incident management. It’s the period from when an event is
identified to the time the incident is managed, remediated, and returned
to normal. An MSSP should be able to quantify performance through
response time, remediation time, and dwell time type metrics.
Another way to evaluate MSSP incident response capabilities is to
examine a providers escalation protocol and procedures. What level of
authority does the MSSP have to respond or remediate threats? There

71
should be clear definition of triggers and authorizations to include
who is contacted when and default actions based on severity.
Team members need to be involved quickly in high-severity events,
whether it’s via a call, email or other method, those communication
channels must be identified and tested. Escalation SLAs need
to be defined during the procurement process. If a provider is
unwilling to define escalation SLAs during procurement, there will
likely be issues meeting managed threat detection and response
expectations in real time. No two providers are alike in this regard.
Some MSSPs won’t provide many options. For instance, they may
insist on use of a ticketing system to ensure SLA fulfillment, but
that option may not fit an organization’s culture resulting in missed
escalations, which can increase dwell time and risk of breach.
How many breaches has the MSSP managed in the past year, and
what is the typical dwell time?

72
companies.
“The best way to evaluate how an MSSP
handles a cyber attack is to have the MSSP
tell you how it happened.”
You can read documents day and night, playbooks, make policy standards, and
interview people, but the best way to evaluate how an MSSP handles a cyber
attack is to have the MSSP tell you how it happened. An MSSP can’t tell you it
hasn’t happened, and if it isn’t able to tell you the details of a breach experienced,
it probably hasn’t been in the business long enough.
It’s important to hear about a real-life example. What went wrong? How did your
company help detect that? What happened after the detection? You need to
watch for certain signs of maturity, such as how the MSSP talks about escalation,
protocols, and notifications. You want to see the level of transparency in the
MSSP’s communications, and the technical depth and rigor of the research. You
want to know how the MSSP would report a breach. Will the Board of Directors
be notified that there was an event? Or is the MSSP going to say there was this
particular type of malware that resulted in this particular type and extent of

73
damage, the escalation that occurred, the velocity of the event,
the time it took to detect it, the time to containment, the time to
notification, and other key metrics? These are the signs of maturity
you are looking for.
Evaluate the MSSP’s technical capabilities to automate some of
this detection, response, and escalation management, but keep
in mind there is a tool for everything. Ask the value of a particular
technology and what risk it mitigates, and gauge how the MSSP
conducts that conversation. If the MSSP can’t articulate in a
convincing way the risk management benefits or the business
value of a cybersecurity technology, it might give you cause to
question the maturity level of the operation.
At the end of the day, you’re going to need both people and tools.
It’s more about understanding the MSSP’s management principles,
operational service levels, management of talent and technology,
innovation management, and evaluation of new technologies. The
key is evaluating how the MSSP approaches the challenge.
Ask the MSSP to explain what happened in an actual cyber
attack experienced. What went wrong? How did your company
help detect that? What happened after the detection?

74
“The cyber security incident response protocol
documentation should be an easily navigated and
manageable set of instructions. Talk with references
to see how the vendor actually responded during a
cyber incident.”
You want to be confident that the MSSP has an actionable protocol.
The protocol documentation should not be a giant tomb, but rather an
accessible and manageable set of instructions and procedures. It is
something that lays out the escalation path. When you detect something,
here’s the second level of confirmation, and here’s another confirmation
level that validates it as a genuine event. At the point that a human has
determined this is not a false positive, what are the follow-up steps?
What is the chain of notification to the organization, and what steps are
initiated upon notification?
The MSSP should be able to work with you to craft a breach response
protocol that fits your business needs. The protocol needs to tie into
your response process, and that may be dictated in part by notification
requirements set by federal or state regulators, or international jurisdictions,
and there may be requirements to notify customers. This can become quite
involved. An MSSP may have a common protocol, but it should also have an
extensive menu of options that cover your business case.
You either have internal skills to help you evaluate the MSSP’s ability to
follow through on its protocol, or you will contract with a consultant to

75
help with that. Either way, you need to walk through the MSSP’s
processes and procedures from beginning to end. It’s also a good
idea to seek feedback from an MSSP’s other customers, although
it can be difficult to find a business willing to acknowledge or
discuss the details of breaches it has experienced.
How many cyber security incidents has the vendor responded to
for your industry, and in total across their client base? Do they
understand the regulatory regulatory reporting requirements
for your industry? Does the vendor perform postmortem joint
customer/vendor reviews to learn from the incident, improve
responses, identify how to prevent/minimize future similar
events?

7676
Singapore.
Renewables Services
“You need to test the MSSP with a breach
challenge to see how it behaves and how it
acts in a real scenario.”
To see how an MSSP would respond to a breach, you have to create an exercise.
You need to test the MSSP with a breach challenge to see how it behaves and
how it acts in a real scenario. For example, what does the MSSP do if you have
a database that is dumped in the public Internet? What is the plan? You can test
that by creating a similar scenario with data dropped around the web and see if
the MSSP can pick it up. This is similar to a real case because when you have a
data breach, the first thing that happens is that people begin to share. The sooner
you can detect and contain that, the better.
You also want to see how the MSSP responds. The response will vary depending
on the level of the breach and the criticality of the information. You want to know
what steps the MSSP will take, who exactly will take those steps, whether it will
involve service interruption, and how that decision will be made. It should all be
well described and documented in the MSSP Breach Detection, Containment and
Response Plan, with all involved staff fully trained and prepared to follow it.
You should evaluate the MSSP’s technology. There are aspects of breach
response that can be automated with technology, but there are some that cannot

77
and still require human involvement. One complements the other.
Machine learning is a great help in focusing on the things that are
most important, but you cannot, based on today’s capabilities,
expect machine learning and other technologies to control
everything. When evaluating breach response capabilities, the
ultimate question is this: how will this reduce your exposure to the
world in the case of an attack happening in your organization?
What processes does the MSSP use to reduce your exposure
to the world in the case of an attack happening in your
organization?

BlueVoyant: 7 Experts Share Key Questions To Ask When Evaluating Providers

More Related Content

What's hot

Similar to BlueVoyant: 7 Experts Share Key Questions To Ask When Evaluating Providers

More from Mighty Guides, Inc.

Recently uploaded

BlueVoyant: 7 Experts Share Key Questions To Ask When Evaluating Providers