Read the article \"Security Controls that Work\" by Dwayne Melancon in the 2007 Issue, Volume 4 of the Information Systems Control Journal (available http://www.isaca.org/Journal/Past-Issues/2007/Volume-4 /Pages/Security-Controls-That- Work1.aspx). Write a report that answers the following questions: 1. What are the differences between high-performing organizations and medium- and low- performing organizations in terms of normal operating performance? Detection of security breaches? Percentage of budget devoted to IT? 2. Which controls were used by almost all high-performing organizations, but were not used by any low- or medium-performers? 3. What three things do high-performing organizations never do? 4. What metrics can an IT auditor use to assess how an organization is performing in terms of change controls and change management? Why are those metrics particularly useful? Security Controls That Work By Dwayne Melançon, CISA Ask the average IT or security manager what measures his/her organization takes to secure its networks, systems, applications and data, and the answer will most likely involve a combination of traditional perimeter protection solutions (such as firewalls, intrusion detection, antivirus and antispyware) together with patch management, business continuance strategies, and access control methods and policies. All of these measures make sense at first glance, yet the deluge of intrusions, data thefts, worms and other attacks continues unabated, with organizations losing productivity, revenue and customers every year. There are many reasons for this gap in controls and effectiveness. Access controls can be taken only so far before they run into legitimate resistance from employees who find their productivity hampered by the very controls designed to protect it. Traditional perimeter protection and access control are not as effective at blocking attacks from inside organizations as they are at blocking external hackers, which says a lot, since the latter manage to breach thousands of company networks every year. And, as the number and frequency of zero-day attacks continue to grow, the effectiveness of patch management and traditional signature-based intrusion detection, antivirus and antispyware solutions is increasingly in doubt. All of this begs a host of questions: How is it possible to determine whether an organization’s security controls actually work? Of all the hundreds of practices and objectives within Control Objectives for Information and related Technology (COBIT), IT Infrastructure Library (ITIL) and the other frameworks an organization may implement, which ones are truly the most effective at helping the organization block and respond to attacks—and which ones merely sound good but do not accomplish all that much in practice? Why are some organizations vastly better than others at preventing and responding to attacks? On which controls should auditors focus to verify that the infrastructure is genuinely protec.

1. Read the article " Security Controls that Work" by Dwayne Melancon below write a report that
answers the following questions.
4. What metrics can an IT auditor use to assess how an organization is performing in terms of
change controls and change management? Why are those metrics particularly useful?
Security Controls That Work By Dwayne Melançon, CISA Ask the average IT or security
manager what measures his/her organization takes to secure its networks, systems, applications
and data, and the answer will most likely involve a combination of traditional perimeter
protection solutions (such as firewalls, intrusion detection, antivirus and antispyware) together
with patch management, business continuance strategies, and access control methods and
policies. All of these measures make sense at first glance, yet the deluge of intrusions, data
thefts, worms and other attacks continues unabated, with organizations losing productivity,
revenue and customers every year. There are many reasons for this gap in controls and
effectiveness. Access controls can be taken only so far before they run into legitimate resistance
from employees who find their productivity hampered by the very controls designed to protect it.
Traditional perimeter protection and access control are not as effective at blocking attacks from
inside organizations as they are at blocking external hackers, which says a lot, since the latter
manage to breach thousands of company networks every year. And, as the number and frequency
of zero-day attacks continue to grow, the effectiveness of patch management and traditional
signature-based intrusion detection, antivirus and antispyware solutions is increasingly in doubt.
All of this begs a host of questions: How is it possible to determine whether an organization’s
security controls actually work? Of all the hundreds of practices and objectives within Control
Objectives for Information and related Technology (COBIT), IT Infrastructure Library (ITIL)
and the other frameworks an organization may implement, which ones are truly the most
effective at helping the organization block and respond to attacks—and which ones merely sound
good but do not accomplish all that much in practice? Why are some organizations vastly better
than others at preventing and responding to attacks? On which controls should auditors focus to
verify that the infrastructure is genuinely protected? Come budget approval time, where should
the company concentrate its security money, and how can it be demonstrated to senior
management that those proposed investments will actually do the job? These are the types of
questions the IT Process Institute (ITPI) set out to answer when it was founded in 2000. One of
the results of ITPI’s work, the “IT Controls Performance Benchmark Study,”1 proves with
empirical evidence that not only are some organizations vastly better than the rest of the pack at
preventing and responding to attacks, but also that the difference between these and other
organizations’ effectiveness boils down to just a few foundational controls. And the most
significant within these foundational controls are not rooted in access control, but in monitoring

2. and managing change. According to Gene Kim, cofounder with Kevin Behr of ITPI, “Security
executives often whine that the business does not value security controls, viewing them as
bureaucratic and burdensome. What the “IT Controls Performance Benchmark Study”
benchmarking proves is that no matter how many access controls you have, you won’t get the
performance or security breakthroughs you really want until you tackle change.” Pareto in
Practice In more than six years of research, the IT Controls Performance Study examined 98 IT
groups across multiple industries to determine whether the Pareto Principle, otherwise known as
the 80/20 rule, applies to IT controls. The Pareto Principle states that, for many phenomena, 80
percent of the consequences stem from 20 percent of the causes. As part of its research, ITPI was
able to identify a small group of very high-performing IT organizations that had the following
outstanding characteristics: • Superior service levels, measured by the mean time between
failures and low mean time to repair • The earliest and most consistent integration of security
controls into IT operational processes, measured by control location, security staff participation
in the IT operations life cycle and number of security incidents resulting in loss • The best
posture of compliance, measured by the fewest number of repeat audit findings and lowest staff
count required to stay compliant • High efficiencies, measured by high serverto-system
administrator ratios and low amounts of unplanned work (i.e., new work that is unexpectedly
introduced when a change is made) Further benchmarks and survey results led to some truly eye-
opening observations regarding security. When it came to preventing and responding to security
incidents, the high performers, which represented 13 percent of the survey respondents,
outperformed their lower-performing peers by a factor of five to 10. When these high performers
experienced a breach, they were markedly better at response than their lower-performing peers,
for example: • High performers typically detected breaches within minutes vs. hours for medium
performers and even days for low performers Why are some organizations vastly better than
others at preventing and responding to attacks? I NFORMATION S YSTEMS C ONTROL J
OURNAL , V OLUME 4, 2007 • High performers were far more likely to detect breaches using
existing automated controls. Medium performers were 60 percent less likely to detect breaches
this way, and low performers were 79 percent less likely to detect breaches with such controls. •
High performers were 29 percent less likely than companies classified as medium performers to
experience financial loss or loss of customers and reputation and 84 percent less likely than
companies classified as low performers The corresponding performance gap in operations was
similarly dramatic. Compared to medium and low performers, high performers: • Completed
eight times as many projects • Managed six times as many applications and IT services •
Authorized and implemented 15 times as many changes • Achieved server-to-system
administrator ratios 2.5 times higher than medium performers and 5.4 times higher than low
performers • Experienced one-half the change failure rate of medium performers and one-third

3. the change failure rate of low performers • Experienced 12 percent less unplanned work than
medium performers and 37 percent less than low performers Another interesting finding was that
top performers allocated three times more budget to IT as a percentage of their total operating
expenses than their lower-performing brethren. This may seem counterintuitive at first, but this
finding actually reflects higher IT satisfaction ratings in the business and, therefore, more
willingness on the part of senior management to spend a higher percentage of the budget on IT
and IT security projects. After all, these organizations have proven they deliver more predictable
results with the money they receive, so they can more easily justify funding for additional
projects. Which Controls? After identifying high-performing organizations, researchers set out to
determine whether there was some consistency in the types of controls most commonly
implemented by the high performers compared to their lower-performing counterparts. This
would, in turn, provide evidence as to which controls were actually the most effective in helping
organizations prevent and respond to security incidents. To do this, researchers identified 63
COBIT control objectives within six ISO 20000 control categories—access, change, resolution,
configuration, release and service levels— representing the places where high-performing
organizations first implement IT controls. They then conducted a survey containing 25
performance indicators spanning audit, operations and security performance measures. These
included security effectiveness, audit compliance disruption levels, IT user satisfaction and
unplanned work. By analyzing relationships between control objectives and corresponding
performance indicators, researchers were able to differentiate which controls are actually most
effective for predictable service delivery, as well as for preventing and responding to security
incidents. The study concluded that the Pareto Principle does apply. Study results showed that 20
percent of the controls provide 80 percent of the benefit. In this case, researchers found that 21
controls, three to four within each of the six control categories, had the same impact on
performance measures as the full set of 63 controls. The next question, however, was whether
using more of the 21 foundational controls actually resulted in better security and higher
performance. To answer this question, researchers employed a statistical technique called
clustering to group similar populations with similar control environments and performance. The
goal of this exercise was to find a cluster that achieved the absolute highest levels of
performance. Figure 1 shows a representation of the controls of the three clusters that emerged.
Each wedge on the polar vector indicates one of the foundational controls, and the size of each
wedge shows the percentage of the cluster members that responded “yes” to questions that
mapped to that control. What is immediately apparent is that nearly all the members of the high-
performing cluster used all of the foundational controls, while almost all the members of the low-
performing cluster used none of them, except those that applied to access and resolution. 2
Figure 1—Three Clusters: Low, Medium and High Performers Low Performers Medium

4. Performers High Performers 5: Resolution 0: Access 1: Change 4: Svdlvl 3: Release 2: Config 5:
Resolution 0: Access 1: Change 4: Svdlvl 3: Release 2: Config 5: Resolution 0: Access 1:
Change 4: Svdlvl 3: Release 2: Config F-Cluster Low F-Cluster Med F-Cluster High I
NFORMATION S YSTEMS C ONTROL J OURNAL , V OLUME 4, 2007 What does this
mean exactly? Low-performing organizations rely almost exclusively on access controls, such as
issuing and revoking passwords, and reactive resolution controls, such as trouble-ticketing
systems, to prevent and respond to security incidents. The study further found that out of the 21
foundational controls high performers used, there were two used by virtually all the high
performers and none of the low or medium performers. Both are highlighted in figure 2, which
overlays the high performers’ cluster controls with those of the medium performers, indicated by
the solid black line. Both of these controls revolve around change management: • Are systems
monitored for unauthorized changes? • Are there defined consequences for intentional
unauthorized changes? These two controls are very significant in that they are “discriminant
controls” in this study, meaning that when they are absent from an organization, that
organization is never a high performer. Rounding out the top six foundational controls were four
change and configuration management controls identified as most present in the high performers
and least present in medium and low performers: • A formal process for IT configuration
management • An automated process for configuration management • A process to track change
success rates (the percentage of changes that succeed without causing an incident, service outage
or impairment) • A process that provides relevant personnel with correct and accurate
information on current IT infrastructure configurations The study found that these top six
controls help organizations manage risks and respond to security incidents by giving them the
means to look forward, averting the riskiest changes before they happen, and look backward,
identifying definitively the source of outages or service issues. Because they have a process that
tracks and records all changes to their infrastructure and their success rates, highperforming
organizations have a more informed understanding of their production environment and can rule
out change as a cause very early in the incident response process. This means they can easily find
the changes that caused the incident and remediate them quickly. Low performers lack the means
to detect unauthorized change in their IT environments and, therefore, expose themselves to
higher security risks and a decreased ability to respond to events quickly. In fact, the study
showed that high performers have fewer security incidents, fewer audit findings and lower
compliance costs than low and medium performers. Further bolstering the observation that
change management is a major differentiator, the study found three things that all high-
performing security and IT organizations never do: • They never let developers make changes in
production. • They never let change management processes get bureaucratic. • They never let
users exceed their role in the change process. What This Means The most impressive aspect of

5. the ITPI study is just how clear and definitive the results are. The organizations that are most
successful in preventing and responding to security incidents are those that have mastered
change management. Those that are least successful focus all their security resources on access
management and reactive resolution controls, and none on change management. The
implications are best described in the Visible Ops Handbook. For an organization to be a high
performer, it must cultivate a “culture” of change management and causality throughout, with
zero tolerance for unauthorized changes. As with any organizational culture, the culture of
change management should start at the top, with leaders establishing a tone that all change must
follow an explicit change management policy and process from the highest to the lowest levels
of the organization, with zero tolerance for unauthorized change. These same executives should
establish concrete, wellpublicized consequences for violating change management procedures,
with a clear, written change management policy. Many of the study’s high performers said their
organization had instituted a policy of “warn once, discipline on second offense,” and involved
top management in the warning process. Those that do not have this culture are likely to show a
higher frequency of security incidents, longer and less-effective incident response, more
unplanned work, lower service quality, and poorer compliance. One of the components of an
effective change management policy is the establishment of a governing body, such as a change
advisory board, that reviews and evaluates all changes for risk before approving them. This
board reinforces the written policy, requiring mandatory testing for each and every change, and
an explicit rollback plan for each in the case of an unexpected result. Postincident reviews are
also crucial, so that the organization protects itself from repeating past mistakes. During these
reviews, change owners should document their findings and work to integrate lessons learned
into future operational practices. Perhaps most important for responding to changes is having
clear visibility into all change activities, not just those that are authorized. Automated controls
that can maintain a change history reduce the risk of human error in managing the 3 Figure
2—High vs. Medium Performer Clusters 5: Resolution 0: Access 1: Change 4: Svdlvl 3: Release
2: Config F-Cluster High I NFORMATION S YSTEMS C ONTROL J OURNAL , V OLUME
4, 2007 process. They also allow IT to take measures such as preimplementation testing or more
rigorous change review to improve change success rates and accurately measure the
effectiveness of those processes and policies. The Role of Auditing High-performing
organizations were able to provide proof that management audited actual practices and enforced
accountability for process and policy adherence. Auditors can play a crucial role in moving an
organization from the low- or medium-performing category to the high-performing category. By
focusing heavily on the following metrics, an IT auditor can get a good picture of how the
organization is performing: • Amount of time devoted to unplanned work—An unplanned work
rate higher than 20 to 25 percent is a sure indication of a lack of effective controls and a cultural

6. problem within IT. It usually means too much time and resources are spent on troubleshooting
and maintaining IT operations and not enough time is spent on improving the business. The
Visible Ops Handbook indicates that high performers spend less than 5 percent of their time on
unplanned work. • Volume of emergency changes—Almost by definition, “emergency” changes
are unauthorized changes that are often used as a way to circumvent the formal change
management process or avoid disciplining employees for violating those processes. If an
organization has a volume of emergency changes that exceeds 15 percent, auditors should take
that as a warning sign that it is not taking change management seriously. The highest performers
tend to have 5 percent or fewer emergency changes. Also, it is important to ensure that there is
an actual process, albeit streamlined, for emergency changes. • Number and causes of failed
changes—The ITPI study found that high performers consistently maintained successful change
rates of 95 percent or more, often as high as 99 percent. Successful changes are those that are
implemented without causing an outage or unplanned work episode. Other things to look out for,
which the study found in medium and low performers, include: • A high frequency of security
incidents, unexplained outages or other system availability events • A lot of late projects and cost
overruns due to unplanned or emergency work • High employee turnover and low morale
Auditors also should examine the automated controls used by the organization to gain visibility
into all change activities, not just authorized changes, to determine if the change management
technology successfully covers all the right foundational controls. Some of these technology
types include: • Preventive—This is usually a change management or authorization system, such
as an IT service or help desk, that can create an audit trail of authorizations, track the status of
changes and guide the overall change process. • Detective—This technology uses automated,
independent detective controls or random change audits to monitor the production environment
for changes, compare changes with authorizations, and detect undocumented changes that
circumvent the change review and authorization process or violate policy. Called “out of band”
changes, these also include extra changes hidden in an authorized work order. • Corrective—This
technology implements processes, such as provisioning or backup and restoration programs, that
can revert unauthorized or troublesome changes and restore the system to a known, authorized,
supported state. Look at the Numbers The results of the IT Controls Performance Study make a
strong case, based on empirical evidence, that most of the value of IT security controls comes
from implementing a small subset of COBIT or other controls centered around change
management. Organizations that focus on access and reactive resolution controls at the expense
of change management are guaranteed to experience more security incidents, more damage from
security incidents, and dramatically longer and less-effective resolution. Organizations that foster
a culture of disciplined change management and causality, with full support from senior
management, and have zero tolerance for unauthorized change, will have a superior security

7. posture with fewer incidents, dramatically less damage to the business from security breaches
and much faster resolution of incidents when they happen. Change management is particularly
effective at detecting internal security breaches, which many existing security strategies and
technologies, such as firewalls and access controls, fail to address adequately. A recent Deloitte
Touche Tohmatsu study found that almost half of all surveyed financial services companies had
experienced an internal breach in the past year.2 Security is not the only benefit of a culture of
change management. Organizations that foster a culture of change management also perform
dramatically better than their less change-oriented counterparts in just about every way, from less
unplanned work to more successful IT projects, higher number of successful changes and much
more efficient use of IT resources. “The security managers who are gaining responsibility and
budget are those who are tackling the harder issues around change,” said Gene Kim. “Those who
don’t will continue to shrink in responsibility or have their air cut off.” Endnotes 1 ITPI, “IT
Controls Performance Benchmark Study,” April 2006, www.itpi.org 2 Deloitte Touche
Tohmatsu, Global State of Information Security, 2005, www.deloitte.com Dwayne Melançon,
CISA is the vice president of corporate and business development at Tripwire. He is a specialist
in strategic partnerships and alliances, and developing professional services and support
organizations. Melançon is certified on both IT management and audit processes, possessing
ITIL Foundations. actually the most effective in helping organizations prevent and respond to
security incidents. High performers were far more likely to detect breaches using existing
automated controls. Medium performers were 60 percent less likely to detect breaches this way,
and low performers were 79 percent less likely to detect breaches with such controls To do this,
researchers identified 63 CoBIT control objectives within six ISO 20000 control categories-
access, change, resolution, configuration, release and service levels representing the places
where high-performing organizations first implement IT controls. They then conducted a survey
containing 25 performance indicators spanning audit, operations and security performance
measures. These included security effectiveness, audit compliance disruption levels, IT user
satisfaction and unplanned work. By analyzing relationships between control objectives and
corresponding performance indicators, researchers were able to differentiate which controls are
actually most effective for predictable service delivery, as well as for preventing and responding
to security incidents . High performers were 29 percent less likely than companies classified as
medium performers to experience financial loss or loss of customers and reputation and 84
percent less likely than companies classified as low performers The corresponding performance
gap in operations was similarly dramatic. Compared to medium and low performers, high
performers Completed eight times as many projects Managed six times as many applications and
IT services Authorized and implemented 15 times as many changes Achieved server-to-system
administrator ratios 2.5 times higher than medium performers and 5.4 times higher than low

8. performers Experienced one-half the change failure rate of medium performers and one-third the
change failure rate of low performers The study concluded that the Pareto Principle does apply.
Study results showed that 20 percent of the controls provide 80 percent of the benefit. In this
case, researchers found that 21 controls, three to four within each of the six control categories,
had the same impact on performance measures as the full set of 63 controls .Experienced 12
percent less unplanned work than mediunm performers and 37 percent less than low performers
The next question, however, was whether using more of the Another interesting finding was that
top performers allocated 21 foundational controls actually resulted in better security and three
times more budget to IT as a percentage of their total operating expenses than their lower-
performing brethren. Thisemployed a statistical technique called clustering to group may seem
counterintuitive at first, but this finding actually reflects higher IT satisfaction ratings in the
business and, therefore, more willingness on the part of senior management to that achieved the
absolute highest levels of performance. spend a higher percentage of the budget on IT and IT
security projects. After all, these organizations have proven they deliver clusters that emerged.
Each wedge on the polar vector more predictable results with the money they receive, so they
indicates one of the foundational controls, and the size of each can more easily justify funding
for additional projects. higher performance. To answer this question, researchers similar
populations with similar control environments and performance. The goal of this exercise was to
find a cluster Figure 1 shows a representation of the controls of the three wedge shows the
percentage of the cluster members that responded "yes" to questions that mapped to that
control. Which Controls? What is immediately apparent is that nearly all the members After
identifying high-performing organizations, researchers of the high-performing cluster used all of
the foundational set out to determine whether there was some consistency in the controls, while
almost all the members of the low-performing types of controls most commonly implemented by
the high performers compared to their lower-performing counterparts and resolution. This would,
in turn, provide evidence as to which controls were researchersof the high-performing cluster
used all of the foundational cluster used none of them, except those that applied to access Figure
1-Three Clusters: Low, Medium and High Performers Low Performers Medium Performers High
Performers 0: Access 5: Resolution 0: Access US 4: Svdlvl 4: Svdlvl ge Hi Ig 1: Change Ch nge
2: Config 3: Release lg 1g INFORMATION SYSTEMS CONTROL JoURNAL, VoLUME 4,
2007
Solution
Metrics Which an IT auditor uses to assess how an organization is performing in terms of change
controls and change management are:

9. These Metrics are helpful because of below mentioned parameters.
- Identifying Poor Vulnerability Management.
- Improving Vulnerability Management.
- How Vulnerability Management Drives Changes to the IT Infrastructure.
- Identification and Validation.
- Risk Assessment and Prioritization.
- Using Past Experience to Guide Future Actions.
- Achieving Efficiency through Automation.