Robert Hoopes - PPL Corporation, Speaker at the marcus evans Transmission & Distribution Summit Fall 2011, Wheeling, IL, delivered his presentation on Successful NERC CIP Compliance
3. 3
Reality CheckReality Check
““The issue is not whether your network is protected.The issue is not whether your network is protected.
We know that your network is protected. The issue isWe know that your network is protected. The issue is
about strict compliance to the plain reading of theabout strict compliance to the plain reading of the
language in the CIP standards.”language in the CIP standards.”
Corporate Risk Solutions, Inc. (CRSI), December 2009Corporate Risk Solutions, Inc. (CRSI), December 2009
Translation…while the real objective is protectingTranslation…while the real objective is protecting
critical assets, the report card is based on compliancecritical assets, the report card is based on compliance
to the minutia in the CIP standards, as judged by theto the minutia in the CIP standards, as judged by the
auditors.auditors.
4. 4
Simple Compliance ModelSimple Compliance Model
Three ElementsThree Elements
Clear requirementsClear requirements
Clear accountabilityClear accountability
Documented programDocumented program
All three elements are necessary for successAll three elements are necessary for success
By far, clear accountability is the most importantBy far, clear accountability is the most important
Accountable individuals make things happenAccountable individuals make things happen
This is the same for executives down to the SubjectThis is the same for executives down to the Subject
Matter ExpertsMatter Experts
5. 5
CIP CredentialsCIP Credentials
Responsible for corporate NERC compliance since late 2006Responsible for corporate NERC compliance since late 2006
Assigned by the CEO as CIP “senior manager”Assigned by the CEO as CIP “senior manager”
Built CIP program for multiple Registered Entities from theBuilt CIP program for multiple Registered Entities from the
ground up, as part of overall NERC compliance programground up, as part of overall NERC compliance program
External consultant supportExternal consultant support
One failed gap analysis – Un-named consultantOne failed gap analysis – Un-named consultant
One gap analysis and two mock audits - CRSIOne gap analysis and two mock audits - CRSI
One CIP audit (covered five GO/GOP Register Entities – 36One CIP audit (covered five GO/GOP Register Entities – 36
requirements); verbal feedback:requirements); verbal feedback:
Advanced documentation provided was “far superior than anything weAdvanced documentation provided was “far superior than anything we
have seen”have seen”
““Best CIP compliance program we have seen”Best CIP compliance program we have seen”
Audit completed in 3.5 daysAudit completed in 3.5 days
Two minor issues identifiedTwo minor issues identified
6. 6
Success Requirements (8)Success Requirements (8)
Successful NERC CIP Compliance requires:Successful NERC CIP Compliance requires:
Leadership engagementLeadership engagement
An organizational culture of complianceAn organizational culture of compliance
An “effective” CIP Senior ManagerAn “effective” CIP Senior Manager
A strong foundational programA strong foundational program
Technically competent Subject Matter ExpertsTechnically competent Subject Matter Experts
Sufficient resourcesSufficient resources
Strong consulting supportStrong consulting support
Extensive audit preparationExtensive audit preparation
7. 7
Success Requirement #1:Success Requirement #1:
Leadership EngagementLeadership Engagement
All compliance is localAll compliance is local
Executive management must communicate to seniorExecutive management must communicate to senior
management that CIP compliance is importantmanagement that CIP compliance is important
Senior management in turn must communicate thisSenior management in turn must communicate this
message to line managementmessage to line management
Line management makes it happenLine management makes it happen
Communicates importance of CIP complianceCommunicates importance of CIP compliance
Provides sufficient resourcesProvides sufficient resources
Sets the prioritiesSets the priorities
Periodically checks on performancePeriodically checks on performance
Clear accountability is essentialClear accountability is essential
8. 8
Success Requirement #2:Success Requirement #2:
Org. Culture of ComplianceOrg. Culture of Compliance
Strict compliance is often counter-intuitive toStrict compliance is often counter-intuitive to
individuals who have not been previously exposed toindividuals who have not been previously exposed to
itit
Mountains of records are required…Why?? “Because…Mountains of records are required…Why?? “Because…
it’s the law.”it’s the law.”
Establishing a culture of compliance takes time.Establishing a culture of compliance takes time.
People watch their leaders for cues…”Do as I do…”People watch their leaders for cues…”Do as I do…”
will help, if leaders are engaged.will help, if leaders are engaged.
Always do the right thing. This sets the tone for theAlways do the right thing. This sets the tone for the
organization.organization.
9. 9
Success Requirement #3:Success Requirement #3:
Effective CIP Senior ManagerEffective CIP Senior Manager
CIP compliance does not just happen. While it is a function ofCIP compliance does not just happen. While it is a function of
smart people wanting to do the right things, absent soundsmart people wanting to do the right things, absent sound
leadership there will be gaps in compliance. Different parts ofleadership there will be gaps in compliance. Different parts of
the organization will do what they think is required but theythe organization will do what they think is required but they
may leave gaps in the “white space” between internal workmay leave gaps in the “white space” between internal work
groups.groups.
CIP-003 Requirement 2 calls for the assignment of a singleCIP-003 Requirement 2 calls for the assignment of a single
manager with overall responsibility and authority for leadingmanager with overall responsibility and authority for leading
and managing adherence to the CIP standards.and managing adherence to the CIP standards.
The CIP founders got this one right.The CIP founders got this one right.
However, the CIP senior manager does not relieve local line leadershipHowever, the CIP senior manager does not relieve local line leadership
of CIP accountabilityof CIP accountability
10. 10
Effective CIP Senior ManagerEffective CIP Senior Manager cont.cont.
Perfunctory assignment of a high level seniorPerfunctory assignment of a high level senior
leader as the required “senior manager” toleader as the required “senior manager” to
meet the CIP-003 R2 requirement ismeet the CIP-003 R2 requirement is
problematic.problematic.
If the assigned senior manager is too high in theIf the assigned senior manager is too high in the
organization to be engaged in the ongoing issuesorganization to be engaged in the ongoing issues
related to CIP compliance, problems will arise andrelated to CIP compliance, problems will arise and
find you at a later time.find you at a later time.
11. 11
Success Requirement #4:Success Requirement #4:
Strong Foundational ProgramStrong Foundational Program
A solid compliance program has three elementsA solid compliance program has three elements
Clear requirements (CIP standards…)Clear requirements (CIP standards…)
Clear accountability (engaged leadership)Clear accountability (engaged leadership)
Documented programmatic controls (policies andDocumented programmatic controls (policies and
procedures)procedures)
Programmatic controls must documentProgrammatic controls must document
Who is responsible for what?Who is responsible for what?
CIP-003 Requirement 1 calls for a cyber security policyCIP-003 Requirement 1 calls for a cyber security policy
that addresses the requirements in Standards CIP-002 thruthat addresses the requirements in Standards CIP-002 thru
-009.-009.
The cyber security policy should document the what and who isThe cyber security policy should document the what and who is
responsible (across the organization) for doing itresponsible (across the organization) for doing it
12. 12
Success Requirement #5:Success Requirement #5:
Technically Competent SMEsTechnically Competent SMEs
Study and understand the CIP requirementsStudy and understand the CIP requirements
Identify how to comply and make it happenIdentify how to comply and make it happen
Accountable to their line leadershipAccountable to their line leadership
Should be responsible for producing and storingShould be responsible for producing and storing
required evidence of compliancerequired evidence of compliance
Explain to the auditors how/why the entity isExplain to the auditors how/why the entity is
compliant to the applicable CIP requirementcompliant to the applicable CIP requirement
13. 13
Success Requirement #6:Success Requirement #6:
Sufficient ResourcesSufficient Resources
How much is enough?How much is enough?
Line leadership must decide, based on competingLine leadership must decide, based on competing
objectives for available resourcesobjectives for available resources
Not enough can lead to painful shortfallsNot enough can lead to painful shortfalls
Can result in expensive violationsCan result in expensive violations
CIP compliance must be part of individuals’ jobCIP compliance must be part of individuals’ job
functionsfunctions
Full time CIP resources are the exceptionFull time CIP resources are the exception
CIP audit preparation is labor intensive, beginning monthsCIP audit preparation is labor intensive, beginning months
before the audit and involving the various CIP SMEs in thebefore the audit and involving the various CIP SMEs in the
business line, IT, security and other support groupsbusiness line, IT, security and other support groups
14. 14
Success Requirement #7:Success Requirement #7:
Strong Consulting SupportStrong Consulting Support
Outside eyes on your CIP program and evidence isOutside eyes on your CIP program and evidence is
absolutely essentialabsolutely essential
They will see and interpret things differently than yourThey will see and interpret things differently than your
SMEsSMEs
Based on their industry experience, they will be right mostBased on their industry experience, they will be right most
of the timeof the time
Can help identify and help fix problem areasCan help identify and help fix problem areas
Choose good CIP consultantsChoose good CIP consultants
Excellent audit support record (based on input from yourExcellent audit support record (based on input from your
peers)peers)
Those that perform CIP audits for Regions have a uniqueThose that perform CIP audits for Regions have a unique
perspective that is invaluableperspective that is invaluable
15. 15
Success Requirement #8:Success Requirement #8:
Extensive Audit PreparationExtensive Audit Preparation
Begin immediately and do it annuallyBegin immediately and do it annually
If you have not yet started, you are lateIf you have not yet started, you are late
Complete the CIP RSAWS and organize supporting evidenceComplete the CIP RSAWS and organize supporting evidence
of complianceof compliance
The audit package for some CIP standards can exceed 1,000The audit package for some CIP standards can exceed 1,000
pagespages
Audit packages should be signed by a Preparer, Reviewer andAudit packages should be signed by a Preparer, Reviewer and
Approver.Approver.
Approver is the local VP or GM - responsible for CIP compliance inApprover is the local VP or GM - responsible for CIP compliance in
their organizationtheir organization
Sometimes more than one preparer and reviewer sign the packages,Sometimes more than one preparer and reviewer sign the packages,
based on distribution of laborbased on distribution of labor
16. 16
Extensive Audit PreparationExtensive Audit Preparation cont.cont.
Recent GO/GOP CIP audit preparation and conductRecent GO/GOP CIP audit preparation and conduct
involved 33 CIP compliance personnel and SMEsinvolved 33 CIP compliance personnel and SMEs
Evidence collectionEvidence collection
Evidence reviewingEvidence reviewing
Evidence packagingEvidence packaging
SME reviewSME review
Legal reviewLegal review
SME audit presentation trainingSME audit presentation training
SMEs standing by during audit to present and/or answerSMEs standing by during audit to present and/or answer
auditors’ questionsauditors’ questions
17. 17
CIP Experience:CIP Experience:
Program Start-up to CIP AuditProgram Start-up to CIP Audit
JAN 2007 – Began development of the required CIP CyberJAN 2007 – Began development of the required CIP Cyber
Security Policy and other program documentsSecurity Policy and other program documents
Laid out internal responsibilities for each CIP requirementLaid out internal responsibilities for each CIP requirement
Identified key CIP compliance individual in each affected organizationIdentified key CIP compliance individual in each affected organization
MAY 2008 – ReviewedMAY 2008 – Reviewed CIP-002 and CIP-006 implementationCIP-002 and CIP-006 implementation
with external consultantswith external consultants
JUN 2008 – Turned on PSP securityJUN 2008 – Turned on PSP security
APR 2009 – Aborted CIP Gap AnalysisAPR 2009 – Aborted CIP Gap Analysis
Consultants were the wrong fitConsultants were the wrong fit
JUL 2009 – ConductedJUL 2009 – Conducted CIP Gap AnalysisCIP Gap Analysis (CRSI)(CRSI)
Numerous issues needed refinementNumerous issues needed refinement
18. 18
CIP Experience:CIP Experience:
Start-up to AuditStart-up to Audit cont.cont.
SEP 2009 – Turned on security for the remaining ESPsSEP 2009 – Turned on security for the remaining ESPs
DEC 2009 – ConductedDEC 2009 – Conducted CIP Mock AuditCIP Mock Audit (CRSI)(CRSI)
One major deficiency, self-reported prior to 1/1/2010 (mandatoryOne major deficiency, self-reported prior to 1/1/2010 (mandatory
enforcement date)enforcement date)
JAN 1, 2010 – CIP Standards mandatory and enforceable forJAN 1, 2010 – CIP Standards mandatory and enforceable for
PPL Registered EntitiesPPL Registered Entities
MAY-OCT 2010 – Self-reported minor CIP violationsMAY-OCT 2010 – Self-reported minor CIP violations
Several were residual issues from prior to January 1, 2010Several were residual issues from prior to January 1, 2010
MAR 2011 – ConductedMAR 2011 – Conducted CIP Mock AuditCIP Mock Audit (CRSI)(CRSI)
Minor issues needed refinementMinor issues needed refinement
MAY 2011 – ConductedMAY 2011 – Conducted CIP AuditCIP Audit
Two minor issues identifiedTwo minor issues identified
19. 19
How Much?How Much?
CIP Gap Analysis and Mock AuditCIP Gap Analysis and Mock Audit
each around consulting 80 man-hours plus travel andeach around consulting 80 man-hours plus travel and
expensesexpenses
CIP Audit Prep – High volume of internal resourcesCIP Audit Prep – High volume of internal resources
expendedexpended
Audit included five Registered Entities (two with CriticalAudit included five Registered Entities (two with Critical
Assets)Assets)
Four compliance personnel and the various Subject MatterFour compliance personnel and the various Subject Matter
Experts put in many hoursExperts put in many hours
Months of preparationMonths of preparation
Post Audit – 33 individuals recognized for theirPost Audit – 33 individuals recognized for their
contributions to the preparation and conduct of the auditcontributions to the preparation and conduct of the audit
20. 20
Audit Prep TimelineAudit Prep Timeline
December 2010December 2010
Began CIP Audit Package DevelopmentBegan CIP Audit Package Development
February 8, 2011February 8, 2011
PPL received 90-day notification letterPPL received 90-day notification letter
February 28 – March 4February 28 – March 4
Conducted Third Party Mock AuditConducted Third Party Mock Audit
March 10March 10
Submitted Pre-Audit Survey and QuestionnaireSubmitted Pre-Audit Survey and Questionnaire
March 30March 30
Submitted RSAW and EvidenceSubmitted RSAW and Evidence
May 9 -13May 9 -13
Conducted RFC Onsite AuditConducted RFC Onsite Audit
21. 21
Audit Package PreparationAudit Package Preparation
One package for each of the eight CIP standardsOne package for each of the eight CIP standards
Most packages > 1,000 pagesMost packages > 1,000 pages
Work began in DecemberWork began in December
Compliance Specialists led this effort with support from the SMEsCompliance Specialists led this effort with support from the SMEs
Compliance staff met weekly with SMEs to review RSAWCompliance staff met weekly with SMEs to review RSAW
language and supporting evidencelanguage and supporting evidence
Audit packages were reviewed by SMEs and OGCAudit packages were reviewed by SMEs and OGC
Two Day Offsite Meeting with all SMEs and Compliance staffTwo Day Offsite Meeting with all SMEs and Compliance staff
to review completed packagesto review completed packages
22. 22
The PayoffThe Payoff
Regional Entity Feedback:Regional Entity Feedback:
CEO: Advanced documentation provided was “farCEO: Advanced documentation provided was “far
superior than anything we have seen”superior than anything we have seen”
Audit Team: “Best CIP compliance program weAudit Team: “Best CIP compliance program we
have seen”have seen”
Audit completed in 3.5 daysAudit completed in 3.5 days
Included the review of more than100 TFEsIncluded the review of more than100 TFEs
Two minor issues identifiedTwo minor issues identified
23. 23
Was It Worth It??Was It Worth It??
Enforcement space is very expensiveEnforcement space is very expensive
Even a minor violation receiving a minor penaltyEven a minor violation receiving a minor penalty
has many thousands of dollars in hiddenhas many thousands of dollars in hidden
processing costsprocessing costs
While the real objective is protecting criticalWhile the real objective is protecting critical
assets, the report card is based on complianceassets, the report card is based on compliance
to the details in the CIP standards, as judgedto the details in the CIP standards, as judged
by the auditorsby the auditors
You be the judge…You be the judge…