Yokogawa & NextNine – Lessons Learned: Global Cybersecurity Management System at a Major Oil & Gas Company

Copyright © Yokogawa Electric Corporation
Copyright © Yokogawa Corporation of America
Copyright © NextNine Inc. All rights reserved.
2015
Jeff Melrose CISSP-ISSEP Yokogawa
Michael Coden CISSP NextNine
Lessons Learned:
First Year of Deployment and
Operation of a Global Cybersecurity
Management System at a Major Oil
and Gas Company

- 2 -
• Overview of Global Cybersecurity System
• 60 sites worldwide
• What went right
• What went wrong
• What processes needed to change
• What technology/process changes needed to be
made
• What new technology was developed
Agenda

- 3 -
Overview of Global
Cybersecurity System

- 4 -Copyright © Yokogawa Electric Corporation
- 4 -
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
ISA / IEC-62443
Automated and Direct Asset
Discovery and Inventory
Automated Patch-AV Delivery
And Compliance/Enforcement Reports
Event and Incident Log Collection, Conditioning,
and Transfer to Centralized SIEM for Analysis
Secure Remote Access and
Secure Remote Device-to-Device Connection
Multi-Site File Transfer Infrastructure
For Multi-Site Backup / Restore
Centralizing, Connecting, & Automating Cybersecurity Processes
Makes the “Cybersecurity Culture” Scalable

- 5 -
Yokogawa/NextNine Secured Remote Solution: System Overview
Central System
IC Systems IC SystemsIC Systems IC Systems
Remote
Site System
VSE
Remote
Site System
VSE
Remote
Site System
VSE
Remote
Site System
VSE
OS Patch
Dist. Server
VSE VM Backup
Anti-Virus
Dist. Server
Central System
Business LAN (L4)
Business LAN (L4)
Remote Site System (L3.5)
PCN(L3)
Service
Center
Anti-Virus
Replication
Server
OS Patch
Replication
Server
Auth.
Server
Remote
Operation
Solution Dashboard
(IP-VPN/Internet VPN/
User Corporate WAN)
Yokogawa
IA System
OT System
Supplier A
OT System
Supplier B
Verified
patches
Secure
Tunnel

- 6 -
What Went Right

- 7 -
Customer Dashboard
• Security Center Dashboard was Created
– Security Policy enforcement
– Policy tracking green/red
– Compliance Reporting
– Patch status tracked

- 8 -
Auto-Asset Discovery & Inventory
DMZ
Central Security
Center
Application
Server
Comm
Server
Real-Time
Database
Server
Network
& Security
Devices
Virtual
Security
Engine™
Devices, Systems, Applications
Remote Site/s
Internet
External Users
Partner / SI / OEM
Full Web UI
Full Web UI
Internal Users
WMI
SNMP
OPC
SSH
HTTP
Telnet (CLI)
SFTP
FTP
Proprietary
Others
Solution supports all versions of:
 Windows (NT, XP, Vista, Win7, 2000,
2003, 2008, 2012)
 Unix (HP-UX, AIX, Solaris, ….)
 Linux (Red Hat, Ubuntu, ….)
 Any other product that can be accessed
via the protocols at the left.

- 9 -
Event & Sys-logs Converted to CEF & Transferred to Central SIEM
VSE Continuously Scans Ports & Services vs. White/Black List
Devices
Systems
Applications
Network
Devices
Virtual
Security
Engineer™
Local
Peronnel
Network
& Security
Devices
Virtual
Security
Engineer™
Local
Personnel
Internet
External Users
Partner / SI / OEM
Field Service
Full Web UI
Cyber Security SIEM and
Analysis Tools, e.g.:
ArcSight, Q-Radar,
Nitro, ….
Detecting Rogue
Devices, Ports, Services
Full Web UI
Internal Users
DMZ
Site
Central Security
Center
Application
Server
Comm
Server
Real-Time
Database
Server
Nigeria
Qatar
VSE continuously collects logs,
converts them to CEF (Common
Event Format) sends logs for
analysis and detection of
malicious activities.
VSE
continuously
scans Ports and
Services –
comparing
against
Whitelist &
Blacklist.
Full Web UI
Cybersecurity Experts

- 10 -
Customer Dashboard
• Data for Security Center Dashboard is Collected
Devices
Business
Criticality Are Patches
up to date?
Is Antivirus
up to date?
Are Logs
being sent
to the SIEM
Is Removable
Media being
used?
Do Ports &
Services
match the
Black/
Whitelist?

- 11 -
Cybersecurity Management System - Governance
• Process can now be implemented for
cybersecurity governance.
– Every plant/facility can now be tracked on an
“as-like” basis
– No more exceptions due to distance or region
– One stop shop for a view of the organizations’
Cyber defensive profile

- 12 -
Assured encrypted access across IT networks world
wide independent of media (satcom)
Virtual Security Engines:
-All remote connectivity is through a single port
outbound only connection to specific IP address
-FIPS 140-2 Compliant & 1024-bit TLS Encrypted.
Remote Site A
Remote Site B
Remote Site C
Secure Center
Certificate
Something I have
Certificate
Something I have
Certificate
Something I have
Certificate
Something I have
Trusted Platform
Module
Trusted Platform
Module
Trusted Platform
Module
– Data is compressed, encapsulated, encrypted
– No possibility of VPN bleed or fake connections
– A secure multipurpose tunnel to customer sites
Only 1 Firewall Rule to Manage for All Remote Connections

- 13 -
Bi-Directional File Transfer – to/from Anywhere: Off-site
Backup/Restore, Production Optimization, Secure File Delivery
Devices
Systems
Applications
Network
Devices
Virtual
Security
Engineer™
Local
Peronnel
Network
& Security
Devices
Virtual
Security
Engineer™
Local
Personnel
Internet
External Users
Partner / SI / OEM
Field Service
Full Web UI
Backup
Location
# 2 With
Auto-Verify
of Backups
Backup
Location
# 1 With
Auto-Verify
of Backups
Full Web UI
Internal Users
DMZ
Houston
Central Security
Center
Application
Server
Comm
Server
Real-Time
Database
Server
Nigeria
California
Amsterdam
Qatar

- 14 -
Audit Trail
• Audit Trail – Insider threat mitigation

- 15 -
Help desks established
• Established both Level 2 and Level 3
Helpdesks established in Europe
– 24/7 coverage
– Full visibility into plants supported
• Personnel
• Lead Contacts
• IT / OT local support
• Escalation contacts
• Vendor lead contacts for each plant

- 16 -
Industrial Controls – more like IT
• Fully Documented system in terms that IT and
Cybersecurity personnel understand
• Plant’s connection to unified TCP/IP network went
well.
• Initial Deployment process went well with IT related
timelines met
• IT hardware delivered on time and in good condition
(IT component procurement works!)
• Signoffs for Acceptance Testing occurred on time with
minimal issues
• Initial Training on Cybersecurity Management was
completed on time and budget

- 17 -
Secure ICS patch management
Centralized vetting of all patches with direct links to suppliers
WSUS
ePO
SEPM
WSUS
ePO
SEPM
Devices
Systems
Applications
Network
Devices
Virtual
Security
Engine™
Network
& Security
Devices
Virtual
Security
Engineer™
Remote Sites
Internet
External Users
Partner / SI / OEM
Field Service
Full Web UI
Your
Product
Patch
Server
Full Web UI
Internal Users
DMZ
Central
Security Center
Application
Server
Comm
Server
Real-Time
Database
Server
Windows
WSUS
Server
McAfee
ePO
Server
Symantec
SEPM
Server

- 18 -
Secure ICS patch management
• Able to show delivery of patches to every
plant and track to completion of patching
effort
Devices
Business
Criticality Are Patches
up to date?
Is Antivirus
up to date?

- 19 -
What went wrong

- 20 -
IT integration is hard
• Review cycles on detailed site sign off were
increased due to more people reviewing (IT
and ICS)
• Delivery synchronization problems between:
hardware, Virtual Hypervisor, OS’s, other
software modules
• Installation and configuration of software
longer than planned.

- 21 -
Regional Issues for Integration
• World wide integration is hard
– Getting personnel
– Legal to work personnel for that region
– Site access (clearance issues)
– Safety certification for personnel at plant
– Extended encryption configuration for remote
sites

- 22 -
ICS personnel NOT familiar with IT integration cycles
• IT integrations cycles are quick reaction
– ICS personnel can’t be called on like a
telephone repair man
– Advanced planning needed to get person
familiar with install to return to plant
– ICS Integration follows more of an Engineering
Process with Configuration Control.

- 23 -
IT components packaging
• IT components at HW level usually had all
components needed
• IT SW however sometimes lacked complete
deployment setup
• Training on the IT related components was
lacking for certain configuration issues
• Handover to support could be more seamless
without being a manual process (probably get
better as more sites are set up)

- 24 -
Plant build out and provisioning
• Some Plant build out was delayed due to
getting proper space to place components
• Provisioning at the network cloud to local
plant was easy
• Last mile inside the plant provisioning was
more complex (laying infrastructure inside an
active plant is time consuming, and only local
people can provide guidance on how long it
may take)

- 25 -
What processes needed to
change

- 26 -
IT vs OT/ICS
• Cats and Dogs need to declare peace ICS to
IT joint meetings
– More advanced warning of deployment plans
to plant personnel
– More information to Plant personnel to smooth
integration
– We plan on more briefings to Plants
– if possible

- 27 -
ICS/Operational Technology (OT) Controls Last Mile
• ICS and Plant Managers normally leads for
Plant last mile
• It is important to have an engineering solution
approach to IT within ICS domains
• Configuration Control, Review Process, Safety
Checks, Pre-Briefs are all processes that need
to be followed.

- 28 -
ICS / OT Runs on Maintenance Cycles
• Maintenance rules at the Plant
• ICS / OT, IT and Cybersecurity personnel
must be understanding on these cycles
• Times and locations convenient to IT, ICS and
Cyber may be completely bad for Plant
operation
• ICS / OT, IT and Cyber personnel need to be
the more flexible party
• Oh and when Plant maintenance says “you’re
done” … you ARE Done for the day!

- 29 -
What technology/process
changes needed to be
made

- 30 -
Help Desk/Service Desk to Plant Communication and
Integration
• Who to talk to when at what part of the Plant
• Who tracked at coordination at Plant level
• Better and more reliable IP based integrated
communication infrastructure to all Plants
– This integration drove
• Larger bandwidth WAN to Plants
• Class of Service management of the WAN
• Partnering with international Telecom for WAN
infrastructure and provisioning

- 31 -
Yokogawa needed to invest in Security Training Course
for Employee Engineers
As of April, 2014,
about 700 certified Yokogawa engineers.
Yokogawa has supported
GICSP program since its first day.
- 31
-

- 32 -
Yokogawa had to marshal and International
Service Organization
Response
Center
Service
Office Network
A worldwide network of Yokogawa Response Centers, service offices,
and service engineers provides a prompt response to all kinds of customer
inquiries on an around-the-clock, 365-day-per-year basis.
Call center services by
specialists
Remote monitoring and patrol
inspection
Supply of information on
hardware/software revisions
Customers
On-site
maintenance
Dispatch of
engineers
Supply of
spare parts
and
components
Remote
maintenance
Data collection
and analysis
Escalation
Technical support
from responsible
department
Korea
32
Singapore China BrazilIndiaBahrain USAThe
Netherlands
TaiwanRussia

- 33 -
What new technology was
developed

- 34 -
Remote Access Device Granularity
• Remote Access Users can be given restricted
access by:
– Site
– Device(s)
– Functions
• View
• Edit
• Delete

- 35 -
Password Vault
• Three opposing problems:
– Many systems using default passwords or same
passwords
– Ease of login required for safety operation
– Third parties had passwords outside plant
• Solution = Password Vault in VSE
– VSE contains credentials for systems with
different privileges
– VSE uses correct password for each device
depending on Remote User’s privileges

- 36 -
Secure Remote Access – Third Parties Can Only Access
Specific Devices at Specific Sites with Site Control
“Virtual Security Engineers:”
– With Remote Access, Cyber Security and 3rd Party
experts can immediately connect to only specific devices
at specific sites determined by your security policies
– Remote Site controls granting of access
– Remote Site can Supervise remote access
Remote Site A
Remote Site B
Remote Site C
Secure Center
End-customer approves
remote access
VSE Interface

- 37 -
Secure Remote Access – Direct to Device
“Virtual Security Engineers:”
– VSE Connects Experts Computer Directly to Target System
– High Speed Real Time Desktop Sharing
-- Device to Device connection for any application
Remote Site A
Remote Site B
Remote Site C
Secure Center
– Sessions are video recorded at
both Remote and Central Sites

- 38 -
Improved Asset Inventory Management
• Device Properties Entry (require someone to input
info about devices, custodian, criticality etc)
• Collected via NextNine VSE:
– Ipv4 Addresses
– MAC Addresses
– OS name and version
– Application software name and
version
– OS patches name and date
– HW manufacturer and model
– AV agent name and version
– AV signatures file version and
date
– AV service status
– WSUS properties
– Device Attributes
• Entered Into NextNine VSE:
– Custodian
– Criticality (C, E, N)
– Type (Monitoring System,
Safety System, Workstation,
Server, Firewall, Router, …)
– Vendor
– Vendor Software
– Function (Metering,
Engineering Station, DCS,
PLC, …)
– Life-cycle (Active, Inactive …)
– Deviation (free text)
Additional items may be added upon request.

- 39 -
Rapid Deployment of Exploit Scanners
Heartbleed
scanner was
delivered in 48
hours!
DMZ
Central Security
Center
Application
Server
Comm
Server
Real-Time
Database
Server
Network
& Security
Devices
Virtual
Security
Engine™
Remote Site/s
Internet
External Users
Partner / SI / OEM
Full Web UI
Full Web UI
Internal Users
• GUI based App Development Environment
• Develop new Apps in a few hours
• Distribute Apps to all VSE’s
• No recompile or reboot of VSE is
required
• App is used immediately
ShellShock
scanner was
delivered in one
week!

- 40 -
Ultimately the deployment yielded better central visibility
into security policy enforcement across all plants
<Document Number>
<date/time>

- 41 -
Thank You
May 2015
Jeff Melrose CISSP-ISSEP Yokogawa
Michael Coden NextNine

Yokogawa & NextNine – Lessons Learned: Global Cybersecurity Management System at a Major Oil & Gas Company

More Related Content

What's hot

Viewers also liked

Similar to Yokogawa & NextNine – Lessons Learned: Global Cybersecurity Management System at a Major Oil & Gas Company

Recently uploaded

Yokogawa & NextNine – Lessons Learned: Global Cybersecurity Management System at a Major Oil & Gas Company