What to Do When You Don’t Know What to Do: Control System Patching Problems and Their Solutions


Published on

FoxGuard Solutions has encountered and resolved a wide variety of problems in our monthly work of patching control systems for our OEM clients and hundreds of power utility sites. In this presentation, we will cover a list of problems you might encounter and some real-world strategies that we have helped our clients implement to deal with them.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

What to Do When You Don’t Know What to Do: Control System Patching Problems and Their Solutions

  1. 1. FoxGuard Solutions 1 Monta Elkins Security Architect -- FoxGuard Solutions www.FoxGuardSolutions.com What to do when you don’t know what to do: Control system patching problems and their solutions
  2. 2. Installed Software FoxGuard Solutions 2 Windows Control Panel – Programs and Features
  3. 3. Installed Software FoxGuard Solutions 3 This powershell command shows the installed software: Get-WmiObject win32_product | Select-Object Name,Vendor,Version
  4. 4. Finding Patches Patch Tuesday FoxGuard Solutions 4
  5. 5. Identifying Patches FoxGuard Solutions 5
  6. 6. Air-gapped FoxGuard Solutions 6 update the wsusscn2.cab manually it usually resides in C:UsersusernameAppDataLocalMicrosoftMBSACachewsu sscn2.cab download the cab file from here and “carry it” http://download.windowsupdate.com/microsoftupdate/v6/wsus scan/wsusscn2.cab Now use MBSA to identify patches
  7. 7. Identifying Patches FoxGuard Solutions 7 CLI options: From the mbsa program folder (c:Program FilesMicrosoft Baseline Security Analyzer) Execute Mbsacli >results.txt
  8. 8. Which are Security Patches FoxGuard Solutions 8
  9. 9. Security Patches FoxGuard Solutions 9
  10. 10. A Patch List FoxGuard Solutions 10 Manually download and carry patches from the final list and install them
  11. 11. Another Approach FoxGuard Solutions 11 Discovering Patches and Downloading them Virtual Environment Approach: Setup virtual machines containing all software identified on your systems, (but not configuration information) Connect virtual machines to the Internet Scan to identify and download appropriate patches Hand carry the validated patches to air gapped machines
  12. 12. Installed Updates FoxGuard Solutions 12
  13. 13. Another Method to Verify Patch Installation FoxGuard Solutions 13 Powershell: Get-WmiObject -Class "win32_quickfixengineering"
  14. 14. Windows Update History FoxGuard Solutions 14
  15. 15. Verifying Patch Installation FoxGuard Solutions 15
  16. 16. Watch for Disk Space Issues Patches will not install if there is not enough disk space. Recommendation: Have at minimum 1 Gigabyte free storage space Troubleshooting FoxGuard Solutions 16
  17. 17. Patch Failure FoxGuard Solutions 17 Microsoft Patch fails to install System Update Readiness Tool “The System Update Readiness Tool can help fix problems that might prevent Windows updates and service packs from installing If your computer is having problems installing an update or a service pack, download and install the tool, which runs automatically. Then, try installing the update or service pack again.”
  18. 18. Missing Patches FoxGuard Solutions 18 Detection Issue: Update KB2645410 for Windows 7 and Windows Server 2008 R2 Historians. Update for Microsoft Visual Studio 2010 Service Pack 1. This update may be required but is not detected by Shavlik (vCenter) Protect. Corrective Action: FoxGuard Solutions recommends that you manually deploy update KB2645410 on all Windows 7 and Windows Server 2008 R2 Historians
  19. 19. FoxGuard Solutions Technical Information Notice Notice#:20140312-01 Notice Title: AVG Virus Warning Reason for Notice: After applying the AVG Anti-Virus 2013 updates from the M1 2014 release the virus “VBS/Downloader.Agent” was found on the system. FoxGuard Solutions has confirmed the two files referenced are automated manufacturing process artifacts used during the HMI manufacturing process that were not removed prior to the system being shipped from the factory. AV Signature Updates Can Cause Problems FoxGuard Solutions 19
  20. 20. The script is used to temporarily turn off User Account Control (UAC) so that manufacturing automation tools can run successfully on the system. FoxGuard Solutions has determined that these scripts are not infected files, but they do contain code that triggers AVG to flag them as a virus. Specifically, the following code is flagged by AVG: If WScript.Arguments.length = 0 Then Set objShell = CreateObject("Shell.Application") objShell.ShellExecute "wscript.exe", Chr(34) & _ WScript.ScriptFullName & Chr(34) & " uac", "", "runas", 1 Else This is effectively equivalent to right-clicking an application and choosing “Run as administrator”. This is a common practice with scripts that require UAC elevation to execute properly, earlier releases did not flag these files as malware. AV Trigger Details FoxGuard Solutions 20
  21. 21. Validation Checklists & Signoffs FoxGuard Solutions 21 Have a set of validation checklists to verify operations after patching. Include testing signoff for record keeping
  22. 22. AV & IDS Signatures FoxGuard Solutions 22 CIP 007-3 R4.2. The Responsible Entity shall document and implement a process for the update of anti-virus and malware prevention “signatures.” The process must address installing and testing the signatures. Use a “virus test file” "EICAR Standard Anti-Virus Test File“ 68 bytes And a “malicious network traffic” file
  23. 23. Ports and Services FoxGuard Solutions 23 Logical Network Accessible Ports – What are they? – Listening ports – Document need • What is it? • Why is it needed? • On this particular device – Or Shut it off • Host based firewall mitigation – RPC port changes – MS DNS 2501 (MS improper docs) – Every 35 days (and patching / updates 010-1) Centralized Ports and Services Auditor (CPSA) White Paper FoxGuardSolutions.com
  24. 24. Improper Documentation for DNS FoxGuard Solutions 24 DNS documentation from Microsoft could cause you to fail an audit We received this acknowledgement of our findings
  25. 25. Test Lab and Rollout FoxGuard Solutions 25 Validation lab equipment should closely mirror production equipment Where direct mirroring isn’t practical, be sure to include a superset of all installed software. Now do it “for real” Use phased rollout approach: •Test lab •Less critical machines •More critical machines •Patch •Verify •Validate •Backup
  26. 26. FoxGuard Patching and Validation Services FoxGuard Solutions 26 FoxGuard Solutions' DisPatch subscriptions provide validated patches and updates plus documentation on a monthly basis. To learn how FoxGuard Solutions can help you with patch and update validation, contact us at requestinfo@foxguardsolutions.com, or by calling 877-446-4732.