Schneider Electric Scada Global Support Provides Troubleshooting and Technical Tips

SCADA & MES Global Support
presents troubleshooting and
technical tips

Introduction

Presenters:
L. Duguid – Senior Global Support Engineer
● Top Five Issues raised with Support (SCADA)

T. Montmayeur – Senior Global Support Engineer
● Best Practices for Upgrading Vijeo Citect / CitectSCADA

SCADA & MES Global Support Portal:
www.citect.schneider-electric.com

Schneider Electric - Industry - O&O Software Activity - SCADA Troubleshooting Tips Copyright © 2012 2

Top Five Issues Raised with Support
Presented by L. Duguid

1. Driver Debugging

2. Web Client / Server Issues

3. Cicode Debugging

4. Clustering

5. Runtime Crashes

Schneider Electric SCADA & MES


Driver Debugging

Symptoms:

Intermittent Communications, #COM, # Graphics, Errors in log files

Diagnosis:

• Check Boards, Ports, I/O Device forms as per documentation
Note: Each I/O Device should use its own port

• Upgrade Driver (DriverWeb for documentation and driver releases)

• Check Syslog.dat for errors check against the Knowledge Base
2012-09-27 09:39:21.874 +10:00 Error: Channel not yet initialized
2012-09-27 09:39:21.874 +10:00 CINIT 0017 PORT1_BOARD4 Melter_CLX Error_in_CMD-2 16
2012-09-27 09:39:21.874 +10:00 Generic 000023 Driver 536870937 (0x20000019)
2012-09-27 09:39:21.877 +10:00 Error: Channel not yet initialized

• Check Syslog.dat for errors check against the Knowledge Base


Driver Debugging
•Enable Driver debugging – best to delete logs first

Older drivers:
[MELSEC, MODNET]
DebugStr = * ALL

New Drivers:
[OPC, DNPr, ABCLX]
DebugCategory=ALL
DebugLevel=ALL
LogFileSize=50000

Search syslog for ‘Error’ and again check Hex number in Knowledge Base

• Test with a simple one tag, one I/O Device project

• Send log files to support with SOE

• Check variable tag addresses against documentation


Web Client / Server Issues
Symptoms:

#COM, # Graphics, software protection failure on Webclient, changes not propagating

Diagnosis:

• Check configuration as per documentation

Q6268 – V7.20 (Windows 7, Server 2008) – has lots of debugging tips
Q4700 – prior to V7.2 (earlier OS)
Note: There are specific settings for Server 2003

• Ensure Service Pack matches between WebServer and Citect Server

• Delete the %tmp% directory

• Check Internet Explorer settings as per Q6268 and Q4464

• Test with Example project – is the issue specific to your project or Citect in general?

• Check the ports are open (Q4946)

•Test Web Client on Server machine first


Cicode Debugging

Symptoms:

Compile Errors, Function help, Cicode Debugging

Diagnosis:

• Read the help

• Compiler will pick up deprecated functions or invalid function calls

• Debug Cicode using break points, stack window, F11

• Ensure Cicode Editor is using correct process [Debug]CodeDebug = Client

• Does running in Single process help? What process should the code be called from?


Implementing Clustering
Symptoms:

#Com, # Graphics

Diagnosis:

Introducing a second cluster will require the use of cluster context in your projects.

• Assign cluster context by either adding prefix to tags (cluster.tag), assign to page in page
properties, or via PageDisplay Cicode function on page call.

• Check Cluster Connections Setup in Computer Setup Wizard to ensure client can see all
required clusters.

• Accumulators should be assigned to Cluster and use Cluster prefix on tags.

• Connecting via OPC and CtAPI to runtime tags will require prefix.

• V7.30 to resolve tag where possible


Debugging Crashes
Symptoms:

Runtime Manager reports process has stopped responding and exception report is
generated.

Diagnosis:

• Exception’s stored in Logs directory, should be prompted to save.

• Exception contains many files, Debug.log is most useful to support.

Exception 0xc0000005 occurred at location 0x74149084 at time Thu Jul 26 21:33:08 2012

(741c0000) - (7425afff) v8.0.50727.4940
C:WindowsWinSxSx86_microsoft.........8cc06a442b34fcMSVCR80.dll

• Debug.log can contain multiple crashes.

• Crash locations can be random, harder to fault find.

• Reproduction steps are very important.

• Check for Events or Cicode being executed, change of shift, operators login, any patterns.


Debugging Crashes
• Debug Diagnostics should be used if no crash log is generated (Q5387).

• Ensure latest Service Pack and Drivers are installed

• Running in multi-process a great indication as to where the issues is.

• I/O Server Crash
Upgrade Drivers, Switch devices to memory mode, Disable devices that are not
operational

• Alarm Server Crash
Check alarm formatting in Alarm Categories, Delete Alarm save files

• Report Server Crash
Debug Cicode used in Report Format file by copying to Cicode Editor, Verify Output
device configuration and file

• Trend Server Crash
Run Trend File Clash Utility from website (Toolbox) to verify trend configuration
‘clashes’, Read trend files using Trend Reader (Toolbox), Delete trend files.

• Support use Bugzilla to search for problems or raise issue with developers for more
involved crash analysis (user.dmp analysis).


Get your cases resolved faster…
• Provide a simple test project that demonstrates the issue

• Report the issue with a clear and concise definition of the issue

• Unless you are sure, faster to let us decide where the problem is

• Tell us what you know, and what you think you know

• Provide concise information where possible (rather than “it doesn’t work”)

• Divide and Conquer – what do you know, try to refine it through trial and error

• Try and use standard terminology (Web Client, IO Server, IO Device, etc.)

• What is the history of the issue (What changed? What have you tried?)

• Keep issues separate with multiple CSR’s

• What are you trying to achieve, let us know from the outset

• Shift times are 7am-3pm, 3pm-11pm, 11pm-7am (EST), there is a staff change over
between shifts. While we endeavour to provide continuous coverage for open cases if you
prefer to speak to your specific case Engineer be sure to call within the corresponding shift
times


Best Practices for Upgrading
Presented by T. Montmayeur

1. Standard process for standalone systems

2. Upgrade on a redundant production system

3. Useful Tips

4. Upgrade tools

5. Potential issues and how to troubleshoot
them


● Why upgrade?
● New features and enhancements
● Performance and product capabilities
● Operating System compliance

Old trend style template against Process Analyst view..


Standard Process
In this presentation, the initial version is 7.0 spk4 and it Paramount
is meant to be upgraded to version 7.30.

1.Backup V7.0 project > Backup Project
2.Uninstall SCADA V7.0 > License Upgrade
3.Install SCADA V7.30
4.Restore your V7.0 project into SCADA V7.30 > Test environment

> Rollout Process

*This is the minimal upgrade process


I want to upgrade to get
the latest features. But I don’t want
any downtime and want to keep all
my historical data.


Redundant system on Production

Example Configuration
Primary Standby
• Primary Server
• Standby Server
• 4 Control Clients

Client Client Client Client


Redundant system on Production
1. Upgrade the keys
2. Set up a development environment and proceed to testing
• Test your project in memory mode
• If possible test on actual device

3. Production environment:
• On both servers: [LAN]AllowLegacyConnections = 1
• Shutdown CitectSCADA on the standby server.
• Upgrade CitectSCADA on this server according to the standard upgrade
procedure.
• Restart you standby server
• Perform testing
• Shutdown the primary
• make sure the standby takes over and perform testing again
• Once standby is running fine Upgrade Citect on the primary server
according to the standard upgrade procedure.
• Restart the primary server and check your hardware alarms to make sure
you don’t have any connection failure.
• Upgrade your clients one by one.


Useful Tips
1. History Data
• Back up trend files
• Backup alarm files ALMSAV.DAT and
ALMINDEXSAVE.DAT

If upgrading hardware as well:

Trends:
• Create the same file hierarchy on the new system
• Place the file in the same folders

Alarms:
• Run the computer setup wizard and edit Alarm paths

2. Additional files to backup, when upgrading hardware

• Data folder
• Report files (User defined folder)
• [Ctedit]<Pathname> check in citect.ini

Upgrade Tools
1. Environment manager

Development Environment
Multiple versions
Multiple projects backup and restore

2. Help
• What’s new in CitectSCADA
• Upgrading to CitectSCADA

3. Knowledgebase
http://www.citect-kb.schneider-electric.com/ScadaKB/KnowledgebaseCategory32.aspx


Potential issues and how to troubleshoot - 1
Obsolete functions
Help: Getting Started > What's New in v7.x > What's New in CitectSCADA 7.20 >
Cicode Functions

Operating Systems (OS) compatibility

• SCADA
http://www.citect.schneider-electric.com/documents/frk/SCADA/SCADA-Compatibility-Matrix.pdf

• Drivers
Not all of 150+ drivers have been fully tested on latest OS. If not compatible, own
testing required.
Once tested OS check can be disabled:
[<DriverName>]OverrideOSProtection


Slow system after upgrade

Hardware & Software. Refer to installation guide
Ex: 7.20 Installation Guide, System Requirements


Downtime
Redundant system is necessary to achieve a “No Downtime” Upgrade.

Upgrading keys

• Make sure authcode is correct
• Make sure the key is showing correct version
• Upgrade CiuSafe
• Upgrade Sentinel Driver

Further issues?



Useful Links
Knowledge base – Upgrade related articles

Installation Guide - Installation Files (DVD / Zip file)

Help
Getting Started > What's New in CitectSCADAV7.x
Getting Started > Upgrading to CitectSCADA V7.20

Website - Upgrading SCADA guidelines
http://www.citect.schneider-electric.com/index.php?option=com_content&view=article&id=1553&Itemid=1540

- Authcode Generator
http://www.citect.schneider-electric.com/index.php?option=com_content&view=article&id=1592&Itemid=942

Driver web
http://www.citect.schneider-electric.com/forum/forumdisplay.php?f=2


Schneider Electric Scada Global Support Provides Troubleshooting and Technical Tips

More Related Content

What's hot

Viewers also liked

Similar to Schneider Electric Scada Global Support Provides Troubleshooting and Technical Tips

Schneider Electric Scada Global Support Provides Troubleshooting and Technical Tips

Editor's Notes