Introduction to Novell ZENworks           ®                      ®



Configuration Management Troubleshooting



Edward Carter                     Kurt Finlayson
Escalation Engineer               Escalation Engineer
Novell Inc./edcarter@novell.com   Novell Inc./kfinlayson@novell.com


Shaun Pond
Escalation Engineer
Novell Inc./spond@novell.com
Objectives Of This Session

                        • Basic troubleshooting

                        • Key logs

                        • Tools

                        • Strategies

                        • Key Processes


2   © Novell, Inc. All rights reserved.
Agent Troubleshooting
Agent Troubleshooting

    •   Services running on the agent

    •   Communication

    •   Agent cache

    •   ZAC commands




4   © Novell, Inc. All rights reserved.
Agent Services

    •   Novell ZENworks Agent Service
                        ®                 ®




    •   Novell Identity store
    •   Remote Management
    •   Novell ZENworks PreAgent




5   © Novell, Inc. All rights reserved.
Communication

    Ports
     80, 443, 2645, 5950, 7628
    Testing listening ports
         –   VNC
         –   Telnet
             telnet <IP address> <port number>
    Testing Quicktask port
         http://<Agent IPaddress>:7628/RURunning




6   © Novell, Inc. All rights reserved.
Agent Cache

    •   Cache location
    •   Cache DBs
    •   ZAC cache command




7   © Novell, Inc. All rights reserved.
ZENworks Administration Command       ®


    Line Interface (ZAC)

    •   What is ZAC?
    •   ZAC usage information
    •   ZAC zeninfo
    •   Most important files to examine.




8   © Novell, Inc. All rights reserved.
ZAC Commands (1)

    •   zac -- help (Lists the commands that can be used)
    •   zac <option> --help (Describes what the command
        does)
         –   zac bundle-list --help
         –   zac register --help
    •   zac policy-list (pl)
    •   zac bundle-list (bl)
    •   zac file-system-guid (fgs)




9   © Novell, Inc. All rights reserved.
ZAC Commands (2)

     •   zac logger (log)
         alternative to using the z-icon
          –   zac logger level debug
          –   zac logger resetlog
     •   zac emp wakeup
          –   make sure all messages are in the zmd-
              messages.log




10   © Novell, Inc. All rights reserved.
ZAC zeninfo

     •   zac zeninfo-collect
          –   zac zeninfo-collect [<targetfile>]
              one-step log collector
     •   Queries the runtime state of the ZenworksWindowsService for
               detailed information.
          –   ZenworksWindowsService must be running for zac zeninfo to
              work.
          –   Collects metadata about cached data on workstation.
     •   Must extract zeninfo-date-time.zip for html links to work
     •   Pre ZCM 10.2, this command was “zac info-collect”
          –   zeninfocollect*.zip still available in
              installdownloadstools folder for use in case agent is not
              functional.

11   © Novell, Inc. All rights reserved.
ZAC zeninfo
     (cont.)




12   © Novell, Inc. All rights reserved.
Contents of zeninfo*.zip

     •   zac zeninfo creates zeninfo-date-time.zip in the
             %TEMP% folder by default
          –   Folders in the zeninfo-date-time.zip
               >   AgentCacheInfo
               >   ConfigInfo
               >   DebugLogs
               >   MDStatus
               >   ProductInfo
               >   PropPageInfo
               >   RefreshInfo
               >   systemInfo


13   © Novell, Inc. All rights reserved.
Contents of zeninfo*.zip
     (cont.)




14   © Novell, Inc. All rights reserved.
configInfo

     •   configInfoZENWORKS_HOMEconf

          –   DeviceData hashed representation of the device
              password

          –   DeviceGuid GUID should match the GUID in ZCC




15   © Novell, Inc. All rights reserved.
debugLogs

     •   Contains Windows Event View Logs
     •   ZENWORKS_HOMElogsLocalStorezmd-
             messages.log
     •   Additional logs collected when zac zeninfo is run on a
              server
         – loader-messages.log

         – services-messages.log

         – zenworks-installation*.log

         – zcc.log




16   © Novell, Inc. All rights reserved.
Agent Logging Setup
     •   (zmd-messages.log) = Contains most of the agent side logging information
     •   (NWGINA.LOG) = Contains information related to GINA startup
     •   (zenlgn.log) = Login process
     •   (casaauthtoken.log) = CASA Client Log
     •   (nalshell.txt) NAL Shell Log = The NAL Shell log contains information related to the ZENworks   ®



         explorer shell extension.
     •   (system-update.log) = logs a SU from the time the systemupdatemodule takes over.




17   © Novell, Inc. All rights reserved.
Enable Agent Logging

     •   Screenshots for zicon
     •   Casa and remote
     •   Log locations
     •   Xml, config, registry




18   © Novell, Inc. All rights reserved.
Server Troubleshooting
Server Troubleshooting

     •   Services running on the server
     •   Communication
     •   Logs




20   © Novell, Inc. All rights reserved.
Server Services

     Novell ZENworks loader
                    ®                      ®




     Novell ZENworks preboot policy service
     Novell ZENworks preboot service
     Novell ZENworks remote management
     Novell ZENworks server
     Novell ZENworks services monitor
     Novell ZENworks tftp service
     Novell ZENworks proxy DHCP service
     Novell ZENworks embedded datastore
     CasaAuthTokenSvc

21   © Novell, Inc. All rights reserved.
Communication
     •   Database ports
          – Sybase 2638
          – Oracle 1521
          – SQL 1543
     •   LDAP
          – 389 and 636
     •   SOAP
          – 80 and 443
     •   PDHCP
          – 67 and 4011
     •   TFTP
          – 69
     •   Preboot policy
          –   13331
     •   Preboot server
          –   998



22   © Novell, Inc. All rights reserved.
Logs

         see TID 3418069 for information on how to obtain these
         logs.
     •   loader-messages.log
          – Log from ZENworks loader       ®




     •   serviceservices-messages.log
     •   CASA logs (ats.log ats.trace)
     •   zcc.log
     •   catalina.out
     •   Preboot stuff
     •   System-update logs
     •   Install log
     •   Windows event logs


23   © Novell, Inc. All rights reserved.
ZEN and The Art of Database Management
Database Types

     •   Embedded Sybase
     •   External Sybase
     •   MS SQL 2005
     •   Oracle 10g




25   © Novell, Inc. All rights reserved.
Database Types
     (cont.)

               Think before you install
               Choose the right database

             Database                      Suitable for
             Sybase (embedded)             As many as 1,000
             Sybase (external)             As many as 5,000
             Microsoft SQL Server 2005     5,000 to 40,000
             Oracle 10g                    5,000 to 40,000


     http://www.novell.com/documentation/zcm10/zcm10_deployment_bp/data/index.html




26   © Novell, Inc. All rights reserved.
zman

     •   Command-line interface for ZCM
     •   Runs on any Primary server
     •   Can be used to script operations
     •   zman
          –   Lists all commands
     •   zman <category> /?
          –   Gives specific commands within the category, e.g.,
              zman policy /?
     •   zman <command> --help
          –   Command-level usage and examples, e.g.,
              zman policy-list --help


27   © Novell, Inc. All rights reserved.
Embedded Sybase



                          zman dgc
       ZENworks 10 Configuration Management Command Line Utilities Reference
                          ®




28   © Novell, Inc. All rights reserved.
Embedded Sybase
     (cont.)

                Maintenance – everyone needs this
          –   Backups
               >   Zone
                   “Backing Up and Restoring the ZENworks Server and Certificate Authority.”
                                                                 ®


                   http://www.novell.com/documentation/zcm10/zcm10_system_admin/data/bb2j7h6.html
               >   Database
                   “Backing Up the Embedded Sybase SQL Anywhere Database”
                   http://www.novell.com/documentation/zcm10/zcm10_system_admin/data/ban0rcu.html

          –   Size
               >   “System Planning, Deployment, and Best Practices Guide”
                   http://www.novell.com/documentation/zcm10/zcm10_deployment_bp/data/index.html




29   © Novell, Inc. All rights reserved.
External Databases

     •   Befriend your DBA
          –   Tools to backup external databases are provided by the
              database company, not by ZENworks   ®




30   © Novell, Inc. All rights reserved.
System Update
System Update Overview

     •   System Update is a mechanism to download ZENworks        ®



         Configuration Management (ZCM) updates from a Novell         ®



         Customer Center (NCC) Server.
     •   The system update architecture is designed to integrate with
         the ZCM content, assignment and agent processes.
     •   The basic flow of an update is outlined in three major
         sections:
          –   Check for update
          –   Download/Import
          –   Apply Downloaded Update.



32   © Novell, Inc. All rights reserved.
Check for Update

     •   System Updates are available to any Primary server but can be set via
         ZCC under Configuration>System Update Settings. This event contacts
         the NCC server and downloads product patches and installations.

                    https://you.novell.com/zenworks/systemupdate-version2/external/zcm/10.1/




33   © Novell, Inc. All rights reserved.
Check for Update
     (cont.)




     update.xml




34   © Novell, Inc. All rights reserved.
Download/Import

     •   The next important process occurs when you initiate a download
         of an update or import of an update. The download can be done
         from ZCC or by using the “zman sudu” command equivalent. The
         process for importing from a manual .zip file can only be done
         from “zman sui <uid>” command.
          –   System update module checks to see if there is sufficient space
              to download the system update.
          –   System update reads the commands and content information
              from the update.xml file and populates the respective tables in
              the database and imports the data into the content repository
              system.
          –   The update is marked with the status “DOWNLOADED” in the
              database. The status change is reflected in ZCC and the
              update is now ready to deploy.

35   © Novell, Inc. All rights reserved.
Apply Update

     •   After downloading an update you must authorize the
         update prior to it being deployed to your devices. This
         happens automatically during deployment in ZCC or
         you can use the “zman suaz” command.

          –   Update is scheduled
          –   Preglobal actions are run against zone by a primary server
          –   Agent takes over and installs update to device.




36   © Novell, Inc. All rights reserved.
Apply Update
     (cont.)

     •   Logging:
          –   When the systemupdate module takes over on the device a log is created under
              logssystem-updateGUID called system-update.log and tracks the remainder of
              the update.
          –   The zmd-messages.log is also very valuable when set to debug and above for
              troubleshooting update issues. These two logs are almost always needed for
              Novell Technical Services to troubleshoot a failed System Update.
          –   An additional file is created under /work/system-update/in-process and is
              populated with the GUID of the system update being apply. This is a status file
              and will move the GUID to the update-completed file after the update is
              completed. The process of moving the GUID triggers an update event to be sent
              to the server stating the update completion.
          –   As the update progresses, individual .msi logs are captured during the msi install
              and are stored in the /logs/system-update/guid folder and are also helpful when
              troubleshooting a failure related to a particular .msi installation.



37   © Novell, Inc. All rights reserved.
Unpublished Work of Novell, Inc. All Rights Reserved.
This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc.
Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope
of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified,
translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc.
Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.


General Disclaimer
This document is not to be construed as a promise by any participating company to develop, deliver, or market a
product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in
making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents
of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any
particular purpose. The development, release, and timing of features or functionality described for Novell products
remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to
make changes to its content, at any time, without obligation to notify any person or entity of such revisions or
changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc.
in the United States and other countries. All third-party trademarks are the property of their respective owners.

Introduction to Novell ZENworks Configuration Management Troubleshooting

  • 1.
    Introduction to NovellZENworks ® ® Configuration Management Troubleshooting Edward Carter Kurt Finlayson Escalation Engineer Escalation Engineer Novell Inc./edcarter@novell.com Novell Inc./kfinlayson@novell.com Shaun Pond Escalation Engineer Novell Inc./spond@novell.com
  • 2.
    Objectives Of ThisSession • Basic troubleshooting • Key logs • Tools • Strategies • Key Processes 2 © Novell, Inc. All rights reserved.
  • 3.
  • 4.
    Agent Troubleshooting • Services running on the agent • Communication • Agent cache • ZAC commands 4 © Novell, Inc. All rights reserved.
  • 5.
    Agent Services • Novell ZENworks Agent Service ® ® • Novell Identity store • Remote Management • Novell ZENworks PreAgent 5 © Novell, Inc. All rights reserved.
  • 6.
    Communication Ports 80, 443, 2645, 5950, 7628 Testing listening ports – VNC – Telnet telnet <IP address> <port number> Testing Quicktask port http://<Agent IPaddress>:7628/RURunning 6 © Novell, Inc. All rights reserved.
  • 7.
    Agent Cache • Cache location • Cache DBs • ZAC cache command 7 © Novell, Inc. All rights reserved.
  • 8.
    ZENworks Administration Command ® Line Interface (ZAC) • What is ZAC? • ZAC usage information • ZAC zeninfo • Most important files to examine. 8 © Novell, Inc. All rights reserved.
  • 9.
    ZAC Commands (1) • zac -- help (Lists the commands that can be used) • zac <option> --help (Describes what the command does) – zac bundle-list --help – zac register --help • zac policy-list (pl) • zac bundle-list (bl) • zac file-system-guid (fgs) 9 © Novell, Inc. All rights reserved.
  • 10.
    ZAC Commands (2) • zac logger (log) alternative to using the z-icon – zac logger level debug – zac logger resetlog • zac emp wakeup – make sure all messages are in the zmd- messages.log 10 © Novell, Inc. All rights reserved.
  • 11.
    ZAC zeninfo • zac zeninfo-collect – zac zeninfo-collect [<targetfile>] one-step log collector • Queries the runtime state of the ZenworksWindowsService for detailed information. – ZenworksWindowsService must be running for zac zeninfo to work. – Collects metadata about cached data on workstation. • Must extract zeninfo-date-time.zip for html links to work • Pre ZCM 10.2, this command was “zac info-collect” – zeninfocollect*.zip still available in installdownloadstools folder for use in case agent is not functional. 11 © Novell, Inc. All rights reserved.
  • 12.
    ZAC zeninfo (cont.) 12 © Novell, Inc. All rights reserved.
  • 13.
    Contents of zeninfo*.zip • zac zeninfo creates zeninfo-date-time.zip in the %TEMP% folder by default – Folders in the zeninfo-date-time.zip > AgentCacheInfo > ConfigInfo > DebugLogs > MDStatus > ProductInfo > PropPageInfo > RefreshInfo > systemInfo 13 © Novell, Inc. All rights reserved.
  • 14.
    Contents of zeninfo*.zip (cont.) 14 © Novell, Inc. All rights reserved.
  • 15.
    configInfo • configInfoZENWORKS_HOMEconf – DeviceData hashed representation of the device password – DeviceGuid GUID should match the GUID in ZCC 15 © Novell, Inc. All rights reserved.
  • 16.
    debugLogs • Contains Windows Event View Logs • ZENWORKS_HOMElogsLocalStorezmd- messages.log • Additional logs collected when zac zeninfo is run on a server – loader-messages.log – services-messages.log – zenworks-installation*.log – zcc.log 16 © Novell, Inc. All rights reserved.
  • 17.
    Agent Logging Setup • (zmd-messages.log) = Contains most of the agent side logging information • (NWGINA.LOG) = Contains information related to GINA startup • (zenlgn.log) = Login process • (casaauthtoken.log) = CASA Client Log • (nalshell.txt) NAL Shell Log = The NAL Shell log contains information related to the ZENworks ® explorer shell extension. • (system-update.log) = logs a SU from the time the systemupdatemodule takes over. 17 © Novell, Inc. All rights reserved.
  • 18.
    Enable Agent Logging • Screenshots for zicon • Casa and remote • Log locations • Xml, config, registry 18 © Novell, Inc. All rights reserved.
  • 19.
  • 20.
    Server Troubleshooting • Services running on the server • Communication • Logs 20 © Novell, Inc. All rights reserved.
  • 21.
    Server Services Novell ZENworks loader ® ® Novell ZENworks preboot policy service Novell ZENworks preboot service Novell ZENworks remote management Novell ZENworks server Novell ZENworks services monitor Novell ZENworks tftp service Novell ZENworks proxy DHCP service Novell ZENworks embedded datastore CasaAuthTokenSvc 21 © Novell, Inc. All rights reserved.
  • 22.
    Communication • Database ports – Sybase 2638 – Oracle 1521 – SQL 1543 • LDAP – 389 and 636 • SOAP – 80 and 443 • PDHCP – 67 and 4011 • TFTP – 69 • Preboot policy – 13331 • Preboot server – 998 22 © Novell, Inc. All rights reserved.
  • 23.
    Logs see TID 3418069 for information on how to obtain these logs. • loader-messages.log – Log from ZENworks loader ® • serviceservices-messages.log • CASA logs (ats.log ats.trace) • zcc.log • catalina.out • Preboot stuff • System-update logs • Install log • Windows event logs 23 © Novell, Inc. All rights reserved.
  • 24.
    ZEN and TheArt of Database Management
  • 25.
    Database Types • Embedded Sybase • External Sybase • MS SQL 2005 • Oracle 10g 25 © Novell, Inc. All rights reserved.
  • 26.
    Database Types (cont.) Think before you install Choose the right database Database Suitable for Sybase (embedded) As many as 1,000 Sybase (external) As many as 5,000 Microsoft SQL Server 2005 5,000 to 40,000 Oracle 10g 5,000 to 40,000 http://www.novell.com/documentation/zcm10/zcm10_deployment_bp/data/index.html 26 © Novell, Inc. All rights reserved.
  • 27.
    zman • Command-line interface for ZCM • Runs on any Primary server • Can be used to script operations • zman – Lists all commands • zman <category> /? – Gives specific commands within the category, e.g., zman policy /? • zman <command> --help – Command-level usage and examples, e.g., zman policy-list --help 27 © Novell, Inc. All rights reserved.
  • 28.
    Embedded Sybase zman dgc ZENworks 10 Configuration Management Command Line Utilities Reference ® 28 © Novell, Inc. All rights reserved.
  • 29.
    Embedded Sybase (cont.) Maintenance – everyone needs this – Backups > Zone “Backing Up and Restoring the ZENworks Server and Certificate Authority.” ® http://www.novell.com/documentation/zcm10/zcm10_system_admin/data/bb2j7h6.html > Database “Backing Up the Embedded Sybase SQL Anywhere Database” http://www.novell.com/documentation/zcm10/zcm10_system_admin/data/ban0rcu.html – Size > “System Planning, Deployment, and Best Practices Guide” http://www.novell.com/documentation/zcm10/zcm10_deployment_bp/data/index.html 29 © Novell, Inc. All rights reserved.
  • 30.
    External Databases • Befriend your DBA – Tools to backup external databases are provided by the database company, not by ZENworks ® 30 © Novell, Inc. All rights reserved.
  • 31.
  • 32.
    System Update Overview • System Update is a mechanism to download ZENworks ® Configuration Management (ZCM) updates from a Novell ® Customer Center (NCC) Server. • The system update architecture is designed to integrate with the ZCM content, assignment and agent processes. • The basic flow of an update is outlined in three major sections: – Check for update – Download/Import – Apply Downloaded Update. 32 © Novell, Inc. All rights reserved.
  • 33.
    Check for Update • System Updates are available to any Primary server but can be set via ZCC under Configuration>System Update Settings. This event contacts the NCC server and downloads product patches and installations. https://you.novell.com/zenworks/systemupdate-version2/external/zcm/10.1/ 33 © Novell, Inc. All rights reserved.
  • 34.
    Check for Update (cont.) update.xml 34 © Novell, Inc. All rights reserved.
  • 35.
    Download/Import • The next important process occurs when you initiate a download of an update or import of an update. The download can be done from ZCC or by using the “zman sudu” command equivalent. The process for importing from a manual .zip file can only be done from “zman sui <uid>” command. – System update module checks to see if there is sufficient space to download the system update. – System update reads the commands and content information from the update.xml file and populates the respective tables in the database and imports the data into the content repository system. – The update is marked with the status “DOWNLOADED” in the database. The status change is reflected in ZCC and the update is now ready to deploy. 35 © Novell, Inc. All rights reserved.
  • 36.
    Apply Update • After downloading an update you must authorize the update prior to it being deployed to your devices. This happens automatically during deployment in ZCC or you can use the “zman suaz” command. – Update is scheduled – Preglobal actions are run against zone by a primary server – Agent takes over and installs update to device. 36 © Novell, Inc. All rights reserved.
  • 37.
    Apply Update (cont.) • Logging: – When the systemupdate module takes over on the device a log is created under logssystem-updateGUID called system-update.log and tracks the remainder of the update. – The zmd-messages.log is also very valuable when set to debug and above for troubleshooting update issues. These two logs are almost always needed for Novell Technical Services to troubleshoot a failed System Update. – An additional file is created under /work/system-update/in-process and is populated with the GUID of the system update being apply. This is a status file and will move the GUID to the update-completed file after the update is completed. The process of moving the GUID triggers an update event to be sent to the server stating the update completion. – As the update progresses, individual .msi logs are captured during the msi install and are stored in the /logs/system-update/guid folder and are also helpful when troubleshooting a failure related to a particular .msi installation. 37 © Novell, Inc. All rights reserved.
  • 39.
    Unpublished Work ofNovell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.