Implementing tivoli data warehouse v 1.2 sg247100
Upcoming SlideShare
Loading in...5
×
 

Implementing tivoli data warehouse v 1.2 sg247100

on

  • 3,076 views

 

Statistics

Views

Total Views
3,076
Views on SlideShare
3,076
Embed Views
0

Actions

Likes
0
Downloads
13
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Implementing tivoli data warehouse v 1.2 sg247100 Implementing tivoli data warehouse v 1.2 sg247100 Document Transcript

    • Front coverImplementing TivoliData Warehouse 1.2A primer for deployments of any sizeand proof of conceptsLatest Version 1.2 featuresincluding Crystal EnterpriseWarehouse enablementpack case studies Edson Manoel Cristiano Colantuono Hans-Georg Köhne Devi Raju Ghufran Shah Sergio Henrique Soares Monteiroibm.com/redbooks
    • International Technical Support OrganizationImplementing Tivoli Data Warehouse 1.2June 2004 SG24-7100-00
    • Note: Before using this information and the product it supports, read the information in “Notices” on page xix.First Edition (June 2004)This edition applies to Version 1.2 of the Tivoli Data Warehouse product.© Copyright International Business Machines Corporation 2004. All rights reserved.Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.
    • Contents Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi The team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiiPart 1. Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1. Introducing Tivoli Data Warehouse 1.2. . . . . . . . . . . . . . . . . . . . 3 1.1 Data warehousing basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.1 Data warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1.2 Data mart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.3 Business intelligence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.1.4 Data mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 Tivoli Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 What is new in Tivoli Data Warehouse 1.2 . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3.1 Crystal Enterprise™ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3.2 IBM DB2 UDB for OS/390 and z/OS support . . . . . . . . . . . . . . . . . . 13 1.3.3 Flexible and extended configuration support . . . . . . . . . . . . . . . . . . 17 1.3.4 Installation enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.3.5 Serviceability and scalability improvements . . . . . . . . . . . . . . . . . . . 19 1.4 Tivoli Data Warehouse architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.4.1 Tivoli Data Warehouse control center server . . . . . . . . . . . . . . . . . . 21 1.4.2 Source databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.4.3 Central data warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.4.4 Data marts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.4.5 Warehouse agents and agent sites. . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.6 Crystal Enterprise Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.5 Benefits of using Tivoli Data Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . 23 Chapter 2. Planning for Tivoli Data Warehouse 1.2 . . . . . . . . . . . . . . . . . . 27© Copyright IBM Corp. 2004. All rights reserved. iii
    • 2.1 Hardware and software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.1.1 Hardware requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.1.2 Software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.1.3 Database requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.4 Crystal Enterprise requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 Physical and logical design considerations . . . . . . . . . . . . . . . . . . . . . . . . 36 2.2.1 Source databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.2.2 Control server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.2.3 Central data warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.2.4 Data marts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.2.5 Single machine installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 2.2.6 Distributed deployment on UNIX and Windows servers . . . . . . . . . . 43 2.2.7 Distributed deployment on z/OS, UNIX, and Windows servers . . . . 45 2.2.8 Warehouse agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 2.2.9 Considerations about warehouse databases on z/OS . . . . . . . . . . . 54 2.2.10 Coexistence with other products . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.2.11 Selecting port numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.3 Database sizing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.4 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 2.4.1 Authority required to install and maintain IBM DB2 UDB . . . . . . . . . 57 2.4.2 Authority required to install Tivoli Data Warehouse . . . . . . . . . . . . . 57 2.4.3 Firewalls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.4.4 Controlling access to data in the warehouse . . . . . . . . . . . . . . . . . . 59 2.4.5 Protecting information in Crystal Enterprise Professional for Tivoli . 59 2.4.6 Multicustomer and multicenter support . . . . . . . . . . . . . . . . . . . . . . . 60 2.5 Network traffic considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 2.5.1 Architectural choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 2.5.2 Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 2.6 Integration with other business intelligence tools . . . . . . . . . . . . . . . . . . . 64 2.7 ETL development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 2.8 Skills required for a Tivoli Data Warehouse project . . . . . . . . . . . . . . . . . 67 2.8.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 2.8.2 Data collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 2.8.3 Data manipulation (ETL1 and ETL2). . . . . . . . . . . . . . . . . . . . . . . . . 68 2.8.4 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running. . . . . . . . . 71 3.1 Preparing for the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.1.1 Ensuring fully qualified host names. . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.1.2 Installing and configuring IBM DB2 client and server . . . . . . . . . . . . 76 3.1.3 Crystal Enterprise installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.2 Tivoli Data Warehouse 1.2 installation . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.3 Quick start deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93iv Implementing Tivoli Data Warehouse 1.2
    • 3.3.1 Quick start deployment: installation and configuration . . . . . . . . . . . 94 3.3.2 Configuring the control database . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 3.3.3 Creating ODBC connections to the data mart databases . . . . . . . . 101 3.4 Distributed deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.4.1 Distributed deployment installation: Windows and UNIX . . . . . . . . 104 3.4.2 Distributed deployment installation: z/OS . . . . . . . . . . . . . . . . . . . . 115 3.4.3 Creating ODBC connections to the data mart databases . . . . . . . . 123 3.5 Installing warehouse agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.5.1 Installing IBM DB2 Warehouse Manager . . . . . . . . . . . . . . . . . . . . 128 3.5.2 Creating the remote agent sites . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 3.6 Verification of the installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 3.6.1 Verifying the remote agent install . . . . . . . . . . . . . . . . . . . . . . . . . . 141 3.7 Installing warehouse enablement packs . . . . . . . . . . . . . . . . . . . . . . . . . 142 Chapter 4. Performance maximization techniques . . . . . . . . . . . . . . . . . 145 4.1 DB2 performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4.2 Operating system performance tuning . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.2.1 Windows environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.2.2 Primary Windows performance factors . . . . . . . . . . . . . . . . . . . . . . 151 4.2.3 AIX environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.3 Tivoli Data Warehouse performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 155Part 2. Case study scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack . . . . . . . . . 161 5.1 Case study overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 5.2 IBM Tivoli NetView WEP overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.3 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.3.1 Verifying prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.3.2 Gathering installation information . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.4 Preparing NetView for data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5.4.1 Enabling NetView to export data for Tivoli Data Warehouse . . . . . 167 5.4.2 NetView SmartSets configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.4.3 Configuring NetView Data Warehouse daemon (tdwdaemon) . . . . 176 5.4.4 Verifying NetView data collection enablement . . . . . . . . . . . . . . . . 178 5.5 Installation of the NetView WEPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 5.5.1 Backing up the TDW environment . . . . . . . . . . . . . . . . . . . . . . . . . 181 5.5.2 Establishing ODBC connections . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 5.5.3 Installing NetView Enablement Pack Software . . . . . . . . . . . . . . . . 185 5.5.4 Defining the authority to the warehouse sources and targets . . . . . 188 5.6 Testing, scheduling, and promoting the ETLs . . . . . . . . . . . . . . . . . . . . . 191 5.6.1 Promoting the ETLs to TEST mode . . . . . . . . . . . . . . . . . . . . . . . . 192 5.6.2 Testing the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 5.6.3 Scheduling the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 Contents v
    • 5.6.4 Promoting the ETLs to Production status . . . . . . . . . . . . . . . . . . . . 197 5.7 Running NetView ETLs on remote agent sites . . . . . . . . . . . . . . . . . . . . 198 5.8 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 5.8.1 Accessing the Crystal ePortfolio feature . . . . . . . . . . . . . . . . . . . . . 206 Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack. . . . . . . 225 6.1 Case study overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 6.2 IBM Tivoli Monitoring WEP overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6.3 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 6.4 Installing the ITM WEP data collector component. . . . . . . . . . . . . . . . . . 232 6.4.1 Activate data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 6.5 Installing and configuring ITM Generic WEP. . . . . . . . . . . . . . . . . . . . . . 241 6.5.1 Backing up the TWH databases . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 6.5.2 Establishing an ODBC connection on the Control Center. . . . . . . . 242 6.5.3 Installing the ITM 5.1.1 AMX ETL processes . . . . . . . . . . . . . . . . . 247 6.5.4 Installing AMX Fix Packs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 6.5.5 Defining the authority to the warehouse sources and targets . . . . . 254 6.5.6 Modifying the ETL for the source table name to the RIM user . . . . 257 6.6 Installing and configuring ITM for OS WEP . . . . . . . . . . . . . . . . . . . . . . . 262 6.6.1 Backing up the TWH databases . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 6.6.2 Installing the ITM 5.1.1 AMY ETL processes . . . . . . . . . . . . . . . . . 262 6.6.3 Installing AMY Fix Packs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 6.6.4 Defining the authority to the warehouse sources and targets . . . . . 265 6.7 Testing, scheduling, and promoting the ETLs . . . . . . . . . . . . . . . . . . . . . 267 6.7.1 Testing the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 6.7.2 Checking that data has been collected . . . . . . . . . . . . . . . . . . . . . . 270 6.7.3 Scheduling the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 6.7.4 Promoting the ETL status to Production mode . . . . . . . . . . . . . . . . 274 6.8 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 6.8.1 Available reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 6.8.2 Accessing the Crystal ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 6.9 Troubleshooting of ITM data collection . . . . . . . . . . . . . . . . . . . . . . . . . . 286 6.9.1 Using itmchk.sh script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 6.9.2 Manual checking of ITM data collection . . . . . . . . . . . . . . . . . . . . . 290 Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack . 297 7.1 Case study overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 7.2 IBM Tivoli Storage Manager WEP overview . . . . . . . . . . . . . . . . . . . . . . 299 7.3 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 7.4 Installing and configuring ITSM WEP 5.2 . . . . . . . . . . . . . . . . . . . . . . . . 301 7.4.1 Changes required on the IBM Tivoli Storage Manager servers . . . 301 7.4.2 Installing the IBM Tivoli Storage Manager ODBC . . . . . . . . . . . . . . 301 7.4.3 Backing up the TWH databases . . . . . . . . . . . . . . . . . . . . . . . . . . . 304vi Implementing Tivoli Data Warehouse 1.2
    • 7.4.4 IBM Tivoli Storage Manager WEP installation . . . . . . . . . . . . . . . . 305 7.4.5 Defining the authority to the warehouse sources and targets . . . . . 313 7.5 IBM Tivoli Storage Manager ETL processes . . . . . . . . . . . . . . . . . . . . . . 314 7.5.1 ANR_C05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 7.5.2 ANR_C10_EXPServer_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 7.5.3 ANR_M05_ETL2_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 7.6 Testing, scheduling, and promoting the ETLs . . . . . . . . . . . . . . . . . . . . . 320 7.6.1 ETL data collection verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 7.7 Reporting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 7.7.1 Available reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 7.7.2 Accessing the Crystal ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . 322Part 3. Appendixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 Appendix A. IBM DB2 UDB administration for other relational DBAs . . 339 Common DBA tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Creating databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Creating databases in IBM DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 Creating databases in Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 Creating databases in Sybase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 Managing space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 DB2 space management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 Oracle space management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 Sybase space management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Creating objects in the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Creating tables in DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Creating tables in Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Creating tables in Sybase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Additional table control parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Appendix B. Tivoli Data Warehouse 1.2 reference . . . . . . . . . . . . . . . . . . 349 Report listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 Measurement sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Appendix C. Warehouse Enablement Packs properties file . . . . . . . . . . 361 The twh_install_props.cfg properties file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 Contents vii
    • Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369viii Implementing Tivoli Data Warehouse 1.2
    • Figures 1-1 IBM DB2 Data Warehouse Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1-2 Crystal Enterprise multi-tier architecture . . . . . . . . . . . . . . . . . . . . . . . . 12 1-3 TDS OS/390 and TDW 1.2 Data flow . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1-4 Distributed and OS/390 Data feeds into Tivoli Data Warehouse . . . . . . 16 1-5 Multiple source applications loading into a central data warehouse . . . 17 1-6 Tivoli Data Warehouse — the big picture . . . . . . . . . . . . . . . . . . . . . . . 20 1-7 Detail Component view of Tivoli Data Warehouse 1.2. . . . . . . . . . . . . . 23 1-8 Integrated Systems Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2-1 Single machine installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2-2 Distributed deployment on Windows and UNIX systems . . . . . . . . . . . . 44 2-3 Operational data sources and the CDW databases on the same server 45 2-4 Data sources, CDW, and data mart databases on a z/OS system . . . . 46 2-5 Operational data sources both on z/OS and on distributed systems . . . 47 2-6 Separate data mart databases on z/OS system and distributed system 48 2-7 Two CDWs on a Windows or UNIX system and on a z/OS system. . . . 49 2-8 Warehouse agent on control server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2-9 Warehouse agents on data targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 2-10 Configuration with a warehouse agent on the source . . . . . . . . . . . . . . 52 2-11 Tivoli Data Warehouse and firewalls . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2-12 Business intelligence integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3-1 Installation process overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3-2 Install DB2 V7 components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 3-3 Create DB2 Services - DB2 Instance db2inst1 . . . . . . . . . . . . . . . . . . . 79 3-4 Create the DB2 fenced user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 3-5 Administration Server window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 3-6 Select DB2 Enterprise Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 3-7 Installation Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 3-8 MSDE security configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3-9 Installation window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 3-10 Completion window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 3-11 Crystal Enterprise Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3-12 Crystal Administration Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3-13 Quick start deployment configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3-14 InstallShield Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3-15 Tivoli common logging directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3-16 Setup window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 3-17 DB2 connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3-18 Crystal connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98© Copyright IBM Corp. 2004. All rights reserved. ix
    • 3-19 Summary window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3-20 Completion window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 3-21 Configuring the IBM DB2 data warehouse center . . . . . . . . . . . . . . . . 100 3-22 Configuring the Warehouse Control Database Management . . . . . . . 101 3-23 Distributed deployment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 3-24 Install Shield Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3-25 Tivoli Common Logging Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3-26 Setup Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 3-27 Before proceeding with TDW 1.2 distributed installation . . . . . . . . . . . 108 3-28 DB2 connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 3-29 Central data warehouse on remote host . . . . . . . . . . . . . . . . . . . . . . . 109 3-30 Central data warehouse database server list. . . . . . . . . . . . . . . . . . . . 110 3-31 Data mart on remote host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3-32 Data mart database server list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 3-33 Crystal connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 3-34 Summary window. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 3-35 Completion window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 3-36 Configuring the IBM DB2 Data Warehouse Center . . . . . . . . . . . . . . . 114 3-37 Configuring the Warehouse Control Database Management . . . . . . . 115 3-38 Adding central data warehouses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 3-39 z/OS IBM DB2 Server information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 3-40 z/OS central data warehouse database configuration . . . . . . . . . . . . . 117 3-41 Central data warehouse server on z/OS . . . . . . . . . . . . . . . . . . . . . . . 118 3-42 Central data warehouse summary window . . . . . . . . . . . . . . . . . . . . . 118 3-43 Central data warehouse on z/OS install. . . . . . . . . . . . . . . . . . . . . . . . 119 3-44 Adding data marts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 3-45 z/OS IBM DB2 Server information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 3-46 z/OS data mart database configuration . . . . . . . . . . . . . . . . . . . . . . . . 121 3-47 Data mart server on z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 3-48 Data mart creation summary window. . . . . . . . . . . . . . . . . . . . . . . . . . 122 3-49 Data mart on z/OS install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 3-50 Distributed environment with agent . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 3-51 Select the DB2 Warehouse Manager components . . . . . . . . . . . . . . . 129 3-52 Install DB2 V7 menu on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 3-53 Create DB2 Service Menu on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 3-54 Setup Window - create warehouse agents . . . . . . . . . . . . . . . . . . . . . 132 3-55 Before proceeding with remote agent sites creation . . . . . . . . . . . . . . 133 3-56 Warehouse agents - specify the TDW control server . . . . . . . . . . . . . 134 3-57 Successful remote agent creation window. . . . . . . . . . . . . . . . . . . . . . 135 3-58 DB2 Data Warehouse services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 3-59 Remote Agent Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 3-60 Verify Remote Agents on Tivoli Data Warehouse Control Center . . . . 141 5-1 Distributed deployment scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163x Implementing Tivoli Data Warehouse 1.2
    • 5-2 IBM Tivoli NetView Warehouse Enablement Pack data flow . . . . . . . . 1645-3 NetView Configure data export to DB2 - Parameters . . . . . . . . . . . . . 1685-4 NetView Configure data export to DB2 - create database . . . . . . . . . . 1685-5 NetView Configure data export to DB2 - register and start tdwdaemon1695-6 NetView SmartSet desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1715-7 Microsoft SmartSet Advanced attributes . . . . . . . . . . . . . . . . . . . . . . . 1725-8 Create Microsoft SmartSet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1735-9 NetView SmartSets - Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1745-10 NetView SmartSets - Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1755-11 SmartSet Microsoft contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1765-12 Create an ODBC data source for NETVIEW . . . . . . . . . . . . . . . . . . . . 1835-13 Add an ODBC data source for NETVIEW . . . . . . . . . . . . . . . . . . . . . . 1835-14 Configure NetView Source database connectivity . . . . . . . . . . . . . . . . 1845-15 NetView WEP installation - List of WEPs to install . . . . . . . . . . . . . . . 1855-16 NetView WEP installation - Properties file . . . . . . . . . . . . . . . . . . . . . . 1865-17 NetView WEP installation - List of WEPs to install NetView . . . . . . . . 1875-18 NetView WEP installation - successful installation . . . . . . . . . . . . . . . 1875-19 Data Warehouse Control Center - check control database . . . . . . . . . 1895-20 Configure NetView data warehouse sources . . . . . . . . . . . . . . . . . . . . 1905-21 Configure NetView data warehouse targets . . . . . . . . . . . . . . . . . . . . 1915-22 Promote ETLs to test mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1925-23 Test ETL process steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1935-24 Work in Progress - Log file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1945-25 Sample contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1955-26 Schedule ANM_c05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . 1965-27 Schedule configuration for ANM_C05_ETL1_Process . . . . . . . . . . . . 1975-28 Promote ANM_c05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . . 1985-29 Select remote agents properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2005-30 Change remote agents properties - sources and targets. . . . . . . . . . . 2015-31 Select ETL process properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2025-32 Demote ETL processes to development mode . . . . . . . . . . . . . . . . . . 2025-33 Change the ETL processes agent site . . . . . . . . . . . . . . . . . . . . . . . . . 2035-34 Work in Progress - Run ETL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2045-35 Work in progress - Check ETL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2055-36 Log Details menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2055-37 Crystal Enterprise - Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2075-38 Crystal Enterprise 9 - ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2085-39 Crystal Enterprise 9 - Log in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2095-40 Crystal Enterprise 9 - Folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2105-41 Crystal Enterprise 9 - Tivoli Reports: IBM Tivoli NetView . . . . . . . . . . 2115-42 Crystal Enterprise 9 - Daily Status Summary by SmartSet . . . . . . . . . 2125-43 Crystal Enterprise 9 - Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2125-44 Crystal Enterprise 9 - Parameters for Schedule Option . . . . . . . . . . . . 213 Figures xi
    • 5-45 Crystal Enterprise 9 - Schedule Parameters . . . . . . . . . . . . . . . . . . . . 214 5-46 Crystal Enterprise 9 - Schedule Parameter Selection . . . . . . . . . . . . . 215 5-47 Crystal Enterprise 9 - Parameters: Specific Time Frame. . . . . . . . . . . 216 5-48 Crystal Enterprise 9 - Report History . . . . . . . . . . . . . . . . . . . . . . . . . . 217 5-49 Failed report generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 5-50 Crystal Enterprise 9 - Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 5-51 Crystal Enterprise 9 - Report (count) . . . . . . . . . . . . . . . . . . . . . . . . . . 220 5-52 Summary of total status changes by SmartSet example . . . . . . . . . . . 221 5-53 Nodes with longest outage times example . . . . . . . . . . . . . . . . . . . . . 222 5-54 Total daily status changes in monitored network example . . . . . . . . . 223 6-1 Environment for our case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6-2 Overview of ITM integration with Tivoli Data Warehouse . . . . . . . . . . 228 6-3 IBM Tivoli Monitoring data flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 6-4 Resource Model Data Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 6-5 Aggregation time line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 6-6 Installing warehouse support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 6-7 RIM setup options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 6-8 Logging option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6-9 Client Configuration Assistant opening dialog . . . . . . . . . . . . . . . . . . . 243 6-10 Add Database Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 6-11 Add System dialog window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 6-12 Select ITM_DB in the dialog window . . . . . . . . . . . . . . . . . . . . . . . . . . 246 6-13 Confirmation dialog window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 6-14 User ID and Password dialog window . . . . . . . . . . . . . . . . . . . . . . . . . 247 6-15 ODBC connection successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 6-16 Install a Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 6-17 Tivoli Common Logging Directory window . . . . . . . . . . . . . . . . . . . . . . 249 6-18 Add Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 6-19 Location of installation properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 6-20 Installation menu window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 6-21 Installation summary window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 6-22 AMX installation completion window. . . . . . . . . . . . . . . . . . . . . . . . . . . 253 6-23 Installation of AMX Fix Pack 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 6-24 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Sources . . . . . . . . 255 6-25 AMX_ITM_RIM_Source user ID information . . . . . . . . . . . . . . . . . . . . 255 6-26 AMX_TWH_CDW_Source user ID information . . . . . . . . . . . . . . . . . . 256 6-27 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Target . . . . . . . . . 256 6-28 AMX_TWH_CDW_Target user ID information. . . . . . . . . . . . . . . . . . . 257 6-29 Tables and views of AMX_ITM_TIM_Source. . . . . . . . . . . . . . . . . . . . 258 6-30 Table name filter specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 6-31 Endpoint tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 6-32 AMX_c05_ETL1 process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 6-33 Selecting new table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261xii Implementing Tivoli Data Warehouse 1.2
    • 6-34 Installation menu window with the AMY pack . . . . . . . . . . . . . . . . . . . 2636-35 AMY installation completion window . . . . . . . . . . . . . . . . . . . . . . . . . . 2646-36 Installation of AMY Fix Pack 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2656-37 AMY_TWH_CDW_Source user ID information . . . . . . . . . . . . . . . . . . 2666-38 AMY_TWH_MART_Target user ID information . . . . . . . . . . . . . . . . . . 2676-39 Change ETL mode to Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2686-40 Manually test the ETLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2696-41 Work in progress window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2706-42 Sample Content of table F_OS_HOUR . . . . . . . . . . . . . . . . . . . . . . . . 2716-43 Schedule AMX_c05_ETL1_Process . . . . . . . . . . . . . . . . . . . . . . . . . . 2726-44 Schedule configuration for AMX_c05_ETL1_Process . . . . . . . . . . . . . 2736-45 Promoting ETLs to Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2746-46 Crystal Enterprise - Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2766-47 Crystal Enterprise 9 - ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2776-48 Crystal Enterprise 9 - Log in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2786-49 Crystal Enterprise 9 - Folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2796-50 Crystal Enterprise 9 - available reports for ITM . . . . . . . . . . . . . . . . . . 2806-51 Scheduling Operating System Busiest System report . . . . . . . . . . . . . 2816-52 Crystal Enterprise 9 - parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2826-53 Crystal Enterprise 9 - Parameters for the report . . . . . . . . . . . . . . . . . 2826-54 Crystal Enterprise 9 - Report History . . . . . . . . . . . . . . . . . . . . . . . . . . 2836-55 Operating System Busiest Systems report . . . . . . . . . . . . . . . . . . . . . 2846-56 Operating System Paging File Utilization. . . . . . . . . . . . . . . . . . . . . . . 2856-57 Operating System Operating System UNIX CPU Statistics . . . . . . . . . 2867-1 TDW 1.2 - distributed deployment scenario . . . . . . . . . . . . . . . . . . . . . 2997-2 ITSM ODBC Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3027-3 ITSM ODBC data source configuration panel . . . . . . . . . . . . . . . . . . . 3037-4 Install a Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3057-5 Tivoli Common Logging Directory window . . . . . . . . . . . . . . . . . . . . . . 3067-6 Add Warehouse Pack window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3067-7 Location of installation properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3077-8 Data mart and remote agent site settings . . . . . . . . . . . . . . . . . . . . . . 3087-9 Central data warehouse and remote agent site settings . . . . . . . . . . . 3097-10 Editing IBM Tivoli Storage Manager ODBC settings . . . . . . . . . . . . . . 3107-11 ITSM ODBC Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3117-12 Installation menu window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3117-13 Installation summary window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3127-14 Installation Progress and Completion window . . . . . . . . . . . . . . . . . . . 3137-15 Sample of Process Model ANR_C05_ETL1_Process . . . . . . . . . . . . . 3167-16 Sample Content of Table D_NODE . . . . . . . . . . . . . . . . . . . . . . . . . . . 3217-17 Crystal Enterprise - Launchpad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3237-18 Crystal Enterprise 9 - ePortfolio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3247-19 Crystal Enterprise 9 - Log in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Figures xiii
    • 7-20 Crystal Enterprise 9 - Folders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 7-21 Crystal Enterprise 9 - available reports for ITSM . . . . . . . . . . . . . . . . . 327 7-22 Scheduling Operating System Busiest System report . . . . . . . . . . . . . 328 7-23 Crystal Enterprise 9 - parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 7-24 Crystal Enterprise 9 - Parameters for the report . . . . . . . . . . . . . . . . . 330 7-25 How Has Clients use of Server Storage Changed Over Time? . . . . . . 331 7-26 How Has Clients Use of Server Storage Changed Over Time? . . . . . 332 7-27 How Has Clients Use of Server Storage Changed by Platform? . . . . . 333 7-28 How Has My Server Storage Space Utilization Changed Over Time? 334 7-29 Which Clients are Using the Most Server Storage?. . . . . . . . . . . . . . . 335 C-1 Location of the twh_install_props.cfg file . . . . . . . . . . . . . . . . . . . . . . . 362xiv Implementing Tivoli Data Warehouse 1.2
    • Tables 2-1 Hardware recommendations for Tivoli Data Warehouse components . 29 2-2 Additional hard disk space requirements . . . . . . . . . . . . . . . . . . . . . . . . 30 2-3 Software requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2-4 Web servers and OS supported by Crystal Web Connector . . . . . . . . . 35 2-5 Requirements for Tivoli Data Warehouse components . . . . . . . . . . . . . 36 2-6 Agent sites placement for data transfers to a central data warehouse . 52 2-7 Where to place agent sites for data transfers to data marts . . . . . . . . . 53 2-8 Default port used in Tivoli Data Warehouse environments . . . . . . . . . . 56 5-1 Environment for NetView integration . . . . . . . . . . . . . . . . . . . . . . . . . . 162 5-2 NetView WEP Prerequisite Check - NetView server platform . . . . . . . 165 5-3 Netview Enablement Pack installation information . . . . . . . . . . . . . . . 166 5-4 Case Study SmartSets attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 5-5 Add database wizard - register TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . 184 5-6 NetView sources and targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 6-1 Hardware and operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 7-1 Environment for NetView integration . . . . . . . . . . . . . . . . . . . . . . . . . . 298 7-2 ITSM WEP Warehouse Object Names . . . . . . . . . . . . . . . . . . . . . . . . 314 C-1 WEP installation properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363© Copyright IBM Corp. 2004. All rights reserved. xv
    • xvi Implementing Tivoli Data Warehouse 1.2
    • Examples 3-1 twh_create_datasource script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3-2 Verification of central data warehouse database on z/OS . . . . . . . . . . 119 3-3 Verification of data mart database on z/OS . . . . . . . . . . . . . . . . . . . . . 123 3-4 twh_create_datasource script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 3-5 Verify control server (twh_list_cs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 3-6 Verify central data warehouse (twh_list_cdws) . . . . . . . . . . . . . . . . . . 137 3-7 Verify data mart databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 3-8 Verify remote agent site (twh_list_agentsites) . . . . . . . . . . . . . . . . . . . 138 3-9 Verify Crystal Enterprise Professional for Tivoli installation . . . . . . . . . 139 3-10 Verify data user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 3-11 twh_configwep command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 5-1 Verify NetView source database updates . . . . . . . . . . . . . . . . . . . . . . 169 5-2 NetView tdwdaemon configuration file tdwdaemon.properties . . . . . . 178 5-3 Restart the NetView data warehouse daemon tdwdaemon . . . . . . . . . 178 5-4 Status of NetView data warehouse daemon (tdwdaemon) . . . . . . . . . 179 5-5 Status of the NetView SNMP collector daemon (snmpcollect) . . . . . . 179 5-6 Check the NetView source database . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6-1 Testing the RIM object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 6-2 Datacollector configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 6-3 wdmlseng command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 6-4 wdmcollect command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 6-5 Sample SQL that check the collection . . . . . . . . . . . . . . . . . . . . . . . . . 240 6-6 Running itmchk.sh tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 6-7 itmchk.sh tool report. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 6-8 Retrieving the date of last data upload into ITM database. . . . . . . . . . 290 6-9 Names of the endpoints collecting data . . . . . . . . . . . . . . . . . . . . . . . . 290 6-10 wrimtest command output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 6-11 Status of resource models distributed on an endpoint . . . . . . . . . . . . . 292 6-12 msg_DataCollector.log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 6-13 trace_tmnt_rimh_eng1.log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 6-14 trace_dmxengine.log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295© Copyright IBM Corp. 2004. All rights reserved. xvii
    • xviii Implementing Tivoli Data Warehouse 1.2
    • NoticesThis information was developed for products and services offered in the U.S.A.IBM may not offer the products, services, or features discussed in this document in other countries. Consultyour local IBM representative for information on the products and services currently available in your area.Any reference to an IBM product, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product, program, or service thatdoes not infringe any IBM intellectual property right may be used instead. However, it is the usersresponsibility to evaluate and verify the operation of any non-IBM product, program, or service.IBM may have patents or pending patent applications covering subject matter described in this document.The furnishing of this document does not give you any license to these patents. You can send licenseinquiries, in writing, to:IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.The following paragraph does not apply to the United Kingdom or any other country where such provisionsare inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDESTHIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimerof express or implied warranties in certain transactions, therefore, this statement may not apply to you.This information could include technical inaccuracies or typographical errors. Changes are periodically madeto the information herein; these changes will be incorporated in new editions of the publication. IBM maymake improvements and/or changes in the product(s) and/or the program(s) described in this publication atany time without notice.Any references in this information to non-IBM Web sites are provided for convenience only and do not in anymanner serve as an endorsement of those Web sites. The materials at those Web sites are not part of thematerials for this IBM product and use of those Web sites is at your own risk.IBM may use or distribute any of the information you supply in any way it believes appropriate withoutincurring any obligation to you.Information concerning non-IBM products was obtained from the suppliers of those products, their publishedannouncements or other publicly available sources. IBM has not tested those products and cannot confirmthe accuracy of performance, compatibility or any other claims related to non-IBM products. Questions onthe capabilities of non-IBM products should be addressed to the suppliers of those products.This information contains examples of data and reports used in daily business operations. To illustrate themas completely as possible, the examples include the names of individuals, companies, brands, and products.All of these names are fictitious and any similarity to the names and addresses used by an actual businessenterprise is entirely coincidental.COPYRIGHT LICENSE:This information contains sample application programs in source language, which illustrates programmingtechniques on various operating platforms. You may copy, modify, and distribute these sample programs inany form without payment to IBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operating platform for which thesample programs are written. These examples have not been thoroughly tested under all conditions. IBM,therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy,modify, and distribute these sample programs in any form without payment to IBM for the purposes ofdeveloping, using, marketing, or distributing application programs conforming to IBMs applicationprogramming interfaces.© Copyright IBM Corp. 2004. All rights reserved. xix
    • TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States,other countries, or both: AIX® IBM® Redbooks™ CICS® IMS™ RMF™ DataJoiner® Informix® S/390® DB2 Universal Database™ Lotus® SP2® DB2® MQSeries® Tivoli Enterprise Console® Domino® MVS™ Tivoli Enterprise™ DRDA® NetView® Tivoli® ^™ NetVista™ WebSphere® ™ OS/390® z/OS® Everyplace® RACF® ibm.com® Redbooks (logo) ™The following terms are trademarks of other companies:Crystal and Crystal Enterprise are trademarks of Business Objects.Intel and Intel Inside (logos) are trademarks of Intel Corporation in the United States, other countries, orboth.Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in theUnited States, other countries, or both.Java and all Java-based trademarks and logos are trademarks or registered trademarks of SunMicrosystems, Inc. in the United States, other countries, or both.UNIX is a registered trademark of The Open Group in the United States and other countries.Other company, product, and service names may be trademarks or service marks of others.xx Implementing Tivoli Data Warehouse 1.2
    • Preface With Tivoli® Data Warehouse, you can analyze historical trends from various Tivoli and customer applications. The Tivoli Data Warehouse infrastructure enables a set of extract, transform, and load (ETL) utilities to extract and move data from Tivoli application data stores to a central repository. The open architecture of Tivoli Data Warehouse also enables data from non-Tivoli applications to be integrated into its central repository. Data from the central repository can be extracted into data marts that pertain to the reporting needs of selected groups. These data marts can also be used to produce cross application reports. This IBM Redbook focuses on planning, installation, customization, use, maintenance, and troubleshooting topics related to the new features of the Tivoli Data Warehouse version 1.2. This is done using a number of case study scenarios and several warehouse enablement packs. The instructions given in this book are very detailed and explicit. These instructions are not the only way to install the products and related prerequisites. They are meant to be followed by anyone to successfully install, configure, and set up Tivoli Data Warehouse environments of any size.The team that wrote this redbook This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Austin Center. Edson Manoel is a Software Engineer at IBM Corporation - International Technical Support Organization, Austin Center, working as an IT Specialist in the Systems Management area. Prior to joining the ITSO, Edson worked in the IBM Software Group as a Tivoli Technology Ambassador and in IBM Brasil Professional Services Organization as a Certified IT Specialist. He was involved in numerous projects, designing and implementing systems management solutions for IBM customers and Business Partners. Edson holds a BSc degree in Applied Mathematics from Universidade de Sao Paulo, Brazil. Cristiano Colantuono is an IT Specialist at IBM Tivoli Laboratory in Rome. He joined IBM in 1999, working in the distributed systems management area. He projected and implemented several Tivoli solutions for the IT infrastructures in Rome as well as in other European IBM development laboratories. Before joining IBM, Cristiano had also some experience as a Web developer and a Web© Copyright IBM Corp. 2004. All rights reserved. xxi
    • administrator. He graduated in Physics at the University of Rome and collaborated with the Italian National Institute of Nuclear Physics developing simulation programs for high energy physics experiments. Dr. Hans-Georg Köhne is a software architect for SerCon in Germany. He graduated in physics at the University of Muenster developing simulation programs for high energy physics experiments. He joined SerCon in 1996, working in the distributed systems management area. He planned and implemented several systems management solutions in the areas software distribution, availability management, and business automation. Devi Raju is a Tivoli Implementation Specialist for IBM India. She started her career with IBM and has been with IBM for 8 years now. Devi has 4 years of experience in Enterprise System Management. She has worked in various large Tivoli customer projects. She is also a Tivoli Certified Consultant on PACO products. Ghufran Shah is an IBM Certified Deployment Professional and an IBM Certified Instructor based in the UK with os-security.com. He holds a degree in Computer Science, and has over 8 years of experience in Systems Development and Enterprise Systems Management. As well as teaching Tivoli courses worldwide, his areas of expertise include Tivoli Systems Management Architecture, Implementation, and Training together with Provisioning and Orchestration. His focus in now on leveraging IBM solutions to provide customers with the vision and reality of an OnDemand environment. Sergio Henrique Soares Monteiro is an IT Specialist in Brazil. He has over 10 years of experience in database administration and development fields. He has worked with Oracle, DB2, Informix and SQL Server on UNIX and Windows, including clustered servers. He currently works as a Database administrator in the CTI’s IBM in Hortolandia, Brazil. His areas of expertise include sizing, performance tuning, and internals of RDBMS. Thanks to the following people for their contributions to this project: Budi Darmawan International Technical Support Organization, Austin Center David Stephenson IBM Global Services, Australia Diana Marcattili IBM Global Services, Italy Georg Holzknecht Senior Systems Consultant, T-Systems CDS GmbH, Germanyxxii Implementing Tivoli Data Warehouse 1.2
    • Jonathan Cook, Brian Jeffrey, Mike Mallo Tivoli Data Warehouse development team, IBM Software Group, Austin Ken Hannigan IBM Tivoli Storage Manager development team, IBM Software Group, Tucson Yvonne Lyon, editor International Technical Support Organization, San Jose CenterBecome a published author Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. Youll team with IBM technical professionals, Business Partners and/or customers. Your efforts will help increase product acceptance and customer satisfaction. As a bonus, youll develop a network of contacts in IBM development labs, and increase your productivity and marketability. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.htmlComments welcome Your comments are important to us! We want our Redbooks™ to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an Internet note to: redbook@us.ibm.com Mail your comments to: IBM® Corporation, International Technical Support Organization Dept. JN9B Building 003 Internal Zip 2834 11400 Burnet Road Austin, Texas 78758-3493 Preface xxiii
    • xxiv Implementing Tivoli Data Warehouse 1.2
    • Part 1Part 1 Fundamentals© Copyright IBM Corp. 2004. All rights reserved. 1
    • 2 Implementing Tivoli Data Warehouse 1.2
    • 1 Chapter 1. Introducing Tivoli Data Warehouse 1.2 This chapter provides a brief introduction to the concepts, technologies, and products behind the Tivoli Data Warehouse, and the new features that can be found in Version 1.2. We cover the following topics: “Data warehousing basics” on page 4 “Tivoli Data Warehouse” on page 8 “What is new in Tivoli Data Warehouse 1.2” on page 10 “Tivoli Data Warehouse architecture” on page 20 “Benefits of using Tivoli Data Warehouse” on page 23© Copyright IBM Corp. 2004. All rights reserved. 3
    • 1.1 Data warehousing basics Data warehousing is the process of managing a data warehouse and its components, called data marts. This management process includes all the ongoing support needs of the refresh cycle, database maintenance, and continual refinements to the underlying data model. In addition to that, data warehousing can be thought of as a tool to enable and support business intelligence. The concept of data warehousing carries several other important terms mentioned in the above paragraph. Such terms will be explained in the sections to follow. They are: Data warehouse Data mart Business intelligence Data mining1.1.1 Data warehouse A data warehouse is the cohesive data model that defines the central data repository for an organization. An important point is that we dont define a warehouse in terms of the number of databases. Instead, we consider it a complete, integrated data model of the enterprise, regardless of how or where the data is stored. A data warehouse is a collection of databases where data is collected for the purpose of being analyzed. This collection of databases can be formed by one or more databases. The defining characteristic of a data warehouse is its purpose. Most data is collected to handle a companys on-going business. This type of data can be called operational data. The systems used to collect operational data are referred to as OLTP. A data warehouse collects, organizes, and makes data available for the purpose of analysis in order to give management the ability to access and analyze information about its business. This type of data can be called informational data. The systems used to work with informational data are referred to as online analytical processing (OLAP). Bill Inmon coined the term data warehouse in 1990. His definition is as follows: “A (data) warehouse is a subject-oriented, integrated, time-variant, and non-volatile collection of data in support of managements decision-making process.”4 Implementing Tivoli Data Warehouse 1.2
    • These are the main types of data: Subject-oriented: Data that gives information about a particular subject instead of about a companys on-going operations Integrated: Data that is gathered into the data warehouse from a variety of sources and merged into a coherent whole Time-variant: All data in the data warehouse that is identified with a particular time period1.1.2 Data mart A data mart is a repository containing data specific to a particular business group in an enterprise. All data in a data mart derives from the data warehouse, and all data relates directly to the enterprise wide data model. Often, data marts contain summarized or aggregated data that the user community can easily consume. Another way to differentiate a data warehouse from a data mart is to look at the datas consumers and format. IT analysts and canned reporting utilities consume warehouse data, whose storage is usually coded and cryptic. The user community consumes data mart data, whose storage is usually in a more readable format. For example, to reduce the need for complex queries and assist business users who might be uncomfortable with the SQL language, data tables could contain the de-normalized code table values. A data mart contains a subset of corporate data that is of value to a specific business unit, department, or set of users. This subset consists of historical, summarized, and possibly detailed data captured from transaction processing systems, or from an enterprise data warehouse. It is important to realize that a data mart is defined by the functional scope of its users, and not by the size of the data mart database. In parallel to increasing data mart usage, the underlying databases will rapidly increase in size.1.1.3 Business intelligence Business intelligence (BI) is not business as usual. It is about making better decisions more quickly and easily. Businesses collect enormous amounts of data every day: Information about orders, inventory, accounts payable, point-of-sale transactions, and, of course, customers. Businesses also acquire data, such as demographics and mailing lists, from outside sources. Unfortunately, based on a recent survey, over 93 percent of corporate data is not usable in the business decision-making process today. This applies also to systems management, where data tends to be of more technical nature. Chapter 1. Introducing Tivoli Data Warehouse 1.2 5
    • Consolidating and organizing data for better business decisions can lead to a competitive advantage, and learning to uncover and leverage those advantages is what business intelligence is all about. The amount of business data is increasing exponentially. In fact, it doubles every two to three years. More information means more competition. In the age of the information explosion, executives, managers, professionals, and workers all need to be able to make better decisions faster. Because now, more than ever, time is money. Much more than a combination of data and technology, BI helps you to create knowledge from a world of information. Get the right data, discover its power, and share the value, BI transforms information into knowledge. Business intelligence is the application of putting the right information into the hands of the right user at the right time to support the decision-making process. Business driving forces It can be noted that there are some business driving forces behind business intelligence, one being the need to improve ease-of-use and reduce the resources required to implement and use new information technologies. Other driving forces behind business intelligence include these: The need to increase revenues, reduce costs, and compete more effectively. Gone are the days when end users could manage and plan business operations using monthly batch reports, and IT organizations had months to implement new applications. Today companies need to deploy informational applications rapidly, and provide business users with easy and fast access to business information that reflects the rapidly changing business environment. Business intelligence systems are focused towards end user information access and delivery, and provide packaged business solutions in addition to supporting the sophisticated information technologies required for the processing of today’s business information. The need to manage and model the complexity of today’s business environment. Corporate mergers and deregulation means that companies today are providing and supporting a wider range of products and services to a broader and more diverse audience than ever before. Understanding and managing such a complex business environment and maximizing business investment is becoming increasingly more difficult. Business intelligence systems provide more than just basic query and reporting mechanisms, they also offer sophisticated information analysis and information discovery tools that are designed to handle and process the complex business information associated with today’s business environment.6 Implementing Tivoli Data Warehouse 1.2
    • The need to reduce IT costs and leverage existing corporate business information. The investment in IT systems today is usually a significant percentage of corporate expenses, and there is a need not only to reduce this overhead, but also to gain the maximum business benefits from the information managed by IT systems. New information technologies like corporate intranets, thin-client computing, and subscription-driven information delivery help reduce the cost of deploying business intelligence systems to a wider user audience, especially information consumers like executives and business managers. Business intelligence systems also broaden the scope of the information that can be processed to include not only operational and warehouse data, but also information managed by office systems and corporate Web servers.1.1.4 Data mining Data mining is the process of extracting valid, useful, previously unknown, and comprehensible information from data and using it to make business decisions. The organizations of today are under tremendous pressure to compete in an environment of tight deadlines and reduced profits. Legacy and lengthy business processes that require data to be extracted and manipulated prior to use will no longer be acceptable. Instead, enterprises need rapid decision support based on the analysis and forecasting of predictive behavior. Data warehousing and data mining techniques provide this capability. Data mining can be defined as the extraction of hidden predictive information from large databases, and is a powerful technology with great potential to help companies focus on the most important information in their data warehouses. Once a Tivoli Data Warehouse has been established, data mining tools can then be used to predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions. Data mining tools can answer business questions that traditionally were too time consuming to resolve. These tools hunt databases for hidden patterns, finding predictive information that experts may miss because it lies outside their expectations. The art of data mining is not trivial, and it can be similar to “finding the needle in the haystack”. In this case, the needle is that single piece of intelligence your business needs, and the haystack is the large data warehouse youve built up over a period of time within your business. Chapter 1. Introducing Tivoli Data Warehouse 1.2 7
    • Most companies already collect and analyze massive quantities of data. Data mining techniques can be implemented rapidly on existing software and hardware platforms to enhance the value of existing information resources, and can be integrated with new products and systems as they are brought on-line. Given databases of sufficient size and quality, data mining technology can generate new business opportunities by providing these capabilities: Automated prediction of trends and behaviors: Data mining automates the process of finding predictive information in large databases. Questions that traditionally required extensive hands-on analysis can now be answered directly from the data and quickly. A typical example of a predictive problem is targeted server performance. Data mining uses data on past critical events to identify the servers most likely to cause future critical problems. Other predictive problems include forecasting server outage and other forms of performance degradation that is likely to occur, given certain events. Automated discovery of previously unknown patterns: Data mining tools sweep through databases and identify previously hidden patterns in one step. An example of pattern discovery is the analysis of IBM Tivoli Monitoring data to identify seemingly unrelated events that are often received together.1.2 Tivoli Data Warehouse The Tivoli Data Warehouse 1.2 is built on an IBM DB2® Data Warehouse. It offers all IBM DB2 Data Warehouse functionality with additional Tivoli specific extensions. The IBM Data Warehouse Management uses the IBM DB2 Universal Database Enterprise Edition and the IBM DB2 Data Warehouse Manager feature. It provides an integrated, distributed, heterogeneous warehouse management infrastructure for designing, building, maintaining, governing, and accessing highly scalable, robust data warehouses, operational data stores, and data marts stored in IBM DB2 databases. IBM DB2 Data Warehouse Manager helps warehouse administrators: To manage data volumes, to move data directly from source to target (also allowing packaged and simplified access to popular partner products such as SAP R/3), and to control the servers on which transformations take place with distributed warehouse agents To speed warehouse and data mart deployment with commonly used, pre-built data cleansing and statistical transformations To build and manage from a central point of control, integrated in IBM DB2, utilizing the Data Warehouse Center graphical user interface8 Implementing Tivoli Data Warehouse 1.2
    • DB2 warehouse management consists of: An administrative client to define and manage data warehousing tasks and objects, and warehouse or data mart operations: the Data Warehouse Center A manager to manage and control the flow of data: the warehouse server Agents residing on IBM DB2 Universal Database Enterprise Edition server platforms to perform requests from the manager or warehouse server: the local or remote warehouse agent A warehouse control database storing the warehouse management metadata on a IBM DB2 database server A metadata administrative and publishing tool with its own administration graphical user interface (GUI): Information Catalog Manager to manage and present both technical and business metadata The different components of the IBM DB2 Data Warehouse Manager are shown in Figure 1-1. Clients Warehouse Warehouse Databases End Users Server Agents Data Warehouse Data Relational Center Message Source Data Message DB2 Data Target Message Non- Data Relational Source Message Metadata Data Metadata Non-DB2 yy Target Data Log Control Editions Database Configuration Data Flat Files, DB2 Web or SAP R/3 Included with IBM DB2Figure 1-1 IBM DB2 Data Warehouse Manager Chapter 1. Introducing Tivoli Data Warehouse 1.2 9
    • 1.3 What is new in Tivoli Data Warehouse 1.2 Tivoli Data Warehouse 1.2 provides a number of enhancements and new features over Version 1.1, such as these: Improved interfaced and Web-based reporting using Crystal Enterprise™ DB2 UDB for OS/390 and z/OS support Flexible and extended configuration support Installation enhancements Serviceability and scalability improvements We now discuss each of these areas.1.3.1 Crystal Enterprise™ Among the enhancements that Tivoli Data Warehouse 1.2 provides, an important change is the new mechanism for producing reports and the user interface. Version 1.1 of Tivoli Data Warehouse used Tivoli Presentation Services and the IBM Console. Tivoli Data Warehouse 1.2 does not use the IBM Console nor Tivoli Presentation Services. The reporting technology is now provided by Crystal Enterprise™, by Business Objects, which is a world standard for high-quality and high-performance reporting. Crystal Enterprise™ provides: Out-of-the-box Web-based reporting and information delivery for all your Tivoli products An extendable, scalable reporting solution to meet the information delivery needs of your IT organization A report scheduling capability An export feature to export reports to variety of formats (Excel, Word, PDF) The capability to change the look and feel of the reports The Tivoli Data Warehouse 1.2 comes supplied with Crystal Enterprise Professional Version 9 for Tivoli (limited use version) to analyze and deliver out-of-the-box Reports from the Tivoli Data Warehouse into the hands of decision-makers using a Web browser. This will allow for a rapid return on investment you have made in your Tivoli solution, by providing out-of-the-box Web based reporting, including scheduling and report export capability. All of this is achieved using a customizable platform for organizing, categorizing, and delivering information.10 Implementing Tivoli Data Warehouse 1.2
    • If we define a report as an entity that visualizes the output of SQL clauses, or an“SQL Pull”, then the Crystal Enterprise Professional Version 9 for Tivoli, which isshipped with the Tivoli Data Warehouse 1.2 product, is supplied with a number ofstandard reports provided by the Tivoli Data Warehouse Enablement Packs(WEPs). When a report is made available by the WEP to Crystal Enterprise, thelayout, legends, colors, and the look-and-feel of the report can all be customized.However, to create a new report (using the definition above), or to modify theSQL pull criteria of an existing report, Crystal Reports and a different version ofCrystal Enterprise™ is required: Crystal Enterprise Version 9 Special Edition.A license for Crystal Enterprise Version 9 Special Edition must be purchasedseparately. The Crystal Enterprise Version 9 Special Edition will allow you to: Extend your reporting capabilities to develop, deliver, and analyze new reports created from your Tivoli Systems Management Data using Crystal Reports version 9 Provide support for approximately 75 concurrent online users Add, modify, and design new reports from your Tivoli Systems Management Data using Crystal Reports version 9For the tasks listed above, Crystal Reports Version 9 Special Edition is requiredand must be purchased separately.Next we present a brief introduction to the Crystal Enterprise architecture. Asshown in Figure 1-2, by using the Crystal Enterprise™ multi-tier architecture, theIBM Tivoli product portfolio has a key partnership developed to ensure thedeepest level of integration and ongoing support for this solution. Please notethat some of the functions may not be available on the Crystal EnterpriseProfessional for Tivoli product. Chapter 1. Introducing Tivoli Data Warehouse 1.2 11
    • Browser or Crystal applications Crystal Management Console eProtfolio Client Tier Crystal Configuration Manager Publishing Wizard Import Wizard Web Server / Web Connector Crystal Enterprise Framework Intelligence Tier Web Component Server File Repository Server Automated Process Scheduler Cache Server Event Server Processing Tier Job Server Page Server Report Application Server Data Tier OLAP Relational ODBC XML, ERP, CRM, COM Figure 1-2 Crystal Enterprise multi-tier architecture In Crystal Enterprise, there are four tiers, each of which can be installed on one machine, or with the Crystal Enterprise Version 9 Special Edition, spread across many. The Crystal Enterprise architecture tiers are as follows: Client tier: Administrators and end users interact with this component directly, which is made up of the applications that enable people to administer, publish, and view reports. Intelligence tier: These components manage the Crystal Enterprise administration system, which consists of maintaining all aspects of the security information, storing report instances, and controlling the flow of requests to the appropriate servers.12 Implementing Tivoli Data Warehouse 1.2
    • Processing tier: These components access the data and generate the reports. This is the only tier that communicates directly with the databases that contain the report data. Data tier: The databases that contain the data used in the reports fall into this tier. These databases are referred as Data Sources in Crystal Enterprise, and a wide range of databases are supported. These databases could contain historic data and/or operational data. This redbook does not go into the details of Crystal Enterprise Professional Version 9 for Tivoli administration and configuration. Refer to the following documentation shipped with the product: Crystal Enterprise 9 Installation Guide Crystal Enterprise 9 Administrator’s Guide Crystal Enterprise 9 Getting Started Guide Crystal Enterprise 9 ePortfolio User’s Guide1.3.2 IBM DB2 UDB for OS/390 and z/OS support On z/OS® systems, operational data is extracted from a Tivoli Decision Support for OS/390® database. As the next generation of historical data reporting and analysis solutions, Tivoli Data Warehouse is a successor to Tivoli Decision Support (TDS) on distributed platforms (Wintel/UNIX) and a companion product for Tivoli Decision Support for OS/390. The following sections explain how Tivoli Data Warehouse relates to and works with Tivoli Decision Support and Tivoli Decision Support for OS/390. Tivoli Decision Support for OS/390 and Tivoli Data Warehouse The following items compare and contrast Tivoli Data Warehouse and Tivoli Decision Support: Both Tivoli Data Warehouse and Tivoli Decision Support collect and analyze data gathered by the system management products in your enterprise. Both provide an infrastructure for reporting and analysis, but do not themselves extract data or provide reports. Each relies on other applications to use the infrastructure to extract and analyze data and to provide reports that satisfy a specific reporting or analysis need. In Tivoli Decision Support, an application that provides a solution to a specific reporting need is called a Tivoli Decision Support Guide. In Tivoli Data Warehouse, the corresponding application is called a Warehouse Enablement Pack. Chapter 1. Introducing Tivoli Data Warehouse 1.2 13
    • Some Tivoli Decision Support Guides require direct access to the data in your operational data stores, which can decrease the performance of the products creating and using those data stores. Tivoli Data Warehouse ensures that your operational data stores are not impacted by users running reports. It also ensures that users can run reports efficiently by accessing databases that are optimized for interactive reporting. By saving historical data in a central location and in a common format, Tivoli Data Warehouse makes it easier to create reports that draw on data collected by more than one product. Tivoli Decision Support stores and accesses data using Cognos Powerplay and Crystal Reports. In contrast, Tivoli Data Warehouse publishes the format of its data, as well as the format of the data in the products that feed the warehouse, allowing the use of various reporting tools. This enables you to use the business intelligence solutions you already know. In addition, Tivoli software uses Crystal Enterprise, which is provided with Tivoli Data Warehouse, as a common reporting solution. Tivoli Data Warehouse provides support for multiple languages. Tivoli Decision Support is available only in English. Tivoli Decision Support for OS/390 is available in English and Japanese. TDS for OS/390 and Tivoli Data Warehouse 1.2 Interaction This section describes how Tivoli Decision Support for OS/390 works with Tivoli Data Warehouse 1.2 to store and aggregate data. Tivoli Decision Support for OS/390 collects system management data from System Management Facility (SMF), Information Management System (IMS™), and other logs. It aggregates and summarizes data on a hourly, daily, and monthly basis and places the data into its own database. The Tivoli Decision Support for OS/390 database contains data primarily from z/OS systems. Tivoli Data Warehouse 1.2 can use the Tivoli Decision Support for OS/390 databases as an operational data source for the z/OS applications. The flow of data between Tivoli Decision Support for OS/390 and Tivoli Data Warehouse 1.2 can be seen in Figure 1-3.14 Implementing Tivoli Data Warehouse 1.2
    • Source Data Consolidated by TDS for OS/390 Data source Logs Data source IMS Data source SMF TDS for Reports OS/390 TDW 1.2 CDW on OS/390 Central Data Data Mart Data Mart Warehouse Crystal Crystal TDW 1.2 Server Server Web Reports Control Center Web Server Web ServerFigure 1-3 TDS OS/390 and TDW 1.2 Data flowData from z/OS systemsData from z/OS systems can only be placed in a central data warehouse onz/OS. A set of Extract Transform and Load (ETL) programs from a warehouseenablement pack will extract data from the TDS for OS/390 database, transform,and place the data into the central data warehouse on the z/OS system.Another set of ETLs will then extract the data from the central data warehousedatabase on the z/OS system and load the data into data mart databases. Thereare no restrictions on the location of the data mart databases. These databasescan be located on z/OS systems or any other supported platforms. You can thenanalyze the data and generate reports from the data contained in the data martdatabases. Chapter 1. Introducing Tivoli Data Warehouse 1.2 15
    • Figure 1-4 shows how data from the distributed environment and the OS/390 or z/OS environment can be encapsulated to produce real, end-to-end enterprise reporting. Source Data Consolidated by TDS for OS/390 IT Infrastructure Management Data Logs IMS TEC ITM SMF TDS for Reports SLA OS/390 Reports ETL ETL Central Data Warehouse on OS/390 Central Data Warehouse distributed Central Data Central Data Central Data Warehouse Warehouse Warehouse ETL Data Mart for reporting functionality Data Mart Gain the competitive edge Crystal Other Business Enterprise Intelligence Tools Other Reporting Other Data Tools Analysis ToolsFigure 1-4 Distributed and OS/390 Data feeds into Tivoli Data Warehouse For additional details on Tivoli Data Warehouse 1.2 components placement, refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27.16 Implementing Tivoli Data Warehouse 1.2
    • 1.3.3 Flexible and extended configuration support You can have up to four central data warehouse databases and up to four data mart databases on Windows®, UNIX®, or z/OS systems, and multiple source application support controlled by Tivoli Data Warehouse 1.2, as shown in Figure 1-5. Data source Application 1 . . Data source Application 2 . Central Data Data Mart Data Mart . Warehouse TDW 1.2 Control Center Data source Application N Data source Application 1 . . Data source Application 2 . Central Data Data Mart Data Mart . Warehouse Data source Application N Data source Application 1 . . Data source Application 2 . Central Data Warehouse Data Mart Data Mart . Data source Application N Data source Application 1 . . Data source Application 2 . Central Data Data Mart Data Mart . Warehouse Data source Application NFigure 1-5 Multiple source applications loading into a central data warehouse Chapter 1. Introducing Tivoli Data Warehouse 1.2 17
    • However, you could also keep data about multiple customers and data centers in one central data warehouse database, while restricting access so that customers can see and work with data and reports based only on their data and not any other customer’s data. For example, support for multiple customer environments enables a service provider to keep historical data about all of its customers in one deployment of Tivoli Data Warehouse. Multiple data center support provides a way to partition data physically by customizable criteria such as location, application, or business purpose.1.3.4 Installation enhancements The following enhancements have been made to the installation process of Tivoli Data Warehouse 1.2: The Tivoli Data Warehouse product and Warehouse Enablement Pack (WEP) installation processes have been separated: – In version 1.1 of the Tivoli Data Warehouse, to install a warehouse enablement pack, it was necessary to launch the installation program of the Tivoli Data Warehouse. In Tivoli Data Warehouse 1.2, the installation processes for the base product and WEPs have been separated. Once the Tivoli Data Warehouse base product has been installed, a shortcut is created, which launches the WEP Installation Wizard. The Warehouse Enablement Pack Installer: – Allows installation and un-installation as well as patch installs from the same GUI – Leaves the warehouse enablement pack in a scheduled state Many manual steps have been automated, such as: – DB2 Warehouse Manager Remote Agent discovery and configuration – Client/Server connections for IBM DB2 or Oracle – Creation of ODBC data sources on Windows, AIX®, and Solaris – Auto-configuration of Data Warehouse Center Metadata, including passwords and schedules18 Implementing Tivoli Data Warehouse 1.2
    • 1.3.5 Serviceability and scalability improvements The following serviceability improvements have been made to Tivoli Data Warehouse 1.2: Centralized logging in XML and log viewer tool: Tivoli applications support a common XML format, in which they log messages and traces. This common format is called Log XML. The LogXML Viewer processes logs in that format, enabling you to view the logs in any Web browser in an easy-to-read format. The viewer is installed with Tivoli Data Warehouse and is located in the Tivoli common logging directory/cdw/tools directory. The LogXML Viewer converts the logged messages into ASCII or HTML for presentation. Visual cues are associated with error and warning messages. HTML viewer that enables you to view your logs online. Flexible tracing capability. First failure data capture: First Failure Data Capture (FFDC) is the ability to capture useful data after a problem has been logged. The purpose of FFDC features is to collect enough data so that a problem can be diagnosed without having to reproduce it a second time. Since it is likely that a failure will happen while the system is in an unattended mode, it is important to capture this data in such a way that it is not overwritten, before it can be gathered and sent to a support center or help desk. Data that is captured as part of FFDC can include trace logs, message logs, dumps of in-memory data structures, and so forth. For scalability, Tivoli Data Warehouse 1.2 now provides: Support for multiple instances of operational data source. A warehouse enablement pack can extract operational data from multiple instances of the same data source. Capability to grow the environment overtime by adding central data warehouses and data marts after the initial installation. Chapter 1. Introducing Tivoli Data Warehouse 1.2 19
    • 1.4 Tivoli Data Warehouse architecture Tivoli Data Warehouse 1.2 is made available by Tivoli to consolidate historical data from different management applications into one or more central data warehouse databases, and to make it easy to analyze comprehensible information from data stored into its data mart databases, thus facilitating the business decision making process. Figure 1-6 shows how the Tivoli Data Warehouse technology together with Crystal Enterprise can be used to provide an end-to-end business intelligence enabler. Tivoli Data Warehouse 1.2 Technology IT Infrastructure Management Data TDW 1.2 Data source TEC Control Center Data source ITM Data source Appl2 Data source SLA Central Data Data source Appl1 Warehouse Central Data Operational Data Data Mart Central Data Warehouse Warehouse OLAP, Business Intelligence & Analysis Tools Crystal Enterprise Server Web Server Crystal Web Interface Web ReportsFigure 1-6 Tivoli Data Warehouse — the big picture A Tivoli Data Warehouse 1.2 architecture can be composed of the following elements: Tivoli Data Warehouse control center server One or more central data warehouse databases One or more data mart databases IBM DB2 warehouse agents and agents sites Crystal Enterprise server20 Implementing Tivoli Data Warehouse 1.2
    • The following sections describe these elements in detail.1.4.1 Tivoli Data Warehouse control center server The control center server is the system that contains the control database for Tivoli Data Warehouse and is the system from which you manage your data. The control database contains metadata for both Tivoli Data Warehouse and for the warehouse management functions of IBM DB2 Universal Database™ Enterprise Edition. There can only be one control server in a Tivoli Data Warehouse 1.2 deployment.1.4.2 Source databases A source databases holds operational data worth to be loaded into the Tivoli Data Warehouse environment. Typically, the source databases are application specific and their number is likely to increase for a data warehouse installation. Most Tivoli products provide a warehouse enablement pack which makes application specific data available in a source database. This can be a dedicated warehouse source database as it is coming with IBM Tivoli Monitoring, or it can be an interface to the applications built in database as provided for IBM Tivoli Storage Manager or IBM Tivoli NetView®. WEP for Tivoli products also include a means to upload data from the source database to the central data warehouse, thus minimizing the efforts for data collection.1.4.3 Central data warehouse The central data warehouse is a set of IBM DB2 databases that contains the historical data for your enterprise. You can have up to four central data warehouse databases in a Tivoli Data Warehouse 1.2 deployment.1.4.4 Data marts A separate set of IBM DB2 databases contains the data marts for your enterprise. Each data mart contains a subset of the historical data from the central data warehouse that satisfies the analysis and reporting needs of a specific department, team, customer, or application. You can have up to four data mart databases in a Tivoli Data Warehouse 1.2 deployment. Each data mart database can contain the data for multiple central data warehouse databases. A WEP for a Tivoli application provides all the necessary means to fill data marts with their specific data. Chapter 1. Introducing Tivoli Data Warehouse 1.2 21
    • 1.4.5 Warehouse agents and agent sites The warehouse agent is the component of IBM DB2 Warehouse Manager that manages the flow of data between data sources and targets that are on different computers. By default, the control center server uses a local warehouse agent to manage the data flow between operational data sources, central data warehouse databases, and data mart databases. You can optionally install the warehouse agent component of IBM DB2 Warehouse Manager on a computer other than the control center server. Typically, you place an agent on the computer that is the target of a data transfer. That computer becomes a remote agent site, which the Data Warehouse Center uses to manage the transfer of Tivoli Data Warehouse data. This can speed up the data transfer as well as reduce the workload on the control server.1.4.6 Crystal Enterprise Server Crystal Enterprise Professional for Tivoli completely replaces the Reports Interface of TEDW 1.1 version. It gives us a new mechanism for obtaining the Reports provided by the Warehouse Enablement Packs. The installation and configuration of a Crystal Enterprise environment is mandatory before you begin installing Tivoli Data Warehouse 1.2. Tivoli Data Warehouse 1.2 supports only the full stand-alone installation of Crystal Enterprise. In the full stand-alone installation, Crystal Enterprise is installed on a single computer that is already running as a Web Server. Crystal Enterprise is dependent of a number of software components that must be up and running prior to its installation: Operating Systems: Windows NT®, Windows 2000, or Windows 2003 Internet Browser: Internet Explorer or Netscape Navigator Web Servers: IBM HTTP Server, Microsoft® IIS, iPlanet Enterprise Server, or Lotus® Domino®22 Implementing Tivoli Data Warehouse 1.2
    • Figure 1-7 shows an overview of the Tivoli Data Warehouse 1.2 architecture and supported software components. Win NT/2000 Web-based Reports Cr TDW 1.2 ys Control Center ta l eP IE 5.5 SP2 & 6.0 or tfo Netscape 6.2.3 lio WM Agent Applications’ DB2 UDB EE & IBM HTTP Server Data Store DB2/390 IIS v4 & v5 iPlanet Lotus Domino ETL1 Central Data Data Mart Web Server Warehouse ETL2 Data Mart Data Mart Data Mart Star Schema Crystal AIX,Sun Solaris, HP-UX, Data Mart Enterprise NT/2K, OS/390, Turbo, Server RedHat and SuSE Linux AIX,Sun Solaris, NT/2K, MVS Win NT/2000/2003Figure 1-7 Detail Component view of Tivoli Data Warehouse 1.21.5 Benefits of using Tivoli Data Warehouse By using Tivoli Data Warehouse, you can access the underlying data about your IT infrastructure, including network devices and connections, desktops, hardware, software, events, and other information. With this information in a data warehouse, you can look at your IT costs, performance, and other trends across your enterprise. Tivoli Data Warehouse can be used to show the value and return on investment of Tivoli and IBM software, and it can be used to identify areas where you can be more effective. Tivoli Data Warehouse provides the following capabilities: It has an open architecture for storing, aggregating, and correlating historical data. Most Tivoli products are shipped with warehouse enablement packs to provide an easy and fast integration of the Tivoli products with the Tivoli Data Warehouse. Chapter 1. Introducing Tivoli Data Warehouse 1.2 23
    • In addition to the data collected by diverse IBM Tivoli software, Tivoli Data Warehouse has the flexibility and extensibility to enable you to integrate your own application data. It offers database optimizations both for the efficient storage of large amounts of historical data and for fast access to data for analysis and report generation. It provides the infrastructure and tools necessary for maintaining the data: – These tools include the Tivoli Data Warehouse application, IBM DB2 Universal Database Enterprise Edition, the Data Warehouse Center, the DB2 Warehouse Manager, and Crystal Enterprise. – It includes the ability to use your choice of data analysis tools to examine your historical data. – In addition to the Crystal Enterprise reporting solution that is shipped with Tivoli Data Warehouse, you can analyze your data using any product that performs online analytical processing (OLAP), planning, trending, analysis, accounting, or data mining. It offers multi-customer and multicenter support: – You can keep data about multiple customers and multiple data centers in one warehouse, but restrict access so that customers can see and work with data and reports based only on their own data and not any other customer’s data. – You can also restrict an individual user’s ability to access data. It includes internationalization support: – Reports can be displayed in the language of the user’s choice. Crystal Enterprise only comes in English, French, German, and Japanese; however, the reports generated by Crystal are translated into Brazilian Portuguese, French, German, Italian, Spanish, Japanese, Korean, Simplified Chinese, and Traditional Chinese. – This means that even if you are running the Crystal Enterprise server in English, you could view a report in Italian through the Crystal Report Viewer Web interface if that language is set as your locale preference.24 Implementing Tivoli Data Warehouse 1.2
    • As shown in Figure 1-8, the Tivoli Data Warehouse 1.2 could be used as a single integration point for all Systems Management data, and it could also be used as both tool and technology to drive business intelligence within your enterprise. Business Impact Management Event Correlation & Automation Tivoli Service Level Advisor Monitoring Reporting and Business Reporting and Business Intelligence Vendors Storage Security Provisioning Third Party Systems Tivoli Data Warehouse Management ApplicationsFigure 1-8 Integrated Systems Management Chapter 1. Introducing Tivoli Data Warehouse 1.2 25
    • 26 Implementing Tivoli Data Warehouse 1.2
    • 2 Chapter 2. Planning for Tivoli Data Warehouse 1.2 This chapter provides planning considerations for Tivoli Data Warehouse. We cover the following topics: “Hardware and software requirements” on page 28 “Physical and logical design considerations” on page 36 “Database sizing” on page 56 “Security” on page 57 “Network traffic considerations” on page 61 “Integration with other business intelligence tools” on page 64 “ETL development” on page 65 “Skills required for a Tivoli Data Warehouse project” on page 67© Copyright IBM Corp. 2004. All rights reserved. 27
    • 2.1 Hardware and software requirements Tivoli Data Warehouse 1.2 can be deployed in two different configurations: Quick start installation, also known as stand-alone installation, with all the components installed on a single Microsoft Windows NT, Windows 2000, or Windows 2003 system. This is convenient for demonstrations, as an educational or test platform, and for companies that do not plan to have many users concurrently accessing the data stored in the Tivoli Data Warehouse databases and/or those that do not need to capture and analyze large amounts of data. Distributed installation, with the components installed on multiple systems in your enterprise, including UNIX and z/OS servers. See “Software requirements” on page 30 to determine the operating systems supporting each component of Tivoli Data Warehouse. The historical reporting for Tivoli Data Warehouse 1.2 is provided by Crystal Enterprise, which can be installed in three different configurations, depending on the version used: Full stand-alone installation on a single Windows system that is already running as a Web server. This provides the quickest way to install Crystal Enterprise. Server-side installation connected to a Web server, which allows you to maintain separation between Crystal Enterprise and the Web server by running them on separate machines. In this scenario the Web server can also be on a UNIX system. Expanded installation, which allows you to install Crystal Enterprise server components on more machines in order to create an Automated Process Scheduler (APS) cluster, to increase available resources and to distribute the processing workload. Note: IBM ships a limited license of Crystal Enterprise Professional Version 9 for Tivoli with Tivoli Data Warehouse 1.2 that only allows the full stand-alone installation option. In the following sections we provide the current hardware and software requirements for a Tivoli Data Warehouse environment, but you should also check the Tivoli Data Warehouse Release Notes, SC32-1399, for possible updates about these requirements.28 Implementing Tivoli Data Warehouse 1.2
    • 2.1.1 Hardware requirements This section provides information about the hardware requirements for installing Tivoli Data Warehouse 1.2 components. Please be aware that as the warehouse enablement packs are added to the Tivoli Data Warehouse 1.2 installation, additional hard disk space is required. See the documentation provided by the Warehouse Enablement Pack for application planning information and additional hard disk space requirements. Table 2-1 lists the recommended hardware requirements for Tivoli Data Warehouse 1.2 for both stand-alone and distributed configurations. Note: The values in Table 2-1 differ from the information provided in the Tivoli Data Warehouse Release Notes, SC32-1399. They represent the hardware used at the time of writing this book and serve as our recommended starting configuration for both stand-alone and distributed configurations. Table 2-1 Hardware recommendations for Tivoli Data Warehouse components Installation Tivoli Data Warehouse Recommended size configuration components Stand-alone installation All 1.0 GB RAM 2.4 GHz processor 30 GB disk Distributed installation Control server 512 MB RAM 1.3 MHz processor 10 GB disk Central data warehouse 512 MB RAM 1.3 MHz processor (400 MHz CPU or equivalent for UNIX) 30 GB disk Data mart 512 MB RAM 1.3 MHz processor (400 MHz CPU or equivalent for UNIX) 30 GB disk Disk space requirements for central data warehouse and data marts can vary greatly according to the amount of actual collected data. See 2.3, “Database sizing” on page 56 to help you in evaluating the storage required by the different metrics you wish to collect in your own environment. Chapter 2. Planning for Tivoli Data Warehouse 1.2 29
    • Table 2-2 lists the hardware requirements for additional components of a Tivoli Data Warehouse 1.2 solution. Table 2-2 Additional hard disk space requirements Component Recommended hardware Crystal Enterprise Professional Version 9 for 1.0 GB RAM Tivoli 2.4 GHz processor 30 GB disk (1 GB for installation alone) Warehouse agent 50 MB disk The storage required by the installation of Crystal Enterprise Professional Version 9 for Tivoli shown in Table 2-2 assumes a full stand-alone installation of Crystal Enterprise Professional Version 9 for Tivoli. Please note that on the Crystal Enterprise server additional storage is required for Web server, database client software and all the reports installed by each warehouse enablement pack.2.1.2 Software requirements Table 2-3 provides information about the software requirements for the Tivoli Data Warehouse 1.2 components. Note: You might receive confusing error messages if your systems do not meet the software requirements listed in this section.30 Implementing Tivoli Data Warehouse 1.2
    • Table 2-3 Software requirements Tivoli Data Warehouse core components Operating Data Warehouse Control Central Data Crystal Crystal system source agents server data mart Back Weba warehouse database End Microsoft® Yes Yes Yes Yes Yes Yes, Yes Windows NT®, also service pack 6 NT4 or higher and Server Microsoft Data SP6a Access Components (MDAC) 2.7 service pack 1 Windows 2000 Server SP2®+ Windows 2000 Advanced Server SP2+ Windows 2003 Server IBM AIX® 4.3.3, Yes Yes No Yes Yes No 4.3.3, 5.1, and 5.2 5.1 only Sun Solaris Yes Yes No Yes Yes No 2.8 only Versions 2.8 and 2.9 RedHat Linux Yes No No No No No Red Version 7.1, 7.2, Hat 7.2, 7.3 and 7.3 only Advanced Server 2.1 SuSE Linux Yes No No No No No SuSe Version 7.2 7.3, 8.0 Turbo Linux 7 Yes No No No No No No zOS 1.2, 1.3, Yes No No Yes Yes No No 1.4 a. The Crystal Enterprise limited edition provided with Tivoli Data Warehouse requires that the Web server is on the same system of Crystal Enterprise server Chapter 2. Planning for Tivoli Data Warehouse 1.2 31
    • 2.1.3 Database requirements Tivoli Data Warehouse 1.2 requires the minimum level of IBM DB2 Universal Database Enterprise Edition 7.2 to be at Fix Pack 8e (that is, Fix Pack 8 plus the e-fix for Fix Pack 8). If you are currently running an IBM DB2 version prior to 7.2, you must upgrade to version 7.2 and apply the Fix Pack 8 and e-fix for Fix Pack 8, at minimum. Note: IBM DB2 Universal Enterprise Edition Version 8 and above are not supported. Even though IBM DB2 Universal Database Enterprise Edition 7.2 Fix Pack 8e is the minimum level of IBM DB2 required, Tivoli Data Warehouse 1.2 is bundled with IBM DB2 Universal Database Enterprise Edition 7.2 at the Fix Pack 10a level (FP10a). It is recommended to use the IBM DB2 Universal Database Enterprise Edition media provided with Tivoli Data Warehouse 1.2 to ensure the use of the correct levels of software and Fix Pack. Note: IBM DB2 Universal Database Enterprise Edition 7.2 must be AT LEAST at the Fix Pack 8e level in order to be able to install TDW 1.2, otherwise the TDW 1.2 installation will detect an inadequate level of IBM DB2 and fail. For warehouse databases on z/OS systems, the supported database version is DB2 Universal Database for OS/390 and z/OS V7.1 with the DB2 Management Clients Package installed (FMID JDB771D). The recommended hard disk space in Table 2-1 on page 29 may not be large enough to accommodate some data growth as transactions are added to the database. However, when you plan for your database requirements, you must consider the following possibilities: Future data growth Addition of warehouse enablement packs It is recommended that you install your central data warehouse on an expandable system with a minimum of 30 GB of data space. The data source support is determined by Open Database Connectivity (ODBC) support in IBM DB2 releases and specified by the warehouse enablement pack product. Note: The actual RDBMSs supported as data sources for a given warehouse enablement pack are documented in the README file for that warehouse enablement pack.32 Implementing Tivoli Data Warehouse 1.2
    • 2.1.4 Crystal Enterprise requirements The server components of Crystal Enterprise Professional Version 9 for Tivoli can be installed on the following operating systems: Microsoft Windows NT4 Server SP6a Microsoft Windows 2000 SP3, Server version Microsoft Windows 2000 SP3, Advanced Server version Microsoft Windows 2000 SP3, Data Center version The Crystal Enterprise Automated Process Scheduler (APS) component requires a database to store information about the system and its users. By default, the Setup program installs and configures its own Microsoft Data Engine (MSDE) database if necessary. The MSDE is a client/server data engine that provides local data storage and is compatible with Microsoft SQL Server. Crystal Enterprise Professional for Tivoli APS clustering is automatically supported by the default MSDE database. If you already have the Microsoft Data Engine or Microsoft SQL Server installed, the installation program creates the APS database using your existing database engine. You can migrate this initial APS database to another database server later. Supported servers for APS database migration are: Microsoft SQL Server 7 SP4 (ODBC) Microsoft SQL Server 2000 SP2 (ODBC) Microsoft MSDE (ODBC) Oracle 9i (native) Oracle 8i (8.1.7) (native) IBM DB2 UDB 7.2 (native) Sybase Adaptive Server 12.5 (ODBC and native) Informix® Dynamic Server 2000 v 9.21 (ODBC) Chapter 2. Planning for Tivoli Data Warehouse 1.2 33
    • Note: if the Microsoft Data Engine (MSDE) or Microsoft SQL Server is already installed on the local machine, you must set up a user account for the Crystal Enterprise Professional for Tivoli APS before installing Crystal Enterprise Professional for Tivoli, as follows: 1. Determine whether the Crystal Enterprise Professional for Tivoli APS should use Windows NT or Microsoft SQL Server authentication when connecting to your local database installation. 2. Using your usual administrative tools, create or select a user account that provides Crystal Enterprise with the appropriate privileges to your database server: – If you want the APS to connect to its database using Windows NT authentication, ensure that the Windows NT user account that you assign to the APS has System Administrator’s role in your SQL Server installation. In this scenario, the Windows NT user account that you assign to the APS is not actually used to create the system database during the installation process. Instead, your own Windows NT administrative account is used to create the database, so verify that your Windows NT account also has the System Administrator’s role in your SQL Server installation. – If you want the APS to connect to its database using SQL Server authentication, the login that you assign to the APS must belong to the Database Creators role in your SQL Server installation. In this scenario, the SQL Server credentials that you assign to the APS are also used to create the database and its tables. 3. Verify that you can log on to SQL Server and carry out administrative tasks using the account you set up for use by the APS. For details about APS database migration, see “Configuring the intelligence tier” in the Crystal Enterprise 9 Administrator’s Guide manual, which is shipped with the product. Note: For a detailed list of environments tested with Crystal Enterprise, please consult the Platforms.txt file included with your product distribution. In case you have acquired a Crystal Enterprise 9 Special Edition license and choose to deploy the server-side installation method using the Crystal server connected to an external Web server, you also need to install and configure the appropriate Web Connector on your Web server machine. The supported Web servers are listed in Table 2-4.34 Implementing Tivoli Data Warehouse 1.2
    • Table 2-4 Web servers and OS supported by Crystal Web Connector Web server Operating systems Microsoft IIS5 / ISAPIY Windows 2000 Server Microsoft IIS5 / CGIY Microsoft IIS4 / ISAPI-Y Windows NT 4.0 Server Microsoft IIS4 / CGI-Y iPlanet 6.0 SP3 / NSAPI Windows 2000 Server iPlanet 6.0 SP3 / CGI Windows NT 4.0 Server iPlanet 4.1 SP10 / NSAPI Sun Solaris 2.7, 2.8 iPlanet 4.1 SP10 / CGI Domino 5.0.8 / DSAPI Windows 2000 Server Windows NT 4.0 Server Domino 5.0.8 / CGI Windows 2000 Server Windows NT 4.0 Server Sun Solaris 2.7, 2.8 IBM AIX 4.3.3, 5.1 Apache 1.3.26 / ASAPI Sun Solaris 2.7, 2.8 Apache 1.3.26 / CGI RedHat 6.2/7.3 (x86) SuSe 7.3/8.0 (x86) IBM AIX 4.3.3, 5.1 IBM HTTP 1.3.19.2 / ASAPI IBM AIX 4.3.3, 5.1 IBM HTTP 1.3.19.2 / CGI Windows 2000 Server IBM AIX 4.3.3, 5.1Web browser requirementsCrystal Enterprise Professional for Tivoli supports the following Web browsers: Microsoft Internet Explorer 6.0 (Windows) Microsoft Internet Explorer 5.5 SP2 (Windows) Netscape 6.2.3 (Windows) Microsoft IE 5.0 on OS9 or OS X’s Classic mode (Macintosh) Note: Crystal Enterprise delivers reports and analytic content to any supported browser using pure DHTML. Chapter 2. Planning for Tivoli Data Warehouse 1.2 35
    • 2.2 Physical and logical design considerations In the previous section we have briefly introduced the basic components of the Tivoli Data Warehouse 1.2 application. Now we discuss how those components can be installed on different servers and what are the advantages or the possible drawbacks of the different configurations. All the possible architectures for a Tivoli Data Warehouse implementation are determined by the location of the following components: Control server Central data warehouse Data marts Crystal Enterprise server Warehouse agents Source databases All these components can coexist on the same server or they can be spread out on different servers. See Table 2-5 to determine which operating systems are the platforms required by each Tivoli Data Warehouse component. Table 2-5 Requirements for Tivoli Data Warehouse components Component Supported operating DB2 components systems platforma required Control server Windows IBM DB2 Universal Database Enterprise Edition Central data warehouse Windows, AIX, Solaris, IBM DB2 Universal databases z/OS Database Enterprise Edition IBM DB2 Warehouse Manager (optional) Data mart databases Windows, AIX, Solaris, IBM DB2 Universal z/OS Database Enterprise Edition IBM DB2 Warehouse Manager (optional) Crystal Enterprise server Windows IBM DB2 Universal Database Enterprise Edition or IBM DB2 database client36 Implementing Tivoli Data Warehouse 1.2
    • Component Supported operating DB2 components systems platforma required Warehouse agents Windows, AIX, Solaris IBM DB2 Universal Database Enterprise Edition IBM DB2 Warehouse Manager a. See the Tivoli Data Warehouse Release Notes, SC32-1399 for complete details about supported operating systems.2.2.1 Source databases The source databases are the fundaments of a data warehouse implementation. Their health is of crucial impact to the whole Tivoli Data Warehouse environment. From an administrative point of view, source databases can be divided into two types: The first type of source database is part of the product from which data should be collected. An example is the IBM Tivoli Storage Manager (ITSM), where the source database is the internal ITSM database. In this case, the application administrator should be responsible for this source database. The second type, such as the IBM Tivoli Monitoring (ITM) source database, is built for the purpose of a source database exclusively. In this sense, you may consider these as part of the data warehouse infrastructure.2.2.2 Control server The control server is the system that contains the control database for Tivoli Data Warehouse and from which you manage your data warehouse environment. Only one control server can be installed in a Tivoli Data Warehouse environment and it must be installed on a Windows platform. Before installing the control server, you must install IBM DB2 Universal Database Enterprise Edition 7.2 locally and have the Crystal Enterprise environment up and running on the same or on a different server. You must also install IBM DB2 Universal Database Enterprise Edition 7.2 or DB2 Universal Database for OS/390 and z/OS V7 on each computer that will contain a central data warehouse database or a data mart database. Chapter 2. Planning for Tivoli Data Warehouse 1.2 37
    • The control server creates a control database (TWH_MD) which contains descriptions of the stored data (known as metadata) for both Tivoli Data Warehouse and for the warehouse management functions. The control servers uses the DB2 Data Warehouse Center to automate the data warehouse processing, to define the ETL processes that move and transform data into the central data warehouse and the data marts. The control server runs also a warehouse agent, the component of IBM DB2 Warehouse Manager that manages the data flow between warehouse sources and targets. In advanced Tivoli Data Warehouse scenarios, you can move the warehouse agent to other locations (see “Warehouse agents” on page 49). When using the configuration with the warehouse agent on the control server, the computer on which you install the control server must also connect to the operational data stores of your enterprise, which potentially reside on other systems and in relational databases other than IBM DB2. To enable the control server to access these data sources, you must install the appropriate database client for each data source on the control server system.2.2.3 Central data warehouse The central data warehouse is a set of IBM DB2 databases containing the historical data for your enterprise. The central data warehouse does not require any Tivoli Data Warehouse software or DB2 Warehouse components. You may choose to install a warehouse agent on the same server containing the central data warehouse to improve the performance of data transfer between warehouse databases (refer to “Warehouse agents” on page 49). With Tivoli Data Warehouse 1.2, it will support up to four central data warehouse databases. You can have one or more central data warehouse databases in a Tivoli Data Warehouse deployment on Windows and UNIX. On z/OS you can have only one central data warehouse database (for example, see Figure 2-7 on page 49). The central databases can be installed in the following configurations: Only 1 central data warehouse database on Windows and UNIX platforms or 1 central data warehouse database on a z/OS system Up to 4 central data warehouse databases on Windows and UNIX platforms Up to 3 central data warehouse databases on Windows and UNIX platforms and 1 central data warehouse database on a z/OS system38 Implementing Tivoli Data Warehouse 1.2
    • Note: If you plan to install warehouse packs that were created to run on Tivoli Enterprise Data Warehouse 1.1, you need to create at least one central data warehouse database on a Windows or UNIX system. These warehouse packs use only the first central data warehouse database that is created on a Windows or UNIX system. This central data warehouse database must be named TWH_CDW. A central data warehouse database on a z/OS system can be populated only by warehouse packs developed for Tivoli Data Warehouse 1.2.On Windows and UNIX platforms, the first central data warehouse databasecreated by Tivoli Data Warehouse 1.2 is named TWH_CDW. Subsequent centraldata warehouse database will be named TCDW1, TCDW2, and TCDW3.On z/OS systems, there are no hard specifications on the name used for thecentral data warehouse database. However, it is a good practice to follow thenaming convention adopted by Tivoli Data Warehouse 1.2.Multiple central data warehouse databases might be useful in the followingsituations: If your Tivoli Data Warehouse deployment contains systems in widely separated time zones or geographies: A central data warehouse ETL typically runs during off-peak hours to avoid impacting the performance of your operational data stores: Having central data warehouse databases located on servers in different time zones enables you to schedule ETLs for each system at an appropriate off-peak time. If your deployment includes z/OS systems. Warehouse packs that use data extracted from z/OS data sources must load their data into a data mart database on a z/OS system. In contrast, warehouse packs that use operational data stores from Windows or UNIX systems can load that data into a data mart database on any supported operating system. Therefore, when you have sources on z/OS and on distributed systems, you must have at least one central data warehouse database on a z/OS system (see Figure 2-5 on page 47 and Figure 2-6 on page 48). You may optionally choose to have also a second central data warehouse database on a distributed system in order to keep distributed applications data completely separate from z/OS applications data (see Figure 2-7 on page 49). Chapter 2. Planning for Tivoli Data Warehouse 1.2 39
    • If you want to distribute the central data warehouse workload. When using different warehouse packs that do not provide cross-application reports, you can have each warehouse pack load its data into separate central data warehouse databases. This allows you to schedule the central data warehouse ETLs for both warehouse packs to run at the same off-peak time without causing database performance problems. When planning to use multiple central data warehouse databases, consider the following information: If you use a set of warehouse packs that collect historical data intended for the same reporting purpose, all of the warehouse packs must write their data into the same central data warehouse database. Note that if a warehouse pack supports extracting data from multiple central data warehouse databases, its documentation contains information about the placement of the central data warehouse databases. Distributed application data may flow through a central data warehouse database either on a z/OS or on a distributed system into a data mart database either on a z/OS or on a distributed system. z/OS application data can flow only through a central data warehouse database on a z/OS system into a data mart database on a z/OS system. Important: Although it is possible for a data analysis program to read data directly from central data warehouse databases without using data marts, this is highly not recommended and not supported. Analyzing historical data directly from the central data warehouse database can cause performance problems for all applications using the central data warehouse.2.2.4 Data marts The data marts for your enterprise are contained in a separate set of IBM DB2 databases. Each data mart contains a subset of the historical data from the central data warehouse that satisfies the analysis and reporting needs of a specific department, team, customer, or application. With Tivoli Data Warehouse 1.2, there are up to four data mart databases supported. You can have one or more central data warehouse databases in a Tivoli Data Warehouse deployment on Windows and UNIX. You can have one or more data mart databases for each central data warehouse deployed on Windows, UNIX, and z/OS systems.40 Implementing Tivoli Data Warehouse 1.2
    • On z/OS systems, Tivoli Data Warehouse 1.2 only supports one data martdatabase per IBM DB2 subsystem. In addition to that, the central datawarehouse and data marts must be in the same IBM DB2 subsystem, and theIBM DB2 subsystem must have an unique location name.On Windows and UNIX platforms, the first data mart database created by TivoliData Warehouse 1.2 is named TWH_MART. Subsequent central datawarehouse database will be named TMART1, TMART2, and TMART3.On z/OS systems, there are no hard specifications on the name for the data martdatabase. However, it is a good practice to follow the naming conventionadopted by Tivoli Data Warehouse 1.2.Each data mart database can contain the data from multiple central datawarehouse databases.The data mart databases do not require any Tivoli Data Warehouse software orDB2 Warehouse components, but you may choose to install a warehouse agenton the servers containing the data mart databases to improve the performance ofdata transfer from central data warehouse databases (refer to “Warehouseagents” on page 49) Note: If you plan to install warehouse packs that were created to run on Tivoli Enterprise Data Warehouse 1.1, you need to create at least one data mart database on a Windows or UNIX system. These warehouse packs use only the first data mart database that is created on a Windows or UNIX system (TWH_MART). A data mart database on a z/OS system can be populated only by warehouse packs for Tivoli Data Warehouse 1.2.Multiple data mart databases might be useful in the following situations: If your deployment includes z/OS systems. Warehouse packs that use data extracted from z/OS data sources must load their data into a data mart database on a z/OS system. In contrast, warehouse packs that use operational data stores from Windows or UNIX systems can load that data into a data mart database on any supported operating system. You might optionally place a data mart database on a Windows or UNIX system to keep data from those systems separate from data from z/OS applications (see Figure 2-6 on page 48). Chapter 2. Planning for Tivoli Data Warehouse 1.2 41
    • If you want to store your enterprise data in different database for security reasons. You can allow each user to access only to the data mart database containing the information which that user is authorized to examine. If you plan to access data marts using different reporting or data analysis programs. You can format the data and you can tune each data mart database according to the programs that is used to analyze it and the expected workload. For an effective planning of data mart databases locations, you should consider these requirements: Each warehouse enablement pack provides its own data structure called star schema. A single data mart database can contain many star schemas. Data from different warehouse enablement packs can be stored in the same data mart database. Each using its separate star schema. Each warehouse pack can write to only one data mart database, and it must pull all of the data for the data mart from a single central data warehouse database. Different star schemas in one data mart database can pull their data from different central data warehouse databases. Data mart databases on a Windows or UNIX system cannot pull z/OS applications data while a data mart database on z/OS can receive data coming all supported platforms (z/OS, UNIX and Windows).2.2.5 Single machine installation You can install all components of Tivoli Data Warehouse on one single machine (see Figure 2-1). This configuration uses the quick start installation method. It is easy to set up and to maintain but is recommended only for test or demonstration environments. It is also useful for those who do not plan to have many users concurrently accessing the data stored in the Tivoli Data Warehouse database and/or those who do not need to capture and analyze large amounts of data. The control server must be on Windows NT, Windows 2000 Server, Windows 2000 Advanced Server, or Windows Server 2003. Thus the single machine configuration cannot be available on UNIX or z/OS systems.42 Implementing Tivoli Data Warehouse 1.2
    • TDW environment Crystal Enterprise (Windows) (Windows) Web Reports Control Server Crystal Metadata Server Data source Web Central Data Warehouse Data Mart ServerFigure 2-1 Single machine installation2.2.6 Distributed deployment on UNIX and Windows servers Most production environments usually require that the Tivoli Data Warehouse components be installed on different servers, in order to distribute workload or leverage preexisting infrastructures. Next we consider some typical scenarios for UNIX and Windows environments: All Tivoli Data Warehouse components on different Windows or UNIX servers: The control server is on a Microsoft Windows server, a central data warehouse database and data mart database on separate large server-class computers (Windows or UNIX), and Crystal Enterprise on a Microsoft Windows server, as seen in Figure 2-2. Chapter 2. Planning for Tivoli Data Warehouse 1.2 43
    • Windows Control Server system Metadata Windows or UNIX Windows or UNIX system system Central Data Warehouse Data Mart Data source Web Reports Data source Windows system Data source Crystal Server Web ServerFigure 2-2 Distributed deployment on Windows and UNIX systems Central data warehouse and source databases on the same server: The central data warehouse database is on the same computer as the database containing the operational data sources. The control server and Crystal Enterprise are on two different Microsoft Windows servers, as seen in Figure 2-3. Because operational data sources usually have a high rate of transactions per hour, it is not recommended to share the same IBM DB2 server for data source and data marts: This configuration may increase the time needed to obtain reports from data marts.44 Implementing Tivoli Data Warehouse 1.2
    • On the other hand, it is possible to have a common IBM DB2 server for data sources and central data warehouse without affecting the performances whenever the ETL1 and ETL2 can be scheduled in off-peak times. Windows Control Server system Metadata Windows or UNIX system Windows or UNIX system Data source Data source Data Mart Central Data Warehouse Web Reports Windows system Crystal Server Web ServerFigure 2-3 Operational data sources and the CDW databases on the same server2.2.7 Distributed deployment on z/OS, UNIX, and Windows servers Next we describe some examples of Tivoli Data Warehouse 1.2 architecture design scenarios that also include z/OS system management data sources: Chapter 2. Planning for Tivoli Data Warehouse 1.2 45
    • Data sources, central data warehouse and data mart database on a z/OS system: The central data warehouse and the data mart database are in a IBM DB2 UDB for OS/390 and z/OS. The control server and Crystal Enterprise are on two different Microsoft Windows servers, as seen in Figure 2-4. This kind of configuration is typically used when all management data source comes from z/OS applications. The reason is that all warehouse enablement packs extracting data from z/OS data sources must load their data into a central data database located at the Z/OS system. z/OS Source system Source Control Server Central Data (Windows) Warehouse Data Mart TDW environment Windows system Crystal Web Reports Server Web ServerFigure 2-4 Data sources, CDW, and data mart databases on a z/OS system Operational data sources both on z/OS and on distributed systems: You can transfer data to a central data warehouse database on a z/OS system even if your operational data sources are distributed among z/OS and distributed systems, as seen in Figure 2-5. In this configuration you can use only warehouse enablement packs for Tivoli Data Warehouse 1.2, because those warehouse enablement packs for version 1.1 do not allow any data transfer to a central data warehouse located on a z/OS system.46 Implementing Tivoli Data Warehouse 1.2
    • The common data mart database on z/OS provides the reports for both data sources on z/OS and distributed systems. You may choose to have the common data mart database on a distributed system instead of z/OS, but in that case you cannot have any reports from operational data sources on z/OS, as seen in Figure 2-6. Windows system Control Server Metadata Z/OS system Windows or UNIX system Data source Data source Central Data Data Mart Data source Warehouse Web Reports Windows system Crystal Server Web ServerFigure 2-5 Operational data sources both on z/OS and on distributed systems Separate data mart databases on a z/OS and on a distributed system: This configuration is typically used to segment the reporting functions into two logical areas, one for the z/OS and the other for the distributed environment. All z/OS application data flows through the central data warehouse database to the data mart database on z/OS, while the distributed applications data is transferred to a data mart on a distributed system. Chapter 2. Planning for Tivoli Data Warehouse 1.2 47
    • Windows system Control Server Metadata Windows or UNIX Z/OS system system Data source Data source Data source Data source Data Mart2 Data Mart Central Data Warehouse Windows or UNIX Web Reports system Windows system Data Mart1 Data Mart Crystal Server Web ServerFigure 2-6 Separate data mart databases on z/OS system and distributed system Two central data warehouse servers, one on z/OS, and one on a UNIX or Windows system: This is a more complex deployment with one central data warehouse and one data mart database in a DB2 UDB for OS/390 and z/OS, a second central data warehouse database on a UNIX or Windows server, and the control server and Crystal Enterprise server on separate Windows systems, as seen in Figure 2-7. This configuration may be chosen by customers who want to keep completely separate z/OS applications data from distributed applications data.48 Implementing Tivoli Data Warehouse 1.2
    • Windows or UNIX system Windows Control Server system Data source Metadata Data source Z/OS system Data source Central Data Data source Warehouse 1 Data Mart2 Data Mart Windows or UNIX Central Data system Warehouse 2 Windows Web Reports system Data Mart1 Data Mart Crystal Windows or UNIX Server system Web ServerFigure 2-7 Two CDWs on a Windows or UNIX system and on a z/OS system2.2.8 Warehouse agents Warehouse agents are IBM DB2 Warehouse Manager components that manage the flow of data between warehouse sources and targets using DB2 CLI or Open Database Connectivity (ODBC) drivers to communicate with different databases. Every system having a warehouse agent installed is called an agent site or a remote agent. Tivoli Data Warehouse supports warehouse agents on Windows and UNIX operating systems, but not on z/OS. Chapter 2. Planning for Tivoli Data Warehouse 1.2 49
    • When you install the Tivoli Data Warehouse control server, a warehouse agent is automatically installed on the Control server machine. In the basic configuration, shown in Figure 2-8, the control server uses its local warehouse agent to manage data flow from the operational data sources to the central data warehouse (ETL1 process) and from that to the data marts (ETL2 process). In case the Tivoli Data Warehouse databases are located on the same system as the control server, the warehouse agent is not used. Central Data Warehouse ETL1 ETL2 ETL1 ETL2 Source Warehouse Data Mart Agent Warehouse Server (Control Server) Figure 2-8 Warehouse agent on control server In a distributed scenario, as shown in Figure 2-9, you might improve the performance of Tivoli Data Warehouse by placing a warehouse agent on each central data warehouse server and data mart server. These remote warehouse agents allow a straight data flow from target to source without passing through the control server, reducing the workload on this server and increasing the speed of data transfer.50 Implementing Tivoli Data Warehouse 1.2
    • Agent site Warehouse Agent Central Data Warehouse ETL2 Data Warehouse ETL1 Agent Data Source Data Mart Agent site Warehouse Server (Control Server)Figure 2-9 Warehouse agents on data targetsTypically the warehouse agent is placed on the target of a data transfer. In thisconfiguration, the warehouse agent performs the following tasks: Passes SQL statements that extract data from the remote source tables Transforms the data if required Writes the data to the target table on the local databaseThis configuration offers the best performance in a distributed environment byoptimizing the DB2 data flow and using block fetching for the extraction. This isthe recommended configuration to use with Tivoli Data Warehouse when sourceand target are on physically separate machines.The warehouse agent may be installed on the system that contains the sourcedatabase. In this configuration, the agent: Passes SQL statements that extract data from the local source tables. Transforms the data if required. Writes the data to the target table on the remote database.This alternative configuration does not optimize the DB2 data flow (DistributedRelational Database Architecture DRDA® or DB2 private protocols) and shouldbe used only if justified by specific architecture requirements. For example, ifyour data source and data mart databases are on distributed systems while thecentral data warehouse database is on a Z/OS system, you are forced to placewarehouse agents on the distributed systems, one of which is the source of datatransfer, as seen in Figure 2-10. Chapter 2. Planning for Tivoli Data Warehouse 1.2 51
    • z/OS system Central Data Warehouse ETL2 Warehouse ETL1 Warehouse Agent Agent Source Data Mart Agent site Agent site Warehouse Server (Control Server) UNIX or Windows UNIX or Windows system systemFigure 2-10 Configuration with a warehouse agent on the source To improve the performance of your ETL process, you should carefully plan where to place agent sites in your environment and which site to associate with each ETL. Table 2-6 suggests where to place the warehouse agents when transferring data from data sources to the central data warehouse database in different scenarios. Table 2-6 Agent sites placement for data transfers to a central data warehouse Operational data source Central data warehouse Warehouse agent location database location location A Windows or UNIX A different Windows or Central data warehouse system UNIX system system The same system No agent required A z/OS system Operational data source system52 Implementing Tivoli Data Warehouse 1.2
    • Operational data source Central data warehouse Warehouse agent location database location location z/OS system The same z/OS location No agent required A different z/OS location Control server A Windows or UNIX Deployment not system supported. Data sources on z/OS can load data only in a central data warehouse on z/OS.Table 2-7 suggests where to place the warehouse agents when transferring datafrom central data warehouse databases to data mart databases in differentscenarios.Table 2-7 Where to place agent sites for data transfers to data marts Central data warehouse Data mart database Warehouse agent database location location location a Windows or UNIX A different Windows or Data mart system system UNIX system The same system No agent required A z/OS system Central data warehouse system z/OS system The same z/OS location No agent required A different z/OS location Control server A Windows or UNIX Data mart systemNote that Tivoli Data Warehouse automatically recognizes when the source andtarget data are on the same computer, and in that case it transfers the datawithout using the warehouse agent.Here are some common situations in which data transfer does not use thewarehouse agent: When using operational data sources, central data warehouse, and data mart in the same IBM DB2 location on a z/OS system. When the operational data and the central data warehouse database are on the same computer running Windows or UNIX. When transferring data between a central data warehouse database and data mart database on the same computer running Windows or UNIX. Chapter 2. Planning for Tivoli Data Warehouse 1.2 53
    • To install warehouse agents, you must install IBM DB2 Warehouse Manager and Tivoli Data Warehouse on each machine that will be an agent site. If you are using operational data stored in databases other than IBM DB2 (Oracle, Informix etc.) you are also required to install on that computer a database client for each type of remote database that the agent needs to access. For example, if the operational data source for a warehouse pack is an Oracle database on another computer, you must install also an Oracle database client on the agent site. For more information about warehouse agents, refer to the IBM DB2 Warehouse Manager Installation Guide, and the redbook, DB2 Warehouse Management: High Availability and Problem Determination Guide, SG24-6544-00.2.2.9 Considerations about warehouse databases on z/OS As already explained in the previous sections, there are some limitations when placing Tivoli Data Warehouse databases on z/OS. Here is a summary: Only one central data warehouse database can be placed on a z/OS system. Data from applications on z/OS or OS/390 systems can be placed only in central data warehouse and data mart databases in DB2 UDB for OS/390 and z/OS. To extract operational data from a Tivoli Decision Support for OS/390 database, the central data warehouse database and the Tivoli Decision Support for OS/390 database must be in the same DB2 subsystem. The central data warehouse database and data mart database can be a storage group that is shared with other applications. Only one data mart can be installed per single IBM DB2/390 location (same host and port). Tivoli Data Warehouse 1.2 do not support warehouse agents on z/OS systems. Before installing a Tivoli Data Warehouse database on a z/OS system, you should know the following parameters about the existing IBM DB2 installation: Host name of the z/OS system TCP/IP port configured for IBM DB2 Database location Storage volume, storage group, and storage VCAT Buffer pool Tablespace name UTF8 tablespace name Tablespace size (primary and secondary)54 Implementing Tivoli Data Warehouse 1.2
    • 2.2.10 Coexistence with other products This section describes coexistence issues between Tivoli Data Warehouse and other applications such as Tivoli Decision Support (distributed), Tivoli Decision Support for OS/390, and other IBM DB2 databases and data warehouses: Coexistence with Tivoli Decision Support: You can continue to use Tivoli Decision Support at the same time and on the same systems as Tivoli Data Warehouse. For the best performance, do not schedule Tivoli Data Warehouse ETLs and Tivoli Decision Support cube builds concurrently, if they access the same databases. Coexistence with Tivoli Decision Support for OS/390: You can continue to use Tivoli Decision Support for OS/390 at the same time and on the same systems as Tivoli Data Warehouse. For the best performance, schedule Tivoli Data Warehouse ETLs after Tivoli Decision Support for OS/390 data collection has occurred. Coexistence with other data warehouses: Do not mix components from different deployments of Tivoli Data Warehouse on any computer. For example, if you can have both a production deployment and a test deployment, you must not place any warehouse components of the test deployment on the same computer as any components of the production deployment. You can use the IBM DB2 Data Warehouse Center to manage multiple data warehouses. However, only one data warehouse can be active (running scheduled jobs) at a time. Scheduled jobs run only for the warehouse whose control database (TWH_MD in case of Tivoli Data Warehouse) is selected in the IBM DB2 Data Warehouse Center. Other warehouses, each identified by a different control database, retain their information but do not run scheduled jobs until their control database is selected in the IBM DB2 Data Warehouse Center. Coexistence with other IBM DB2 database applications: You can have other applications using the same IBM DB2 instances used by Tivoli Data Warehouse, provided that they do not use the same database names required by warehouse components (TWH_MD, TWH_CDW, TCDW1, TCDW2, TCDW3, TWH_MART, TMART1, TMART2, TMART3, and any other database names defined on z/OS systems). If you install Tivoli Data Warehouse using an existing IBM DB2 instance, all other clients that are connected to that instance lose their IBM DB2 connections during the installation process. Therefore, remember to stop all existing connection before starting Tivoli Data Warehouse installation. Chapter 2. Planning for Tivoli Data Warehouse 1.2 55
    • 2.2.11 Selecting port numbers You must allocate port numbers for Tivoli Data Warehouse for communication between the Tivoli Data Warehouse control server and the remote IBM DB2 databases (data sources, central data warehouse, data marts) and for communications with Crystal Enterprise. Table 2-8 lists the default port numbers that are used by Tivoli Data Warehouse. You can use the same or different port numbers for the control server, data mart database computers, and central data warehouse database computers. Table 2-8 Default port used in Tivoli Data Warehouse environments Component Default port DB2 databases on UNIX and Windows 50000 DB2 databases on z/OS 5000 Crystal Enterprise server 6400 Internal communication between the Crystal APS and other 6401 Crystal components, such as the Crystal Web Connector running on the HTTP Server HTTP server for Crystal 80 Enterprise2.3 Database sizing A correct database sizing is one of the most important issues you have to face when planning a Tivoli Data Warehouse deployment. The growth rate of your central data warehouse and data mart databases is related to the number of measured items in your environment (hosts, database instances, events and so on), the measurements you require for each item and the sampling frequency. You may find detailed instructions for estimating the amount of required storage in the implementation guide of each warehouse pack you install in your environment. If the implementation guides do not provide this kind of information, refer to the planning chapter of Installing and Configuring Tivoli Data Warehouse, GC32-0744-02. In this publication, some calculations are illustrated for estimating the central data warehouse database storage requirements for a warehouse pack.56 Implementing Tivoli Data Warehouse 1.2
    • You can also find useful information about database sizing in the redbook Planning a Tivoli Enterprise Data Warehouse Project, SG24-6608, even if the estimates indicated in that redbook are for warehouse enablement packs running on Tivoli Enterprise™ Data Warehouse version 1.1.2.4 Security The following sections describe security considerations for installing and using Tivoli Data Warehouse.2.4.1 Authority required to install and maintain IBM DB2 UDB For UNIX machines, the user must be logged in with root authority. User and group accounts will be created for the instance owner, Administration Server, fenced user defined functions, and fenced stored procedures. It is recommended that you use different user IDs for UDFs and the instance owner. For Windows NT and 2000 machines, the user must be part of the administrators group. Specifically, the user must have these advanced user rights: Act as part of the operating system Create token object Increase quotas Replace a process level token More detailed user account information can be obtained from the DB2 UDB Quick Beginnings EEE for NT V7, GC09-2963 or DB2 UDB Quick Beginnings EEE for UNIX V7, GC09-2964.2.4.2 Authority required to install Tivoli Data Warehouse On Windows systems, to install Tivoli Data Warehouse or to install, start, or stop the Crystal Enterprise Professional for Tivoli server, you must be logged on as a locally defined administrator. On UNIX-based systems, to install Tivoli Data Warehouse you must be logged as the root user. On z/OS systems, to add or remove a central data warehouse database or data mart SYSADM authority is required, while DBADM authority on central data warehouse and data mart databases is required to install warehouse packs. Chapter 2. Planning for Tivoli Data Warehouse 1.2 57
    • 2.4.3 Firewalls You cannot have a firewall between any Tivoli Data Warehouse components. This includes the control server and the Data Warehouse Agent sites. However, it is possible to have configurations with a firewall between source databases and a central data warehouse, or a firewall between a central data warehouse and data marts, if the source database vendor supports communication through firewalls, and the Data Warehouse Agent resides on the central data warehouse. Figure 2-11 shows both a valid configuration and a non-functional configuration. Only the ODBC communication can be transported through the firewall. Warehouse Warehouse Control Server Control Server Warehouse Warehouse Agent Agent Central Data Central Data Source Source Warehouse Warehouse Firewall Firewall ODBC Data Data warehouse Control DataFigure 2-11 Tivoli Data Warehouse and firewalls Crystal Enterprise can deliver a broad range of reporting and analytic content to any browser using pure DHTML. Unlike plug-in based technologies, DHTML requires no software downloads and no special configuration to enable viewing, making it ideal for deployments through a firewall without compromising security.58 Implementing Tivoli Data Warehouse 1.2
    • 2.4.4 Controlling access to data in the warehouse The following techniques can be used to control access to data in Tivoli Enterprise Data Warehouse: Restricting which DB2 users have access to specific views in the central data warehouse and in the data mart database. This is done using DB2 administrative tools. Using multicustomer or multicenter support, as described in “Multicustomer and multicenter support”2.4.5 Protecting information in Crystal Enterprise Professional for Tivoli Crystal Enterprise Professional for Tivoli allows you to choose one of three types of user authentications: Enterprise Windows NT LDAP The Enterprise authentication is based on Crystal proprietary internally defined accounts and groups used only for Crystal Enterprise. The Windows NT authentication uses already existing NT and 2000 user accounts and groups for authentication in Crystal Enterprise Professional for Tivoli, eliminating the need to create new additional accounts within the application. The LDAP authentication can be chosen if you have already an LDAP directory server in your environment. In this case, you can use existing LDAP user accounts and groups also for Crystal Enterprise Professional for Tivoli. Crystal Enterprise Professional for Tivoli not only authenticates its users, but also allows an access control to all the objects within the application using the Crystal Management Console (CMC). The base unit for controlling consists of specific object rights that provide a user or a group with permission to perform a particular action on an object. Each object right can be Explicitly Granted, Explicitly Denied, or Not Specified. The Crystal Enterprise Professional for Tivoli object security model is designed according a “denial base” in order to guarantee that users and groups do not automatically acquire rights that are not explicitly granted. All rights that are not specified are denied by default. Additionally, in case of contradictory settings that grant and deny the same right to the same user or group, the right is denied by default. Chapter 2. Planning for Tivoli Data Warehouse 1.2 59
    • Crystal Enterprise includes a set of predefined access levels that allow you to set common security levels quickly and facilitate administration and maintenance. Each access level grants a set of rights that combine to allow users to accomplish common tasks such as viewing reports, scheduling reports, etc. Users can inherit rights as the result of group membership, subgroups can inherit rights from parent groups, and both users and groups can inherit rights from parent folders. However, you can always disable inheritance or customize security levels for particular objects, users, or groups. Refer to the manual, Crystal Enterprise Professional Version 9 for Tivoli Administrator’s Guide, which is provided with Crystal Enterprise Professional Version 9 for Tivoli, for further information about security aspects of Crystal Enterprise Professional for Tivoli.2.4.6 Multicustomer and multicenter support Tivoli Data Warehouse allows you to keep data about multiple customers and centers in one central data warehouse, while restricting access so that customers can see and work with data and reports based only on their data and not any other customer’s data. Important: If you intend to have multicustomer and multicenter support in your Tivoli Data Warehouse, you must plan and configure for this before importing any data into your central data warehouse. If you decide to enable multicustomer and multicenter support after you have already imported some data, you must set up a new central data warehouse for the new multicustomer environment and, if necessary, migrate your old data into the new environment. In order to support a multicustomer environment, you must be sure that your application data correctly discriminates between different customers. Various applications can use different data fields to distinguish between customers, but within applications, you should make the data fields consistent. You must determine which field in a data source is applicable for identifying multiple customer data. An application might use a single data field to determine customer uniqueness, while other applications might use two or more data fields to distinguish customer data. Either case is permitted as long as the data fields used are consistent within a single application.60 Implementing Tivoli Data Warehouse 1.2
    • When the central data warehouse ETL is run, data from applications is assigned valid customer account codes by matching certain data fields in the incoming data with pre-identified values in a matching database table. Because each application can use different fields and different numbers of fields to identify customers, each application has its own matching table that it uses during the central data warehouse ETL process. To configure the central data warehouse for multicustomer functionality, you must manually define your customers into the TWG.CUST and Product_Code.CUST_LOOKUP tables, where Product_Code is the code univocally associated to the warehouse pack you intend to use. In the same way, you can configure the multicenter support simply defining your centers into the TWG.CENTR and Product_Code.CENTR_LOOKUP tables on the central data warehouse database. See the multicustomer and multicenter support information in the manual, Enabling an Application for Tivoli Data Warehouse, GC32-0745-02, for details on how to configure the central data warehouse for multicustomer or multicenter support.2.5 Network traffic considerations An accurate planning of a Tivoli Data Warehouse environment should also consider the possible impact on the network. When the ETLs run, large amounts of data are transferred, and this can greatly affect the network performance. However, a correct scheduling of the ETLs and a proper design of Tivoli Data Warehouse implementation can reduce the impact on the network. Note: The ETLs are based on Structured Query Language (SQL). The SQL processing exhibits a typical request/response pattern, meaning that requests send small amounts of data, whereas responses return large amounts of data. This can have an adverse effect on the network. Every customer environment will have unique network requirements. Therefore, Tivoli Data Warehouse must be tailored to fit each customer’s specific needs. In this section, we discuss some of the decisions that a customer will face when planning for Tivoli Data Warehouse. Chapter 2. Planning for Tivoli Data Warehouse 1.2 61
    • 2.5.1 Architectural choices The single machine installation of Tivoli Data Warehouse (Figure 2-1 on page 43) affects the network far less than any other distributed installation (for example, see Figure 2-2 on page 44). Simply put, there is less data transferred across the network in a stand-alone environment than on a distributed environment. On the other hand, a single server cannot handle the entire workload of a Tivoli Data Warehouse for most production environments and then a distributed installation is very often required. In this case the network performance can be improved by using remote warehouse agents and carefully placing the data marts and the Crystal Enterprise Professional for Tivoli server. Remote warehouse agents During a typical distributed installation, a local warehouse agent is installed on the system with the control server. Utilizing the local agent is the easiest and fastest in terms of installation, but this configuration is the least efficient in terms of performance and network traffic. In an advanced distributed Tivoli Data Warehouse, remote warehouse agents are usually located on the target databases. In this setup, communication between the source or target and the control server is minimized. Please refer to 2.2.8, “Warehouse agents” on page 49 for more details on remote warehouse agents and their possible locations. Data marts and Crystal Enterprise server Crystal Enterprise Professional for Tivoli server uses an IBM DB2 client connection to retrieve data from data mart databases. You may reduce the network traffic by simply installing the data mart databases on the same machine where the Crystal Enterprise Professional for Tivoli server is located. On the other hand, this configuration requires that the data mart databases be on a Windows server, while you may prefer to have them on a UNIX or z/OS system.62 Implementing Tivoli Data Warehouse 1.2
    • 2.5.2 Scheduling As stated earlier, ETLs transfer potentially large amounts of data over the network. In a production environment, it is advised not to run these during normal business hours. Instead, ETLs should be scheduled to run when network traffic is low. Production versus maintenance windows The production and maintenance windows should be the first parameters defined when determining when to schedule ETLs. Generally, the production window consists of normal business hours. This usually means the hours representing the highest traffic volume on the network. This is also the time during the day that end users view and run reports against the data marts. ETLs and database backups should not be run during the production window, as they can both negatively impact report performance and other users across the network as well. In contrast, the maintenance window consists of non-business hours, which represent the off-peak hours on the network. ETLs and backups should run during this time. If your environment is spread over wide geographical areas, you may choose to have different central data warehouse and databases for different time zones. This configuration allows you to schedule ETLs separately during each maintenance window in the different time zones (see “Physical and logical design considerations” on page 36). ETLs After establishing the production window, the ETLs may now be scheduled. Once again, different strategies may be implemented to fit the needs of the customer. There are a number of different variables that come into play. Two important variables to consider are the number of warehouse enablement packs (WEPs) and the types of WEPs installed in the Tivoli Data Warehouse. One or two WEPs that extract large amounts of data can take much longer than the combination of many WEPs that extract small amounts of data. If it seems that the amount of time needed to run the ETLs will be longer than the time allocated, a couple of strategies could be implemented. You can refer to the redbook Planning a Tivoli Enterprise Data Warehouse Project, SG24-6608 for estimates about the time required for each ETL to run. This redbook provides information about ETLs times running on a Tivoli Enterprise Data Warehouse version 1.1 environment. Chapter 2. Planning for Tivoli Data Warehouse 1.2 63
    • ETL grouping Another consideration when planning the scheduling of ETLs should be grouping. When you have several warehouse packs installed in your environment, you can use two basically different approaches for ETLs schedule: First run all the ETL1s (from operational data sources to central data warehouse), then all the ETL2s (from central data warehouse to data mart). Run sequentially the ETL1 and ETL2 for each warehouse pack. The first approach will place most of the network load at the beginning of your maintenance window when ETL1s are run. This is because the ETL1s query the application databases and must transfer data across the network when doing so. ETL2s extract data from the central data warehouse and load it into the data marts. If these two databases reside on the same machine, network traffic is minimal. Therefore if the total time of all ETLs runs into the production window, there is relatively little impact on the network. The drawback is that if the ETL2s carry over into the production window, the reports will be unavailable during normal business until the ETLs complete. The second approach guarantees that at least some data marts would be already updated at the start of business even if the total time of all ETLs runs into the production window. The drawback is that an ETL1 might still be running at the start of business, which might put a heavy load on the network at a high peak time. Tip: ETL execution times are largely dependent upon the amount data they query. Always try to schedule ETLs that handle large amounts of data before your ETLs that handle less data in the maintenance window when considering either strategy.2.6 Integration with other business intelligence tools Tivoli Data Warehouse 1.2 natively provides Web reports through Crystal Enterprise Professional Version 9 for Tivoli. However, if you have existing business intelligence infrastructures already in place or if you prefer using other tools to analyze your data, you can easily integrate them with Tivoli Data Warehouse. However, the installation of Crystal Enterprise Professional Version 9 for Tivoli is a prerequisite for Tivoli Data Warehouse 1.2 and is mandatory. You can connect your preferred applications to the data marts and extract your own reports without using Crystal Enterprise Professional for Tivoli. Technically, these applications may report directly from central data warehouse, but this is a non-supported configuration, provides very poor performance, and may interfere with the ETL processes (see Figure 2-12).64 Implementing Tivoli Data Warehouse 1.2
    • Central Data Warehouse Data Data Mart 1 Data Mart 3 Mart 2 Do not connect directly to CDW Crystal Other BI Enterprise Applications Figure 2-12 Business intelligence integration By having only a subset of the data in the data mart, the database system can manage the data faster and easier. Also, since the filtering has already been applied at the ETL2 run time, the reporting queries become much smaller. Smaller queries performing on a small subset of data are, of course, easier to tune. Therefore, the end customer will experience better reporting performance. Refer to the redbook, Tivoli Data Warehouse Report Interfacing, SG24-6084 for details on how to integrate with other business intelligence and reporting tools.2.7 ETL development A typical organization generally relies on very different tools to generate data for its business and stores historical information on separate repositories and in a variety of formats, including relational databases, spreadsheet data, log files, etc. These repositories have different data schemas, since they were designed by different vendors, and they may have metrics specific to that platform. While platform-specific reports are usually required, it is often useful to create enterprise level reports, which mix information from the various platforms. An example of this is to report performance utilization statistics of servers in a single report, regardless of the platform architecture. Chapter 2. Planning for Tivoli Data Warehouse 1.2 65
    • Tivoli Data Warehouse allows mapping of the existing data sources into one or more central data repository the repository, so that reports having a common look and feel can be easily generated, thus improving the understanding of different platforms. In addition to having different data repositories caused by different platforms, different tools are frequently deployed on the same platforms, but on different servers. This is usually the case when these servers are supported by different personnel and then consolidated at some future time (for example, companies that have grown through acquisition, where the servers have come from different companies, or where different support organizations have been allowed to select their own system management tools). By mapping these tools to the central data repository, common reports can be generated, masking the different tools used to create the data. Lastly, one may want to convert existing infrastructure to Tivoli based products. By using Tivoli Data Warehouse, historical data from existing servers can be loaded into the central data repository without any data loss. This will allow one to convert over to a Tivoli product that uses the central data repository and not lose any data. While servers are in the process of converting, both the new and old data sources can be used to generate reports in the new common format. The ETL1 programs take the data from these sources and places it in the central data warehouse, while the ETL2 programs extract from the central data warehouse a subset of historical data that contains data tailored to and optimized for a specific reporting or analysis task. This subset of data is used to create one or more data marts, a subset of the historical data that satisfies the needs of a specific department, team, or customer. A data mart is optimized for interactive reporting and data analysis. The format of a data mart is specific to the reporting or analysis tool you plan to use. Customers can then use Crystal Enterprise Professional for Tivoli or other analysis program to analyze a specific aspect of their enterprise using the data in one or more data marts. Whenever you need to extract data from sources not supported by existing Tivoli warehouse packs or if you decide to use customized data marts, you have to develop your own ETLs. The guide, Enabling an Application for Tivoli Data Warehouse, GC32-0745-02, provides all the information needed to develop customized ETLs.66 Implementing Tivoli Data Warehouse 1.2
    • 2.8 Skills required for a Tivoli Data Warehouse project Implementation of a complete Tivoli Data Warehouse solution requires different skills and usually involves different departments inside a company. A Tivoli Data Warehouse project can be schematically divided into four phases: 1. Implementation 2. Data collection 3. Data manipulation (ETL1 and ETL2) 4. Reporting2.8.1 Implementation The setup of a Tivoli Data Warehouse environment is highly automated and therefore does not require a very specific training. The Tivoli Data Warehouse administrator should have DB2 administrative skills on distributed platform in order to manage and optimize all the database used in the data warehouse. If the managed environment includes also z/OS systems, the administrator should have administrative skills also in DB2 UDB for OS/390 and z/OS or he should be supported by a person with these skills at least during the databases setup on z/OS.2.8.2 Data collection The most important phase of a warehouse project consist of defining all the data sources involved and which is the really relevant data in our business. Generally IT departments are responsible for the collection of metrics concerning availability, performances and utilization of IT resources (systems, applications, networks). Other departments could provide additional data, such as financial statistics or inventories. Skills required to select and actually perform data collection vary according the type of data under scope. Generally a system administrator or an application administrator is directly responsible for the implementation of IT resources monitors or he can provide the information required to implement them. Tivoli software products already offer a wide range of solutions for collecting data about IT infrastructure, but if a company has already implemented different data collection processes before Tivoli Data Warehouse installation, there is no need to modify them. The data manipulation phase is in charge of retrieving data from legacy sources and to convert it into a common format. Chapter 2. Planning for Tivoli Data Warehouse 1.2 67
    • Let us consider the following scenario to explain this further. Suppose that a company plans to store in a Tivoli Data Warehouse all data retrieved by Tivoli monitoring applications as well as by some preexisting and highly customized applications. Tivoli Data Warehouse allows that company to correlate data coming indifferently from Tivoli and non Tivoli sources without affecting the preexisting processes: that can be obtained simply customizing the SQL code necessary to transfer data from the old application database to the central data warehouse (source ETL) and that required to populate the data marts (target ETL) that will be used to generate reports. Please note that the ETLs perform not only a plain data transfer between different databases, but they are also in charge of all the transformation tasks required to have a common format for all data independently from the sources. A typical example of data transformation may be the time stamp of a measurement: each monitoring application generally uses the local time and that could generate confusion whenever we examine data produced in different areas of the world. Therefore the ETLs are also required to convert the times referring to a common standard, such as the Greenwich Mean Time. Another example of data transformation concerns the standardization in the denomination of the measured components: different applications may use different names to indicate the same object and the ETLs must correct any possible mismatch in order to produce always coherent reports also when comparing data from different sources.2.8.3 Data manipulation (ETL1 and ETL2) Once the data sources are established, the next steps are as follows: 1. Loading legacy sources data into central data warehouse and transforming data into a standard format (ETL1 or Source ETL process) 2. Selecting data subsets from a central data warehouse according different business areas, and loading them into data marts (ETL2 process or Target ETL process) 3. Scheduling convenient time frames to run ETL processes Tivoli already provides free ETLs for most Tivoli products, These ETLs can be immediately utilized to manipulate data from Tivoli applications or they can be used as models to create internally developed ETLs to manage other products.68 Implementing Tivoli Data Warehouse 1.2
    • Customized ETLs can be packaged and shipped to customers or colleagues,who can install them using the Tivoli Enterprise Data Warehouse installationwizard. You can find out how to integrate your own components into the TivoliData Warehouse in the manual Enabling an Application for Tivoli DataWarehouse, GC32-0745-02.Source and target ETLs are developed with the same method and slightvariations in design considerations. Skills required to develop ETLs are: Standard SQL Some experience in Data Warehousing and fair amount of DB2 skills Knowledge of source databases involvedETL developers should totally understand their end users’ requirements in orderto project proper data marts, while each data source administrator is requestedto provide ETL developers with the structure of their databases. No interface isprovided to allow users unfamiliar with the data and with SQL to simply movedata from any application’s data store to the data warehouse.In a Tivoli-only environment with Tivoli WEPs, the following skills arerecommended: Knowledge of the source application Knowledge of the used Tivoli product to collect the date (IBM Tivoli Monitoring or IBM Tivoli Storage Manager) Basic Knowledge of Tivoli Data WarehouseTo set up data collection for a new application, the following skills are needed: Knowledge of the source application Knowledge of Standard SQL Knowledge of Tivoli Data Warehouse and its underlying data modelOne of the most critical aspects of the data manipulation phase in a largeenvironment is the scheduling of all the different ETL processes running in aTivoli Enterprise Data Warehouse. The person responsible for scheduling ETLsshould have a thorough knowledge about: The timing of data source updates The requirements of end users for all reports The workload for each server in the Tivoli Data Warehouse environment The impact of ETLs on the networkYou can find a discussion about the last two items in “Network trafficconsiderations” on page 61. Chapter 2. Planning for Tivoli Data Warehouse 1.2 69
    • 2.8.4 Reporting The final step of a Tivoli Data Warehouse process is producing timely updated reports according different end users’ specifications. Tivoli warehouse enablement packs already provide out-of-the-box reports for the Crystal Enterprise Professional for Tivoli Limited Edition bundled with Tivoli Data Warehouse, but if there is a need to define additional customized reports, then either Crystal Enterprise Professional for Tivoli Professional Edition or another Business Intelligence tool is required. These are the skills required to implement customized reports for Tivoli Data Warehouse: Basic knowledge of how to connect the data mart databases via ODBC Basic knowledge of standard SQL Knowledge of the data mart structure and data Experience with Crystal Enterprise products or other Business Intelligence tools Report designers usually interact with the ETL2 developers, who are providing all of their requirements about relevant metrics, details on information, aggregation times, preservation of old data, and so on, in order to optimize the star schemas according to their reporting needs.70 Implementing Tivoli Data Warehouse 1.2
    • 3 Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running This chapter describes the installation steps for Tivoli Data Warehouse. You can install the application with all components on a single system or with the components distributed throughout your network. Always check the Tivoli Data Warehouse Release Notes, SC32-1399 for details about required Fix Packs, patches, or operating system-specific configurations. Look for late-breaking information about the required service at the TDW support Web site at the following address: http://www-3.ibm.com/software/sysmgmt/products/support/TivoliDataWarehouse.html Topics covered include: “Installing and configuring IBM DB2 client and server” on page 76 “Crystal Enterprise installation” on page 86 “Quick start deployment” on page 93 “Distributed deployment” on page 103 “Installing warehouse agents” on page 126 “Verification of the installation” on page 135 “Installing warehouse enablement packs” on page 142© Copyright IBM Corp. 2004. All rights reserved. 71
    • 3.1 Preparing for the installation Tivoli Data Warehouse 1.2 requires additional supporting applications, such as IBM DB2 Universal Database Enterprise Edition and Crystal Enterprise Professional Version 9 for Tivoli. The entire installation process is depicted in Figure 3-1. Even though this diagram refers to an installation process for a distributed Tivoli Data Warehouse 1.2 environment, the same applies also for a single machine environment. The remainder of this section contains tasks that need to be performed prior to installing the Tivoli Data Warehouse 1.2 environment. Depending on the architecture to be used, you may need to perform some or all of these tasks. Figure 3-1 also provides a guide to which tasks should be performed and the machines they should be performed on.72 Implementing Tivoli Data Warehouse 1.2
    • Phase 1 Install the Crystal Enterprise Server Install a Web Server Define whether Crystal Enterprise will use Microsoft SQL Server or Windows NT authentication Crystal Enterprise Install IBM DB2 Client for access to the Data Mart databases Server Install Crystal Enterprise Professional version 9 for Tivoli Phase 2 Prepare the IBM DB2 database Servers Install IBM DB2 Universal Database Enterprise Edition 7.2 and MINIMUM Fixpack 8 or upgrade existing IBM DB2 Servers to version 7.2 at least Fixpack 8 level on the TDW Control Server, and all Servers that will host Central Data Warehouse and TDW Control Data Mart databases on Windows / UNIX platforms Server Central Data Warehouse Data Mart Install or upgrade IBM DB2 Universal Database for OS/390 and z/OS V7.1 on the mainframe that will host Central Data Warehouse and Data Mart databases Data Mart Central Data Warehouse Phase 3 Perform TDW installation from the Control Server Start the installation process from the TDW Control Server machine. During the Data Mart install process, provide user IDs, password and Port number so that the Control Server can access and create all the Central Data Warehouse and Data Mart databases on the remote systems Central Data Warehouse On the TDW Control Server, using the DB2 Warehouse Manager, configure IBM Data Mart DB2 to use the TDW Control Database (TWH_MD) From the TDW Control Server, install and configure all the Central Data Warehouse and Data Mart databases on the z/OS systems TDW Control Server Central Data Warehouse Phase 4 Create Warehouse agent sites (optional) On the systems that will act as Warehouse Agent Sites: Warehouse Agent Site 1 Install the IBM DB2 Warehouse Manager Warehouse Agent Install the Warehouse Agent component Site 2 TDW Control Server Warehouse Agent Site 3Figure 3-1 Installation process overview Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 73
    • 3.1.1 Ensuring fully qualified host names Your operating system must be configured to provide Tivoli Data Warehouse with a fully qualified computer name rather than a short name. This is especially important in environments with many different operating systems. To ensure that your system is configured to provide a fully qualified computer name, follow the instructions in the next sections. On AIX systems The default domain name search order is as follows: 1. Domain Name Server (DNS) Server 2. Network Information Service (NIS) 3. Local /etc/hosts file If the /etc/resolv.conf does not exist, the /etc/hosts file is used. If only the /etc/hosts file is used, the fully qualified computer name must be first one that is listed after the IP address. Verify that the /etc/resolv.conf file exists and contains the appropriate information, such as: domain mydivision.mycompany.com nameserver 123.123.123.123 If NIS is installed, the /etc/irs.conf file overrides the system default. It contains the following information: hosts = bind, local The /etc/netsvc.conf file, if it exists, overrides the /etc/irs.conf and the system default. It contains the following information: hosts = bind,local On Linux systems Verify that the /etc/resolv.conf file exists and contains the appropriate information, such as: domain mydivision.mycompany.com nameserver 123.123.123.123 A short name is used if the /etc/nsswitch.conf file contains a line that begins as follows and if the /etc/hosts file contains the short name for the computer: hosts: files74 Implementing Tivoli Data Warehouse 1.2
    • To correct this, follow these steps:1. Change the line in the /etc/nsswitch.conf file to: hosts:dns nis files2. Stop the network services.3. Restart the network service.On Solaris systemsVerify that the /etc/resolv.conf file exists and contains the appropriateinformation, such as:domain mydivison.mycompany.comnameserver 123.123.123.123A short name is used if the /etc/nsswitch.conf file contains a line that begins asfollows and if the /etc/hosts file contains the short name for the computer: hosts: filesTo correct this, follow these steps:1. Change the line in the /etc/nsswitch.conf file to: hosts: dns nis files2. Enter the following command to stop the inet service: /etc/init.d/inetsvc stop3. Enter the following command to restart the inet service: /etc/init.d/inetsvc startOn Microsoft Windows NT systemsTo verify that a primary domain system (DNS) suffix is set, follow these steps:1. From the Windows task bar, click Start -> Settings -> Control Panel.2. In the Control Panel windows, double-click Network.3. Click the Protocols tab.4. Select the TCP/IP Protocol and then click Properties.5. Click the DNS tab.6. Ensure that the field Domain contains a domain suffix. If it does not, type the suffix, click OK, and restart the computer when prompted. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 75
    • On Windows 2000 systems To verify that a primary domain name system (DNS) suffix is set, follow these steps: 1. On the desktop, right-click My computer -> Properties. 2. Click the Network Identification tab. 3. Ensure that the field Full Computer Name contains a fully qualified domain name. If it does not, follow these steps. a. Click Properties. b. Click More. c. In the field Primary DNS suffix for this computer, type the primary DNS suffix, and restart the computer when prompted.3.1.2 Installing and configuring IBM DB2 client and server As described in 2.1.3, “Database requirements” on page 32, Tivoli Data Warehouse 1.2 is dependent of IBM DB2 Universal Database Enterprise Edition server for a single system installation, or for the control server, central data warehouse, and data mart components of a distributed installation. For the Crystal Enterprise Professional for Tivoli in a distributed Tivoli Data Warehouse installation, you can use IBM DB2 client at the proper level to access the data mart databases. This section provides information about the following topics: “Using an existing installation of DB2 client or server” on page 76 “IBM DB2 Server installation on AIX” on page 78 “IBM DB2 Fix Pack 8 installation on AIX” on page 82 “IBM DB2 Server installation on Windows 2000” on page 82 “IBM DB2 Fix Pack 8 installation on Windows 2000” on page 84 “Installing IBM DB2 Client” on page 85 Using an existing installation of DB2 client or server To use an existing installation of IBM DB2 client or server, make sure it is at the version and patch level specified by the Tivoli Data Warehouse 1.2 software requirements. Refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27 and to the Tivoli Data Warehouse Release Notes, SC32-1399. Use the db2level command to determine the Fix Pack Level of the existing IBM DB2 server environment.76 Implementing Tivoli Data Warehouse 1.2
    • On Windows systems, at DB2 7.2 Fix Pack 8, db2level command returns amessage similar to the following:DB21085I Instance "db2admin"uses DB2 code release "SQL07026"with levelidentifier "03070105"and informational tokens "DB2 v7.1.0.75","n021110" and"WR21314".The string "DB2 v7.1.0.75" indicates that the system is at IBM DB2 V7.2 Fix Pack8 level. The last two information tokens ("n021110" and "WR21314") vary byoperating system type and the specific patches that are installed.If the IBM DB2 client or server is installed but does not have Fix Pack 8 or higher,it is required to upgrade to Version 7.2 Fix Pack 8 (at minimum) before runningthe Tivoli Data Warehouse installation wizard. You cannot install Tivoli DataWarehouse with a lower Fix Pack level.In addition to checking the IBM DB2 proper version and level, consider thefollowing possibilities: If you have Lightweight Directory Access Protocol (LDAP), make sure that it is disabled for this IBM DB2 instance. In a IBM DB2 command window, run the following command: db2set -all | more Examine the value of DB2_ENABLE_LDAP settings. If this value is listed and is set to YES, disable LDAP and restart the IBM DB2 server by running the following commands in a IBM DB2 command windows: db2set db2_enable_ldap=NO db2stop force db2start If the value is not listed or is set to NO, no action is required. On UNIX systems, make sure that the IBM DB2 administration client is installed. If you can start the DB2 Control Center, the client is installed. On Windows Systems, if you did not perform a typical installation of IBM DB2 Universal Database, make sure you have the following components: – Administration and configuration Tool – Application Development Interfaces – Data Warehousing Tools If you selected the Typical installation when installing IBM DB2 Universal Database on a Windows system, all of the necessary components are available. Make sure that other applications using the IBM DB2 instance do not have database names that duplicate those of Tivoli Data Warehouse. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 77
    • IBM DB2 Server installation on AIX This section describes the IBM DB2 Universal Database Enterprise Edition Server Version 7.2 installation process on AIX. 1. Log in as a user with root authority, move to the directory where the DB2 7.2 Server for AIX CDROM is mounted, and start the DB2 setup utility, as follows: # ./db2setup 2. The Install DB2 V7.2 window, shown in Figure 3-2, appears. Select DB2 Administration Client and DB2 UDB Enterprise Edition. Figure 3-2 Install DB2 V7 components78 Implementing Tivoli Data Warehouse 1.2
    • 3. A new DB2 instance should be created for the Administration Server database. We specified the DB2 instance name db2inst1, as shown in Figure 3-3. You should also specify /home/db2inst1 as the instance owner directory.Figure 3-3 Create DB2 Services - DB2 Instance db2inst1 Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 79
    • 4. The installation process creates the DB2 fenced user. We specified the DB2 instance name db2fenc1, as shown in Figure 3-4. Figure 3-4 Create the DB2 fenced user80 Implementing Tivoli Data Warehouse 1.2
    • 5. Next, Figure 3-5 shows the values we used to create the user ID for the DB2 Administration Server.Figure 3-5 Administration Server window6. The installation process creates and sets the values of several environment variables, for example, DB2SYSTEM.7. At the end of the installation process, you may check the installation log file created at /tmp/db2setup.log.8. The installed JDBC code level needs to be upgraded to Version 2.0. You should log on to the system with a valid DB2 user ID, and issue the following commands: – For Bash, Bourne, or Korn shell: # . INSTHOME/sqllib/db2profile # cd /INSTHOME/sqllib/java12/ # . ./usejdbc2 Where INSTHOME is the home directory of the instance. – Verify that the JDBC level is correct by entering the following command: # echo $CLASSPATH The output must include the following path: INSTHOME/sqllib/java12/db2java.zip Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 81
    • IBM DB2 Fix Pack 8 installation on AIX This session describes the installation of DB2 Fix Pack 8 on AIX. Here are the steps for installing IBM DB2 Fix Pack 8: 1. Stop all database activity before applying this Fix Pack. To stop all database activity, issue the commands: # db2stop # db2admin stop 2. Unzip the Fix Pack using the following command to get a tar file: # gzip FP8_U484610.tar.Z 3. Un-tar the Fix Pack using the following command to extract the Fix Pack files. # tar -xvf FP8_U484610.tar 4. Run the following command to install the Fix Pack from the location where you un-tar the Fix Pack files. # ./installFixpack 5. Provide the DB2 instance password if prompted. 6. The installation wizard copies the files and finishes the installation of the Fix Pack. 7. Un-tar the efix for Fix Pack 8 tar -xvf special_U484610.tar 8. Make a backup of files db2bp and db2level, in directory /usr/lpp/db2_07_01/bin directory 9. Copy the files db2bp and db2level to the /usr/lpp/db2_07_01/bin directory. Note: If you are using a 32-bit IBM DB2 Server, make sure to install the 32-bit Fix Pack 8. Or if you are using a 64-bit IBM DB2 Server, make sure to install the 64-bit Fix Pack 8. IBM DB2 Server installation on Windows 2000 This section describes the IBM DB2 Universal Database Enterprise Edition Server Version 7.2 installation process on Windows. Note: Use the installation media provided with the TDW product. This ensures that you install the correct version and Fix Pack of DB2. 1. Load the DB2 installation media.82 Implementing Tivoli Data Warehouse 1.2
    • 2. Select Start -> Run. Type in D:setup.exe and click OK to start the installation. From the Installation window, select Install.3. The Select Products window opens. From this window you can select the component(s) of DB2 for Windows you would like to install. Select DB2 Enterprise Edition as shown in Figure 3-6. Click Next.Figure 3-6 Select DB2 Enterprise Edition4. The Select Installation Type window opens. Select the installation type you prefer. We selected Typical.5. Select the installation directory. In our environment, we used C:DB2SQLLIB.6. The installation prompts for the DB2 administrative user ID. We selected and set the password for the db2admin user ID.7. After the installation wizard copies the DB2 files onto the machine, the Install OLAP Starter Kit window opens. Select Do not install the OLAP Starter Kit and then click Continue.8. When the setup complete the installation process, click Finish.9. Update Java™. The installed JDBC code level needs to be upgraded to Version 2.0. You should open a DOS-command prompt window and issue the following commands, where DB2_DIR is the DB2 installation directory: cd DB2_DIRjava12 usejdbc2 The usejdbc2 command will copy the appropriate version of db2java.zip into the DB2_DIRjava12 directory.10.Reboot the machine. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 83
    • IBM DB2 Fix Pack 8 installation on Windows 2000 This section describes the installation of IBM DB2 Fix Pack 8 on Windows. Note: IBM DB2 Universal Database Enterprise Edition 7.2 Fix Pack 8 is the minimum requirement to install Tivoli Data Warehouse 1.2 on top. This section describes how to install Fix Pack 8 on an IBM DB2 Universal Database Enterprise Edition. If you use the IBM DB2 version 7.10 Fix Pack 10 supplied on Tivoli Data Warehouse 1.2 installation media, you must skip this section! If you are installing the Fix Pack by using the Administrator account of Windows 2000 Advanced Server, please make sure you complete the following steps: 1. Click Start -> Programs -> Administrative Tools -> Local Security Policy In the popup window, click Local Security Settings -> User Rights Assignment. 2. In the window, you will see lists of user rights. Make sure the Administrator account has the following rights: – Act as part of operating system – Create a token object – Increase quotas – Replace a process level token Note: Once you have installed a Fix Pack, you won’t be able to un-install it. 3. Stop all database activity before applying this Fix Pack. To stop all database activity, on a DB2 command window, run: c:db2sqllibbin:>db2stop or c:db2sqllibbin:>db2stop force and c:db2sqllibbin:>db2admin stop 4. Unzip and extract the Fix Pack files to a temporary directory. 5. Run the following command to install Fix Pack from the Fix Pack directory: c:fp8_wr21314setup.exe 6. Key in the DB2 instance owner password if the setup prompts for it and click Next. 7. The wizard shows the selection window. Click Next to continue. 8. Extract the efix for Fix Pack 8 into a temporary directory. Take a backup of the files db2bp.exe and db2level.exe84 Implementing Tivoli Data Warehouse 1.2
    • 9. Copy the files db2bp.exe and db2level.exe from the temporary directory to the <DB2DIR>bin directory, where <DB2DIR> is the IDM DB2 installation directory: c:special_wr21314copy db2bp.exe c:db2sqllibbin c:special_wr21314copy db2level.exe c:db2qllibbin10.Reboot the machine.Installing IBM DB2 ClientThis section describes the installation of IBM DB2 Client on Windows. Note: The Server of IBM DB2 Universal Database Enterprise Edition has a client included.If using an existing installation of the IBM DB2 client, ensure that it is at V7.2 FixPack 8 level or higher. Otherwise, upgrade it to at least V7.2 Fix Pack 8.The IBM DB2 client software can be obtained by installing the IBM DB2Administration client package, that comes with the IBM DB2 installation mediaprovided by Tivoli Data Warehouse 1.2. Use the IBM DB2 installation mediaprovided with the Tivoli Data Warehouse. This ensures that you get the correctversion.The IBM DB2 Administration client package can be installed by performing thefollowing actions:1. Insert the IBM DB2 installation media into the CD-ROM drive. If the program does not start automatically when you insert the DB2 installation CD, run the setup.exe program in the root directory of the CD. Click Install, provide the following information when prompted:2. Select DB2 Administration Client. Make sure that the DB2 Enterprise Edition product is not selected.3. Select the Typical Installation.4. You can change the default destination folder (optional). However, we used c:db2sqllib in our installation. Accept the default values for the remaining items in this panel.5. For the Control center server user name, specify a user that does not already exist on your system. When the DB2 installation wizard creates the user for you, it ensures that the user has the correct roles and privileges. If the user already exists, consider deleting it and letting the IBM DB2 installation recreate it. For more information about DB2 naming rules, refer to IBM DB2 Universal Database for Windows Quick Beginnings, GC09-2971. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 85
    • In a safe place, record the IBM DB2 user name and password that was specified for the IBM DB2 client. 6. On the Start Copying Files panel, select Next. and Finish to complete the setup.3.1.3 Crystal Enterprise installation Crystal Enterprise Professional Version 9 for Tivoli, a reporting solution provided with Tivoli Data Warehouse 1.2. Crystal Enterprise, typically resides on a system that does not have other Tivoli Data Warehouse components installed on it. Users access Crystal Enterprise reports using a Web browser from any system. Crystal Enterprise is integrated with Tivoli Data Warehouse 1.2 to provide historical reporting. Warehouse enablement packs provide reports designed for use with Crystal Enterprise. You access the reports with a Web browser that connects to the Crystal Enterprise server. Crystal Enterprise Automated Process Scheduler The Crystal Enterprise Automated Process Scheduler (APS) requires a database to store information about the system and its users. By default, the Setup program installs and configures its own Microsoft Data Engine (MSDE) database if necessary. APS clustering is automatically supported by the default MSDE database. The MSDE is a client/server data engine that provides local data storage and is compatible with Microsoft SQL Server. If you already have the MSDE or SQL Server installed, the installation program creates the APS database using your existing database engine. If the Microsoft Data Engine (MSDE) or Microsoft SQL Server is already installed on the local machine, you must first set up an account for the APS, as follows: 1. Determine whether the APS should use Windows NT or SQL Server authentication when connecting to your local database installation. 2. Using your usual administrative tools, create or select a user account that provides Crystal Enterprise with the appropriate privileges to your database server: – If you want the APS to connect to its database using Windows NT authentication, ensure that the Windows NT user account that you assign to the APS belongs to the System Administrator’s role in your SQL Server installation. In this scenario, the Windows NT user account that you assign to the APS is not actually used to create the system database during the installation process. Instead, your own Windows NT administrative account is used to create the database, so verify that your Windows NT account also belongs to the System Administrator’s role in your SQL Server installation.86 Implementing Tivoli Data Warehouse 1.2
    • – If you want the APS to connect to its database using SQL Server authentication, the login that you assign to the APS must belong to the Database Creators role in your SQL Server installation. In this scenario, the SQL Server credentials that you assign to the APS are also used to create the database and its tables.3. Verify that you can log on to SQL Server and carry out administrative tasks using the account you set up for use by the APS.Installing and configuring Crystal Enterprise ServerYou must install and configure Crystal Enterprise Professional Version 9 forTivoli prior to installing Tivoli Data Warehouse 1.2. Tivoli Data Warehousesupports the full stand-alone installation of Crystal Enterprise ProfessionalVersion 9 for Tivoli. Crystal Enterprise requires that a compatible Web server berunning on the same machine for a full stand-alone installation.Ensure that the following components are installed and configured correctlybefore you install Crystal Enterprise Professional for Tivoli. Refer also to 2.1.4,“Crystal Enterprise requirements” on page 33 for additional information. Windows NT or Windows 2000 SP3 at minimum. Web Server software, such as Microsoft IIS, iPlanet Enterprise Server, or IBM HTTP Server. Internet Explorer or Netscape Navigator Web browser. IBM DB2 Client, in order to have access to the Data Mart for reporting. Refer to 3.1.2, “Installing and configuring IBM DB2 client and server” on page 76 for details.In order to install Crystal Enterprise Professional Version 9 for Tivoli, completethe following steps: Note: Use the installation media provided with the Tivoli Data Warehouse 1.2 product. This ensures that you install the correct version of Crystal Enterprise.1. Run setup.exe, from the win32 directory of your product distribution. On the Welcome window, click Next to proceed.2. Accept the license agreement. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 87
    • 3. As shown in Figure 3-7, change the destination folder (optional) and choose the installation type New - Install a new Crystal Enterprise System. Click Next. Figure 3-7 Installation Type 4. The setup program checks to see whether or not the Microsoft Data Engine (MSDE) or Microsoft SQL server is installed on the local machine. If the setup program detects a database, use the Microsoft SQL Server Authentication window to provide the credentials that correspond to the database account you setup for the APS. The default user ID for the database account is named sa. For more information refer to “Crystal Enterprise Automated Process Scheduler” on page 86. If the setup program does not detect an existing database, the installation process will install Microsoft Data Engine and will create the credentials for the default SQL administrator account (sa) user ID. The installation wizards prompts you for the password to be used by the sa user ID. The setup program later configures the APS to connect to its system database using the sa account and the password you create here. Click Next.88 Implementing Tivoli Data Warehouse 1.2
    • This is shown in Figure 3-8.Figure 3-8 MSDE security configuration5. Figure 3-9 shows the Crystal Enterprise Professional Version 9 for Tivoli installation in progress.Figure 3-9 Installation window Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 89
    • 6. Click Finish on the completion window, as shown in Figure 3-10. Figure 3-10 Completion window 7. If the Web Server installed on the local machine is a supported version, then the setup program installs and configures the appropriate Crystal Enterprise Web Connector. Therefore, when the installation is complete, you can access the Crystal Enterprise Professional for Tivoli server by opening your Web browser and pointing it to: http://<CRYSTALSERVER>/crystal/enterprise9/ In this URL, <CRYSTALSERVER> is the host name of the Crystal Enterprise Professional for Tivoli server machine.90 Implementing Tivoli Data Warehouse 1.2
    • 8. Crystal Enterprise Launchpad is launched, as shown in Figure 3-11. Click Administrative Tool Console.Figure 3-11 Crystal Enterprise Launchpad Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 91
    • 9. Logon as Administrator, as shown in Figure 3-12 to verify that the setup program installed and configured the appropriate Crystal Enterprise Web connector. The default password for the Administrator account is set to blank (no password). Figure 3-12 Crystal Administration Tools For details on how to use the Crystal Enterprise Professional for Tivoli Web interface, refer to the Crystal Enterprise Professional Version 9 for Tivoli Administrator’s Guide manual, which is provided with Crystal Enterprise Professional Version 9 for Tivoli.3.2 Tivoli Data Warehouse 1.2 installation The basic prerequisite components of the Tivoli Data Warehouse 1.2 application have been introduced in the previous sections. We now cover the different architectures of a Tivoli Data Warehouse 1.2 installation, for example, how the components can be reasonably placed on one or more machines, and how they work together.92 Implementing Tivoli Data Warehouse 1.2
    • As described in 2.2, “Physical and logical design considerations” on page 36, there are several ways to deploy a Tivoli Data Warehouse 1.2 environment. Next, we go into detail on the installation steps and initial configuration of the environment, as follows: In 3.3, “Quick start deployment” on page 93 we provide installation steps for a stand-alone system installation with all the components installed on a single Windows 2000 Server system. In 3.4, “Distributed deployment” on page 103 we describe an example scenario of a Tivoli Data Warehouse 1.2 distributed environment and provides the installation steps for a TDW control server, one or more central data warehouse, and data mart databases on Windows, UNIX, and z/OS systems. Such deployment is best suited for large enterprises, enterprises distributed across widely-separated time zones, and enterprises containing z/OS systems collecting systems management data.3.3 Quick start deployment This type of installation quickly deploys Tivoli Data Warehouse 1.2, with the TDW control server and its database (TWH_MD), one central data warehouse database (TWH_CDW), and one data mart database (TWH_MART) on a single Windows system. Since the TDW control server must be on a Windows 2000, Windows 2003, or Windows NT machine, thus the single machine configuration cannot be installed on a UNIX system. Before proceeding with the quick start installation of Tivoli Data Warehouse, ensure that the following is setup and installed: 1. Check all the prerequisites. Refer to 2.1, “Hardware and software requirements” on page 28. 2. Ensure that the system’s DNS settings resolve fully qualified host names. Refer to 3.1.1, “Ensuring fully qualified host names” on page 74. 3. Make sure the Crystal Enterprise Professional Version 9 for Tivoli Server is functional. Refer to 3.1.3, “Crystal Enterprise installation” on page 86. 4. Install and/or upgrade IBM DB2 Universal Database Enterprise Edition to version 7.2 and minimum of Fix Pack 8 level. Refer to 3.1.2, “Installing and configuring IBM DB2 client and server” on page 76. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 93
    • Figure 3-13 shows a typical quick start configuration, mapped to an existing Tivoli environment. All the Tivoli Data Warehouse 1.2 components (TDW control server, central data warehouse, and data mart databases) are installed on a single Windows 2000 machine. TWH_MART Tivoli Environment TWH_CDW TWH_MD Stand alone architecture - Windows 2000 Server Crystal Enterprise Server - TDW Control Server - Central data warehouse - Windows 2000 Server - Data Mart - IIS Web Server - Crystal Enterprise Professional version 9 for Tivoli Figure 3-13 Quick start deployment configuration3.3.1 Quick start deployment: installation and configuration This section provides instructions for installing Tivoli Data Warehouse 1.2 on a single system. This will install and configure the following components: TDW control server Control center, central data warehouse, and data mart databases. Settings on how to connect to the Crystal Enterprise server Attention: Make sure you have completed all the prerequisite tasks before proceeding with the Tivoli Data Warehouse 1.2 installation. In order to perform the Tivoli Data Warehouse 1.2 installation, use the following steps: 1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD.94 Implementing Tivoli Data Warehouse 1.2
    • 2. Figure 3-14, the InstallShield Wizard, is displayed. Click Next to proceed.Figure 3-14 InstallShield Wizard3. Figure 3-15 displays the directory of the Tivoli common logging. The default location is %ProgramFiles%IBMTivolicommon. Each product stores logging information in a separate subdirectory within the Tivoli Common Logging directory. Click Next.Figure 3-15 Tivoli common logging directory Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 95
    • 4. Figure 3-16 displays the Setup Window. Select Quick start. Specify the installation directory (Optional); the default is %ProgramFiles%TWH; and click Next. Figure 3-16 Setup window96 Implementing Tivoli Data Warehouse 1.2
    • 5. Figure 3-17, the information window, specifies the existing IBM DB2 instance owner user ID and password to be used to create and connect to the Tivoli Data Warehouse databases. Click Next.Figure 3-17 DB2 connection6. Figure 3-18 specifies the following connection information for the Crystal Enterprise server, and then click Next. – Host name: The fully qualified host name of the machine name where Crystal Enterprise Professional Version 9 for Tivoli is installed. – User name: The user credentials Tivoli Data Warehouse will use on the connection to the Crystal Enterprise server. Defaults to Administrator. – Password: Password for the Administrator user ID. Defaults to blank (no password). The default values above are based on a new Crystal Enterprise Professional Version 9 for Tivoli Server limited edition shipped with Tivoli Data Warehouse 1.2. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 97
    • Figure 3-18 Crystal connection 7. Figure 3-19 indicates the components to be installed and their location. In our case, one TDW control server, one central data warehouse database, and one data mart database on the local computer. Click Install to start the installation. Figure 3-19 Summary window98 Implementing Tivoli Data Warehouse 1.2
    • 8. After the installation is over, a completion window is displayed, as shown in Figure 3-20. It has a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors. Click Next. If you are prompted to restart, click Yes, restart my system. Figure 3-20 Completion window3.3.2 Configuring the control database This section contains instructions for configuring the control database. This database contains metadata for Tivoli Data Warehouse and the IBM DB2 Data Warehouse Center. Therefore we need to specify the Tivoli Data Warehouse control database (TWH_MD) to be the main control database for both the IBM DB2 Data Warehouse Center and the IBM DB2 Warehouse Control Database Management components. In order to configure the IBM DB2 Data Warehouse Center, complete the following steps: 1. On the Windows task bar, click Start -> Programs -> IBM DB2 -> Control Center. 2. From the IBM DB2 Control Center, start the IBM DB2 Data Warehouse Center by choosing Tools -> Data Warehouse Center. 3. In the Data Warehouse Center Logon window, click Advanced. There is no need to provide login information. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 99
    • 4. In the Advanced window, shown in Figure 3-21, type TWH_MD in the control database field. Click OK to return to the logon window. Figure 3-21 Configuring the IBM DB2 data warehouse center 5. Click Cancel to exit from the Data Warehouse Center login screen. In order to configure the IBM DB2 Warehouse Control Database Management, complete the following steps: 1. Open the Data Warehouse Center - Control Database Management window by selecting Start -> Programs -> IBM DB2 -> Warehouse Control Database Management. 2. Type the TWH_MD in new control database. 3. Do not change the schema name. 4. Type the IBM DB2 instance owner user ID and password for the control database, and then click OK. In our scenario, we used the db2admin user ID. 5. When the message, Processing has completed, appears as shown in Figure 3-22, click Cancel.100 Implementing Tivoli Data Warehouse 1.2
    • Figure 3-22 Configuring the Warehouse Control Database Management3.3.3 Creating ODBC connections to the data mart databases The Crystal Enterprise server needs to have access to the information stored in the data mart database for reporting. After the quick start install of Tivoli Data Warehouse 1.2 finished, there is only one data mart database created (TWH_MART), and an ODBC connection for this database must be set up. You may further expand this environment by either creating additional data mart databases, or installing warehouse enablement packs that use different data marts. In any case, there must be an ODBC connection to these data marts defined on the Crystal Enterprise server. Tivoli Data Warehouse 1.2 provides the twh_create_datasource script that sets up ODBC connections to the data mart databases. You may use this script or create the ODBC connections manually. In order to create an ODBC connection to the TWH_MART database using the twh_create_datasources script, perform the following tasks: 1. On the Crystal Enterprise server machine, open an IBM DB2 command window: Start-> Programs-> IBM DB2-> Command Window. 2. Copy both the twh_create_datasource.bat and ODBCcfg.exe files from the disk1tools directory of the Tivoli Data Warehouse 1.2 installation media to a temporary directory on the Crystal Enterprise server machine. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 101
    • 3. Run the twh_create_datasource script using the following syntax as shown in Example 3-1: twh_create_datasource <DBtype> <ID> <odbcname> <DBname> <SRVname> <port> In this script: <DBtype> Can be set to DB2UDB or DB390 depending on the location of the database. <ID> An unique identifier for the local node name. The script creates node names following the naming convention: TDWCS%ID%. In our example, we used ID=10 resulting in node name TDWCS10. <odbcname> The ODBC data source name. <DBname> The data mart database name. <SRVname> The IBM DB2 server in which the data mart database resides. This must be the fully qualified host name. <port> The port number to connect to the IBM DB2 server. Example 3-1 twh_create_datasource script C:Temp>twh_create_datasource.bat DB2UDB 10 TWH_MART TWH_MART tdw001.itsc.austin.ibm.com 50000 Creating DB2/UDB datasource TWH_MART C:Temp>db2cmd /w /c /i db2 catalog tcpip node TDWCS10 remote tdw001.itsc.austin.ibm.com server 50000 DB20000I The CATALOG TCPIP NODE command completed successfully. DB21056W Directory changes may not be effective until the directory cache is refreshed. C:Temp>db2cmd /w /c /i db2 catalog database TWH_MART at node TDWCS10 authentication server DB20000I The CATALOG DATABASE command completed successfully. DB21056W Directory changes may not be effective until the directory cache is refreshed. C:Temp>C:TempODBCcfg.exe DB2 TWH_MART TWH_MART No Username was provided. Skipping connection test. C:Temp>102 Implementing Tivoli Data Warehouse 1.2
    • 3.4 Distributed deployment A distributed deployment, has a central data warehouse or data mart database that is not on the control server, or has more than one central data warehouse or data mart databases. Before proceeding with the installation of Tivoli Data Warehouse on a distributed environment, ensure that the following software is set up and installed: 1. Check all the prerequisites. Refer to 2.1, “Hardware and software requirements” on page 28. 2. Ensure that the DNS settings on ALL the systems resolve fully qualified host names. Refer to 3.1.1, “Ensuring fully qualified host names” on page 74. 3. Make sure the Crystal Enterprise Professional Version 9 for Tivoli Server is functional. Refer to 3.1.3, “Crystal Enterprise installation” on page 86. 4. Install and/or upgrade IBM DB2 Universal Database Enterprise Edition to version 7.2 and minimum Fix Pack 8 on the Windows and UNIX systems that serve as: – TDW control server – Central data warehouse database servers – Data mart database servers Refer to 3.1.2, “Installing and configuring IBM DB2 client and server” on page 76. 5. Install and/or upgrade DB2 UDB for OS/390 and z/OS to version 7.1 on the z/OS and OS/390 systems that host a central data warehouse database and Data Mart databases. Figure 3-23 shows a distributed deployment scenario that will illustrate the installation steps presented in this section. This distributed environment is mapped to an existing Tivoli environment and consists of z/OS and distributed systems. In this scenario we have all the Tivoli Data Warehouse components distributed across different systems, that is, the TDW control server is on a Windows 2000 machine, two central data warehouse database servers are on Windows 2000 and z/OS, and 2 data mart databases on AIX and z/OS. Such an environment can be installed using a two-step approach. The first step installs all Tivoli Data Warehouse components on distributed systems. The second step installs the Tivoli Data Warehouse components on the z/OS system. The installation steps based on this scenario are provided next in 3.4.1, “Distributed deployment installation: Windows and UNIX” on page 104 and 3.4.2, “Distributed deployment installation: z/OS” on page 115. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 103
    • TWH_CDW Central Data Warehouse Windows 2000 Server Hostname: TDW004 Data Mart TWH_MD TWH_MART TDW Control Server Data Mart Tivoli Environment Windows 2000 Server Hostname: TDW003 AIX 5 Hostname: TDW009 Central Data Data Mart Warehouse z/OS environment Data Source Crystal Enterprise Server Data Sources Central Data Warehouse Data Source Windows 2000 Server Data Mart IIS Web Server Data Source Hostname: wtsc66oe Crystal Enterprise Professional 9 for Tivoli Hostname: TDW002 Figure 3-23 Distributed deployment scenario3.4.1 Distributed deployment installation: Windows and UNIX This section provides instructions for installing the following components of the Tivoli Data Warehouse environment shown in Figure 3-23 as follows: TDW control server on machine TDW003 One central data warehouse database on machine TDW004 One data mart database on machine TDW009 The installation steps described here must be performed on the TDW control server machine.104 Implementing Tivoli Data Warehouse 1.2
    • We assume that the Crystal Enterprise Professional for Tivoli Server has alreadybeen deployed. In this case, the Tivoli Data Warehouse installation process willconnect to the Crystal Enterprise Professional for Tivoli Server and install onlythe Crystal Publishing Wizard on the TDW control server system.In order to install the distributed environment portion of the scenario presented inFigure 3-23, perform the following steps:1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD.2. The panel in shown in Figure 3-24, the InstallShield Wizard, is displayed. Click Next to proceed.Figure 3-24 Install Shield Wizard Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 105
    • 3. Figure 3-25 shows the directory of the Tivoli common logging. The default location is %ProgramFiles%IBMTivolicommon. Each product stores logging information in a separate subdirectory within the Tivoli Common Logging directory. Click Next. Figure 3-25 Tivoli Common Logging Directory106 Implementing Tivoli Data Warehouse 1.2
    • 4. Select Custom or Distributed, for distributed deployment. You can use the default installation directory %ProgramFiles%TWH or change it to your needs. Then click Next. However, we changed the installation directory name to C:TWH as shown in Figure 3-26.Figure 3-26 Setup Window5. A warning message is displayed to emphasize the need to fulfill all of the prerequisite tasks required for installing Tivoli Data Warehouse 1.2 in a distributed architecture. This is shown in Figure 3-27. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 107
    • Figure 3-27 Before proceeding with TDW 1.2 distributed installation 6. Specify the existing IBM DB2 instance owner user ID and password to be used to create and connect to the TDW control server database (TWH_MD), as shown in Figure 3-28. Click Next. Figure 3-28 DB2 connection108 Implementing Tivoli Data Warehouse 1.2
    • 7. You are prompted to provide the list of servers that will host the central data warehouse database in your environment. Click Add and type the IBM DB2 Server information on the system that will host the remote central data warehouse database, as shown in Figure 3-29. Click Next.Figure 3-29 Central data warehouse on remote host8. Make sure that all of the central data warehouse database servers are included in your list. The list for our case study environment is shown in Figure 3-30. Click Next. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 109
    • Figure 3-30 Central data warehouse database server list 9. You are prompted to provide the list of servers that will host the data mart database in your environment. Click Add and type the IBM DB2 Server information on the system that will host the remote data mart database, as shown in Figure 3-31. Click Next. Figure 3-31 Data mart on remote host110 Implementing Tivoli Data Warehouse 1.2
    • 10.Make sure that all of the data mart database servers are included in your list. The list for our case study environment is shown in Figure 3-32. Click Next.Figure 3-32 Data mart database server list11.Figure 3-33 specifies the following connection information for the Crystal Enterprise server: – Host name: The fully qualified host name of the machine name where Crystal Enterprise Professional Version 9 for Tivoli is installed. – User name: User credentials Tivoli Data Warehouse will use on the connection to the Crystal Enterprise server. Default to Administrator. – Password: Password for the Administrator user ID. Default to blank (no password). The default values above are based on a new Crystal Enterprise Professional Version 9 for Tivoli Server limited edition shipped with Tivoli Data Warehouse 1.2. Click Next. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 111
    • Figure 3-33 Crystal connection 12.An installation summary is then displayed showing which Tivoli Data Warehouse component is going to be installed on which server, as shown in Figure 3-34. Click Install, or click Back to make changes. Components location Figure 3-34 Summary window112 Implementing Tivoli Data Warehouse 1.2
    • 13.A completion window is displayed as shown in Figure 3-35. It has a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors. Click Next. If you are prompted to restart, click Yes, restart my system.Figure 3-35 Completion windowConfiguring the control databaseThis section contains instructions for configuring the control database. Thisinstructions should be performed on the TDW Control Center machine.It is necessary to specify the Tivoli Data Warehouse control database(TWH_MD) to be the main control database for both the IBM DB2 DataWarehouse Center and the IBM DB2 Warehouse Control DatabaseManagement components.In order to configure the IBM DB2 Data Warehouse Center, complete thefollowing steps:1. On the Windows task bar, click Start -> Programs -> IBM DB2 -> Control Center.2. From the IBM DB2 Control Center, start the IBM DB2 Data Warehouse Center by choosing Tools -> Data Warehouse Center.3. In the Data Warehouse Center Logon window, click Advanced. There is no need to provide login information. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 113
    • 4. In the Advanced window, shown in Figure 3-36, type TWH_MD in the control database field. Click OK to return to the logon window. Figure 3-36 Configuring the IBM DB2 Data Warehouse Center 5. Click Cancel to exit from the Data Warehouse Center login screen. In order to configure the IBM DB2 Warehouse Control Database Management, complete the following steps: 1. Open the Data Warehouse Center - Control Database Management window by selecting Start -> Programs -> IBM DB2 -> Warehouse Control Database Management. 2. Type TWH_MD in the new control database. 3. Do not change the schema name. 4. Type the IBM DB2 instance owner user ID and password for the control database, and then click OK. 5. When the Processing has completed message appears, as shown in Figure 3-37, click Cancel.114 Implementing Tivoli Data Warehouse 1.2
    • Figure 3-37 Configuring the Warehouse Control Database Management3.4.2 Distributed deployment installation: z/OS This section contains instructions for configuring completing the installation of the distributed deployment scenario that was depicted in Figure 3-23 on page 104. Before proceeding with the installation, refer to 2.1, “Hardware and software requirements” on page 28 to ensure that the z/OS environment is at the proper levels. Here we show the steps used to define the central data warehouse and the data mart on the z/OS system. This configuration is particularly needed if you plan to use Tivoli Decision Support for 390, which requires that the central data warehouse and data marts databases be created on the same system. Before installing central data warehouse or data mart databases on z/OS, ensure that your DB2 on z/OS is correctly configured. Make sure you have completed the following tasks: 1. Ensure that the DB2 Universal Database for OS/390 and z/OS V7.1 has the DB2 Management Clients Package installed (FMID JDB771D). 2. Consult your DB2 UDB for OS/390 and z/OS administrator for the following information: – User ID and password with SYSADM authority – Database location – Storage volume, storage group, and storage VCAT – Buffer pool – Database names – Tablespace names – UTF8 tablespace names – Tablespace size (primary and secondary) Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 115
    • Creating the central data warehouse on z/OS The following steps should be performed on the TDW Control Center machine. The installation process will trigger scripts on the z/OS system that will create and configure the database: 1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive of the control server. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD. 2. The install process displays the welcome panel. Click Next. It also provides the location of the common log files. Click Next. 3. The install proceeds by verifying the current Tivoli Data Warehouse environment and configuration. You will be prompted with two options, as shown in Figure 3-38: You can create one or more central data warehouse databases, or you can create one or more data mart databases. In this case, select Add central data warehouses and click Next. Figure 3-38 Adding central data warehouses 4. The installation wizard searches for the existing Tivoli Data Warehouse configuration and displays a list of central data warehouse locations found. It also allows you to add to the existing list of systems in which central data warehouse databases will be created. Click Add on the install window. 5. With the help of your z/OS system administrator, specify the IBM DB2 configuration information on the z/OS system, as shown in Figure 3-39. Specify database type DB2 for z/OS and S/390®, a fully qualified host name, port number, and a user ID with SYSADM authority. Click Next.116 Implementing Tivoli Data Warehouse 1.2
    • Figure 3-39 z/OS IBM DB2 Server information6. With the help of your z/OS system administrator, specify the central data warehouse database configuration information as shown in Figure 3-40. Click Next. Refer to 2.2.9, “Considerations about warehouse databases on z/OS” on page 54 for details.Figure 3-40 z/OS central data warehouse database configuration Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 117
    • 7. Make sure that the new central data warehouse database server is included in your list. The list for our case study environment is shown in Figure 3-41. Figure 3-41 Central data warehouse server on z/OS 8. As displayed in Figure 3-42, the Summary window shows the Tivoli Data Warehouse components that will be installed and configured. Click Install or click Back to make changes. Figure 3-42 Central data warehouse summary window118 Implementing Tivoli Data Warehouse 1.2
    • 9. The completion window is displayed in Figure 3-43. It has a successful completion notice, or messages describing problems. Make sure the window does not list any warnings or errors and that the installation of the central data warehouse databases was successful. Click Finish.Figure 3-43 Central data warehouse on z/OS install10.You can verify your installation by issuing the commands listed in Example 3-2 on the z/OS system. These commands will confirm the creation of the central data warehouse database on the z/OS system.Example 3-2 Verification of central data warehouse database on z/OSselect * from sysibm.sysdatabase;select * from sysibm.systablespace;select * from sysibm.sysstogroup;select * from sysibm.systables where dbname = TCDW1;-- where TCDW1 is the new database name specified during the instalationCreating the data mart on z/OSThe following steps should be performed on the TDW Control Center machine.The installation process will trigger scripts on the z/OS system that will createand configure the data mart database. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 119
    • These are the steps for creating the data mart: 1. Insert the Tivoli Data Warehouse 1.2, installation CD into the CD-ROM drive of the control server. If the installation wizard does not start up, run the setup.exe program, which is located in the root directory of the CD. 2. The install process display the welcome panel. Click Next. It also provides the location of the common log files. Click Next. 3. The install proceeds by verifying the current Tivoli Data Warehouse environment and configuration. You will be prompted with two options, as shown in Figure 3-44. You can to create one or more central data warehouse databases, or you can create one or more data mart databases. In this case, select Add data marts and click Next. Figure 3-44 Adding data marts 4. The installation wizard searches for the existing Tivoli Data Warehouse configuration and display a list of found data mart locations. It also allows you to add to the existing list of systems in which data mart databases will be created. Click Add on the install window. 5. With the help of your z/OS system administrator, specify the IBM DB2 configuration information on the z/OS system, as shown in Figure 3-45. Specify database type DB2 for z/OS and S/390, a fully qualified host name, port number, and a user ID with SYSADM authority. Click Next.120 Implementing Tivoli Data Warehouse 1.2
    • Figure 3-45 z/OS IBM DB2 Server information6. With the help of your z/OS system administrator, specify the data mart database configuration information as shown in Figure 3-46. Click Next. Refer to 2.2.9, “Considerations about warehouse databases on z/OS” on page 54 for details.Figure 3-46 z/OS data mart database configuration Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 121
    • 7. Make sure that the new data mart database server is included in your list. The list for our case study environment is shown in Figure 3-47. Figure 3-47 Data mart server on z/OS 8. The Summary window, as displayed in Figure 3-48, shows the Tivoli Data Warehouse components that will be installed and configured. Click Install or click Back to make changes. Figure 3-48 Data mart creation summary window122 Implementing Tivoli Data Warehouse 1.2
    • 9. The completion window is displayed in Figure 3-49. It has a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors and that the installation of the data mart database was successful. Click Finish. Figure 3-49 Data mart on z/OS install 10.You can verify your installation by issuing the commands listed in Example 3-3 on the z/OS system. These commands will confirm the creation of the data mart database on the z/OS system. Example 3-3 Verification of data mart database on z/OS select * from sysibm.sysdatabase; select * from sysibm.systablespace; select * from sysibm.sysstogroup; select * from sysibm.systables where dbname = TCMART1 -- where TMART1 is the new database name specified during the instalation3.4.3 Creating ODBC connections to the data mart databases The Crystal Enterprise server needs to have access to the information stored in the data mart databases for reporting. After completing the distributed installation of Tivoli Data Warehouse 1.2, you must set up ODBC connections from the Crystal Enterprise server machine to the data mart databases. You may further expand your environment by either creating additional data mart databases, or installing warehouse enablement packs that use different data marts. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 123
    • In any case, there must be ODBC connections to these data marts defined on the Crystal Enterprise server. Tivoli Data Warehouse 1.2 provides the twh_create_datasource script that sets up ODBC connections to the data mart databases. You may use this script or create the ODBC connections manually. In our case study scenario, we need to create ODBC connections to the TWH_MART data mart database located in the AIX machine TDW009 and to the TMART1 data mart database running on the z/OS system WTSC66. In order to create an ODBC connection to the TWH_MART database using the twh_create_datasources script, perform the following tasks: 1. On the Crystal Enterprise server machine, open an IBM DB2 command window: Start-> Programs-> IBM DB2-> Command Window. 2. Copy both the twh_create_datasource.bat and ODBCcfg.exe files from the disk1tools directory of the Tivoli Data Warehouse 1.2 installation media to a temporary directory on the Crystal Enterprise server machine. 3. Run the twh_create_datasource script using the following syntax and shown in Example 3-1. twh_create_datasource <DBtype> <ID> <odbcname> <DBname> <SRVname> <390LocalDBName> <port> Where: <DBtype> Can by set to DB2UDB or DB390 depending on the location of the database. <ID> An unique identifier for the local node name. The script creates node names following the naming convention: TDWCS%ID% <odbcname> The ODBC data source name <DBname> The data mart database name <SRVname> The IBM DB2 server in which the data mart database resides. This must be the fully qualified host name. <390LocalDBName> For databases on z/OS only. Specifies the local database name. <port> The port number to connect to the IBM DB2 server Example 3-4 twh_create_datasource script C:Temp>twh_create_datasource.bat DB2UDB 09 TWH_MART TWH_MART tdw009.itsc.austin.ibm.com 50000 Creating DB2/UDB datasource TWH_MART124 Implementing Tivoli Data Warehouse 1.2
    • C:Temp>db2cmd /w /c /i db2 catalog tcpip node TDWCS09 remotetdw009.itsc.austin.ibm.com server 50000DB20000I The CATALOG TCPIP NODE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.C:Temp>db2cmd /w /c /i db2 catalog database TWH_MART at node TDWCS09authentication serverDB20000I The CATALOG DATABASE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.C:Temp>C:TempODBCcfg.exe DB2 TWH_MART TWH_MARTNo Username was provided. Skipping connection test.C:Temp>C:Temp>twh_create_datasource.bat DB2390 66 TMART1 TMART1 wtsc66oe.itso.ibm.com TMART1 33768Creating DB2/390 datasource TMART1C:Temp>db2cmd /w /c /i db2 catalog tcpip node TDWCS66 remotewtsc66oe.itso.ibm.com server TMART1DB20000I The CATALOG TCPIP NODE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.C:Temp>db2cmd /w /c /i db2 catalog database TMART1 at node TDWCS66authentication dcsDB20000I The CATALOG DATABASE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.C:Temp>db2cmd /w /c /i db2 catalog dcs database TMART1 as 33768 parms,,INTERRUPT_ENABLED,,,,,DB20000I The CATALOG DCS DATABASE command completed successfully.DB21056W Directory changes may not be effective until the directory cache isrefreshed.C:Temp>C:TempODBCcfg.exe DB2 TMART1 TMART1No Username was provided. Skipping connection test.C:Temp Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 125
    • 3.5 Installing warehouse agents The warehouse agent is the component of IBM DB2 Warehouse Manager that manages the flow of data between data sources and targets databases that are on different computers. By default, the TDW control server uses a local warehouse agent to manage the data flow between operational data sources, central data warehouse, and data mart databases. You can optionally install the warehouse agent component of IBM DB2 Warehouse Manager on a computer other than the TDW control server. Typically, you place an agent on the computer that is the target of a data transfer. From the Tivoli Data Warehouse perspective, that computer will become a remote warehouse agent site or, simply an agent site, after registering and enabling the warehouse agent to run ETLs. IBM DB2 Data Warehouse Center will then use the remote warehouse agent site machine to manage the transfer of Tivoli Data Warehouse data. This can speed up the data transfer as well as reduce the workload on the control server. The computer on which you install a warehouse agent is called an agent site. In our case study installation scenario, we decided to have agent sites on both the central data warehouse and data mart servers running on distributed platforms, as shown in Figure 3-50. Note: Make sure the warehouse agent site machines that will run the ETL1 can connect to the operational data source databases. Make sure the warehouse agent site machines that will run the ETL2 can connect to the correspondent central data warehouse databases and data mart databases.126 Implementing Tivoli Data Warehouse 1.2
    • TWH_CDW Agent Site Operational Central Data Warehouse data flow Operational Windows 2000 Server data flow Hostname: TDW004 Data Mart TWH_MD Agent Site TWH_MART TDW Control Server Data Mart Tivoli Environment Windows 2000 Server Hostname: TDW003 AIX 5 Hostname: TDW009 Central Data Data Mart Warehouse z/OS environment Data Source Crystal Enterprise Server Data Sources Central Data Warehouse Data Source Windows 2000 Server Data Mart IIS Web Server Data Source Hostname: wtsc66oe Crystal Enterprise Professional 9 for Tivoli Hostname: TDW002Figure 3-50 Distributed environment with agentThere are two steps to be performed in order to create remote warehouse agentsites:1. Install IBM DB2 Warehouse Manager on every server that will become warehouse agent sites. In our case study scenario, IBM DB2 Warehouse Manager must be installed on the central data warehouse server (tdw004) and on the data mart server (tdw009).2. On the machine to become the warehouse agent site, use the Tivoli Data Warehouse installation wizard to register and enable the warehouse agent to run ETLs.After the warehouse agents have been registered with the control server, thefollowing steps should be performed:1. On the remote agents site: Catalog the databases that the remote agent is supposed to use. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 127
    • 2. On the remote agents site: Make available the warehouse enablement pack files. 3. On the data warehouse control servers site: Configure the ETL processes to use the remote agent. Chapter 5, “IBM Tivoli NetView Warehouse Enablement Pack” on page 161 provides details on the steps needed to enable the WEP to use the remote agent site, described above. Warehouse agents are supported on Windows and UNIX systems only. If you are using IBM DB2 databases on a z/OS system, you must use the warehouse agent on another computer in your deployment.3.5.1 Installing IBM DB2 Warehouse Manager In this section we provide the steps required for installing IBM DB2 Warehouse Manager on Windows. The installation process is described in the manual, Installing and Configuring Tivoli Data Warehouse, GC32-0744-02. Attention: See the IBM DB2 Warehouse Manager installation media provided with Tivoli Data Warehouse 1.2. This ensures that you install the correct version and Fix Pack level. Make sure you stop ALL IBM DB2 processes before proceeding with the installation. Installing IBM DB2 Warehouse Manager on Windows platform 1. Stop ALL IBM DB2 processes before proceeding with the installation. Open a IBM DB2 command window and issue the following commands: db2stop force db2admin stop Also, on the Windows Services panel, stop the following services: – DB2-DB2CTLSV – DB2 JDBC Applet Server – DB2 License Server – DB2 Security Server – Warehouse server – Warehouse logger 2. Load the IBM DB2 Warehouse Manager installation media. 3. Select Start -> Run. Type in D:setup.exe and click OK to start the installation. From the Installation window, select Install.128 Implementing Tivoli Data Warehouse 1.2
    • 4. The Select Products window opens. Make sure that DB2 Warehouse Manager is selected. Click Next.5. The Select Installation Type window opens. Select Custom.6. On the Select Components window, make sure that only Warehouse Agent and Documentation are selected, as shown in Figure 3-51. Click Next.Figure 3-51 Select the DB2 Warehouse Manager components7. At the Start Copying Files window, you can review the installation options. Click Next.8. When the setup complete the installation process, click Finish.9. Reboot the machine.Installing IBM DB2 Warehouse Manager on AIX platform1. Stop ALL IBM DB2 processes before proceeding with the installation. Log in as DB2 administrator (in our case the DB2 administrator is named db2inst1) and issue the following commands: db2stop force db2admin stop2. Log in as user with root authority.3. Mount the IBM DB2 Warehouse Manager installation media.4. Change to the directory where the installation media is mounted.5. Enter the ./db2setup command. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 129
    • 6. Within the Install DB2 V7 Menu select DB2 Data Warehouse Agent as shown in Figure 3-52. Figure 3-52 Install DB2 V7 menu on AIX 7. In the Configuration - DB2 Data Warehouse Agent Services menu, accept the default service names and port numbers. 8. In the Create DB2 Services menu, select: Do not create a DB2 Instance. and Do not create the Administration Server. as shown in Figure 3-53. Then click OK.130 Implementing Tivoli Data Warehouse 1.2
    • Figure 3-53 Create DB2 Service Menu on AIX 9. In the DB2 Setup Utility window your settings are summarized. Click Continue and the installation process starts. 10.After the installation process check the DB2 Setup Utility window for error messages. Click OK to finish the installation of the DB2 warehouse agent.3.5.2 Creating the remote agent sites Perform the following steps on each computer that will be a remote agent site. In order to create a remote agent site, IBM DB2 Warehouse Manager must have been installed on each computer. This procedure will register existing IBM DB2 warehouse agents on the TDW control server and will enable the warehouse agent to run ETL processes. Note: Do not perform this step on the TDW control server. It should be performed on each computer that will become a remote agent site. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 131
    • Creating agent sites on a Windows system and a UNIX system To create a remote agent site, perform the following steps: 1. Insert the Tivoli Data Warehouse 1.2 CD into your CD-ROM drive. 2. Start the Tivoli Data Warehouse installation wizard using the command for your operating system: On Windows If the TDW installation wizard does not start automatically, run the setup.exe program, which is located in the root directory of the CD. On UNIX run the setup_unix.sh program. 3. In the Welcome window, click Next. 4. The Tivoli Common Logging Directory window displays the name of the Tivoli common logging directory. Click Next. 5. Select Create a remote agent site. Change the installation directory (Optional); the default directory is %ProgramFiles%TWH. We have changed it to C:TWH as shown in Figure 3-54. Click Next. Figure 3-54 Setup Window - create warehouse agents132 Implementing Tivoli Data Warehouse 1.2
    • 6. A warning message is displayed to emphasize the need to fulfill all of the prerequisite tasks required for creating remote agent sites. This is shown in Figure 3-55.Figure 3-55 Before proceeding with remote agent sites creation7. Specify the existing IBM DB2 instance owner user ID and password to connect to the local IBM DB2 database. Click Next.8. In the Connection to Remote control server window, specify the following information, as shown in Figure 3-56: – The fully qualified host name of the TDW control server of the Tivoli Data Warehouse 1.2 environment. – The port number for IBM DB2 on the TDW control server – The IBM DB2 instance owner user ID and password to connect to IBM DB2 on the TDW control server Then click Next. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 133
    • Figure 3-56 Warehouse agents - specify the TDW control server 9. The Summary window indicates that you are creating an agent site. Click Install to start the installation. 10.In the Progress Window, review the progress of the program. When the program completes, the Installation Results windows, as shown in Figure 3-57, contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors and then click Next.134 Implementing Tivoli Data Warehouse 1.2
    • Figure 3-57 Successful remote agent creation window 11.Click Finish to exit. You must restart the system before the agent site can be used by any warehouse packs.3.6 Verification of the installation In this section we describe some means to verify the installation of various Tivoli Data Warehouse products. Note: We assume a successful IBM DB2 Universal Database Enterprise Edition installation as base for the Tivoli Data Warehouse installation. Therefore we do not explain how to verify a IBM DB2 Universal Database Enterprise Edition installation. Verify Tivoli Data Warehouse control server. In order to verify the Tivoli Data Warehouse control server, issue the command twh_list_cs.bat. If the control server was installed successfully, this command gives you the host name where the control server was installed and the name of the control server database. However, the control server database must be on the same host as the control server. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 135
    • For our case study installation Example 3-5 gives the output of the twh_list_cs.bat command. You will find host name and database name in Figure 3-50, which gives an overview of our case study scenario. Example 3-5 Verify control server (twh_list_cs) C:TWHtoolsbin>twh_list_cs.bat Listing the control server information in the Tivoli Data Warehouse registry. Control Server: Control Server Database Server Information: Host name: tdw003.itsc.austin.ibm.com Vendor: DB2 UDB Port: 50000 Database name: TWH_MD Control Server Database Client Information: Node name: Not applicable. Database alias: TWH_MD ODBC connection name: TWH_MD Tivoli Data Warehouse component version: 1.2.0.0 The Tivoli Data Warehouse is built on top of the IBM DB2 Universal Database Enterprise Edition Data Warehouse. To check whether the DB2 warehouse services are running enter Start -> Programs -> Administrative Tools -> Services and look for the services named Warehouse logger and Warehouse server. Both must be up and running. Figure 3-58 shows the services windows focusing on the DB2 data warehouse services. Figure 3-58 DB2 Data Warehouse services Verify Tivoli Data Warehouse central data warehouse databases Use the batch twh_list_cdws.bat to display information about the central data warehouse databases. Example 3-6 shows the output for the case study installation. Both central data warehouses are displayed. The second installed central data warehouse on Z/OS has TCDW1 as database alias and TWH_CDW1 as ODBC connection name assigned.136 Implementing Tivoli Data Warehouse 1.2
    • Example 3-6 Verify central data warehouse (twh_list_cdws)C:TWHtoolsbin>twh_list_cdws.batListing the central data warehouse information in the Tivoli Data Warehouseregistry.Central Data Warehouse: Central Data Warehouse Database Server Information: Host name: tdw004.itsc.austin.ibm.com Vendor: DB2 UDB Port: 50000 Database name: TWH_CDW Control Server Database Client Information: Node name: TDW1 Database alias: TWH_CDW ODBC connection name: TWH_CDW Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: ECentral Data Warehouse: Central Data Warehouse Database Server Information: Host name: wtsc66oe.itso.ibm.com Vendor: DB2 390 Port: 33768 Database name: DB2E Control Server Database Client Information: Node name: TDW3 Database alias: TCDW1 ODBC connection name: TWH_CDW1 Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: E Verify Tivoli Data Warehouse Data Mart databases. To check on the data warehouse data mart databases use the twh_list_marts.bat command. Example 3-7 shows the output for our case study scenario. You notice the difference between DB2 UDB for OS/390 and z/OS databases and IBM DB2 Universal Database Enterprise Edition DB2 databases. However, you see no differences between Windows and UNIX based databases. In our case study the TWH_MART database resides on an AIX box (refer to Figure 3-50 on page 127).Example 3-7 Verify data mart databasesC:TWHtoolsbin>twh_list_marts.batListing the data mart information in the Tivoli Data Warehouse registry.Data Mart: Data Mart Database Server Information: Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 137
    • Host name: tdw009.itsc.austin.ibm.com Vendor: DB2 UDB Port: 50000 Database name: TWH_MART Control Server Database Client Information: Node name: TDW2 Database alias: TWH_MART ODBC connection name: TWH_MART Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: E Data Mart: Data Mart Database Server Information: Host name: wtsc66oe.itso.ibm.com Vendor: DB2 390 Port: 33768 Database name: DB2E Control Server Database Client Information: Node name: TDW3 Database alias: TMART1 ODBC connection name: TWH_MART1 Tivoli Data Warehouse component version: 1.2.0.0 Enabled/Disabled: E Verify remote agents sites. To verify the remote agent sites enter the command twh_list_agentsites.bat. Example 3-8 shows the output for our case study installation. The command does not return the remote agents for internal use! Example 3-8 Verify remote agent site (twh_list_agentsites) C:TWHtoolsbin>twh_list_agentsites.bat Listing the agent site information in the Tivoli Data Warehouse registry. Local Agent Site: Host name: tdw003.itsc.austin.ibm.com Version: 1.2.0.0 Enabled/Disabled: Enabled Warehouse Pack Usage: Local Agent Site: Host name: tdw004.itsc.austin.ibm.com Version: 1.2.0.0 Enabled/Disabled: Enabled Warehouse Pack Usage: Local Agent Site: Host name: tdw009.itsc.austin.ibm.com138 Implementing Tivoli Data Warehouse 1.2
    • Version: 1.2.0.0 Enabled/Disabled: Enabled Warehouse Pack Usage: To view the data warehouse remote agent sites you may also select from the windows desktop Start -> Programs -> IBM DB2 -> Control Center to open the DB2 control center. From the DB2 Control Center select Tools -> Data Warehouse Center to open the Data Warehouse Center where you select Administration -> Agent Sites. Figure 3-59 shows the result for the case study installation. All remote agents are listed in this view. However, in addition to the agents listed by the twh_list_agentsites in Figure 3-59, the agents for internal use are also listed.Figure 3-59 Remote Agent Sites Verify the installation of the Crystal Enterprise Professional for Tivoli Server (twh_update_rptsrv) To verify the installation of the Crystal Enterprise Professional for Tivoli Server and its registration with the Tivoli Data Warehouse control server enter the command twh_update_rptsvr -l from DOS-shell. Figure 3-9 shows the output of this command for the case study installation.Example 3-9 Verify Crystal Enterprise Professional for Tivoli installationC:TWHtoolsbin>twh_update_rptsrv -lReport Server Host Name: tdw002.itsc.austin.ibm.comReport Server User ID: Administrator Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 139
    • Verify users, sources, and targets. Enter the command twh_update_userinfo -l to get an overview of the used user names and the available sources and targets. Example 3-10 Verify data user C:TWHtoolsbin>twh_update_userinfo -l tdw003.itsc.austin.ibm.com: 50000: TWH_MD: db2admin: ANM_TWH_MD_Target CDW_TWH_MD_Source tdw004.itsc.austin.ibm.com: 50000: TWH_CDW: db2admin: AN1_TWH_CDW_Target ANM_TWH_CDW_Source ANM_TWH_CDW_Target CDW_TWH_CDW_Source CDW_TWH_CDW_Target tdw009.itsc.austin.ibm.com: 50000: TWH_MART: db2inst1: ANM_TWH_MART_Target CDW_TWH_MART_Source CDW_TWH_MART_Target wtsc66oe.itso.ibm.com: 33768: DB2E: tivo01: CDW_TWH_CDW1_Source CDW_TWH_CDW1_Target CDW_TWH_MART1_Source CDW_TWH_MART1_Target CDWTD0002I The command that changes user IDs and passwords (twh_update_userinfo) completed.140 Implementing Tivoli Data Warehouse 1.2
    • 3.6.1 Verifying the remote agent install Remote agents register with the central data warehouse control server. In order to verify the installation and thus the successful registration with the central data warehouse control server, perform the following steps: 1. Login to the computer running the central data warehouse control server. 2. Select Start -> IBM DB2 -> Control Center. 3. Within the DB2 Control Center window, select the Data Warehouse Center button from the toolbox. 4. Within the Data Warehouse Center window select Warehouse -> Administration -> Agent Sites in the browser tree. The all registered data warehouse agents are displayed. Figure 3-60 shows this view for our test environment. Here the agent sites list includes tdw004.itsc.austin.ibm.com and tdw009.itsc.austin.ibm.com, installations that were described previously in 3.5.1, “Installing IBM DB2 Warehouse Manager” on page 128 and 3.5.2, “Creating the remote agent sites” on page 131. Figure 3-60 Verify Remote Agents on Tivoli Data Warehouse Control Center Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 141
    • 3.7 Installing warehouse enablement packs A warehouse enablement pack is the part of a Tivoli software product that provides warehouse functionality. It can be provided on the installation media for the product, on a separate CD, or in a collection of warehouse enablement packs. When provided on the installation media for the product, a warehouse enablement pack is located in a subdirectory named tedw_apps_etl. Part 2, “Case study scenarios” on page 159 provides instructions on how to install and configure a warehouse enablement pack in great detail. However, before installing any warehouse enablement pack, we recommend that you complete the following tasks: 1. Review the implementation guide manual for each warehouse enablement pack you plan to install. The implementation guide manual is located on the installation media of the warehouse enablement pack. For warehouse enablement packs designed for Tivoli Enterprise Data Warehouse 1.1, it is usually in the tedw_apps_etl/<productcode>/pkg/version/doc subdirectory, and for warehouse enablement packs designed for Tivoli Data Warehouse 1.2, it is typically in the <productcode>/doc subdirectory of the warehouse enablement pack installation media, where <productcode> is the AVA code of the product. 2. Define and create warehouse agent sites. See 3.5, “Installing warehouse agents” on page 126. 3. If the warehouse enablement pack provides an ETL1, which reads data from operation data sources, make sure the warehouse agent site that will run the ELT1 can connect to the operational data source databases. 4. Back up your TDW deployment to ensure that you can return to a known valid state if you encounter an unrecoverable error. 5. Check if any other ETL process steps for any other warehouse enablement packs that are already installed are scheduled to run during the warehouse enablement pack installation. Change the mode of the existing ETL steps to Test to prevent them from running. 6. Verify the WEP installation by issuing the following command on the Tivoli Data Warehouse control server: twh_configwep -u db2admin -p <your password> -f list Example 3-11 shows the output for the IBM Tivoli NetView Warehouse Enablement Pack.142 Implementing Tivoli Data Warehouse 1.2
    • Example 3-11 twh_configwep command outputC:TWHtoolsbin>twh_configwep -u db2admin -p <your password> -f listCDWCW0002I The twh_config_wep.pl program started.Installed warehouse enablement packs:CODE VERSION ROLE NAME---- ------------------ ---- ----------------------------------------------AN1 Version 1.1.0 V11 IBM Tivoli NetviewANM Version 1.1.0 V11 IBM Tivoli Netview---- ------------------ ---- ----------------------------------------------Source datasources used by warehouse enablement packs:CODE DATASOURCE CLIENT_HOSTNAME---- ---------------- ------------------------------------------------------AN1 ANM_SOURCE localhostANM ANM_SOURCE localhost---- ---------------- ------------------------------------------------------Central data warehouse datasources used by warehouse enablement packs:CODE DATASOURCE CLIENT_HOSTNAME---- ---------------- ------------------------------------------------------AN1 TWH_CDW localhostANM TWH_CDW localhost---- ---------------- ------------------------------------------------------Data mart datasources used by warehouse enablement packs:CODE DATASOURCE CLIENT_HOSTNAME---- ---------------- ------------------------------------------------------ANM TWH_MART localhost---- ---------------- ------------------------------------------------------CDWCW0001I The twh_config_wep.pl program completed successfully. Chapter 3. Getting Tivoli Data Warehouse 1.2 up and running 143
    • 144 Implementing Tivoli Data Warehouse 1.2
    • 4 Chapter 4. Performance maximization techniques This chapter provides an overview, tips, and techniques you can use to improve the performance of your Tivoli Data Warehouse 1.2. We cover the following topics: “DB2 performance” on page 146 “Operating system performance tuning” on page 150 “Tivoli Data Warehouse performance” on page 155 The following sections provide performance tuning tips for small, medium, and large configurations. Because these may not be the best settings for your environment, a suggested approach would be to use these values as a starting point and then to monitor system performance, making changes where necessary.© Copyright IBM Corp. 2004. All rights reserved. 145
    • 4.1 DB2 performance Depending on the size of your environment, there are many options available when tuning the performance of your IBM DB2 Universal Database Enterprise Edition servers. The following topics introduce some of the factors affecting DB2 performance. This is based on the information provided by the redbook Database Performance Tuning on AIX, SG24-5511. Buffer pools A buffer pool is an area of memory. In this area, database pages of user table data, index data, and catalog data are temporarily moved from disk storage. DB2 agents read and modify data pages in the buffer pool. The buffer pool is a key influence of overall database performance, because data can be accessed much faster from memory than from a disk. If most of the data required by applications is present in the buffer pool, then less time is needed to access this data. This improves performance. Buffer pools can be defined with varying page sizes including 4K, 8K, 16K and 32K. Prefetchers Prefetchers are present to retrieve data from disk and move it into the buffer pool before applications need the data. For example, applications needing to scan through large volumes of data would have to wait for data to be moved from disk into the buffer pool if there were no data prefetchers. Prefetchers are designed to improve the read performance of applications as well as utilities such as backup and restore, since they prefetch index and move data pages into the buffer pool, thereby reducing the time spent waiting for I/O to complete. The number of prefetchers may be controlled by the database configuration parameter NUM_IOSERVERS. Page cleaners Page cleaners are present to make room in the buffer pool, before agents and prefetchers read pages from disk storage and move them into the buffer pool. For example, if an application has updated a large amount of data in a table, many of the updated data pages in the buffer pool may not yet have been written on to disk storage. Such pages are called dirty pages. Since prefetchers cannot place fetched data pages onto the dirty pages in the buffer pool, these dirty pages must first be flushed to disk storage and become clean pages, so that prefetchers can find room to place fetched data pages from disk storage. The number of page cleaners may be controlled by the database configuration parameter NUM_IOCLEANERS.146 Implementing Tivoli Data Warehouse 1.2
    • The NUM_IOCLEANERS parameter allows you to specify the number ofasynchronous page cleaners for a database. These page cleaners write changedpages from the buffer pool to disk before the space in the buffer pool is requiredby a database agent. As a result, database agents should not have to wait forchanged pages to be written out so that they may use the space in the bufferpool. This improves overall performance of the database applications.If you set the parameter to zero (0), no page cleaners are started and as a result,the database agents will perform all of the page writes from the buffer pool todisk. This parameter can have a significant performance impact on a databasestored across many physical storage devices, since in this case there is a greaterchance that one of the devices will be idle. If no page cleaners are configured,your applications may encounter periodic log full conditions.LogsChanges to data pages in the buffer pool are logged. Agent processes updatinga data record in the database update the associated page in the buffer pool, andwrite a log record into a log buffer. To optimize performance, neither the updateddata pages in the buffer pool, nor the log records in the log buffer are written todisk immediately. They are asynchronously written to disk by page cleaners, andthe logger, respectively. The logger and the buffer pool manager cooperate andensure that an updated data page is not written to disk storage before itsassociated log record is written to the log. This ensures database recovery to aconsistent state from the log in the event of a crash such as a power failure. Anumber parameters can be used here: logfilsiz: This parameter represents the size of the log files. logprimary: The primary log files establish a fixed amount of storage allocated to the recovery log files. This parameter allows you to specify the number of primary log files to be pre-allocated. logsecond: This parameter specifies the number of secondary log files that are created and used for recovery log files (only as needed). logbufsz: This parameter specifies the amount of database shared memory to use a buffer log before writing these records to disk.Java heap parametersIn general, increasing the size of the Java heap improves throughput until theheap no longer resides in physical memory. After the heap begins swapping todisk, Java performance deteriorates drastically. Chapter 4. Performance maximization techniques 147
    • Therefore, the maximum heap size needs to be low enough to contain the heap within physical memory. MON_HEAP_SZ indicates the amount of memory (in 4K pages) which is allocated for database monitor data (at db2start). The amount of memory needed will depend on the number of snapshot switches which are turned on and active Event Monitors. If the memory heap is insufficient, an error will be returned when trying to activate a monitor and it will be logged to the db2diag.log file. Database heap There is one database heap per database, and the database manager uses it on behalf of all applications connected to the database. It contains control block information for tables, indexes, table spaces, and buffer pools. It also contains space for event monitor buffers, the log buffer (logbufsz), and temporary memory used by utilities. Therefore, the size of the heap will be dependent on a large number of variables. The control block information is kept in the heap until all applications disconnect from the database. Sort heap size (sortheap) If the rows to be sorted occupy more than the space available in the sort heap, several sort passes are performed, where each pass sorts a subset of the entire set of rows. Each sort pass is stored in a temporary table in the buffer pool, which might be written to disk. When all the sort passes are complete, these sorted subsets are merged into a single sorted set of rows. A sort is considered to be piped if it does not require a temporary table to store the final, sorted list of data. That is, the results of the sort can be read in a single, sequential access. Piped sorts result in better performance than non-piped sorts and will be used if possible. I/O servers Configuring enough I/O servers with the num_ioservers configuration parameter can greatly enhance the performance of queries for which prefetching of data can be used. To maximize the opportunity for parallel I/O, set num_ioservers to at least the number of physical disks in the database. It is better to overestimate the number of I/O servers than to underestimate. If you specify extra I/O servers, these servers are not used, and their pages are paged out. As a result, performance does not suffer. Next, we describe some further performance features that can affect the SQL performance with indexes.148 Implementing Tivoli Data Warehouse 1.2
    • RUNSTATSThis command updates statistics about the physical characteristics of a table andthe associated indexes. The optimizer uses these statistics when determiningaccess paths to the data.This utility should be called when a table has had many updates, or afterreorganizing a table.This command should be run on all tables, and a typical usage would be:runstats on table <table name> with distribution and detailed indexes allDuring our testing, we developed a script which will perform runstats on everytable, in every database, in the instance pointed to by the environment variableDB2INSTANCE.REORGCHKThis command line utility calculates statistics on the database to determine iftables or indexes, or both, need to be reorganized or cleaned up. It can eitheruse existing statistics or perform runstats to create up-to-date statistics.REORGThis command reorganizes an index or a table. The index option reorganizes allindexes defined on a table by rebuilding the index data into un-fragmented,physically contiguous pages. The table option reorganizes a table byreconstructing the rows to eliminate fragmented data, and by compactinginformation. If you specify an index as part of the table reorganization, then thetable will be physically ordered by that index. With Version 7 of DB2, both tableand index reorganization cannot take place online.Data distributionIf you have a system that has multiple physical disk drives available to DB2, youwould probably want to split out the DB2 data across these multiple disks toreduce I/O contention as much as possible. In an ideal environment this couldmean separate disks for: Database logs Mirrored database logs (if this option has been set up in the database configuration file) Temporary space (used for sorting data and storing intermediate result sets) Table data Index data Chapter 4. Performance maximization techniques 149
    • Due to cost and resource constraints, this ideal scenario may not be completely possible. However, you should try to maximize the utilization of the available disks in your system. As an example, if you only have two physical drives, keep the logs on one drive and create the database and the physical data on the other. Tablespace type Once we have created a DB2 system and chosen our disk distribution, then we can decide how we want to store the data. DB2 has two types of tablespaces to store data, System Managed Space (SMS) and Database Managed Space (DMS). SMS is the default tablespace type, and three tablespaces of this type are created after the installation of DB2. These tablespaces allow the operating system to allocate and manage the space where the table data resides. DMS tablespaces give DB2 the ability to control storage space. The amount of space allocated to a DMS tablespace must exist upon creation, since these files are allocated upon creation, as opposed to SMS tablespaces, which grow within the specified file system. Both types of tablespaces have their own advantages and disadvantages, but in general SMS is easier to manage, whereas DMS can be faster. DMS is also more flexible. The regular data, large data, and indexes can be split between different DMS tablespaces. DMS can be created on raw unformatted disks. Recent enhancements to DMS tablespaces mean that it is now possible to add containers to a tablespace, remove containers, and reduce the size of containers. The Create Tablespace Wizard in the Control Center guides you through creating a tablespace.4.2 Operating system performance tuning Any application running in your environment will be subject to the performance and optimization settings of your operating system. It is important for these settings to be tuned before deployment to the Tivoli Data Warehouse and be monitored on a regular basis, possibly using IBM Tivoli Monitoring.4.2.1 Windows environments If you are using a Windows NT or Windows 2000 server for any Tivoli Data Warehouse components, you can improve performance by optimizing your system traffic. Perform the following steps to make these changes:150 Implementing Tivoli Data Warehouse 1.2
    • 1. Navigate through the path Start -> Settings -> Control Panel -> Network and Dial-up Connections -> Local Area Connection. 2. Click Properties on the Local Area Connection window, and select File and Printer Sharing for Microsoft Networks. 3. Select the option Maximize data throughput for networking applications.4.2.2 Primary Windows performance factors We begin our discussion on system performance by taking into consideration the primary Windows performance factors. As with any system, performance begins by laying a good foundation of well balanced hardware resources that can be exploited by the operating system, and eventually, application specific software such as DB2 UDB. In this section we focus on two primary Windows performance factors: system hardware and system software. Specifically, we cover system hardware, memory, processor, and storage. System hardware: The performance goals of system hardware selection and configuration are to produce a well balanced system that can sustain high rates of overall system utilization. To achieve a well balanced system, we must adequately size individual hardware resources so that no single resource results in a system bottleneck. In this section we look at primary Windows performance factors from a system hardware perspective. You should take into consideration how you plan to scale your system, when planning for additional capacity requirements. Scaling-out is achieved by adding more resources, such as memory, processors, and storage across multiple systems. In general, scaling-out is much easier in terms of hardware planning, as you scale-out by simply adding additional hardware resources (processors, memory, etc.) by adding additional servers. A scale-up approach typically requires a little more planning, as you may reach hardware system limitations in terms of the amount of resources that can be added to the server. Memory: Since accessing data in memory is faster than accessing data on hard drives, the primary factor in terms of memory is quantity. Although memory speeds are also factors, this is seldom an option when configuring a system, unless you are willing to select and configure another system altogether. The amount of physical memory is a critical system hardware resource that can have a huge impact on overall performance. In general, the cost of memory on commodity servers is usually an insignificant factor when compared to the cost of other hardware resources. Chapter 4. Performance maximization techniques 151
    • Today most systems based on the 32-bit Intel® Architecture (IA-32) support the Physical Address Extensions (PAE) capabilities of the IA-32. PAE provides operating system software with an instruction set to address physical memory above 4 GB. Operating systems that take advantage of the PAE can address up to 64 GB physical memory. Given the memory extensions of the IA-32, the primary factor in memory selection will most likely not be the cost of the memory itself, but rather the incremental cost moving from one edition of the Windows operating system to the next in order to address more physical memory. Processor: Most systems are limited by the total number of central processing units (CPU) they can support. Typically a 4-way system cannot be upgraded to an 8-way system unless it is indeed a true 8-way that was populated with only 4 processors. There are a few systems on the market today, such as the IBM x440 and the Unisys ES7000, that can be expanded beyond the total number of original processors by adding additional processor expansion modules. Besides quantity and speed, another important consideration in terms of processor selection is the size of the internal L2 cache. Slower processors with larger internal caches have shown significant throughput advantages for database applications over faster processors with smaller internal caches. Another factor to consider when selecting the number of processors is the operating system software costs. There are incremental licensing costs associated with each Windows 2000 or Windows Server 2003 edition to support more processors. Storage: The disk subsystem has been an area of much debate over the last several years. Most disk subsystems will implement some form of redundancy that has always favored recoverability over performance. In recent years improvements in technology have been able to overcome many of the performance limitations imposed by implementing redundant disk arrays. Performance characteristics of disk controllers include speed, throughput, channels, and cache. Care should be taken in the placement of disk controllers in the system. Although most disk adapters are backwards compatible, you want to match the disk controller speed with that of the systems PCI bus. You should avoid placing faster 64-bit 66 Mhz disk controllers in slower 32-bit 33 Mhz PCI slots. You should also consider the number of disk controllers in your system as well. Attaching a single disk controller with several I/O channels might be capable of driving your subsystem, but can quickly saturate a single PCI bus, not to mention introduce a single point of failure into your system. If possible, you should also avoid placing disk controllers on PCI buses populated with other I/O intensive resources.152 Implementing Tivoli Data Warehouse 1.2
    • Performance characteristics of disk subsystems include disk speed, size, cache, and the number of physical disks in the subsystem. You should favor a subsystem with a large number of small drives over a small number of large drives. If this is impractical, plan for growth by choosing a large number of large drives. Best performance will be achieved for database applications with a large number of physical disks (5-10) per processor. For example, a large 32-way SMP server should be attached to a disk subsystem with a minimum of 160 physical disks, not including parity disks or hot spares. With an average 18-GB disk you would have almost 3 TB of total storage. At first this might seem impractical for a 1 TB database, but you need to consider space requirements for loading files and storing the most recent backup image before copying to tape. Hardware implementations of disk arrays are now commonplace on Intel based servers. Modern disk controllers support RAID levels 0, 1, 5, and 10, sometimes referred to as 0+1. As with most performance decisions, there is always a give (cost) and take (performance) associated with choosing which RAID level to implement.Operating system softwareIn this section we look at primary Windows performance factors from a systemsoftware perspective. We explore the Windows 2000 and Windows Server 2003operating system offerings.Windows 2000 serversThe Windows 2000 family of servers consists of Windows 2000 Server, Windows2000 Advanced Server, and Windows 2000 Datacenter Server. All Windows2000 server products build upon the already rich features of Windows NT 4.0Server and Windows NT 4.0 Server Enterprise Edition.Windows 2000 Server EditionWindows 2000 Server is the entry level edition of the operating system. Itprovides the same basic support functionality of its predecessor Windows NT 4.0Server.The Windows 2000 Server edition is licensed to support 32-bit Intel based serverfrom one (1) to four (4) processors and can address up to 4 GB of physicalmemory. Although limited to 4 GB of addressable physical memory, Windows2000 server as well as all other versions of the operating system support thePhysical Address Extensions (PAE) provided by the IA32 architecture.The Microsoft specific implementation provides a set of APIs called AddressWindowing Extensions (AWE). These APIs allows 32-bit applications to addressreal physical memory above the 4 GB line using a windowing approach that wediscuss in detail later in this chapter. Chapter 4. Performance maximization techniques 153
    • Windows 2000 Advanced Server Edition Windows 2000 Advanced Server provides the same basic server functionality of its predecessor, Windows NT 4.0 Server Enterprise Edition. It includes all of the existing features of the base Windows 2000 Server, is licensed to support 32-bit Intel servers with one (1) to eight (8) processors. It can address up to 8 GB of physical memory using the Physical Address Extensions (PAE) provided by the IA32 architecture. Windows 2000 Advanced Server also includes 4 GB Tuning support. 4 GB Tuning, introduced by Microsoft in Windows NT 4.0 Enterprise Edition Service Pack 3, allows the operating system to make an additional 1 GB of memory available to applications. Windows 2000 Advanced Server also include the Microsoft Clustering Service (MSCS). This optionally installed service provides failover clustering support for up to two nodes in the cluster, as did Windows NT 4.0 Server Enterprise Edition. Windows 2000 Datacenter Server Edition Windows 2000 Datacenter Server includes all of the features of Windows 2000 Advanced Server. This edition of the Windows 2000 operating system can be licensed to support 32-bit Intel servers with 1-8, 1-16 or 1-32 processors. It can address up to 64 GB of physical memory using the Physical Address Extension provided by the IA32 architecture. Windows 2000 Datacenter Server extends the MSCS support to include up to 4 nodes in a single cluster. Windows 2000 Datacenter Server also includes a feature called Winsock Direct. It enables unmodified Windows Sockets (Winsock) applications that use TCP/IP to exploit the performance benefits of system area networks (SANs). Winsock Direct enables efficient high-bandwidth, low-latency messaging that conserves processor time for application use. High-bandwidth and low-latency inter-process communication (IPC) and network system I/O allow more users on the system and provide faster response times and higher transaction rates. Windows 2000 Server or Windows 2000 Advanced Server cannot be upgraded to Windows 2000 Datacenter Server. This is a rather new and unique approach that Microsoft has taken with Windows 2000 Datacenter Server, as the operating system cannot be purchased directly from Microsoft, but instead must be acquired as a complete solution including hardware, software, and services, from a Windows 2000 Datacenter Server OEM provider. Windows Server 2003 Windows Server 2003 is the follow-on to the Windows 2000 family of operating systems. There are three editions of Windows Server 2003.154 Implementing Tivoli Data Warehouse 1.2
    • For entry level systems, Windows Server 2003 Standard Edition will provide support for 32-bit Intel servers with up to 2 CPUs and 4 GB memory. Windows Server 2003 Enterprise Edition will provide support for both 32-bit and 64-bit Intel servers with up to 8 CPUs. The 32-bit version will support 32 GB of memory and the 64-bit version will support up to 64 GB of memory. The Microsoft Cluster Service will be included and provide support for up to 8 nodes in a single cluster. Windows Server 2003 Datacenter Edition will provide support for both 32-bit and 64-bit Intel servers with up to 32 CPUs. The 32-bit version will support 64 GB of memory and the 64-bit version will support up to 128 GB of memory. The Microsoft Cluster Service will be included and provide support for up to 8 nodes in a single cluster.4.2.3 AIX environments To enable large file support, ensure that the Soft FILE size value is set to -1 for the IBM DB2 Universal Database Enterprise Edition administrative user. This parameter defines the largest soft file size, in 512-byte blocks, that this user can create or extend when invoking a process. Perform the following steps to make this change: From the SMIT menu, navigate through the path Security & Users -> Users -> Change/Show Characteristics of a User. Type in the name of the IBM DB2 administrative user in the entry field. Scroll down to the Soft FILE size line and change the value listed to -1.4.3 Tivoli Data Warehouse performance The overall performance of the Tivoli Data Warehouse will be very much dependent on how well the hardware, operating system, and DB2 environment has been configured. A number of Tivoli Data Warehouse components which can be tuned are discussed next. Distributed install The Tivoli Data Warehouse components can be, but do not need to be, installed on the same systems as other Tivoli software or on the systems where the operational data stores reside. The operational data stores are on a system that is not part of the Tivoli Data Warehouse deployment. As an example, the TEC database will have operational data. Chapter 4. Performance maximization techniques 155
    • Crystal Enterprise should reside on a system that does not have other Tivoli Data Warehouse or Tivoli software products on it. Users access Crystal Enterprise reports using a Web browser from any system Other types of data analysis tools, should be located on systems outside your Tivoli Data Warehouse deployment. Control server and warehouse agent The DB2 warehouse agent is the component of IBM DB2 Warehouse Manager that manages the flow of data between data sources and targets that are on different computers. By default, the control server uses a local warehouse agent to manage the data flow between operational data sources, central data warehouse databases, and data mart databases. In some environments, this configuration is sufficient. Optionally, you can create agent sites on other Windows and UNIX systems in your environment. The control server can use the agents on these computers to manage data flow. In a distributed deployment, you can improve the performance of Tivoli Data Warehouse by creating an agent site on the computer that is the target of the data transfer for a central data warehouse or data mart ETL routine. That computer becomes a remote agent site, which the Data Warehouse Center uses to manage the transfer of Tivoli Data Warehouse data. This can speed up the data transfer as well as reduce the workload on the control server. The control server is the system that contains the control database for Tivoli Data Warehouse and is the system from which you manage your data. The control database contains metadata for both Tivoli Data Warehouse and for the warehouse management functions of IBM DB2 Universal Database Enterprise Edition. You can have only one control server in a Tivoli Data Warehouse deployment. On Windows and UNIX systems, using a warehouse agent on the computer that contains the central data warehouse database or a data mart database can potentially improve performance. ETL routines The scheduling of data warehouse ETLs should be done during off-peak hours to avoid impacting the performance of your operational data stores. For distributed environments across geographic locations, you may consider putting central data warehouse databases at each location, as each location may have different off-peak hours.156 Implementing Tivoli Data Warehouse 1.2
    • For example, if your operational data is on systems in the United Kingdom andthe United States, you might put a central data warehouse database on a systemin each location. This enables you to schedule the central data warehouse ETLfor each system at an appropriate off-peak time.The time taken for ETLs to complete depends on many factors, including theamount of data they have to process, the speed of the database in which thesource and target data reside, the performance of the network, and so forth.Ensure that the default scheduling interval is changed to an appropriate intervalfor your environment and data level.The ETL processes that update tables in the central data warehouse should notall be scheduled at the same time. There might be unknown dependencies in thedata, and updates to the same tables might cause performance problems,depending on your environment.Data Analysis programs can read directly from central data warehousedatabases without using data marts; but this use is not supported. Analyzinghistorical data directly from the central data warehouse database can causeperformance problems for all applications using the central data warehouse.Tivoli Decision Support and Tivoli Data WarehouseYou can continue to use Tivoli Decision Support at the same time and on thesame systems as Tivoli Data Warehouse. For the best performance, do notschedule Tivoli Data Warehouse ETLs and Tivoli Decision Support cube buildsconcurrently, if they access the same databases.You can continue to use Tivoli Decision Support for OS/390 at the same time andon the same systems as Tivoli Data Warehouse. For the best performance,schedule Tivoli Data Warehouse ETLs after Tivoli Decision Support for OS/390data collection has occurred.Sample tuningIn a recent deployment of Tivoli Data Warehouse, approximately 85% of thecurrent execution time of the ETL processes was consumed by the triggers andcorresponding sequences.With optimal tuning (without changes of SQL-statements), performance wasincreased by 82%, and with changes to SQL statements, performance can beincreased by 90%. Chapter 4. Performance maximization techniques 157
    • Each environment is different, and these performance improvements cannot be guaranteed, but our recommendations are as follows: Distribute the database to the three available hard disks (the test server in this scenario had three hard disks). Use DMS tablespaces instead of SMS tablespaces. Increase caches of the sequences (approximately 5000 may be enough, but this has to be tested). Execute runstats after loading data to the staging tables. Distribute the temporary tablespace also to the three available hard disks.158 Implementing Tivoli Data Warehouse 1.2
    • Part 2Part 2 Case study scenarios© Copyright IBM Corp. 2004. All rights reserved. 159
    • 160 Implementing Tivoli Data Warehouse 1.2
    • 5 Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack In this chapter we look at what is required to provide network availability reporting in an IBM Tivoli Netview 7.1.4 and Tivoli Data Warehouse 1.2 environment using the IBM Tivoli NetView Warehouse Enablement Pack (WEP). We cover the following topics: “Case study overview” on page 162 “IBM Tivoli NetView WEP overview” on page 163 “Prerequisites” on page 165 “Preparing NetView for data collection” on page 167 “Installation of the NetView WEPs” on page 181 “Testing, scheduling, and promoting the ETLs” on page 191 “Running NetView ETLs on remote agent sites” on page 198 “Reporting” on page 206© Copyright IBM Corp. 2004. All rights reserved. 161
    • 5.1 Case study overview Table 5-1 gives a brief summary of the distributed Tivoli Data Warehouse environment we started with to install the IBM Tivoli NetView Warehouse Enablement Pack. Table 5-1 Environment for NetView integration Host name Component Database Operating system TDW003 TDW Control DB2 Server W2000 FP4 Server 1.2 7.2FP10a TDW004 TDW Central DB2 Server W2000 FP4 Warehouse 1.2 7.2FP10a (remote agent site) TDW009 TDW Data Mart 1.2 DB2 Server AIX 5.2 7.2FP10a TDW002 Crystal Enterprise DB2 Client W2000 FP4 Server 9 7.2FP10a TDW008 NetView 7.1.4 - W2000 FP4 Figure 5-1 summarizes the distributed Tivoli Data Warehouse environment used in this chapter.162 Implementing Tivoli Data Warehouse 1.2
    • Agent site TWH_CDW Central Data Warehouse Windows 2000 Server Hostname: TDW004 TWH_MD TDW Control Server Tivoli NetView Windows 2000 Server Hostname: TDW003 Windows 2000 Server Hostname: TDW008 Data Mart TWH_MART Crystal Enterprise Server Data Mart Windows 2000 Server IIS Web Server AIX 5 Crystal Enterprise Professional 9 for Tivoli Hostname: TDW009 Hostname: TDW002 Figure 5-1 Distributed deployment scenario5.2 IBM Tivoli NetView WEP overview The IBM Tivoli Netview warehouse enablement pack enables Tivoli Data Warehouse reporting of network availability and performance information. The latter is provided by the IBM Tivoli Netview snmpcol daemon. Network node, SmartSet, layer 2, and performance information is recorded to an availability database by IBM Tivoli Netview (hereafter referred to as NetView), which is used as input to Tivoli Data Warehouse. Note: Layer 2 information is only collected when the IBM Tivoli Switch Analyzer product is installed. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 163
    • The IBM Tivoli NetView Warehouse Enablement Pack code is provided with the IBM Tivoli Netview 7.1.4 software. In Figure 5-2 the data flow of an integration of NetView in a Tivoli Data Warehouse environment is illustrated. We start with a brief description of the processes and their control. Node availability information is stored by the NetView process tdwdaemon into the NetView source database. The NetView snmpcol daemon writes performance information gathered by SNMP polling (CPU load, number of processes, etc.) into the NetView source database. The data upload to the NetView source database is controlled by the NetView server that is illustrated in Figure 5-2 by broken lines. The data flow within the Tivoli Data Warehouse is controlled by the Tivoli Data Warehouse control server. The generation and publishing of the NetView specific reports is controlled by the Crystal Enterprise server. snmpcol tdwdaemon IP Network NetView IP Network Server NetView Source Database ETL1 Central Data Warehouse ETL2 Datawarehouse Control Server Data Mart Web Server Crystal Server Data Control Web Reports Figure 5-2 IBM Tivoli NetView Warehouse Enablement Pack data flow164 Implementing Tivoli Data Warehouse 1.2
    • 5.3 Prerequisites The stated prerequisites, as per the IBM Tivoli NetView Warehouse Enablement Pack Implementation Guide - Version 1.1.0, SC32-1237-00, which can be found in the tedw_apps_etlanmpkgv110doc directory of the enablement pack software, are: IBM Tivoli NetView Version 7.1.4 IBM DB2 Universal Database Enterprise Edition Version 7.2 IBM DB2 Universal Database Enterprise Edition Version 7.2 Fix Pack 6 Tivoli Enterprise Data Warehouse required e-fixes to IBM DB2 UDE v7 Fix Pack 6 (1.1-TDW-0002) Tivoli Enterprise Data Warehouse Version 1.1 Tivoli Enterprise Data Warehouse 1.1 Fix Pack 2 (1.1-TDW-FP02) In this case study scenario chapter we will be using Tivoli Data Warehouse 1.2 in a previously built distributed environment, as described in Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71. Here is a list of the products used and their releases in the case study scenario: IBM Tivoli NetView 7.1.4 IBM Tivoli Netview 7.1.4 early fix PJ29597 IBM DB2 Universal Database Enterprise Edition Version 7.2 Fix Pack 10a (shipped with Tivoli Data Warehouse 1.2) IBM Tivoli NetView Warehouse Enablement Pack version 1.1.0 Tivoli Data Warehouse Version 1.25.3.1 Verifying prerequisites From the stated prerequisites, we determine what action (if any) is required on our NetView Server platform, in our case study environment. These actions are described in Table 5-2. Table 5-2 NetView WEP Prerequisite Check - NetView server platform Prerequisite Comment Action required IBM Tivoli Netview 7.1.4 Install IBM Tivoli Netview 7.1.4 IBM DB2 Universal We will be using a DB2 Install the DB2 Client Database Enterprise UDB 7.2 Fix Pack 10a level version 7.2 fix pack 10a on Edition Version 7.2 Fix on a separate server from the NetView server Pack6 the NetView server. machine. We used the installation media shipped with Tivoli Data Warehouse 1.2 Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 165
    • 5.3.2 Gathering installation information Table 5-3 provides the information to be used during the installation of the NetView warehouse enablement pack. Table 5-3 Netview Enablement Pack installation information Information Description Default value Used value required NetView availability Name of the database NETVIEW NETVIEW DB name that will be created during installation to store the NetView availability data. DB2 user ID A DB2 user ID with db2admin db2inst1 create authority. DB2 user ID The DB2 user ID As required password password Data purge days The number of days the 90 Leave as default availability data will be retained before being purged. SmartSet The time of day to load 23 Leave as default membership SmartSet information. retrieval time List of SmartSets The name of the Routers Change to All in SmartSets, for which order to produce availability data will be more data. collected. Note: The Router SmartSet is required. Warehouse pack The directory name usrovdwpack Leave as default install directory where the enablement pack will be installed. NetView availability Name of the database NETVIEW NETVIEW DB name that will be created during installation to store the NetView availability data. DB2 user ID A DB2 user ID with db2admin db2inst1 create authority. DB2 database The DB2 password. password166 Implementing Tivoli Data Warehouse 1.2
    • 5.4 Preparing NetView for data collection NetView uses a DB2 database as source database for its availability and performance data. In this section we discuss how to set up NetView to fill the source database with data. Before performing any configuration on the NetView server, we need to perform the following steps: Ensure that the DB2 Server to host the Netview source database is at proper level. The minimum required by the NetView warehouse enablement pack is IBM DB2 Universal Database Enterprise Edition Version 7.2 Fix Pack 6. Install and configure the DB2 Client on the NetView Server machine in case the NetView source database will be placed in a separate machine. Ensure the DB2 client version and level is compatible with the DB2 server. Note: We recommend to use the IBM DB2 version and level shipped with Tivoli Data Warehouse 1.2.5.4.1 Enabling NetView to export data for Tivoli Data Warehouse In this section we describe how to enable NetView to fill out its source database for use with the Tivoli Data Warehouse 1.2. NetView includes an easy-to-use administration tool that: Creates and populates the netView source database Configures the ODBC connection to the DB2 source database Modifies the SNMP Collection daemon to store performance data on the source database Registers and starts the tdwdaemon which stores network availability data to the source database Perform following steps configure enable NetView to collect data: 1. Log in to your NetView server with administrator authority. 2. Start the installer for data export from NetView to the DB2 source database by selecting Start -> Programs -> Tivoli NetView -> Administration -> Configure data Export to DB2 for use in Tivoli Enterprise Data Warehouse from the windows desktop. 3. A configuration menu, as shown in Figure 5-3 opens. Fill in the appropriate data as collected in 5.3.2, “Gathering installation information” on page 166. Figure 5-3 shows this menu filled with data according to our case study scenario. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 167
    • 4. Select OK to proceed. In our case study installation, we used a remote DB2 server residing on AIX host tdw009. Figure 5-3 NetView Configure data export to DB2 - Parameters 5. Select Create Database in the popup shown in Figure 5-4. The NETVIEW source database and its tables are created. Figure 5-4 NetView Configure data export to DB2 - create database 6. Select Yes in the popup window shown in Figure 5-5 to register and start the Data Warehouse daemon (tdwdaemon).168 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-5 NetView Configure data export to DB2 - register and start tdwdaemonDuring the configuration a number of logs may be written, depending on whetherany problems are encountered during installation. The log files we discoveredwere: usrovdwpackDWP_Install_stdout.log usrovlogTDWError_mmddhh.log usrovlogtdwdaemon.logCheck these logs for unusual messages.Verifying db updatesWe verified that data was actually being written to the availability database byissuing the following command: db2 select count(*) from netview.netview_nodesThe greater than zero count returned (as shown in Example 5-1) indicated thatdata was in fact being written.Example 5-1 Verify NetView source database updatesC:DB2SQLLIBBIN>db2 connect to NETVIEW user db2inst1 using <DB password> Database Connection Information Database server = DB2/6000 7.2.8 SQL authorization ID = DB2INST1 Local database alias = NETVIEWC:DB2SQLLIBBIN>db2 select count(*) from netview.netview_nodes1----------- 349 1 record(s) selected. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 169
    • If the count had been zero, we could have tried the following sequence of actions: 1. Stop the tdwdaemon daemon: ovstop tdwdaemon 2. Stop the netmon daemon: ovstop netmon 3. Start the tdwdaemon daemon: ovstart tdwdaemon 4. Start the netmon daemon: ovstart netmon 5. Verify that both daemons are now active: ovstatus tdwdaemon ovstatus netmon 6. Preload the current NetView status to the availability db: netmonaction.bat 500 7. Verify that data is being written by checking the count on the netview.netview_nodes table again: db2 select count(*) from netview.netview_nodes5.4.2 NetView SmartSets configuration Some of the reports make use of NetView SmartSets. In this section we give a short introduction to the SmartSets concept and show how to create and configure them. SmartSets are containers that are populated with NetView objects. Dynamic or static filters define which objects are mapped to a SmartSet. You can filter by the objects attributes. A fresh NetView installation brings already a number of predefined SmartSets like CriticalNodes, which contain all network nodes that are unavailable, or like Routers, which contain all router nodes.170 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-6 shows the SmartSet desktop of NetView. Some of the network nodesare not available. Therefore, the CriticalNodes SmartSet is displayed as red.Figure 5-6 NetView SmartSet desktopWe now explain how to create new NetView SmartSets. However, you may needto vary the steps to meet your requirements. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 171
    • From the NetView desktop, select Submap -> New Smartset ... from the toolbar to open the SmartSets configuration window shown in Figure 5-7. There, select the folder Advanced. Figure 5-7 Microsoft SmartSet Advanced attributes In the Combine Find Condition field you may insert conditions which meet your needs. We insert the phrase (refer to Figure 5-7): (“SNMP sysObjectID = “1.3.6.1.4.1.311.1.1.3.1.2”) Thus the SmartSet is populated by all network nodes, whose SNMP enterprise ID is equal to the specified value. In this case all Microsoft Windows 2000 servers with SNMP option populate this SmartSet.172 Implementing Tivoli Data Warehouse 1.2
    • Table 5-4 shows the configuration for all three SmartSets we created for our casestudy. The SmartSet IBM contains all Nodes running on an IBM operatingsystem. (However, in our case study, NetView found only AIX hosts.) TheSmartSet TDW contains all nodes whose host names start with the letters tdw.All private enterprise object IDs are listed at the following Web site: http://www.iana.org/assignments/enterprise-numbersTable 5-4 Case Study SmartSets attributes SmartSet Name Combine Find Condition field Microsoft (“SNMP sysObjectID = “1.3.6.1.4.1.311.1.1.3.1.2”) IBM “SNMP sysObjectID ~ “1.3.6.1.4.1.2.*”) TDW “SNMP sysObjectID ~ “^tdw.*”)After filling the Combine Find Condition field shown in Figure 5-7, select CreateSmartSet ... . A popup appears as shown in Figure 5-8. Insert the name of yourSmartSet and optionally a descriptive text. We used the name Microsoft for theSmartSet in our case study. Then click OK to actually create the SmartSet.Figure 5-8 Create Microsoft SmartSet Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 173
    • If you open the folder SmartSets in the SmartSet configuration window again, the newly created SmartSets are now listed as shown in Figure 5-9. The highlighted line shows the new Microsoft SmartSet. Figure 5-9 NetView SmartSets - Attributes174 Implementing Tivoli Data Warehouse 1.2
    • The new SmartSets are also displayed on the NetView SmartSet desktop asshown in Figure 5-10.Figure 5-10 NetView SmartSets - OverviewDouble-click the Microsoft SmartSet icon to open the container as shown inFigure 5-11. NetView polls the enterprise object ID as specified in Table 5-4 onpage 173 via SNMP. Therefore, only SNMP managed nodes populate theMicrosoft SmartSet. Windows 2000 server without SNMP option or with differentSNMP community names are not included. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 175
    • Figure 5-11 SmartSet Microsoft contents5.4.3 Configuring NetView Data Warehouse daemon (tdwdaemon) The NetView tdwdaemon fills the NetView source database with availability data from the network nodes. It also controls the expiration of data within the NetView data warehouse source database. The behavior of the tdwdaemon can be changed by editing its configuration file, which is located by default under the name usrovconftdwdaemon.properties. Most of the parameters are set by the configuration process explained in the previous 5.4.1, “Enabling NetView to export data for Tivoli Data Warehouse” on page 167 and seldom need to be changed. The configuration parameters defined in the tdwdaemon.properties file are: SMARTSETS Here you can specify a comma-separated list of the NetView SmartSets you want to report. Here are some notes on this issue: – The SmartSet names are separated by commas and no white characters. – SmartSet names are case sensitive.176 Implementing Tivoli Data Warehouse 1.2
    • – The default value is Routers.– The Routers SmartSet is required. If you change the SMARTSETS settings, make sure that Routers is included in your list.– If you specify All, availability data of all network nodes will be stored in the source database.– In our case study installation, we have created new NetView SmartSets called TDW, IBM and Microsoft. We selected our self-made SmartSets along with the required SmartSet Routers as shown in Example 5-2.SMARTSET_LOAD_TIMEGives the hour when the NetView SmartSet population is loaded to theNetView data warehouse source database. The default value 23 means, thedata is loaded every day at 11 pm. Note: It is recommended to schedule the ETL1 ANM_c05_ETL1_Process within the Tivoli data warehouse at least 1 hour after the SmartSet load timeOUTAGE_STORAGE_TIMEThis parameter specifies the number of days before availability data expiresin the NetView data warehouse source database. The default value is 90days. Note: The SmartSet data is loaded once a day. Therefore the NetView source database contains a snapshot of this particular point in time. SmartSets which population changes rapidly like for instance CriticalNodes may not be suitable for reporting purposesDBPASSWORDThe encrypted db2 password Note: To change the DB2 password, execute the updateDBPassword.bat command in the /usr/ov/bin directory. An example of this follows: updateDBPassword.bat c:usrovconftdwdaemon.properties newpwdDB2USERThe db2 user ID (db2 administrator or any user ID with create authority).DBNAMEThe name of the NetView source database. Default is NETVIEW. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 177
    • Port DB2 database IP-port of the NetView source database. Default is 50000. HOSTNAME Hostname of the server which hosts the NetView source database. NODENAME DB2 database node of the NetView source database. Default is TDWNODE. Example 5-2 NetView tdwdaemon configuration file tdwdaemon.properties #Thu Mar 18 14:36:46 CST 2004 PORT=50000 HOSTNAME=tdw009.itsc.austin.ibm.com DBUSER=db2inst1 SMARTSET_LOAD_TIME=23 NODENAME=TDWNOTE SMARTSETS=Routers,TDW,Microsoft,IBM DBPASSWORD=c3dqNDNy OUTAGE_STORAGE_TIME=90 DBNAME=netview After changing the tdwdaemon.properties file, you have to restart the tdwdaemon to apply the changes. Use the NetView commands ovstop and ovstart for this purpose. Example 5-3 shows the command shell dialog. Example 5-3 Restart the NetView data warehouse daemon tdwdaemon C:usrovbin>ovstop tdwdaemon Done C:usrovbin>ovstart tdwdaemon Done5.4.4 Verifying NetView data collection enablement In order to verify that the data collection is taking place, perform these tasks: Check the tdwdaemon. Check the snmpcollect daemon. Check the existence of data in the NetView source database. Verifying NetView Data Warehouse daemon (tdwdaemon) The NetView daemon named tdwdaemon is used to perform availability data recording. We verify that this daemon started successfully by using the following command: ovstatus tdwdaemon178 Implementing Tivoli Data Warehouse 1.2
    • Example 5-4 shows the command shell output executing this command.Example 5-4 Status of NetView data warehouse daemon (tdwdaemon)C:usrovbin>ovstatus tdwdaemon object manager name: tdwdaemon behavior: OVs_WELL_BEHAVED state: RUNNING PID: 2092 last message: Initialization complete. exit status: -DoneThe log file for the tdwdaemon is named tdwdaemon.log and can be found in theusrovlog directory. Verify that no apparent errors were being reported on thetdwdaemon.log file. You will also find DB2 errors in this log file that thetdwdaemon has encountered while communicating with the NetView datawarehouse source database. Note: If no tdwdaemon.log exists for tdwdaemon to write to, it will create a new one. Deleting it prior to restarting the tdwdaemon makes it easier to review, because all the old entries are removed.Verifying NetView SNMP collector daemon (snmpcollect)Performance data is stored to the NetView data warehouse source database viathe NetView snmp collector daemon named snmpcollect. We verify that thisdaemon started successfully by using the following command: ovstatus snmpcollectExample 5-5 shows the command shell output executing this command.Example 5-5 Status of the NetView SNMP collector daemon (snmpcollect)C:usrovbin>ovstatus snmpcollect object manager name: snmpcollect behavior: OVs_WELL_BEHAVED state: RUNNING PID: 1616 last message: Initialization complete. exit status: -Done Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 179
    • The log file for the tdwdaemon is named snmpCol.trace and can be found in the usrovlog directory. Verify that no apparent errors were being reported on the snmpCol.trace log file in the directory usrovlog. You will also find DB2 errors in this log file that the snmpcollect daemon has encountered while communicating with the NetView data warehouse source database. Verifying data import to the Netview source database To verify whether data is imported to the NetView source database, you can use the DB2 control center to view the contents of the source databases tables. However, we give an example using the DB2 command line. Log in to the Netview database using the command: db2 connect to NETVIEW user db2inst1 using <password> Where <password> is your password for the db2inst1 database user. Here is a list of the items to check and the commands to use to check them: Availability data: db2 select count(*) from netview.netview_nodes Performance data: db2 select count (*) from netview.snmpcollection NetView Smartsets: db2 select count (*) from netview.smartsets In all three cases the count must be greater than 0. Example 5-6 shows the results of these commands for our test study environment. Note: You may have to wait until the SmartSet data upload has taken place. The time for this upload is specified in the configuration file tdwdaemon.properties. Example 5-6 Check the NetView source database C:DB2SQLLIBBIN>db2 connect to NETVIEW user db2inst1 using <password> Database Connection Information Database server = DB2/6000 7.2.8 SQL authorization ID = DB2INST1 Local database alias = NETVIEW C:DB2SQLLIBBIN>db2 select count(*) from netview.netview_nodes 1 -----------180 Implementing Tivoli Data Warehouse 1.2
    • 349 1 record(s) selected. C:DB2SQLLIBBIN>db2 select count(*) from netview.snmpcollection 1 ----------- 40787 1 record(s) selected. C:DB2SQLLIBBIN>db2 select count(*) from netview.smartsets 1 ----------- 5 1 record(s) selected. C:DB2SQLLIBBIN>5.5 Installation of the NetView WEPs As described in 5.2, “IBM Tivoli NetView WEP overview” on page 163, NetView can collect availability and performance information into the NetView data source database. In order to gather this data into the Tivoli Data Warehouse 1.2 environment, the following two warehouse enablement packs must be installed: NetView availability warehouse enablement pack (ANM) NetView SNMP performance warehouse enablement pack (AN1) In this section we explain how to install and configure both of them. Furthermore, we run the SNMP performance WEP on a data warehouse remote agent site.5.5.1 Backing up the TDW environment We strongly advise that you back up your data warehouse environment prior to any installation activity. In order to back up the Tivoli Data Warehouse 1.2 environment, stop the DB2 Warehouse logger service (vwlogger) and back up the control server database (TWH_MD), all central data warehouse databases, and all data mart databases. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 181
    • For example, in order to back up the TWH_MD, TWH_CDW, and TWH_MART databases, perform the following commands: db2 backup database TWH_MD to C:backup db2 backup database TWH_CDW to C:backup db2 backup database TWH_MART to C:backup5.5.2 Establishing ODBC connections In this section we show how to set up database connections to the NetView source database and how to configure the data warehouse sources and targets to make use of the ODBC data sources. Note: In difference to the IBM Tivoli NetView Warehouse Enablement Pack manual, we have not used ANM_SOURCE but NETVIEW as ODBC Data Source name. AIX DB2 instances can only use a maximum of 8 characters in names for database aliases. ANM_SOURCE has 10 characters and therefore cannot be used on AIX DB2 instances. Creating a NETVIEW ODBC data source By default, the data warehouse agent on the data warehouse control server is used to execute the NetView ETLs. So you have to create the ODBC data source on the data warehouse control server. However, if you plan to execute the ETLs or parts of them on a remote data warehouse agent, you must also create an ODBC data source for the NetView source database on the remote agent site. In order to create a NetView ODBC data source for the NetView source database Log in to the data warehouse control server or the remote agent site server with administrator authority and select Start -> Control Panel-> Administrative Tools -> Data Sources (ODBC). Then Select System DSN to view ODBC data source administration window as shown on the left side in Figure 5-12. Select Add... to open the Create New Data Source window as shown on the right side in Figure 5-12. Here you select the IBM DB2 ODBC DRIVER and proceed with selecting Finish.182 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-12 Create an ODBC data source for NETVIEWFigure 5-13 shows the ODBC IBM DB2 Driver - Add window. You can add anoptional description. If your NETVIEW source database is local or alreadycataloged within the DB2 client, you need to select NETVIEW from the Databasealias dialog.If you use a remote database, select Add.... This will open the Add DatabaseWizard window.Figure 5-13 Add an ODBC data source for NETVIEWHere are the different steps we have to perform in the Add Database Wizard(these steps are also shown in Figure 5-14):1. Source We selected Manually configure connection to database.2. Protocol We selected TCP/IP. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 183
    • 3. TCP/IP We inserted data as shown in Table 5-5. Table 5-5 Add database wizard - register TCP/IP Selection inserted values remark Host name tdw009.itsc.austin.ibm.com Hostname of the database server Port number 50000 default DB2 service port Service name - optional left blank 4. Database We inserted NETVIEW as the database name. Select Finish to start the registration. We wanted to maually configure our database connection The protocol standard used in our environment is TCP/IP The hostname of the DB2 UDB Server i.e. where our NetView availability data is kept, along with the default DB2 port number.Figure 5-14 Configure NetView Source database connectivity184 Implementing Tivoli Data Warehouse 1.2
    • 5.5.3 Installing NetView Enablement Pack Software In this section we describe how we installed the NetView availability warehouse enablement pack (ANM) in our case study scenario environment. The installation of the NetView SNMP performance warehouse enablement pack (AN1) can be performed in a similar way. Warehouse enablement pack installations must be performed on the control server machine. In order to install the NetView availability warehouse enablement pack (ANM), perform the following steps: 1. While logged on to the control server as a user ID with administrator authority, select Start -> Programs -> Tivoli Data Warehouse -> Install a Warehouse Pack. The InstallShield wizard is started. 2. On the welcome window, select Next to continue. In the next window, the path to your Common Logging Directory is displayed. In the case study installation, the path is C:Program Filesibmtivolicommon. 3. The InstallShield wizard checks the Tivoli data warehouse environment. This might take a few minutes. The next window allows you to select the warehouse packs to install. This list is empty, as shown in Figure 5-15. Figure 5-15 NetView WEP installation - List of WEPs to install Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 185
    • 4. Select Add and insert the location of the NetView warehouse pack installations properties file. This file is named twh_install_props.cfg. The properties file for NetView availability WEP with code ANM can be found on the installation media under directory tedw_apps_etlanmpkg, as shown in Figure 5-16. The properties file for NetView SNMP performance WEP with code AN1 can be found on the installation media under directory snmp_etlan1pkg. Figure 5-16 NetView WEP installation - Properties file 5. Select OK and/or Next to get back to List of Warehouse Packs to install. In contrast to Figure 5-15, the list is now populated with the NetView WEP as shown in Figure 5-17.186 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-17 NetView WEP installation - List of WEPs to install NetView6. Select Next to proceed with the installation. The installation takes a few minutes. Figure 5-18 shows the window which is displayed after successful installation.Figure 5-18 NetView WEP installation - successful installationTo install both NetView WEPs, you have to perform the installation steps twiceusing the different properties files of the two WEPs. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 187
    • 5.5.4 Defining the authority to the warehouse sources and targets The IBM Tivoli NetView Warehouse Enablement Pack adds NetView specific data warehouse sources and targets to the Tivoli Data Warehouse environment. After the initial installation of the NetView WEPs, these sources and targets need to be configured. (Refer to the IBM Tivoli NetView Warehouse Pack Implementation Guide, SG32-1237-00). The installation manual suggests using ANM_SOURCE as the name for the data source. In our case study we preferred NETVIEW for this purpose, as we explain in 5.5.2, “Establishing ODBC connections” on page 182. The IBM Tivoli NetView Warehouse Enablement Pack includes two WEPs: the availability WEP with code ANM and the SNMP performance WEP with code AN1. The latter does not include ETL2, data mart structure, or reports. Therefore neither data mart target nor central data warehouse source exists for this WEP. Table 5-6 shows an overview of all NetView based sources and targets. In the last column, the appropriate user IDs for our case study installation are shown. The databases TWH_MART and NETVIEW are running on the AIX platform and their default user is db2inst1. All other databases are running on Windows2000 platform with DB2 default user db2admin.Table 5-6 NetView sources and targets Name in data warehouse Description Database Used by User ANM_AVAIL_Source ODBC data source for the NETVIEW ETL1 db2inst1 NetView source database ANM_TWH_CDW_Source ODBC data source for the TWH_CDW ETL2 db2admin central data warehouse ANM_TWH_CDW_Target ODBC data target for the TWH_CDW ETL1 db2admin central data warehouse ANM_TWH_MART_Target ODBC data target for the data TWH_MART ETL2 db2admin mart ANM_TWH_MD_Target ODBC data target control TWH_MD ETL2 db2admin server database AN1_SNMP_Source ODBC data source for the NETVIEW ETL1 db2inst1 NetView source database AN1_TWH_CDW_Target ODBC data source for the TWH_CDW ETL1 db2admin Tivoli NetView warehouse source database188 Implementing Tivoli Data Warehouse 1.2
    • To configure the Tivoli data warehouse sources and targets for Netview, performthe following steps:1. Open the DB2 control center by selecting Start -> Programs -> IBM DB2 -> Control Center.2. From the DB2 control center, open the data warehouse center by selecting Tools-> Data Warehouse Center from the toolbar.3. In the data warehouse logon window, type the user ID of the data warehouse administrator (default to db2admin) and the appropriate password. Select Advanced to ensure that the control database is set to TWH_MD as shown in Figure 5-19.Figure 5-19 Data Warehouse Control Center - check control databaseIf you have opened the data warehouse control center, you will see a browsertree as shown in Figure 5-20. There are among others the leaves WarehouseSources and Warehouse targets. In this section we first discuss the configurationof the data warehouse authorities for the data warehouse sources and then thedata warehouse targets for use with the NetView WEPs.Authority for data warehouse sources1. Open the Warehouse sources tree and select Properties in the context menu of a source. In Figure 5-20 we selected ANM_AVAIL_Source for example.2. In the Properties window of the selected data warehouse source, select Data Source, also shown for ANM_AVAIL_Source in Figure 5-20. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 189
    • 3. As Data source name, select the specific data source for your environment. For our case study installation the data sources for the NetView data warehouse sources are listed in Table 5-6 on page 188. Therefore we inserted NETVIEW as data source for our ANM_AVAIL_Source example in Figure 5-20. 4. Insert the User ID and appropriate password to the NetView source database. In our case study, we used a DB2 database on AIX for which the default user ID is db2inst1, as shown in Figure 5-20. 5. Select OK to finish the data warehouse source configuration. Repeat these steps for all NetView data warehouse sources as listed in Table 5-6. Figure 5-20 Configure NetView data warehouse sources Authority for data warehouse targets For the NetView data warehouse target configuration, open the data warehouse control center. Starting with the control center, perform the following steps: 1. Open the Warehouse targets tree and select Properties in the context menu of a target. In Figure 5-21, for example, we choose ANM_TWH_CDW_Target. 2. In the properties window of the selected data warehouse target select Database, as also shown for ANM_TWH_CDW_Target in Figure 5-21.190 Implementing Tivoli Data Warehouse 1.2
    • 3. As Database name, select the specific database for your environment. For our case study installation, the databases for the NetView data warehouse targets are listed in Table 5-6 on page 188. TWH_CDW was already configured as needed, so we can leave it as is for our case study (refer to Figure 5-21). 4. Insert the User ID and appropriate password to the target database. Because in our case study the TWH_CDW database is on a Windows machine, we used db2admin, as shown in Figure 5-21. 5. Select OK to finish the data warehouse source configuration. Repeat these steps for all NetView data warehouse targets as listed in Table 5-6 on page 188. Figure 5-21 Configure NetView data warehouse targets5.6 Testing, scheduling, and promoting the ETLs After successfully installing and configuring the Netview warehouse enablement packs, all the ETL1 and ETL2 processes can now be tested, scheduled, and promoted to production. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 191
    • There are three known modes for ETL processes in Tivoli Data Warehouse: Development: In this mode, process steps can be changed and their schedule can be configured. However, they do not run on their scheduled times and they cannot be tested. Test: In this mode, process steps are not scheduled, but they can be tested and their schedule can be changed. Production: In this mode, the processes run as scheduled. Neither the process steps nor their schedules can be changed.5.6.1 Promoting the ETLs to TEST mode In order to test ETLs we must promote them to test or production mode first. For the case study environment, we choose the test mode. On the control server machine, open the Data Warehouse Control Center and select Subject Areas -> ANM_IBM_Tivoli_Netview_v1.1.0_Subject_Area -> Processes -> ANM_c05_ETL1_Process to open the list with the process steps. Right-click the all selected ANM_c05_ETL1_Process processes and select Mode -> Test in its context menu, as shown in Figure 5-22. Figure 5-22 Promote ETLs to test mode You have to promote all process steps you want to test into Test mode.192 Implementing Tivoli Data Warehouse 1.2
    • 5.6.2 Testing the ETLs You can test the ETLs manually first before scheduling and promoting to production. After you have promoted the process steps to Test mode, right-click the ANM_c05_s010_extractNodeInfo process and select Test, as shown in Figure 5-23. The process step is executed immediately. Figure 5-23 Test ETL process steps To view the results, select Warehouse -> Work in progress from the Data Warehouse Control Center. The Work in Progress window opens displaying a line for each executed process step as shown in Figure 5-24. You can right-click the process and select Show Log from the context menu to open the log window. There you can see additional information regarding the process step execution. In case of failure, that’s where you will find the error messages. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 193
    • Figure 5-24 Work in Progress - Log file Important: Do not change the sequence of the processing steps. You might cause great damage to the data within the Tivoli Data Warehouse databases. ETLs: data collection verification Once all processes are tested, you need to verify data collection on the Tivoli Data Warehouse databases: central data warehouse and data mart. There are several ways to accomplish this task. Here we show one of them: 1. On the control server machine, open the DB2 Control Center: Click Start -> Programs -> DB2 -> Control Center. 2. Double-click the desired database, for example, the TWH_MART database. 3. Double-click Tables; on the right hand pane, all tables are displayed. In the case of the TWH_MART database, select the D_L3NODES table. Right-click and select Sample Contents. You will see a window as shown in Figure 5-25, displaying the sample content. This verifies that your ETLs executions are successful.194 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-25 Sample contents5.6.3 Scheduling the ETLs There are two processes that need to be scheduled for the Netview WEP: ANM_c05_ETL1_Process AN1_c05_SNMP_ETL1_Process Note: Do not schedule the ANM_m05_ETL2_Process to run. This process is automatically started by the ANM_c05_ETL1_Process. The following steps are similar for both processes, and we will use ANM_c05_ETL1_Process as an example. On the Tivoli Enterprise Data Warehouse Control Center server using expand Subject Areas, select ANM_IBM_Tivoli_Netview_v1.1.0_Subject_Area -> Processes and right-click AMN_c05_ETL1_Process. Choose Schedule, as shown in Figure 5-26. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 195
    • Figure 5-26 Schedule ANM_c05_ETL1_Process Selecting Schedule will open up a dialog as shown Figure 5-27. Here you have to define the date and time, for this process to run. Note: Changes apply only when the process is in development mode. If you use NetView SmartSets as described in 5.4, “Preparing NetView for data collection” on page 167, you have to synchronize the hour when the SmartSet data is loaded to the NetView source database specified in the tdwdaemon.properties file and the schedule of the NetView WEP ETLs. We recommend that the ETLs be scheduled at least one hour later then the SmartSets loading time. In our case study installation, the SmartSet data load is done at 11pm as shown in Example 5-2 on page 178; one hour later at 0am, the ANM_c05_ETL1_Process is scheduled as shown in Figure 5-27.196 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-27 Schedule configuration for ANM_C05_ETL1_Process5.6.4 Promoting the ETLs to Production status All of the processes are composed by components that have Development or Test status set as default. In order for them to run, their status needs to be changed from Development to Production status. They are: ANM_c05_ETL1_Process: – ANM_c05_s010_extractNodeInfo – ANM_c05_s020_transformNodeInfo – ANM_c05_s030_loadNodeInfo – ANM_m05_s010_metric (This step is a link and cannot be promoted to Test, Development or Production) ANM_m05_ETL2_Process: – ANM_m05_s010_metric – ANM_m05_s020_fact – ANM_m05_s030_outage_rollUp – ANM_m05_s040_transition_rollup – ANM_m05_s050_ss_trans_rollup – ANM_m05_s060_out_rollup – ANM_m05_s070_total Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 197
    • AN1_c05_SNMP_ETL1_Process: – AN1_c05_s010_extractsnmpdata – AN1_c05_s020_transformsnmpdata – AN1_c05_s030_loadsnmpdata The following steps must be performed for all processes described above. Here we use ANM_c05_ETL1_Process to describe it. On the control server, using Data Warehouse Center tool, select the above processes and right-click them. Choose Mode -> Production, as shown in Figure 5-28. Figure 5-28 Promote ANM_c05_ETL1_Process After promoting the processes to production mode, they are scheduled for the configured times and they are visible in the Work in progress list.5.7 Running NetView ETLs on remote agent sites By default, the NetView ETLs are scheduled on the default data warehouse agent named Default DWC AgentSite running on the control server. In Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71, we explain how to install a data warehouse remote agent and create agent sites.198 Implementing Tivoli Data Warehouse 1.2
    • In this section we explain how to make use of such a remote agent site. For ourcase study installation, we move the execution of the NetView SNMPperformance warehouse enablement pack (AN1) ETLs from the control server toa remote agent site.Here is an overview of the steps required to move the execution of the ETLs to aremote agent site:1. Make the ETLs code available on the remote agent site.2. Make all data warehouse data sources and targets, which are used by the ETLs, available at the remote agent site.3. Assign the remote agent the necessary data warehouse sources and targets.4. Demote the moving ETL processes to mode development.5. Configure the ETL processes to use the remote agent.6. Promote the moved ETL processes to mode production.7. Verify ETLs running on remote agent.Making the ETL code available on the remote agent siteThe installation of the NetView SNMP performance WEP creates a new directorywith path %InstallDir%TWHappsan1. (The NetView availability WEP createsthe directory %InstallDir%TWHappsanm). Under this directory are all filesstored used by the NetView ETLs.However, the installation of the NetView SNMP performance WEP does notinstall this directory on remote agent sites. We also found no installationmechanism to install the warehouse enablement pack on the remote agent sites.Therefore we mapped the warehouse folder on the data warehouse controlserver (tdw003.itsc.austin.ibm.com) to the remote agent site on the central datawarehouse server (tdw004.itsc.austin.ibm.com) and copied the an1 subdirectoryto the %InstallDir%TWHapps directory on the remote agent site.Making data sources and targets available for remote agentOn the agent site all necessary databases must be available or in DB2 termscataloged. Table 5-6 on page 188 shows that the AN1 ETLs need access to thedatabases TDH_CDW and NETVIEW. In our case study installation the remoteagent is on the central data warehouse server where the TWH_CDW ODBCdatabase source is available already. We created an ODBC data source namedNETVIEW on the remote agent site machine to the NETVIEW database. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 199
    • Assigning remote agent sources and targets Table 5-6 on page 188 shows that the AN1 ETLs make use of both: The source AN1_SNMP_Source The target AN1_TWH_CDW_Target On the control server machine, from the Data Warehouse Center, we assign both objects to the remote agent site, which in our case study is named tdw004.itsc.austin.ibm.com. Select Administration -> Agent Sites -> <remote agent name>, as shown in Figure 5-29. Figure 5-29 Select remote agents properties In the properties window of the remote agent site, available sources and targets are displayed on the left pane. Assigned sources and targets are displayed on the right pane. Move the proper warehouse sources and targets related to the warehouse enablement pack in question. In our case, shown in Figure 5-30, we choose only the SNMP performance ETL relevant AN1_SNMP_Source and AN1_TWH_CDW_Target to assign to the remote agent site. Click OK to finish the assignment.200 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-30 Change remote agents properties - sources and targetsDemoting the moving ETL processes to Development modeThe ETLs are still configured to run on the default agent on the control server. Inorder to change the processes configuration, they must be demoted to theDevelopment mode.On the control server machine, open the Data Warehouse Center and demotethe related processes. in our case study scenario, select Subject Areas ->AN1_IBM_Tivoli_NetviewSNMP_v1.1.0_Subject_Area -> Processes ->AN1_c05_SNMP_ETL1_Process. On the right pane of the window, a list withprocess steps and their data warehouse sources and targets is displayed asshown in Figure 5-31. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 201
    • Figure 5-31 Select ETL process properties Right-click all AN1 processes in the list: AN1_c05_s010_extractsnmpdata AN1_c05_s020_transformsnmpdata AN1_c05_s030_loadsnmpdata Then select Mode -> Development from the processes context menu as shown in Figure 5-32. Figure 5-32 Demote ETL processes to development mode202 Implementing Tivoli Data Warehouse 1.2
    • Configuring the ETL processes to use the remote agentAfter demoting the ETL processes to Development, their properties can bechanged. To configure the processes to run on a different warehouse agent site,right-click a process and select Properties from the context menu to open theprocesses Properties window as shown for the AN1_c05_s010_extractsnmpdataprocess in Figure 5-33. In the Agent Site drop-down menu, all agent sites areavailable that have the required data warehouse sources and targets assigned.In our case study installation, we see only the default agent and thetdw004.itsc.austin.ibm.com remote agent. We select the latter one, then selectOK to finish the configuration.Perform this step for each process in the list: AN1_c05_s010_extractsnmpdata AN1_c05_s020_transformsnmpdata AN1_c05_s030_loadsnmpdataFigure 5-33 Change the ETL processes agent sitePromoting the moving ETL processes to Production modeAfter applying changes to the process configuration, the processes must bepromoted to Production mode in order to schedule them again.As explained in 5.6.4, “Promoting the ETLs to Production status” on page 197,right-click a process and select Mode -> Production from the processes contextmenu as shown in Figure 5-32. Perform this step for each process in the list: AN1_c05_s010_extractsnmpdata AN1_c05_s020_transformsnmpdata AN1_c05_s030_loadsnmpdataThe AN1_c05_s010_extractsnmpdata is scheduled now. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 203
    • Verifying ETLs running on remote agent To verify the ETLs running on the remote agent, we run the AN1 ETL at once from the data warehouse Work in Progress window. On the control server machine, open the Data Warehouse Center and select Warehouse -> Work in Progress. The Work in Progress window opens as shown in Figure 5-34. In this presentation, scheduled processes are marked with a clock in their icon (as the highlighted AN1_c05_s010_extractsnmpdata process), failed processes with a red cross, and successful processes with a green check mark. Right-click the scheduled process AN1_c05_s010_extractsnmpdata to open its context menu. There, select Run now as shown in Figure 5-34. Figure 5-34 Work in Progress - Run ETL The AN1_c05_s010_extractsnmpdata process is executed on the remote agent now and its subprocesses are started there automatically. If all AN1 processes execute successfully, the Work in Progress window looks like that shown in Figure 5-35. You find new lines for each AN1 subprocess, AN1_c05_s010_extractsnmpdata, AN1_c05_s020_transformsnmpdata, and AN1_c05_s030_loadsnmpdata, which are all check marked after their successful completion. Additionally, the process AN1_c05_s010_extractsnmpdata is scheduled again marked with a clock. This line is highlighted in Figure 5-35.204 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-35 Work in progress - Check ETLDouble-click one of the lines in Figure 5-35 to open the related Log Detailswindow in Figure 5-36 for the ANM_c05_s010_extractNodeInfo process step.Figure 5-36 Log Details menu Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 205
    • In our example, the process is successful and no error messages are displayed in the detailed view. With the buttons Previous and Next you can navigate the log details for all process steps that are displayed in the Log window shown in Figure 5-35.5.8 Reporting In this section we show how to set up, configure, and use some of the reports provided by the NetView availability warehouse enablement pack (ANM). Here is a list of the predefined reports provided by the IBM Tivoli NetView Warehouse Enablement Pack: Daily Status Summary By SmartSet Nodes With Longest Outage Time In Routers SmartSet Nodes With Most Status Changes In Routers SmartSet Nodes With The Longest Outage Times Nodes With The Most Daily Status Changes Summary Of Daily Network Status Summary Of Total Outage Time By SmartSet Summary Of Total Status Changes By SmartSet Total Daily Status Changes In Monitored Network Total Daily Status Changes In Routers SmartSet Crystal Enterprise Professional version 9 for Tivoli has in comparison to a full Crystal license reduced configuration options. If the reports shipped with IBM Tivoli NetView Warehouse Enablement Pack do not match your needs and you want to develop additional reports, you have to upgrade your Crystal Enterprise installation. Note: As described in Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71, an ODBC connection to the data mart database needs to be defined on the Crystal Enterprise server before we can work with the reports. Please refer to that chapter for details.5.8.1 Accessing the Crystal ePortfolio feature All reports provided by Tivoli Data Warehouse 1.2 must be accessed using the Crystal Enterprise ePortfolio feature. The ePortfolio can be accessed from the Crystal Enterprise launchpad. To access the Crystal Enterprise launchpad, start an Internet browser and open the following URL: http://<hostname>/crystal/enterprise9206 Implementing Tivoli Data Warehouse 1.2
    • Here, <hostname> represents the hostname of the Crystal Enterprise reportserver, as shown in Figure 5-37.Figure 5-37 Crystal Enterprise - LaunchpadIn this section, we concentrate on viewing NetView reports, and we do notexplain the configuration of Crystal Enterprise to its full extent. For details onconfiguration and administration tasks, refer to the following manuals shippedwith the product: Crystal Enterprise 9 Installation Guide Crystal Enterprise 9 Administrator’s Guide Crystal Enterprise 9 Getting Started Guide Crystal Enterprise 9 ePortfolio User’s Guide Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 207
    • From the Crystal Enterprise Launchpad, proceed by selecting the ePortfolio link, which will bring you to the window shown in Figure 5-38. In the top bar, you can see that we are authorized as user guest. By default, the guest user has no access to the NetView reports, as indicated by the words No Folders on the left side of the window. Figure 5-38 Crystal Enterprise 9 - ePortfolio The installation process of the first warehouse enablement pack on the Tivoli Data Warehouse environment creates a user ID on the Crystal Enterprise environment named Tivoli. This user ID is to be used to access the reports provided by any IBM Tivoli software.208 Implementing Tivoli Data Warehouse 1.2
    • To log in as the Tivoli user ID, select the Log On button in the upper right cornerof the ePortfolio window in Figure 5-38 on page 208. The Log On window, asshown in Figure 5-39, is presented. The Tivoli user ID has no password bydefault. We use the Enterprise authentication method as we have specifiedduring the Crystal Enterprise installation.Figure 5-39 Crystal Enterprise 9 - Log in Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 209
    • After entering the required data, select Log On to proceed. Now we are back at the ePortfolio window in Figure 5-40, but now with user Tivoli authority. Instead of No Folders in the guest users ePortfolio window in Figure 5-38, there is now a link visible with the name IBM Tivoli NetView in the Tivoli user ePortfolio window shown in Figure 5-40. Figure 5-40 Crystal Enterprise 9 - Folders210 Implementing Tivoli Data Warehouse 1.2
    • We follow this link by selecting IBM Tivoli NetView and proceed to the IBM TivoliNetView reports as shown in Figure 5-41. All reports provided by the IBM TivoliNetView Warehouse Enablement Pack are listed there. As already mentioned,there are only reports on availability and no reports on performance.Figure 5-41 Crystal Enterprise 9 - Tivoli Reports: IBM Tivoli NetViewWe open the reports context menu by left-clicking on the desired report name, asshown in Figure 5-42. We are presented with a menu which contains the items: View: Generate report instantaneously. View latest instance: View last report. Schedule: Change or create a new schedule for report generation. History: View already generated reports. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 211
    • Figure 5-42 Crystal Enterprise 9 - Daily Status Summary by SmartSet We continue by selecting Schedule from the Daily Status Summary by SmartSet report, for example. The Schedule window, as shown in Figure 5-43, is opened. Figure 5-43 Crystal Enterprise 9 - Schedule212 Implementing Tivoli Data Warehouse 1.2
    • The Customize your options toolbar offers three selection buttons: Schedule: Pressing this button starts a new schedule with the current options and parameters. Cancel: Pressing this button closes the Schedule window and you get back to the reports window without adding a new schedule for the report. Help: Opens the help window.Figure 5-44 shows the selection of parameters for the schedule option. Here youcan select the frequency the reports should be generated.Figure 5-44 Crystal Enterprise 9 - Parameters for Schedule OptionWe want to schedule the report to run now.Next, it is necessary to provide the required parameters of the report. From theCustomize your options pull-down menu, select Parameters, as shown inFigure 5-45. We left the other options settings to the default values. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 213
    • Figure 5-45 Crystal Enterprise 9 - Schedule Parameters The Schedule windows changes to Figure 5-46 and we are presented with three selection fields: Time Filter General Time Frame Specific Time Frame Note: Schedule requirements may differ for each report. The schedule selections presented here are for the Daily Status Summary by SmartSet report.214 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-46 Crystal Enterprise 9 - Schedule Parameter SelectionFor the items Time Filter and General Time Frame we select the default valuesNone by pressing the Add button at each selection.Thus we specify a lower and an upper bound for the specific time frame byselecting the Start of range and End of range parameters. Select Add Range toaccept the settings. Figure 5-47 shows the parameters window after theselections for our case study report. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 215
    • Figure 5-47 Crystal Enterprise 9 - Parameters: Specific Time Frame Now all required parameters are specified. Start the report generation by pressing the Schedule button from the toolbar.216 Implementing Tivoli Data Warehouse 1.2
    • As we have left the Schedule parameter set to Now, as shown in Figure 5-43, thereport is scheduled immediately, and the reports history window is opened asshown in Figure 5-48.Figure 5-48 Crystal Enterprise 9 - Report HistoryThe report just scheduled is still running, and therefore it is in status Pending. Note: The History window is not updated automatically. Press the Refresh button to view the current state.Figure 5-48 shows four different status, as follows: Pending: Report generation is still running. Success: Report is generated successfully. Click the link Instance Time in the left column of the table to view the report. Recurring: Report is scheduled to be generated certain times. Refer back to Figure 5-43 on page 212. Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 217
    • Failed: Report generation failed. Click the link Failed to open the log window as shown in Figure 5-49. The error message, Information is needed before this report can be processed, means that your parameter settings are not valid. Go back to the window shown in Figure 5-46 on page 215 and reenter your parameter settings. Figure 5-49 Failed report generation To view successfully generated reports from the history window, as shown before in Figure 5-48 on page 217, click the link Instance Time in the left column of the table to view the associated report. On the following pages, Figure 5-50 and Figure 5-51 show the report Daily Status Summary by SmartSet for our case study scenario. The report is text based and is split into two pages. At the top of the report window are buttons to navigate through multi-page reports. In the NetView data warehouse daemon properties file, tdwdaemon.properties, all specified SmartSets are populated and included in this report (refer back to Example 5-2 on page 178). Some nodes are members of one or more SmartSets, and therefore the availability of these nodes is used multiple times to calculate the total availability.218 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-50 Crystal Enterprise 9 - Report Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 219
    • Figure 5-51 Crystal Enterprise 9 - Report (count)220 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-52, Figure 5-53, and Figure 5-54 show more examples of reports provided by the IBM Tivoli NetView Warehouse Enablement Pack.Figure 5-52 Summary of total status changes by SmartSet example Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 221
    • Figure 5-53 Nodes with longest outage times example222 Implementing Tivoli Data Warehouse 1.2
    • Figure 5-54 Total daily status changes in monitored network example Chapter 5. IBM Tivoli NetView Warehouse Enablement Pack 223
    • 224 Implementing Tivoli Data Warehouse 1.2
    • 6 Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack This chapter describes a case study for integrating data collected from IBM Tivoli Monitoring, Version 5.1.1 with Tivoli Data Warehouse 1.2 using the IBM Tivoli Monitoring Generic Warehouse Enablement Pack and the IBM Tivoli Monitoring for Operating Systems Warehouse Enablement Pack (WEP). We cover the following topics: “Case study overview” on page 226 “IBM Tivoli Monitoring WEP overview” on page 227 “Prerequisites” on page 231 “Installing the ITM WEP data collector component” on page 232 “Installing and configuring ITM Generic WEP” on page 241 “Installing and configuring ITM for OS WEP” on page 262 “Testing, scheduling, and promoting the ETLs” on page 267 “Reporting” on page 275 “Troubleshooting of ITM data collection” on page 286© Copyright IBM Corp. 2004. All rights reserved. 225
    • 6.1 Case study overview This case study shows all the steps required to configure IBM Tivoli Monitoring, Version 5.1.1 in order to gather historical data, process the ITM data into a Tivoli Data Warehouse 1.2, and produce end user reports. The tasks performed in this case study can be summarized as follows: Installing and configuring IBM Tivoli Monitoring Enablement Pack, Version 5.1 on the Tivoli server Activating ITM data collection Installing and configuring ITM Generic Warehouse Enablement Pack (AMX) on the Tivoli Data Warehouse server Installing and configuring ITM Warehouse Enablement Pack for OS (AMY) on the Tivoli Data Warehouse server Managing the AMX and AMY ETLs Reporting The environment used for our case study is composed of the following servers, as shown in Figure 6-1: TDW001 Tivoli Data Warehouse single server installation TDW015 Crystal Enterprise Server TDW009 TMR server, Tivoli gateway, and ITM data source TDW0xx Tivoli endpoints (for xx=01,02,...08 Windows endpoints, for xx=09,10,11 AIX endpoints)226 Implementing Tivoli Data Warehouse 1.2
    • TDW001 Tivoli Data Warehouse Control Server Metadata Data Mart Data Mart Central Data Warehouse Warehouse TDW015 Web Server Crystal Server TDW009 ITM_DB Tivoli Management Data source Server Monitored Endpoints Web Reports TDW0XX Figure 6-1 Environment for our case study Hardware and operating systems used in our case study environment are listed in Table 6-1. Table 6-1 Hardware and operating systems Hostname OS Model Memory Disk size TDW001 Window 2000 NetVista™ 512 MB 40 GB Server SP4 8305 TDW009 AIX 5.1.0 7028 6E1 2048 MB 18 GB TDW015 Window 2000 NetVista 8305 512 MB 40 GB Server SP46.2 IBM Tivoli Monitoring WEP overview IBM Tivoli Monitoring can generate operational data and store it into a central repository called IBM Tivoli Monitoring Middle Layer Database (the default database name is ITM_DB). A program called ETL (Extract, Transform, and Load) takes the operational data, elaborates it, and places the output in the central data warehouse, which can contain historical data from diverse sources Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 227
    • (not only IBM Tivoli Monitoring). A second ETL program extracts a subset of data from the CDW and transfers it into a data mart database specifically designed for reporting (see Figure 6-2). Control Server Metadata Central Data Data Mart Warehouse Crystal Server Web Server Web Reports Figure 6-2 Overview of ITM integration with Tivoli Data Warehouse The IBM Tivoli Monitoring 5.1.1 is shipped with two Warehouse Enablement Packs (WEP) that provide the ETLs required by the Tivoli Data Warehouse integration: IBM Tivoli Monitoring Warehouse Enablement Pack Version 1.1 (also called AMX pack) IBM Tivoli Monitoring for Operating Systems Warehouse Enablement Pack Version 1.1 (also called AMY pack) The AMX warehouse enablement pack provides warehouse functionality for all the applications based on IBM Tivoli Monitoring 5.1.1. It is intended as a generic tool, driven by the metadata provided by each monitoring application. It can be used to extract the data stored into the ITM Middle Layer Repository (ITM_DB) and to load it into the Tivoli Data Warehouse central data warehouse database (TWH_CDW). Due to this “generic” nature, the IBM Tivoli Monitoring WEP does not provide either data mart ETL process, or star schemas. These functions are provided by the WEP of the specific monitoring application.228 Implementing Tivoli Data Warehouse 1.2
    • The ETL process provided by the IBM Tivoli Monitoring WEP (also called genericETL1) is able to load the correct data into the central data warehouse commonschema looking at the rules defined by the specific monitoring applicationmetadata. It is also able to detect and trace any exceptions in the operationaldata (any data which is not properly described by the application metadata).Instead, the IBM Tivoli Monitoring for Operating Systems warehouse pack (AMYWEP) provides a set of metadata to drive the IBM Tivoli Monitoring, Version5.1.1 warehouse pack (AMX or Generic ETL1) to retrieve data collected by theIBM Tivoli Monitoring 5.1.1 Operating Systems Resource Models. It alsoprovides a sample star schema definition used to build some data marts andgeneral-purpose reports.Figure 6-3 depicts the role of the two WEPs in IBM Tivoli Monitoring 5.1.1integration with Tivoli Data Warehouse. ITM for OS WEP (AMY) Control Server Metadata ITM WEP Central Data ETL 2 (AMY) Warehouse Data Mart Data Mart (AMX) Warehouse ETL 1 (AMX) Tivoli Agent RIM Tivoli Agent ITM collect ITM_DB Tivoli AgentFigure 6-3 IBM Tivoli Monitoring data flowThe primary link between IBM Tivoli Monitoring and Tivoli Data Warehouse is theresource model data logging. When you select the Tivoli Enterprise DataWarehouse logging option for a given Resource Model, the Aggregation option isgrayed out. You can still select the Raw data logging option (Figure 6-4) for thesame Resource Model. If you select both options (TEDW Data and Raw Data),the data will be aggregated at the top of the hour, and both raw data andaggregated data will be visible in the Web health console. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 229
    • Aggregation data is processed at each cycle time. The IBM Tivoli Monitoring engine aggregates the current data with the “previous” data (that is already a result of previous aggregation), so that, only the data resulting from the aggregation is kept in memory. This data is logged in the endpoint database when the aggregation period expires. The Tivoli Data Warehouse data aggregation algorithm is exactly the same, but the aggregation period is fixed to 1 hour. The IBM Tivoli Monitoring engine calculates the Min, Max, Avg, and Total values. However, the Total value is not applicable for all metrics. For example, the Web Health console will only show the Min, Max, Avg data. The total is the sum of the metrics value for each cycle time in the aggregation period. A thread is spawned every hour and the endpoint database is queried for all the aggregated data saved in the previous hour. Data are then saved in XML files. The thread is spawned with a fixed delay time (presently not possible to change, that is, 20 minutes) with respect to the full hour. Figure 6-4 shows the data logging options dialog in an IBM Tivoli Monitoring profile. Figure 6-4 Resource Model Data Logging230 Implementing Tivoli Data Warehouse 1.2
    • If Raw Data logging is enabled, the IBM Tivoli Monitoring engine aggregates data collected during each cycle time and stores the resulting value in the endpoint database when the aggregation period expires. For TEDW Data logging option, the algorithm is exactly the same, but the aggregation period is fixed to 1 hour. In both cases, data logging will start at the next full hour after the profile distribution time. Figure 6-5 shows an example of an aggregation time line.Figure 6-5 Aggregation time line The hourly aggregated data is exported from the endpoint database into a well-formed XML file, 20 minutes after the expiration of the aggregation period. For more information about IBM Tivoli Monitoring, you can refer to the redbook: IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring, SG24-5519.6.3 Prerequisites Before installing the warehouse enablement packs for IBM Tivoli Monitoring, Version 5.1.1, the following software must be installed: IBM Tivoli Monitoring Version, 5.1.1 IBM Tivoli Monitoring Version, 5.1.1 Fix Pack 6 (5.1.1-ITM-FP06) IBM DB2 Universal Database Enterprise Edition 7.2 IBM DB2 Universal Database Enterprise Edition 7.2 Fix Pack 8 with its eFix (This is the minimum level of IBM DB2 supported by Tivoli Data Warehouse 1.2. Fix Pack 10a is the recommended level and it is shipped with Tivoli Data Warehouse 1.2). Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 231
    • Tivoli Data Warehouse 1.2 Crystal Enterprise Professional Version 9 for Tivoli IBM DB2 Warehouse Manager, Version 7.2 Fix Pack 8 with its eFix, on the remote agent sites in case of a distributed deployment. (This is the minimum level of IBM DB2 Warehouse Manager supported by Tivoli Data Warehouse 1.2. Fix Pack 10a is the recommended level and it is shipped with Tivoli Data Warehouse 1.2).6.4 Installing the ITM WEP data collector component In this section we describe the steps to install and configure the IBM Tivoli Monitoring Enablement Pack, Version 5.1.1, and to start collecting data from the monitored endpoints. This component should be installed on the TMR server and all gateways in your Tivoli environment where you want to collect data from endpoints and make the data available for Tivoli Data Warehouse. We have only one gateway in our environment and it is installed on the TMR server, therefore we need to install the component exclusively on the TMR server TDW009. As a requirement, the IBM Tivoli Monitoring 5.1.1 product must be already installed and a RIM (RDBMS Interface Module) host should be defined. You can choose to install the ITM database on a Tivoli managed node defined as RIM host or on a server outside the Tivoli environment. In our case study we adopted the first configuration, with the ITM data repository (ITM_DB) on the TMR server (TDW009) that is also a RIM host. The installation method described here uses the Tivoli Desktop, as follows: Note: You can also install the Tivoli Enterprise Data Warehouse Support 5.1.1 from the command line. Use the following to install it from the command line: winstall –c path /cdrom -i DM511TED managed_node Here, DM511ED is the index file for Tivoli Enterprise Data Warehouse Support 5.1.1 and managed_node is the name of the managed node to be installed on. 1. From the Tivoli Desktop, select Desktop -> Install -> Install Product. The Install Product dialog shows the products that are available to install as shown in Figure 6-6.232 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-6 Installing warehouse support2. Select IBM Tivoli Monitoring - Tivoli Enterprise Data Warehouse Support, Version 5.1.1, then select your Tivoli Management Region server and the gateways that you want to have it installed on.3. RIM configuration is required to proceed with the installation, as shown in Figure 6-7. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 233
    • Figure 6-7 RIM setup options The installation process will create a RIM object named itm_rim_<nodename>, where <nodename> is the RIM host in your Tivoli environment (tdw009 for ours). The RIM object can also be created at a later time using the following command, for instance, assuming that you have a DB2 database server: wcrtrim -v DB2 -h tdw009 -d itm_db -u db2inst1 -H /usr/lpp/db2_07_01 -s TCPIP -I /home/db2inst1 itm_rim_tdw009 This RIM object by default has itmitm as a password, which must be changed to match the password of your database instance owner. Use the wsetrimpw command as follows: wsetrimpw itm_rim_<nodename> itmitm <newpw> Here, <newpw> is the database instance owner password. 4. Click Set and then select Install and follow the normal installation dialogs.234 Implementing Tivoli Data Warehouse 1.2
    • 5. The physical database for the Warehouse Support component name is ITM_DB and needs now to be created. This process can be accomplished by either using a provided shell script or using SQL scripts. If you intend to use the provided shell script, make sure you grant the RDBMS administrator (or database instance owner) user ID with Administrator (root) and Tivoli_Admin_Privileges and run the script logged in as your user ID. The reason is that the shell script collects information from the previously created RIM object in order to create both the database and its structure. The shell script name is cr_itm_db.sh, and it is located in the $BINDIR/TME/Tmw2k/Warehousecfg directory. As an alternative method, you can use the SQL scripts. These scripts are also located in the $BINDIR/TME/Tmw2k/Warehousecfg directory and have the following name standard: cr_db.<DBext> and cr_tbl.<DBext>, where <DBext> is the database vendor designator (db2 in our case). The following sequence describes the creation process for DB2 using the SQL scripts: a. On the RIM host machine, login as your instance owner (in our case, db2inst1). b. Only perform this step if the RIM host machine does not have the Warehouse Support component installed. Copy the cr_db.db2 and cr_tbl.db2 files from the $BINDIR/TME/Tmw2k/Warehousecfg directory from your TMR server to the RIM host machine. c. Move to the directory where the SQL scripts are located and rename cr_db.db2 to cr_db_db2.sql and rename cr_tbl.db2 to cr_tbl_db2.sql. Edit cr_db_db2.sql and replace “CREATE DATABASE _xz_db” with “CREATE DATABASE itm_db”. d. Run the following command to create the itm_db database: db2 -td$ -vf cr_db_db2.sql e. In order to have the itm_db database structure created, run the following commands, where <db2inst1pw> is the database instance owner password. db2 connect to itm_db user db2inst1 using <db2inst1pw> db2 -td$ -vf cr_tbl_db2.sql6. On the TMR server, test the RIM object connection: wrimtest -l itm_rim_<rimhost> The output should be similar to Example 6-1. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 235
    • Example 6-1 Testing the RIM object tdw009:/>wrimtest -l itm_rim_tdw009 Resource Type : RIM Resource Label : itm_rim_tdw009 Hostname : tdw009 User Name : db2inst1 Vendor : DB2 Database : itm_db Database Home : /usr/lpp/db2_07_01 Server ID : tcpip Instance Home : /home/db2inst1 Opening Regular Session...Session Opened RIM : Enter Option > Type x and press Enter to release the session. 7. The data collection process of the warehouse support component needs to be configured. The configuration file is named .config and it is located in the $DBDIR/dmml directory. The warehouse support entries in the.config file have the prefix datacollector. Such entries should be added/modified using the wdmconfig command, and it is important to notice that this file must not be modified manually. For details on the wdmconfig command, refer to the IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569 manual. To set the collection parameters, issue the following command: wdmconfig -m <nodename> -D datacollector.rim_name=itm_rim_<rimhost> -D datacollector.db_purge_interval=30 -D datacollector.db_purge_time=0 -D datacollector.delay=30 -D datacollector.sleep_time=1 -D datacollector.max_retry_time=6 Here, <node name> is the gateway specified for the monitored endpoints. You can check if the entries were correctly set by issuing: wdmconfig -m <nodename> -G datacollector* The output should be similar to Example 6-2. Example 6-2 Datacollector configuration # wdmconfig -m tdw009 -G datacollector* Managed Node: tdw009: ======================= datacollector.db_purge_interval = 30 datacollector.db_purge_time = 0 datacollector.delay = 30 datacollector.sleep_time = 1 datacollector.max_retry_time = 6 datacollector.rim_name = itm_rim_tdw009236 Implementing Tivoli Data Warehouse 1.2
    • datacollector.rim_name Specifies the name of the RIM object that the data collection process will use to load data to the database. The default is itm_rim_<RIM host name>. datacollector.db_purge_interval Specifies the number of days the data is kept on the database: older data is automatically removed from the database. The value can range from 10 to 60. The default is 30 days. datacollector.db_purge_time Specifies the time during the day for the data removal operation. The value can range from 0 (midnight) to 23. The default is 0 (midnight). datacollector.delay Specifies the time delay (in minutes, compared to the hour) after which the data collector process uploads data from the endpoints. The value can range from 1 to 60 minutes. The default is 30 minutes. datacollector.sleep_time Specifies the time interval (in minutes) between two consecutive requests of data uploading generated by the data collector processor. The value can range from 1 to 60 minutes. The default is 1 minute. datacollector.max_retry_time Specifies the maximum number of times an XML data file must be processed before being archived when an error occurs. The default is 6 times.6.4.1 Activate data collection The following steps are required to start monitors and data collection into ITM_DB database: 1. Open the Tivoli desktop and create a profile manager for storing the data warehouse collection profiles. The policy region containing the profiles must have Tmw2kProfile as Managed Resource. 2. Create the Tmw2kProfile objects in the profile manager and add the appropriate resource models. For details about resource models, please refer to the manual IBM Tivoli Monitoring - Resource Model Reference, SH19-4570. 3. For each resource models, open it by clicking Edit. In the Edit resource model window, click Logging. The logging window will be shown as in Figure 6-8. You need to check the Enable Data Logging and TEDW Data check boxes. Click Apply Changes and Close when done. Click Modify & Close from the Edit resource model window. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 237
    • Note: By using the Logging window, you can specify how to log data collected by a resource model. If you select the Raw Data option, data will be written exactly as it is collected by the resource model. All the monitored values are collected and copied in the database. This data later can be used by the Health Console. Selecting the TEDW Data option allows data to be collected and copied in the database for later use by Tivoli Enterprise Data Warehouse. If you choose the Aggregated Data option, data will be collected and aggregated at fixed intervals you define (Aggregation Period). Then only the aggregated values are written in the database. The Historical Period option is used to specify the period for which data is to be stored in the database. For the Tivoli Data Warehouse you need to check TEDW Data option. In addition, you can also check the Raw Data option (this does not affect data collection for Tivoli Data Warehouse). Figure 6-8 Logging option 4. Now, you can specify the subscribers. Each resource model can only have specific subscriber types. You may want to create multiple profiles which each profile contains resource models for specific object types. For example, in our ITSO environment we created two different profiles, one for Windows 2000 endpoints and the other for AIX endpoints.238 Implementing Tivoli Data Warehouse 1.2
    • 5. Distribute the profile to your subscribers. You can either do it from the Tivoli desktop, or using the wdmdistrib command. 6. Check the distribution of the profile and the execution of it using the wdmlseng command. Ensure that all the profiles are having the state of Running, as shown in Example 6-3. Example 6-3 wdmlseng command output # wdmlseng -e tdw002 Forwarding the request to the engine... The following profiles are running: tmw2kDefProfile#tdw009-region TMW_EventLog :Running TMW_PhysicalDiskModel :Running TMW_TCPIP :Running TMW_LogicalDisk :Running TMW_MemoryModel :Running TMW_Process :Running TMW_Processor :Running 7. You need to tell the gateways, to which the endpoint reports to, to start collecting historical information to be put into the RIM object. Use the wdmcollect command. The syntax of the command is: wdmcollect -e <endpoint> -s <time> -m <managed_node> Here, <endpoint> is the endpoint name, <time> is the collection interval in hours, and <managed_node> is the managed node to be defined as the data collector node. After you run this command for all the endpoints, you can check the result using the command wdmcollect -q. The managed node will pull the data from the endpoint every interval and store it under $DBDIR/dmml/tedw. Example 6-4 shows the command output in our case study environment. It means that data is being collected from nine endpoints (tdw001, tdw002, tdw003, tdw004, tdw005, tdw006, tdw008, tdw010, tdw011) via the managed node tdw009 every hour (3600 seconds).Example 6-4 wdmcollect command output# wdmcollect -qProcessing ManagedNode tdw009...<Requests> <Request Id="235d1afd" Endpoint="tdw011" RefreshTime="3600" LastCall="Fri Aug 114:32:00 CDT 2003"/> <Request Id="545a2a6b" Endpoint="tdw010" RefreshTime="3600" LastCall="Fri Aug 114:32:00 CDT 2003"/> Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 239
    • <Request Id="52f58445" Endpoint="tdw004" RefreshTime="3600" LastCall="Fri Aug 114:32:00 CDT 2003"/> <Request Id="cc9111e6" Endpoint="tdw003" RefreshTime="3600" LastCall="Fri Aug 114:32:00 CDT 2003"/> <Request Id="bb962170" Endpoint="tdw002" RefreshTime="3600" LastCall="Fri Aug 114:32:00 CDT 2003"/> <Request Id="229f70ca" Endpoint="tdw001" RefreshTime="3600" LastCall="Fri Aug 114:32:01 CDT 2003"/> <Request Id="25f2b4d3" Endpoint="tdw005" RefreshTime="3600" LastCall="Fri Aug 114:39:04 CDT 2003"/> <Request Id="bcfbe569" Endpoint="tdw006" RefreshTime="3600" LastCall="Fri Aug 114:39:12 CDT 2003"/> <Request Id="5b43c86e" Endpoint="tdw008" RefreshTime="3600" LastCall="Fri Aug 114:39:23 CDT 2003"/></Requests> 8. Before you can start checking on the RIM database to see whether the data has been collected, you have to wait for a time that depends on the selected collection interval and the data collector delay. We check the collection by verifying that the ENDPOINTS table has been populated and checking the timekey_dttm in metricsdata table as shown in Example 6-5. Please note that the time reported in timekey_dttm is a GMT time. Example 6-5 Sample SQL that check the collection db2 => connect to ITM_DB Database Connection Information Database server = DB2/6000 7.2.6 SQL authorization ID = DB2INST1 Local database alias = ITM_DB db2 => select host_name from endpoints HOST_NAME ------------------------------------------------------------------------------- tdw001 tdw002 tdw003 tdw004 tdw005 tdw006 tdw007 tdw008 tdw009 tdw010 tdw011240 Implementing Tivoli Data Warehouse 1.2
    • 11 record(s) selected. db2 => select max(timekey_dttm) from metricsdata 1 -------------------------- 2003-07-31-11.00.56.000000 1 record(s) selected.6.5 Installing and configuring ITM Generic WEP This section describes how we installed and configured the IBM Tivoli Monitoring V 5.1.1 Generic Warehouse Enablement Pack (AMX) in our lab environment. The step required for AMX installation and configuration are: 1. Backup TWH databases 2. Configure DB2 connectivity to data sources 3. Installing the ITM 5.1.1 AMX ETL processes 4. Installing AMX Fix Packs 5. Define the authority to the warehouse sources and targets 6. Modify the ETL for source table name to the RIM user6.5.1 Backing up the TWH databases Before installing any warehouse enablement pack (WEP), it is highly recommended to back up all the warehouse databases to ensure that you can return to a known valid state if you encounter an unrecoverable error during installation. In our case study we used a single server installation of Tivoli Data Warehouse, so that we have to back up the three databases TWH_CDW, TWH_MART, and TWH_MD (if you use different configurations, refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27 for warehouse database naming conventions). In order to do so, we created a local directory structure on our DB2 server to store our backups by creating the following three directories: C:tdwbackupspre-amxtwh_cdw C:tdwbackupspre-amxtwh_mart C:tdwbackupspre-amxtwh_md Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 241
    • After creating these directories, we completed the following steps on the DB2 server from a command window: 1. Stop and restart DB2: db2stop force db2start 2. Back up the TWH_CDW database: db2 backup db TWH_CDW to C:tedwbackupspre-amxtwh_cdw 3. Back up the TWH_MART database: db2 backup db TWH_MART to C:tedwbackupspre-amxtwh_mart 4. Back up the TWH_MD database: db2 backup db TWH_MD to C:tedwbackupspre-amxtwh_md6.5.2 Establishing an ODBC connection on the Control Center Because the IBM Tivoli Monitoring Version 5.1.1 Generic ETL1 retrieves its data from the ITM database (ITM_DB), we have to create an ODBC connection to that database in our control server. To create the ODBC connection to ITM_DB database you can use the command line or the DB2 Client Configuration Assistant. If you are using a DB2 command line window on the control server, issue the following commands: db2 catalog tcpip node CAT_ODBC remote <REMOTE_SVR> server <DB2_PORT> db2 catalog database ITM_DB at node CAT_ODBC db2 catalog system odbc data source ITM_DB Where <REMOTE_SVR> is the hostname of the DB2 server that contains the ITM database (TDW009 in our case) and <DB2_PORT> is the TCP/IP port used by DB2 (the default is 50000). If you prefer using DB2 Client Configuration Assistant, execute the following steps: 1. On the Windows task bar, click Start -> Programs -> IBM DB2 -> Client Configuration Assistant. The Client Configuration Assistant is displayed (Figure 6-9).242 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-9 Client Configuration Assistant opening dialog2. Click Add and the Add Database Wizard shows (Figure 6-10). Select Search the network and then click the Database name tab. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 243
    • Figure 6-10 Add Database Wizard 3. In the Database name tab, click Add System. An Add System dialog appears (Figure 6-11). Selected TCPIP and complete the Host name information. Click OK.244 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-11 Add System dialog window 4. A list of DB2 databases should expand from the DB2 server you added (Figure 6-12). Select ITM_DB and click Finish. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 245
    • Figure 6-12 Select ITM_DB in the dialog window 5. A Confirmation window appears (Figure 6-13). It is recommended to test the ODBC connection to verify a healthy link. Click Test Connection. Figure 6-13 Confirmation dialog window 6. Fill out the User ID and Password information (Figure 6-14). Click OK.246 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-14 User ID and Password dialog window 7. The connection should be successful (Figure 6-15). If not, review your settings and change them accordingly. Figure 6-15 ODBC connection successful6.5.3 Installing the ITM 5.1.1 AMX ETL processes To install the AMX warehouse pack, we completed the following steps on the control server: 1. Click Start -> Programs -> TDW -> Install a Warehouse Pack, this starts the warehouse pack installation wizard (see Figure 6-16 on page 248). In the Welcome windows, click Next. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 247
    • Figure 6-16 Install a Warehouse Pack window 2. The following window shows the location of the Tivoli common logging directory (Figure 6-17 on page 249) which will contain all TDW log files. In our installation we use the default location C:Program FilesibmTivolicommon. Click Next.248 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-17 Tivoli Common Logging Directory window3. In the windows as shown in Figure 6-18 on page 249, click Add to add the AMX warehouse pack.Figure 6-18 Add Warehouse Pack window Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 249
    • 4. In the Location of installation properties file window, as shown Figure 6-19 on page 250, specify the location of the AMX warehouse pack installation properties file, twh_install_props.cfg: you can find this file in the IBM Tivoli Monitoring version 5.1 media, under the tedw_apps_etlAMXpkg directory. Click Next. Figure 6-19 Location of installation properties 5. The installation menu window (Figure 6-20 on page 251) now lists the IBM Tivoli Monitoring warehouse enablement pack. Select it and click Next to continue.250 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-20 Installation menu window6. Click Install in the summary window (Figure 6-21 on page 252) to start the warehouse pack installation. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 251
    • Figure 6-21 Installation summary window 7. View the progress of installation through the messages that are shown in windows until its completion. The final installation window (Figure 6-22 on page 253) contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors, and then click Next. If warnings are listed, check the logs to ensure that the warnings can safely be ignored. Click Finish to exit the wizard.252 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-22 AMX installation completion window.6.5.4 Installing AMX Fix Packs In Tivoli Data Warehouse 1.2 the installation process for a WEP patch is exactly the same used for the WEP installation. We installed Fix Pack 6 for AMX following the steps described in “Installing the ITM 5.1.1 AMX ETL processes” on page 247 and specifying the location of file twh_install_props.cfg under of the AMX subdirectory in the Fix Pack media (see Figure 6-23 on page 254). Important: Verify that all the ETL processes belonging to the enablement pack for which you are installing a Fix Pack are in development mode. This is required for preventing that any ETL process may run during the Fix Pack installation. If you have already configured your ETLs with DB accounts information before installing the Fix Pack, you have to configure them again after the Fix Pack installation (see “Defining the authority to the warehouse sources and targets” on page 254). Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 253
    • Figure 6-23 Installation of AMX Fix Pack 66.5.5 Defining the authority to the warehouse sources and targets You should inform the Tivoli Data Warehouse Control Center server of user access information for every Source and Target ETL process installed by the IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1. Follow these steps: 1. Start the IBM DB2 Control Center utility by selecting Start -> Programs -> IBM DB2 -> Control Center. 2. On the IBM DB2 Control Center utility, start the IBM DB2 Data Warehouse Center utility by selecting Tools -> Data Warehouse Center. The Data Warehouse Center logon windows appears. 3. Log into the IBM DB2 Data Warehouse Center utility using the local DB2 administrator user ID. In our case, db2admin. 4. In the Data Warehouse Center window, expand the Warehouse Sources folder. As shown in Figure 6-24, there are two entries for the IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 programs that need to be configured, as follows: – AMX_ITM_RIM_Source – AMX_TWH_CDW_Source254 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-24 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 SourcesYou should edit the properties of each one of the above entries. In order to dothat, right-click it and select Properties and then select the Data Source tab. Fillin the database instance owner user ID information (Figure 6-25 andFigure 6-26).Figure 6-25 AMX_ITM_RIM_Source user ID information Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 255
    • Figure 6-26 AMX_TWH_CDW_Source user ID information 5. For the IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Target ETL, shown in Figure 6-27, expand the Warehouse Target folder, right-click the AMX_TWH_CDW_Target, select Properties and then select the Database tab. Fill in the user ID information.Figure 6-27 IBM Tivoli Monitoring, Version 5.1.1 Generic ETL1 Target256 Implementing Tivoli Data Warehouse 1.2
    • For our environment the values are shown in Figure 6-28. Figure 6-28 AMX_TWH_CDW_Target user ID information6.5.6 Modifying the ETL for the source table name to the RIM user The AMX ETL assumes that the schema name of ITM_DB RIM is set to db2admin. This is the default name for databases created on a Windows machine, but if you are using DB2 on other platforms or you have a different instance name, you must modify the source table name into the DB2 Warehouse Manager. In our case, we have the ITM_DB on a UNIX server and we use db2inst1 as the user ID and instance name. Therefore we have to modify the source table name, executing the following steps: 1. From the Data Warehouse Center, from the Warehouse sources list, select the AMX_ITM_RIM_Source and open the Property page. 2. Go to the Tables and views tab as shown in Figure 6-29. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 257
    • Figure 6-29 Tables and views of AMX_ITM_TIM_Source 3. Expand the Tables folder and you will get a dialog asking for the name filter, such as shown in Figure 6-30. We only need to get the table called ENDPOINTS. The schema name is the RIM user ID. Figure 6-30 Table name filter specification258 Implementing Tivoli Data Warehouse 1.2
    • 4. When the DB2.ENDPOINTS has been found, move it from the available tables and views to the selected tables and views box by clicking the > button. You will have two ENDPOINTS tables as shown in Figure 6-31. Click OK.Figure 6-31 Endpoint tables5. Now from the Data Warehouse Center, expand the Subject Area and find the process called AMX_c05_ETL1_Process. Right-click it and select Open. The Process Modeller window shown is similar to Figure 6-32. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 259
    • Figure 6-32 AMX_c05_ETL1 process260 Implementing Tivoli Data Warehouse 1.2
    • 6. Click the tables icon and click the work area; a dialog box will be presented, as shown in Figure 6-33. Select the DB2INST1.ENDPOINTS table and click the > button. Then click OK.Figure 6-33 Selecting new table7. The new table is now shown in the process modeller window; now we need to connect the tables to the first step. Use the link icon and select data links. Drag the cursor from the ENDPOINTS table to the AMX_c05_s010_RIM_Extract step, and a new link is created.8. Remove the old link by selecting the link, right-click and select Remove. Remove also the old DB2ADMIN.ENDPOINTS table by selecting it, right-click and select Remove.9. Save the process model using the menu Process -> Save and close the window. Attention: At this time, we bypass scheduling the AMX ETL and changing the status of the ETL to production. We do this because we are not ready to run the AMX ETL until an ETL2 is installed (AMY). Running the AMX ETL prematurely will result in errors and prevent you from gathering data in future collections. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 261
    • 6.6 Installing and configuring ITM for OS WEP This section describes how we installed and configured the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack for Operating Systems (AMY) in our lab environment. The steps required for AMY installation are the same required by AMX, as described in “Installing and configuring ITM Generic WEP” on page 241, but in this case we are not required to configure a new DB2 connectivity to the data source, because AMY uses the central data warehouse database as data source.6.6.1 Backing up the TWH databases Once again, before installing any warehouse enablement pack (WEP), it is highly recommended to back up all the Tivoli Data Warehouse databases. We created different directories to store these backups. This enables us to restore Tivoli Data Warehouse at the last known workable level should any unforeseen problems occur during installation or afterwards. C:tedwbackupspre-amytwh_cdw C:tedwbackupspre-amytwh_mart C:tedwbackupspre-amytwh_md After creating these directories, we completed the following steps on the DB2 server from a command window: 1. Stop and restart DB2: db2 force applications all db2stop force db2start 2. Back up the TWH_CDW database: db2 backup db TWH_CDW to C:tedwbackupspre-amytwh_cdw 3. Back up the TWH_MART database: db2 backup db TWH_MART to C:tedwbackupspre-amytwh_mart 4. Back up the TWH_MD database: db2 backup db TWH_MD to C:tedwbackupspre-amytwh_md6.6.2 Installing the ITM 5.1.1 AMY ETL processes To install the IBM Tivoli Monitoring for Operating Systems v. 1.1 warehouse pack (AMY), we followed the same steps we executed on the control server for the AMX pack:262 Implementing Tivoli Data Warehouse 1.2
    • 1. Click Start -> Programs -> TDW -> Install a Warehouse Pack, this starts the warehouse pack installation wizard (see Figure 6-16 on page 248). In the Welcome windows, click Next.2. The following window shows the location of the Tivoli common logging directory (Figure 6-17 on page 249) which will contain all TDW log files. In our installation we use the default location C:Program FilesibmTivolicommon. Click Next.3. In the windows as shown in Figure 6-18 on page 249, click Add to add the AMY warehouse pack.4. In the Location of installation properties file window, as shown Figure 6-19 on page 250, specify the location of the AMY warehouse pack installation properties file, twh_install_props.cfg: you can find this file in the IBM Tivoli Monitoring version 5.1 media, under the tedw_apps_etlAMYpkg directory. Click Next.5. The installation menu window now lists the IBM Tivoli Monitoring warehouse enablement pack (Figure 6-34). Select it and click Next to continue.Figure 6-34 Installation menu window with the AMY pack6. Click Install in the summary window (Figure 6-21 on page 252) to start the warehouse pack installation. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 263
    • 7. View the progress of installation through the messages that are shown in windows until its completion. The final installation window contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors, and then click Next. If warnings are listed, check the logs to ensure that the warnings can safely be ignored. Click Finish to exit the wizard. Figure 6-35 AMY installation completion window6.6.3 Installing AMY Fix Packs Similarly to the installation procedure of the Fix Pack 6 for the AMX ETL, we have to install Fix Pack 6 for AMY by following the steps described in 6.6.2, “Installing the ITM 5.1.1 AMY ETL processes” on page 262 and specifying the location of file twh_install_props.cfg under of the AMY subdirectory in the Fix Pack media (see Figure 6-36).264 Implementing Tivoli Data Warehouse 1.2
    • Important: Verify that all the ETL processes belonging to the warehouse enablement pack for which you are installing a Fix Pack are in “development” mode. This is required for preventing the possibility that any ETL process may run during the Fix Pack installation. If you have already configured your ETLs with DB accounts information before installing the Fix Pack, you have to configure them again after the Fix Pack installation (see “Defining the authority to the warehouse sources and targets” on page 254). Figure 6-36 Installation of AMY Fix Pack 66.6.4 Defining the authority to the warehouse sources and targets We now have to provide Tivoli Data Warehouse Control Center with user access information for every Source and Target ETL process installed by the IBM Tivoli Monitoring, Version 5.1.1 AMY ETL. To do so, we performed the following steps: 1. Start the IBM DB2 Control Center utility by selecting Start -> Programs -> IBM DB2 -> Control Center. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 265
    • 2. On the IBM DB2 Control Center utility, start the IBM DB2 Data Warehouse Center utility by selecting Tools -> Data Warehouse Center. The Data Warehouse Center logon windows appears. 3. Log into the IBM DB2 Data Warehouse Center utility using the local DB2 administrator user ID. In our case, db2admin. 4. In the Data Warehouse Center window, expand the Warehouse Sources folder. Update the database sources that relates to the application that you want to configure. In this example, this is AMY_TWH_CDW_Source. You should edit the properties of this warehouse source. In order to do that, right-click and select Properties and then select the Data Source tab. Fill in the database user ID and password information. For our environment the values are shown in Figure 6-37. Figure 6-37 AMY_TWH_CDW_Source user ID information 5. Again, for the AMY warehouse targets, you need to modify the user ID information from the property pages. You should edit the properties of each one of those warehouse targets. In order to do that, right-click it and select Properties and then select the Database tab. Fill in the database user ID and password information. For our environment the values are shown in Figure 6-38, using the AMY_TWH_MART_Target as example.266 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-38 AMY_TWH_MART_Target user ID information6.7 Testing, scheduling, and promoting the ETLs After successfully installing and configuring the IBM Tivoli Monitoring, Version 5.1.1 Generic Warehouse Enablement Pack (ETL1) and the IBM Tivoli Monitoring, Version 5.1.1. Warehouse Pack for Operating Systems (ETL2), we can now test all the processes from both ETLs. If they run correctly, we can schedule and then promote them into production mode.6.7.1 Testing the ETLs To test the ETLs manually, the following steps are required: 1. Select the ETL processes in the Data Warehouse Center windows, right-click them and choose Mode -> Test as shown in Figure 6-39. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 267
    • Figure 6-39 Change ETL mode to Test 2. Right-click the process you want to test and choose Test (see Figure 6-40 on page 269). The process must be run sequentially according the dependency described in the ETL process model (see Figure 6-32 on page 260). The sequence for all AMX and AMY processes is as follows: a. AMX_c05_ETL1_Process i. AMX_c05_s05_Pre_Extract ii. AMX_c05_s010_Rim_Extract iii. AMX_c05_s020_Parsing iv. AMX_c05_s030_Exception v. AMX_c05_s040_Comp_Msmt b. AMX_c10_Rim_Prune_Process i. AMX_c10_s010_Rim_Prune c. AMY_c05_ETL1_Data_Update_Process i. AMY_c05_s010_Update d. AMY_m05_ETL2_Process i. AMY_m05_s05_Mart_Prepare_Stage ii. AMY_m05_s010_Mart_Pre_Extract iii. AMY_m05_s020_Mart_Extract iv. AMY_m05_s030_Mart_Load v. AMY_m05_s040_Mart_Rollup vi. AMY_m05_s050_Mart_Prune268 Implementing Tivoli Data Warehouse 1.2
    • The following processes should only be tested and executed in case you need to reset the OS data mart. All the data will be cleaned out of the data mart database. e. AMY_m10_Reset_ETL2_Process i. AMY_m10_s010_Reset_OS_Data_MartFigure 6-40 Manually test the ETLs 3. To verify the status of processes (Successful, Failed, In Progress, or Scheduled), select in the Data Warehouse Center Warehouse -> Work in Progress. The Work in Progress window (see Figure 6-41 on page 270) not only shows the status of all processes for all ETLs with status set to Successful, Failed, In Progress, or, if promoted to production, Scheduled, but also allows you to run a process again by simply right-clicking on it and selecting Run now. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 269
    • Figure 6-41 Work in progress window6.7.2 Checking that data has been collected Once all the processes are tested, we checked that the data mart database has been populated with ITM data. To do this, we used the DB2 Control Center to connect to TWH_MART and show a sample content of one of the tables, for example, the TABLE F_OS_HOUR, as shown in Figure 6-42. If you have some data in the table, you have a proof that all the ETL processes are running correctly. If that table is empty, you should verify that your data source is note empty and then run all the ETL processes again.270 Implementing Tivoli Data Warehouse 1.2
    • Figure 6-42 Sample Content of table F_OS_HOUR Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 271
    • 6.7.3 Scheduling the ETLs The following four processes need to be scheduled for the IBM Tivoli Monitoring, Version 5.1.1, Generic ETL1 and IBM Tivoli Monitoring, Version 5.1.1. Warehouse Pack for Operating Systems: AMX_c05_ETL1_Process AMX_c10_Rim_Prune_Process AMY_c05_ETL1_Data_Update_Process AMY_m05_ETL2_Process The following steps are similar for both processes and we will use AMX_c05_ETL1_Process to describe them. On the DB2 Control Center server, using the Data Warehouse Center window, expand Subject Areas. Select AMX_IBM_TIVOLI_Monitoring_v5.1.1_Subject_Area -> Processes and right-click AMX_c05_ETL1_Process. Choose Schedule, as shown in Figure 6-43.Figure 6-43 Schedule AMX_c05_ETL1_Process272 Implementing Tivoli Data Warehouse 1.2
    • Selecting Schedule will open up a dialog as shown in Figure 6-44.Figure 6-44 Schedule configuration for AMX_c05_ETL1_Process Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 273
    • 6.7.4 Promoting the ETL status to Production mode All of the processes are composed of components that have Development status set as default. In order for them to run, their status need to be changed from Development (or Test) to Production. To do this, we used the Data Warehouse Center window again, selecting all the AMX and AMY processes listed in “Scheduling the ETLs” on page 272, right-clicking them and choosing Mode -> Production, as shown in Figure 6-45 on page 274 for the AMX_c05_ETL1_Process.Figure 6-45 Promoting ETLs to Production274 Implementing Tivoli Data Warehouse 1.2
    • 6.8 Reporting Next we show how to set up, configure, and use some of the reports provided by the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack. Note: As described in Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71, an ODBC connection to the data mart database needs to be defined on the Crystal Enterprise server before we can work with the reports. Please refer to that chapter for details.6.8.1 Available reports Here is a list of predefined reports shipped with the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack: Operating System Busiest Systems Operating System Health of a Backup Server Operating System Memory Utilization Operating System Network Statistics Operating System Paging File Utilization Operating System UNIX CPU Statistics Operating System Usage of Domain Controller Operating System Windows CPU Statistics Crystal Enterprise Professional Version 9 for Tivoli has reduced configuration options in comparison to a full Crystal license. If the reports shipped with IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack do not match your needs and you want to develop additional reports, you have to upgrade your Crystal Enterprise installation.6.8.2 Accessing the Crystal ePortfolio All reports provided by Tivoli Data Warehouse 1.2 must be accessed using the ePortfolio feature of the Crystal Enterprise. The ePortfolio can be accessed from the Crystal Enterprise launchpad. In order to access the Crystal Enterprise launchpad, start an Internet browser and open the following URL: http://<hostname>/crystal/enterprise9 Here, <hostname> represents the hostname of the Crystal Enterprise report server, as shown in Figure 6-46. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 275
    • Figure 6-46 Crystal Enterprise - Launchpad In this section, we concentrate on viewing IBM Tivoli Monitoring, Version 5.1.1 reports and we do not explain the configuration of Crystal Enterprise to its full extent. For details on configuration and administration tasks, refer to the following manuals shipped with the product: Crystal Enterprise 9 Installation Guide Crystal Enterprise 9 Administrator’s Guide Crystal Enterprise 9 Getting Started Guide Crystal Enterprise 9 ePortfolio User’s Guide276 Implementing Tivoli Data Warehouse 1.2
    • From the Crystal Enterprise Launchpad, proceed by selecting the ePortfolio link,which will bring you to the window shown in Figure 6-47. In the top bar, you cansee that we are authorized as user guest. By default, the guest user has noaccess to the NetView reports as indicated by the words No Folders on the leftside of the window.Figure 6-47 Crystal Enterprise 9 - ePortfolioThe installation process of the first warehouse enablement pack on the TivoliData Warehouse environment creates a user ID on the Crystal Enterpriseenvironment named Tivoli. This user ID is to be used to access the reportsprovided by any IBM Tivoli software. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 277
    • To log in as the Tivoli user ID, select the Log On button in the upper right corner of the ePortfolio window in Figure 6-47 on page 277. The Log On window as shown in Figure 6-48 is presented. The Tivoli user ID has no password by default. We use the Enterprise authentication method as we have specified during the Crystal Enterprise installation. Figure 6-48 Crystal Enterprise 9 - Log in278 Implementing Tivoli Data Warehouse 1.2
    • After entering the required data, select Log On to proceed. Now we are back atthe ePortfolio window in Figure 6-49, but now with user Tivoli authority. Instead ofNo Folders in the guest users ePortfolio window in Figure 6-47, there is now alink visible with the name IBM Tivoli Monitoring for Operating Systems in theTivoli user ePortfolio window in Figure 6-49 .Figure 6-49 Crystal Enterprise 9 - Folders Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 279
    • We follow this link by selecting IBM Tivoli Monitoring for Operating Systems and proceed to the IBM Tivoli Monitoring, Version 5.1.1 reports as shown in Figure 6-50. All reports provided by the IBM Tivoli Monitoring Warehouse Enablement Pack are listed there. Figure 6-50 Crystal Enterprise 9 - available reports for ITM280 Implementing Tivoli Data Warehouse 1.2
    • In order to obtain the reports, select the desired report, for example, OperatingSystem Busiest Systems, and select Schedule, as shown in Figure 6-51.Figure 6-51 Scheduling Operating System Busiest System reportThe schedule report panel starts. In order to run the reports at this time, selectNow under the Run Report option. As this report requires additional parameters,such as time frame, select Parameters under the Customize your Optionsoption, as shown in Figure 6-52. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 281
    • Figure 6-52 Crystal Enterprise 9 - parameters Figure 6-53 shows the selection of parameters for the report. Select Schedule when ready to run the report. Figure 6-53 Crystal Enterprise 9 - Parameters for the report282 Implementing Tivoli Data Warehouse 1.2
    • Because we selected to run this report now, the report is scheduled immediatelyand the reports history window is opened. The just-scheduled report will run andits initial status is set to Pending. Note: The History window is not updated automatically. Click the Refresh button to view the current state.Figure 6-54 Crystal Enterprise 9 - Report HistoryTo view successful generated reports from the history window as shown inFigure 6-54, click the link Instance Time in the left column of the table to viewthe associated report. The report is shown in Figure 6-54. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 283
    • Figure 6-55 Operating System Busiest Systems report284 Implementing Tivoli Data Warehouse 1.2
    • Next we present some more examples of reports provided by the IBM Tivoli Monitoring, Version 5.1.1 Warehouse Enablement Pack. Figure 6-56 shows the Operating System Paging File Utilization report.Figure 6-56 Operating System Paging File Utilization Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 285
    • Figure 6-57 shows the Operating System UNIX CPU Statistics report.Figure 6-57 Operating System Operating System UNIX CPU Statistics6.9 Troubleshooting of ITM data collection In this section we discuss the commands and techniques that can be used whenever the expected data is not collected into the IBM Tivoli Monitoring database used as a data sources for the Tivoli Data Warehouse. First we show how to use the itmchk.sh tool provided by Tivoli Support to check the ITM environment automatically, then we discuss all the steps required for doing a manual check of the ITM data collection instead.286 Implementing Tivoli Data Warehouse 1.2
    • 6.9.1 Using itmchk.sh script Tivoli Software Support provides a script (itmchk.sh) to check automatically most of the configuration setting required to have data successfully uploaded into the ITM database using a RIM Object. This script can be downloaded from the IBM Software Support Web site at the following link: http://www-1.ibm.com/support/docview.wss?uid=swg24003854 To access that Web site and download the script, you must provide an IBM ID: If you do not have it, you can apply on the same Web link to receive one. The itmchk.sh script is bundled in a package called ITM - TEDW Analysis Tools together with another script (twhchk.sh), which can be used to check the installed components and their installation order in a Tivoli Data Warehouse 1.2 environment and to provide a summary of meaningful data from both the ITM and TEDW databases. To run the itmchk.sh script, follow these steps: 1. Download the ITM-TEDW-Health.tar in the Tivoli Manage Node where your ITM RIM object is available (in our case study scenario, the TMR server TDW009 is also the RIM host for ITM database). 2. Extract all the files and directories contained in the archive file. 3. Go into the ITM-TEDW-Health directory. 4. Setup the Tivoli environment (. /etc/Tivoli/setup_env.sh for a UNIX system). 5. Run the command ./itmchk.sh 6. The program prompts for the ITM RIM name as shown in Example 6-6. Choose the number corresponding to the RIM name used for ITM_DB database and press ENTER. Example 6-6 Running itmchk.sh tool # ./itmchk.sh ======================================================== IBM Tivoli Monitoring - Tivoli Enterprise Data Warehouse Configuration and Status Snapshot (c) IBM - Tivoli ======================================================== ==> Analysis started at: Aug 14 2003 11:46:10 ==> Setting Debug Mode... (I) Debug Disabled ==> Including Help Messages Routines... (I) Done Including Help Messages Routines Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 287
    • ==> Looking up TME 10 System Information... (I) Valid User Found ==> Getting Environmental Configuration... (I) Environment Successfully Imported ==> Looking for RIM Objects definition... (I) Discovered the following RIM Objects: 1) itm_rim_tdw009 2) mdist2 3) spr_rim -n (?) Select the One used by Warehouse Component: 7. The tool checks ITM configuration and then prints a report as shown in Example 6-7. Example 6-7 itmchk.sh tool report (I) Using RIM Object (itm_rim_tdw009) ==> Retrieving RIM Object Information... (I) Valid RIM Object Type Found (I) RIM Object Configuration is: <> Database Vendor: DB2 <> Database Home: /usr/lpp/db2_07_01 <> Database Name: itm_db <> Database User: db2inst1 <> DB2 Instance Home: /home/db2inst1 (I) The RIM Host Trace is turned OFF ==> Testing RIM Object (itm_rim_tdw009) Connectivity... (I) Attempting to Connect... (I) RIM Object (itm_rim_tdw009) is Working Properly ==> Retrieving Data Collector Configuration... (I) RIM Object (itm_rim_tdw009) Matches the Data Collector Configuration (I) The XML files will be processed 6 (default) times before being archived. (I) The time between two consecutive requests of data upload is 1 (default) minutes. (I) Data will be uploaded 30 minutes past the full hour. (I) Old data will be removed from the ITM database at 0 (default) oclock. (I) The ITM database will not retain data older then 30 (default) days. ==> Retrieving Middle Layer Trace Configuration... (I) The profile distribution trace is at level 1 (default) and has a size of 1 (default) bytes. ==> Retrieving the Status of the EP cache... (I) Cache status is: Alive for Endpoint: tdw001 (I) Cache status is: DMEngineOff for Endpoint: tdw002288 Implementing Tivoli Data Warehouse 1.2
    • (I) Cache status is: Alive for Endpoint: tdw003 (I) Cache status is: DMEngineOff for Endpoint: tdw004 (I) Cache status is: Alive for Endpoint: tdw005 (I) Cache status is: Alive for Endpoint: tdw006 (I) Cache status is: Alive for Endpoint: tdw007 (I) Cache status is: Alive for Endpoint: tdw008 (I) Cache status is: Alive for Endpoint: tdw009 (I) Cache status is: Alive for Endpoint: tdw010 (I) Cache status is: DMEngineOff for Endpoint: tdw011==> Retrieving Info about the Queued Requests... (I) Endpoint="tdw001" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw002" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw003" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw004" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw005" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw006" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw007" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw008" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw009" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw010" LastCall="Wed Aug 13 14:39:21 CDT 2003" (I) Endpoint="tdw011" LastCall="Wed Aug 13 14:39:21 CDT 2003"==> Done. ======================================================== ================== Analysis Completed ================== ========================================================The itmchk.sh report shows the following information: RIM Object configuration RIM connection status Data Collector configuration parameters (see “Installing the ITM WEP data collector component” on page 232) Middle Layer Trace Configuration (used for debugging) Status of the endpoint cache The last data upload request to each endpoint Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 289
    • 6.9.2 Manual checking of ITM data collection In this section we show how to check manually every step of ITM data collection. We suggest the following approach: 1. Retrieve the date of last data upload into ITM database 2. Test the RIM connection 3. Check the status of distributed ITM resource models 4. Review data collection parameters 5. Analyze trace files Retrieving date of last data upload into ITM database Connect to ITM_DB, as shown in Example 6-8. Example 6-8 Retrieving the date of last data upload into ITM database db2 => connect to ITM_DB user db2inst1 using <password> Database Connection Information Database server = DB2/NT 7.2.0 SQL authorization ID = DB2INST1 Local database alias = ITM_DB Then select the date of the last data upload: db2 => select max(timekey_dttm) from metricsdata 1 -------------------------- 2003-08-12-05.00.21.000000 1 record(s) selected. The date of the last data upload allows an estimation of the time frame in which the data collection process stopped. This estimate can be very useful when examining the monitoring trace files to help in understanding the reasons for data upload failure. You can also check the hostnames of all endpoints that provided data, as shown in Example 6-9. Example 6-9 Names of the endpoints collecting data db2 => select host_name from endpoints HOST_NAME ----------------------------------------------------------290 Implementing Tivoli Data Warehouse 1.2
    • tdw001tdw002tdw003tdw004tdw005tdw006tdw007tdw008tdw009tdw010tdw011 11 record(s) selected.This information can be used to track which monitored endpoints are notproviding data.Testing the connection between RIM host and ITM databaseOnce you have ascertained that ITM_DB does not receive data from monitoredendpoints, you should check the connection between the RIM host and thedatabase. From any managed node of your Tivoli region having Tivolienvironment configured, run the wrimtest -l <rim_object> command as inExample 6-10. If you do not remember the name of your RIM object, use thecommand wlookup -ar RIM to show all the RIM objects defined in your Tivolienvironment.If the wrimtest command detects a connection failure, you should review all theparameters specified during RIM object creation.Example 6-10 wrimtest command output# wrimtest -l itm_rim_tdw009Resource Type : RIMResource Label : itm_rim_tdw009Host Name : tdw009User Name : db2inst1Vendor : DB2Database : itm_dbDatabase Home : /usr/lpp/db2_07_01Server ID : tcpipInstance Home : /home/db2inst1Instance Name :Opening Regular Session...Session OpenedRIM : Enter Option > Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 291
    • Often a connection failure is simply caused by a change in the database password without the needed update of the RIM object. To change the RIM password use the command: wsetrimpw rim_name <old_password> <new_password> Checking the status of distributed resource models The command wdmlseng -e <endpoint_name> can be used to list the status of all resource models distributed to the specified endpoint. Only the resource models with status Running are collecting data on the endpoint. Example 6-11 Status of resource models distributed on an endpoint # wdmlseng -e tdw006 Forwarding the request to the engine... The following profiles are running: tmw2kDefProfile#tdw009-region TMW_EventLog :Running TMW_PhysicalDiskModel :Running TMW_TCPIP :Running TMW_LogicalDisk :Running TMW_MemoryModel :Running TMW_Process :Running TMW_Processor :Running To change the status of resource models, modify the original monitoring profile using the Tivoli desktop and redistribute it to the endpoint. If the monitoring engine is not running correctly on the endpoint, you can try to restart it using the command: wdmcmd -restart -e <endpoint_name> Reviewing data collection parameters The data collector process populates the ITM_DB database with the monitoring value metric from the endpoints at every interval you specified in the wdmcollect command. The collected metrics are then temporarily cached in the managed node in the $DBDIR/dmml/tedw/<endpoint> directory. The file is a zip file that contains the metrics information in XML format. Once you issue the wdmcollect command, after the interval is expired, you can check whether the file is created.292 Implementing Tivoli Data Warehouse 1.2
    • If your ITM database does not receive data even if your resource models arecorrectly running on the monitored endpoints and the connection between RIMhost and database is working, you should verify that:1. Enable Data Logging and TEDW Data in the Logging option of your monitoring profiles are checked (see “Activate data collection” on page 237).2. The data collection parameters are correct (use wdmconfig -m <nodename> -G datacollector* command).3. The gateway data collection process was started with the wdmcollect command (use wdmcollect -m all -q to list the active collection processes for all managed nodes).Checking trace filesIf you went through all the previous steps but your ITM database still does notreceive data, you can examine the log files. There are several trace and log filesrelated to the data collector process under $DBDIR/AMW/logs directory on themanaged nodes: msg_DataCollector.log trace_tmnt_datacollector_eng1.log trace_tmnt_hb_eng1.log trace_tmnt_profile_core1.log trace_tmnt_rimh_eng1.log trace_tmnt_rm_eng1.log trace_tmnt_task_eng1.logThe message file contains operational messages, while the trace files containerror messages. Each function of the data collector has its own trace file. Thesefunctions are the main data collector engine, heartbeat engine, RIM interface,resource model engine, and task execution engine.For TEDW interface, the important files are msg_DataCollector.log andtrace_tmnt_rimh_eng1.log. In Example 6-12 we show the output ofmsg_DataCollector.log for a successful data upload from endpoint tedw1: youcan easily follow all the steps from the distribution to the upload to the database.Example 6-12 msg_DataCollector.log<F>1035246605000<F>Tue Oct 22 00:30:05 2002GMT<F>AMW<F>DataCollector<F>eastham<F>12488<F>AMW<F> - AMW0181I -The distribution with ID 19431921771035246602 succeeded on the endpointtedw1.<F>1035247476000<F>Tue Oct 22 00:44:36 2002GMT<F>AMW<F>DataCollector<F>eastham<F>12488<F>AMW<F> - AMW0189I - Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 293
    • The data related to the request 520207df has been successfully received <F>1035247480000<F>Tue Oct 22 00:44:40 2002 GMT<F>AMW<F>DataCollector<F>eastham<F>19074<F>AMW<F> - AMW0202I - The file /var/spool/Tivoli/eastham.db/dmml/tedw/tedw1/1035246605.zip is going to be processed for upload. <F>1035247481000<F>Tue Oct 22 00:44:41 2002 GMT<F>AMW<F>DataCollector<F>eastham<F>19074<F>AMW<F> - AMW0198I - The data related to file: /var/spool/Tivoli/eastham.db/dmml/tedw/tedw1/1035246605.zip.dir/ITM_WH@2002#10 #22#0 #20.xml have been successfully loaded into the DataBase In Example 6-13, instead, we show the occurrence of a failure in the data collection process. In this case the problem was caused by a stop of Tivoli Framework processes on the RIM host. Example 6-13 trace_tmnt_rimh_eng1.log <F>1037723936000<F>Tue Nov 19 16:38:56 2002 GMT<F>AMW<F>datacollector<F>eastham<F>18332<F>MIN<F>../../../../.. /src/objects/DataCollector/platform/StoreData/RIMConnectionHandler.cxx<F>RIMCon nectionHandler::connect()<F>537 026664<F>FRWSL0005E A communications failure occurred: FRWOG0014E destination dispatcher unavailable Please refer to the TME 10 Framework Planning and Installation Guide, "TME Maintenance and Troubleshooting" for details on diagnosing communication errors or contact your Tivoli support provider. The IBM Tivoli Monitoring has important log files also at the endpoints: For Windows platform, most information is stored in Tmw2k.log file under the $LCF_DATDIR/LCFNEW/Tmw2k directory. For the UNIX platform, the logs are under the $LCF_DATDIR/LCFNEW/AMW/logs directory. These log files are: – msg_dmxengine.log – trace_dmxengine.log – trace_dmxeu.log – trace_dmxntv.log As for the managed nodes logs, the message file contains operational messages, while the trace files contain error messages. In Example 6-14 we show how the trace_dmxengine.log reports a problem with a distributed resource model (in this case, the Network Interface resource model in ITM_Unix_monitor profile).294 Implementing Tivoli Data Warehouse 1.2
    • Example 6-14 trace_dmxengine.log<F>1036710042642<F>Thu Nov 07 17:00:42 CST2002<F>AMW<F>Engine<F>yarmouth<F>12240<F>MIN<F>ReferenceModel profile=ITM_Unix_monitor#eastham-regionmodel=DMXNetworkInterface<F><F>Thread[TmrSrvAction_RMTimer,5,main]<F>No NICsfound!<F>NoneFor further information about IBM Tivoli Monitoring troubleshooting, refer toAppendix C of IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569-01. Chapter 6. IBM Tivoli Monitoring Warehouse Enablement Pack 295
    • 296 Implementing Tivoli Data Warehouse 1.2
    • 7 Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack This chapter provides an overview of the tasks to be performed to install IBM Tivoli Storage Manager, Version 5.2 Enablement Pack for Tivoli Data Warehouse 1.2. We cover the following topics: “Case study overview” on page 298 “IBM Tivoli Storage Manager WEP overview” on page 299 “Prerequisites” on page 300 “Installing and configuring ITSM WEP 5.2” on page 301 “IBM Tivoli Storage Manager ETL processes” on page 314 “Testing, scheduling, and promoting the ETLs” on page 320 “Reporting” on page 322© Copyright IBM Corp. 2004. All rights reserved. 297
    • 7.1 Case study overview Table 7-1 gives a brief summary of the distributed Tivoli Data Warehouse 1.2 environment we started with in order to install the IBM Tivoli Storage Manager Warehouse Enablement Pack Version 1.1.0. Table 7-1 Environment for NetView integration Hostname Component Database Operating system TDW003 TDW Control DB2 Server W2000 FP4 Server 1.2 7.2FP10a TDW004 TDW Central DB2 Server W2000 FP4 Warehouse 1.2 7.2FP10a (remote agent site) TDW009 TDW Data Mart 1.2 DB2 Server AIX 5.2 7.2FP10a TDW002 Crystal Enterprise DB2 Client W2000 FP4 Server 9 7.2FP10a TSMSRV01 IBM Tivoli Storage - W2000 FP4 Manager 5.2298 Implementing Tivoli Data Warehouse 1.2
    • Figure 7-1 gives a brief summary of the distributed Tivoli Data Warehouse environment used in this chapter. Agent site TWH_CDW Central Data Warehouse Windows 2000 Server Hostname: TDW004 TWH_MD TDW Control Server IBM Tivoli Storage Manager 5.2 Windows 2000 Server Hostname: TDW003 Windows 2000 Server Hostname: TSMSVR01 Data Mart TWH_MART Crystal Enterprise Server Data Mart Windows 2000 Server IIS Web Server AIX 5 Crystal Enterprise Professional 9 for Tivoli Hostname: TDW009 Hostname: TDW002 Figure 7-1 TDW 1.2 - distributed deployment scenario7.2 IBM Tivoli Storage Manager WEP overview The ITSM WEP extracts historical information from one or more IBM Tivoli Storage Manager servers and loads it into the Tivoli Data Warehouse 1.2 environment. Subsets of the historical information stored in the central data warehouse database may be loaded into data mart databases to enable reporting in specific areas of interest. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 299
    • The IBM Tivoli Storage Manager Warehouse Enablement Pack extracts information related to the following components: IBM Tivoli Storage Manager servers Nodes that are managed by the servers Filespaces belonging to the nodes Amount of data stored on the servers on behalf of nodes and their filespaces Storage pools in which the data is stored. IBM Tivoli Storage Manager itself does not record historical information about these entities. As a result, the IBM Tivoli Storage Manager Warehouse Enablement Pack records the current state of each entity as it extracts information from the IBM Tivoli Storage Manager servers. The current state, collected regularly over a period of time, becomes the historical record of the IBM Tivoli Storage Manager activity.7.3 Prerequisites Before installing the IBM Tivoli Storage Manager Warehouse Enablement Pack, you must install the following software: IBM Tivoli Storage Manager 5.2. IBM DB2 Universal Database Enterprise Edition, Version 7.2. IBM DB2 Universal Database Enterprise Edition, Version 7.2 Fix Pack 8e, 9, 10, or 10a. IBM DB2 Universal Enterprise for z/OS and OS/390, Version 7. Tivoli Data Warehouse 1.2. Crystal Enterprise Professional Version 9 for Tivoli. IBM DB2 Warehouse Manager, Version 7.2 Fix Pack 8 with its eFix, on the remote agent sites in case of a distributed deployment. (This is the minimum level of IBM DB2 Warehouse Manager supported by Tivoli Data Warehouse 1.2. Note that Fix Pack 10a is the recommended level and it is shipped with Tivoli Data Warehouse 1.2). IBM Tivoli Storage Manager 5.2 ODBC Driver. The IBM Tivoli Storage Manager Warehouse Enablement Pack supports central data warehouses and data mart databases on DB2 UDB for zOS and for OS390, or on DB2 UDB for Windows and UNIX systems. Regardless of the platform on which data warehouses or data marts reside, the warehouse enablement pack doesn’t support multiple central data warehouse or data mart databases.300 Implementing Tivoli Data Warehouse 1.2
    • 7.4 Installing and configuring ITSM WEP 5.2 In this section we discuss in detail the installation and configuration of the ITSM WEP. The steps required for the IBM Tivoli Storage Manager warehouse enablement pack installation are as follows: Ensure that the IBM Tivoli Storage Manager servers are properly configured. Install and configure the IBM Tivoli Storage Manager 5.2 ODBC on the control server or on the remote agent sites that will access the IBM Tivoli Storage Manager databases (Windows only). Back up the Tivoli Data Warehouse 1.2 environment. Install the IBM Tivoli Storage Manager Warehouse Enablement Pack processes.7.4.1 Changes required on the IBM Tivoli Storage Manager servers Before installing the IBM Tivoli Storage Manager Warehouse Enablement Pack, it is necessary to configure the IBM Tivoli Storage Manager servers from which information will be extracted. The following items must be configured on the servers for the warehouse enablement pack to operate correctly. The servers’ SERVERNAME must be changed from the default value to a value that is distinct from other servers. The SERVERNAME value is used to distinguish between the different servers; otherwise, multiple servers with the same name will result in corruption of the information stored in the central data warehouse. The servers’ SERVERHLADDRESS must be set to one of the following: – The fully qualified TCP/IP hostname of the computer on which the server is running – The IP address of the computer on which the server is running7.4.2 Installing the IBM Tivoli Storage Manager ODBC IBM Tivoli Storage Manager uses a proprietary database. However, it provides a Windows ODBC client and an SQL ‘97 compatible interface by which information about the server may be extracted. As it extracts information from one or more ITSM servers, the central data warehouse ETL (ETL1) analyzes and then stores the information in the central data warehouse. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 301
    • The data mart ETL (ETL2) extracts a subset of the information stored in the central data warehouse database and stores it in a data mart database. The data mart is optimized for reporting on specific areas of interest, and may be used by a number of different reporting tools. Restriction: Since the IBM Tivoli Storage Manager ODBC client runs only on Microsoft Windows platform, it is important that you don’t configure Tivoli Data Warehouse remote agent sites to run on non-Windows platform. In this section we will be installing the IBM Tivoli Storage Manager ODBC on the control center server, which in our case study environment is the TDW003 server. Perform the following tasks to install the IBM Tivoli Storage Manager ODBC on the control server: 1. Locate the TSM520C_GA_ODBC_.exe file in the IBM Tivoli Storage Manager 5.2 installation Media. 2. Double-click the file, and on the Welcome screen, click Next. 3. Select a temporary directory to save installation files and click Next twice to confirm the installation. 4. Select the installation directory for ODBC and click Next. 5. Select Complete setup type and click Next, as shown in Figure 7-2. Figure 7-2 ITSM ODBC Installation 6. Click Install to begin the installation and Finish to complete.302 Implementing Tivoli Data Warehouse 1.2
    • The Tivoli Data Warehouse warehouse pack installer cannot configure IBM TivoliStorage Manager data sources. It will recognize an existing IBM Tivoli StorageManager ODBC data source named ITSMSRC, and allow it to be used with thewarehouse enablement pack.To create the IBM Tivoli Storage Manager ODBC data source on the controlserver. perform the following steps:1. Click Start ->Control Panel -> Administrative Tools -> Data Source (ODBC).2. Go to the System DSN tab and click Add. Select TSM ODBC Driver and Click Finish: The configuration Panel is opened.Figure 7-3 ITSM ODBC data source configuration panel Fill the Data source name with ITSMSRC, fill the Administrator name with the IBM Tivoli Storage Manager administrator name and the TCP/IP with the full qualified hostname of IBM Tivoli Storage Manager server (the name must match with the SERVERHLADDRESS value configured on the IBM Tivoli Storage Manager), leave other parameters as default, and then click OK.If you plan on using the IBM Tivoli Storage Manager Warehouse EnablementPack with multiple data sources, then you may need to follow this process toconfigure the ODBC data sources: Create a dummy DB2 database on the control server and create as many ODBC connections to the dummy database as you have IBM Tivoli Storage Manager servers. Create the ODBC connections as if you were creating the IBM Tivoli Storage Manager ODBC connections (same name). Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 303
    • Do not configure any IBM Tivoli Storage Manager ODBC connections to the real IBM Tivoli Storage Manager servers’ databases. Proceed with the installation of the IBM Tivoli Storage Manager warehouse enablement pack as described in the next section. When the installer prompts for the IBM Tivoli Storage Manager data sources, configure the data sources to point to the ODBC connections to the dummy DB2 database. After the IBM Tivoli Storage Manager warehouse enablement pack installation has completed, replace the DB2 data sources with IBM Storage Manager ODBC data sources with the same name. Manually update the user name and passwords associated with each source in the DB2 Data Warehouse Center.7.4.3 Backing up the TWH databases Before installing any warehouse enablement pack (WEP), it is highly recommended to back up all the warehouse databases to ensure that you can return to a known valid state if you encounter an unrecoverable error during installation. In our case study, we have to back up the three databases, TWH_CDW, TWH_MART, and TWH_MD (if you use different configurations, refer to Chapter 2, “Planning for Tivoli Data Warehouse 1.2” on page 27 for warehouse database naming conventions). In order to do so, we created a local directory structure on our DB2 server to store our backups by creating the following three directories: C:tdwbackupspre-itsmtwh_cdw C:tdwbackupspre-itsmtwh_mart C:tdwbackupspre-itsmtwh_md After creating these directories, we completed the following steps on the DB2 server from a command window: 1. Stop and restart DB2: db2stop force db2start 2. Back up the TWH_CDW database: db2 backup db TWH_CDW to C:tedwbackupspre-itsmtwh_cdw 3. Back up the TWH_MART database: db2 backup db TWH_MART to C:tedwbackupspre-itsmtwh_mart304 Implementing Tivoli Data Warehouse 1.2
    • 4. Back up the TWH_MD database: db2 backup db TWH_MD to C:tedwbackupspre-itsmtwh_md 5. Restart Warehouse logger and Warehouse server services on the control server.7.4.4 IBM Tivoli Storage Manager WEP installation To install the AMX warehouse pack, we completed the following steps on the control server: 1. Click Start -> Programs -> TDW -> Install a Warehouse Pack, this starts the warehouse pack installation wizard (see Figure 7-4 on page 305). In the Welcome window, click Next. Figure 7-4 Install a Warehouse Pack window 2. The following window (Figure 7-5) shows the location of the Tivoli common logging directory which will contain all TDW log files. In our installation we use the default location C:Program FilesibmTivolicommon. Click Next. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 305
    • Figure 7-5 Tivoli Common Logging Directory window 3. In the window shown in Figure 7-6, click Add to add the IBM Tivoli Storage Manager Warehouse Enablement Pack. Figure 7-6 Add Warehouse Pack window306 Implementing Tivoli Data Warehouse 1.2
    • 4. In the Location of installation properties file window, as shown Figure 7-7, specify the location of the IBM Tivoli Storage Manager warehouse enablement pack installation properties file, twh_install_props.cfg. You can find this file in the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack media, under the tedw_apps_etlAMN directory. Click Next.Figure 7-7 Location of installation properties Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 307
    • 5. The installer will prompt for the data mart database to be used by the processes of the IBM Tivoli Storage Manager Warehouse Enablement Pack. It also prompts for the remote agent site that will run the ETL2 processes, as shown in Figure 7-8. On our case study scenario, the data mart is on TDW009 and we will be using the default agent site on the control server on TDW003. Figure 7-8 Data mart and remote agent site settings308 Implementing Tivoli Data Warehouse 1.2
    • 6. The installer will prompt for the central data warehouse database to be used by the processes of the IBM Tivoli Storage Manager Warehouse Enablement Pack. It also prompts for the remote agent site that will run the ETL1 processes, as shown in Figure 7-9. In our case study scenario, the central data warehouse is on TDW004 and we will be using the default agent site on the control server on TDW003. At this time you can opt to perform the scheduling settings for the ETL1 processes. In case you opt to do so, the installation process will schedule the processes to run at the specified time and promote the processes to Production status. You can also opt not to schedule the ETLs at installation time and perform the foregoing tasks manually. The installer will also define the user authority for each one of the warehouse sources and targets processes.Figure 7-9 Central data warehouse and remote agent site settings Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 309
    • 7. The installer will investigate the existence of an IBM Tivoli Storage Manager ODBC connection named ITSMSRC. Click Edit to specify its settings, as shown in Figure 7-10.Figure 7-10 Editing IBM Tivoli Storage Manager ODBC settings 8. As shown in Figure 7-11, set the User ID of IBM Tivoli Storage Manager Administrator and Password. The Administrator name should be the same as the IBM Tivoli Storage Manager ODBC system data source you created prior to the installation. Click Next. The installer will test the ODBC connection and return to ODBC Data Source Properties Panel. Click Next again.310 Implementing Tivoli Data Warehouse 1.2
    • Figure 7-11 ITSM ODBC Settings9. The installation menu window (Figure 7-12) now lists the IBM Tivoli Storage Manager Warehouse Enablement Pack. Select it and click Next to continue.Figure 7-12 Installation menu window Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 311
    • 10.Click Install in the summary window (Figure 7-13) to start the IBM Tivoli Storage Manager Warehouse Enablement Pack installation. Figure 7-13 Installation summary window 11.View the progress of installation through the messages that are shown in windows until its completion. The final installation window (Figure 7-14) contains either a successful completion notice or messages describing problems. Make sure the window does not list any warnings or errors, and then click Next. If warnings are listed, check the logs to ensure that the warnings can safely be ignored. Click Finish to exit the wizard.312 Implementing Tivoli Data Warehouse 1.2
    • Figure 7-14 Installation Progress and Completion window7.4.5 Defining the authority to the warehouse sources and targets Tivoli Data Warehouse 1.2 has simplified the installation process of warehouse enablement packs to the extent that there is no need to define the user authority to the warehouse sources and targets of warehouse enablement packs’ processes developed for Version 1.2. The installer sets the correct user ID and password information required for each source and target. However, we recommend for you to certify that the processes have proper authorization by following the steps defined in the previous chapters of this redbook describing warehouse enablement packs developed for Version 1.1. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 313
    • Table 7-2 provides a list of the warehouse sources and targets whose authority should be checked. Table 7-2 ITSM WEP Warehouse Object Names Warehouse targets ANR_TWH_CDW_Target ANR_TWH_MART_Target Warehouse sources ANR_ITSMRC_Source ANR_TWH_CDW_Source ANR_TWH_MART_Source Subject area ANR_IBM_Tivoli_Storage_Manager_v1.1.0_Subject_Area Processes ANR_C05_ETL1_Process ANR_C10_EXPServer_Process ANR_M05_ETLS2_Process Steps ANR_C05_S010_Preextract ANR_C05_S020_Extract ANR_C05_S030_Transform ANR_C05_S040_SRVR_LOAD ANR_C05_S050_STGP_LOAD ANR_C05_S060_NODE_LOAD ANR_C05_S070_FILESP_LOAD ANR_C05_S080_OCCUP_LOAD ANR_C10_S010_EXPServer ANR_m05_s010_spbuildmart ANR_m05_s020_sprollup ANR_m05_s030_spupdatestats ANR_m05_s040_fspuildmart ANR_m05_s050_fsrollup ANR_m05_s060_fsupdatestats7.5 IBM Tivoli Storage Manager ETL processes Figure 7-2 shows that the IBM Tivoli Storage Manager Warehouse Enablement Pack has the following processes: ANR_C05_ETL1_Process ANR_C10_EXPServer_Process ANR_M05_ETLS2_Process314 Implementing Tivoli Data Warehouse 1.2
    • 7.5.1 ANR_C05_ETL1_Process This process is the main central data warehouse ETL. It extracts information from an IBM Tivoli Storage Manager server and loads it into the central data warehouse database. Run this process once each 24 hour period to collect information about the previous day’s processing. You should choose a time of low activity on the ITSM server in which to run this process. For example, you might choose to schedule the process in the early morning, after your nightly backups have completed. but before the daily server maintenance processes begin. Figure 7-15 illustrates the Process Model of ANR_C05_ETL1_Process. In order to obtain the content of Figure 7-15 on page 316, perform the following steps: Important: Do not modify or make any changes on the Process Model. If you are prompted to Save, click No. 1. Start the IBM DB2 Control Center utility by selecting Start -> Programs -> IBM DB2 -> Control Center. 2. On the IBM DB2 Control Center utility, start the IBM DB2 Data Warehouse Center utility by selecting Tools -> Data Warehouse Center. The Data Warehouse Center logon windows appears. 3. Log into the IBM DB2 Data Warehouse Center utility using the local DB2 administrator user ID, in our case, db2admin. 4. In the Data Warehouse Center Window, expand Subject Areas -> Processes and double-click ANR_ITSM_C05_ETL1_Process. Similar procedures can be followed for the other processes, ANR_C10_EXPServer_Process and ANR_M05_ETLS2_Process. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 315
    • Figure 7-15 Sample of Process Model ANR_C05_ETL1_Process The process ANR_C05_ETL1_Process has the following steps. ANR_c05_s010_preextract This step prepares for the subsequent extraction step by creating staging tables in the central data warehouse database. It is a separate step so that ANR_C05_S020_Extract may be run multiple times to extract information from multiple IBM Tivoli Storage Manager servers.316 Implementing Tivoli Data Warehouse 1.2
    • ANR_c05_s020_extractThis step extracts information from an IBM Tivoli Storage Manager server andstores it in staging tables in the central data warehouse. Typically, there is aone-to-one correspondence between the table from which data is extractedand the staging table the central data warehouse. For example, informationextracted from the IBM Tivoli Storage Manager server’s STATUS table isstored in a table called ANR.STG_STATUS in the central data warehousedatabase.ANR_c05_s030_transformThis step transforms some of the data in the staging tables created byANR_c05_s020_extract for use by subsequent steps. In particular, this stepanalyzes the information stored in the server’s SERVER_HLA field in theSTATUS table to determine if it is a fully qualified TCP/IP host name, aTCPI/IP address, or some other value. A new staging table,ANR.STG_SERVER is created by this step.ANR_c05_s040_srvr_loadThis step loads information about the IBM Tivoli Storage Manager server andthe computer on which is running into the central data warehouse.Information about the server and its host computer is obtained from twostaging tables: ANR.STG_STATUS and ANR.STG_SERVER.This step inserts or updates, as necessary, component entries in the centraldata warehouse database. It defines a parent-child relationship between thehost computer and the server. Finally, it inserts or updates any attributes thatare associated with the components.ANR_c05_s050_stgp_loadThis step loads information about the server’s storage pools into the centraldata warehouse. Information primarily comes from one staging table,ANR.STG_STGP.This step inserts or updates, as necessary, component entries in the centraldata warehouse database. It defines a parent-child relationship between theserver and storage pools. Storage pools that were removed from the serverare detected and expired from the central data warehouse database. Thisstep inserts or updates any attributes that are associated with thecomponents and loads new measurements collected since the last time theprocess ran.ANR_c05_s060_node_loadThis step loads information into the central data warehouse about the clientnodes that are registered to the IBM Tivoli Storage Manager server.Information primarily comes from one staging table, ANR.STG_NODE. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 317
    • This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between the server and the nodes. Nodes that were removed from the server are detected and expired from the central data warehouse database. This step inserts or updates any attributes that are associated with the components and loads new measurements collected since the last time the process ran. ANR_c05_s070_filesp_load This step loads information about the client nodes’ file spaces into the central data warehouse. Information primarily comes from one staging table, ANR.STG_FILESP. This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between the file space and its owning node. File spaces that were removed from the server are detected and expired from the central data warehouse database. This step inserts or updates any attributes that are associated with the components and loads new measurements collected since the last time the process ran. ANR_c05_s080_occup_load This step loads client occupancy information into the central data warehouse. Occupancy information describes the amount of server storage that is being used for a given node’s file spaces. It is broken down by node, file space, storage pool and data type (backup, archive, or space management). Information primarily comes from one staging table, ANR.STG_OCCUP. The IBM Tivoli Storage Manager warehouse enablement pack uses an abstract component type to represent occupancy information. The component type, ANR_FS_OCCUPANCY, represents the server storage being used by a file space in a particular storage pool for a given type of file. It is a child in a parent-child relationship with the file space. ANR_FS_OCCUPANCY components are named after the storage pool that owns the storage and the type of storage being described. Names are generated by concatenating the storage pool name with the storage type and separating the two with a colon. This step inserts or updates, as necessary, component entries in the central data warehouse database. It defines a parent-child relationship between a file space and the ANR_FS_OCCUPANCY component. Occupancy components that were removed from the server (due to storage pool migration, for example) are detected and expired from the central data warehouse database. This step inserts or updates any attributes that are associated with the components and loads new measurements collected since the last time the process ran. If invalid data is detected, the ETL process creates the exception table ANR.EXCEPT_SRVR.318 Implementing Tivoli Data Warehouse 1.2
    • 7.5.2 ANR_C10_EXPServer_Process This process is used to expire information about IBM Tivoli Storage Manager servers and their subcomponents from the central data warehouse. Run this process manually whenever information about a specific IBM Tivoli Storage Manager server must be expired from the central data warehouse. Do not schedule this process to run automatically, as it will expire data from the central data warehouse that you may wish to keep. The ANR_C10_EXPServer_Process process expires servers based on the contents of table ANR.EXPServer_List. ANR.EXPServer_List contains a single column, SERVER_NAME, which names the server to be expired. Multiple servers may be expired by inserting a row for each server into the ANR.EXPServer_List table. If ANR.EXPServer_List contains no rows, then ANR_C10_EXPServer_Process will not expire any servers. This process has the following step: ANR_c10_s010_expserver This step expires each of the servers listed in the ANR.EXPServer_List table. The servers and their attributes are marked as expired. Then all of the servers’ subcomponents and their attributes are expired, as are relationships between the components. After all components related to a given server have been expired, the entry for that server is deleted from the ANR.EXPServer_List table.7.5.3 ANR_M05_ETL2_Process This process loads data from the central data warehouse into the Storage Pool Occupancy Data Mart and the Filespace Occupancy Data Mart. Run this process once very 24 hours to load new data into the Storage Pool Occupancy and Filespace Occupancy data marts. This process should be run only after the central data warehouse process ANR_C05_ETL1_Process has completed. This process has the following steps: ANR_m05_s010_spbuildmart This step loads data from the central data warehouse into the Hourly Storage Pool star schema in the Storage Pool Occupancy Data Mart. ANR_m05_s020_sprollup This step aggregates the hourly data loaded by the ANR_m05_s010_spbuildmart step and loads it into Daily, Weekly, Monthly, Quarterly and Yearly star schemas. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 319
    • ANR_m05_s030_spupdatestats This step updates statistics about fact tables loaded in the previous steps. The statistics are used by reports to determine which set of facts are the most recent for any given server. ANR_m05_s040_fsbuildmart This step loads data from the central data warehouse into the Hourly Filespace star schema in the Filespace Occupancy Data Mart. ANR_m05_s050_fsrollup This step aggregates the hourly data loaded by the ANR_m05_s040_fsbuildmart step and loads it into Daily, Weekly, Monthly, Quarterly and Yearly star schemas. ANR_m05_s060_fsupdatestats This step updates statistics about fact tables loaded in the previous steps. The statistics are used by reports to determine which set of facts are the most recent for any given server.7.6 Testing, scheduling, and promoting the ETLs Tivoli Data Warehouse 1.2 has simplified the installation process of warehouse enablement packs to the extent that the scheduling and promoting of ETLs developed for Version 1.2 are set during the warehouse enablement pack installation time. You can still demote the ETLs to Test mode and test the execution of the processes in order to verify accuracy. The process of promoting and demoting of ETLs is similar to the steps defined in previous chapters of this redbook describing warehouse enablement packs developed for Version 1.1.7.6.1 ETL data collection verification Once the processes run at scheduled time, or run manually (in test mode), you can verify data collection on the data mart database, as follows: 1. On the control server, start the DB2 Control Center tool: Click Start -> Programs -> DB2 -> Control Center. See the example in Figure 7-16. 2. Type the User ID and password, when prompted, in our case it was db2inst1. 3. Double-click Tables, on the right hand pane, all tables are displayed. Select Table D_NODE, right-click, and select Sample Contents. You will see a window displaying the content. This verifies your ETL execution as successful.320 Implementing Tivoli Data Warehouse 1.2
    • Figure 7-16 Sample Content of Table D_NODE Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 321
    • 7.7 Reporting In this section we show how to set up, configure, and use some of the reports provided by the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack. Note: As described in Chapter 3, “Getting Tivoli Data Warehouse 1.2 up and running” on page 71, an ODBC connection to the data mart database needs to be defined on the Crystal Enterprise server before we can work with the reports. Please refer to that chapter for details.7.7.1 Available reports Here is a list of predefined reports shipped with the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack: How Has Clients Use of Server Storage Changed Over Time? How Has Clients Use of Server Storage Changed Over Time by Platform? How Has My Server Storage Space Utilization Changed Over Time? Which Clients are Using the Most Server Storage/ Crystal Enterprise Professional Version 9 for Tivoli has reduced configuration options in comparison to a full Crystal license. If the reports shipped with IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack do not match your needs and you want to develop additional reports, you have to upgrade your Crystal Enterprise installation.7.7.2 Accessing the Crystal ePortfolio All reports provided by Tivoli Data Warehouse 1.2 must be accessed using the ePortfolio feature of the Crystal Enterprise. The ePortfolio can be accessed from the Crystal Enterprise launchpad. In order to access the Crystal Enterprise launchpad, start an Internet browser and open the following URL: http://<hostname>/crystal/enterprise9 Here, <hostname> represents the hostname of the Crystal Enterprise report server, as shown in Figure 7-17.322 Implementing Tivoli Data Warehouse 1.2
    • Figure 7-17 Crystal Enterprise - LaunchpadIn this section, we concentrate on viewing IBM Tivoli Storage Manager 5.2reports and we do not explain the configuration of Crystal Enterprise to its fullextent. For details on configuration and administration tasks, refer to thefollowing manuals shipped with the product: Crystal Enterprise 9 Installation Guide Crystal Enterprise 9 Administrator’s Guide Crystal Enterprise 9 Getting Started Guide Crystal Enterprise 9 ePortfolio User’s Guide Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 323
    • From the Crystal Enterprise Launchpad, proceed by selecting the ePortfolio link, which will bring you to the window shown in Figure 7-18. In the top bar, you can see that we are authorized as user guest. By default, the guest user has no access to the NetView reports, as indicated by the words No Folders on the left side of the window. Figure 7-18 Crystal Enterprise 9 - ePortfolio The installation process of the first warehouse enablement pack on the Tivoli Data Warehouse environment creates a user ID on the Crystal Enterprise environment named Tivoli. This user ID is to be used to access the reports provided by any IBM Tivoli software.324 Implementing Tivoli Data Warehouse 1.2
    • To log in as the Tivoli user ID, select the Log On button in the upper right cornerof the ePortfolio window in Figure 7-18 on page 324. The Log On window asshown in Figure 7-19 is presented. The Tivoli user ID has no password bydefault. We use the Enterprise authentication method as we have specifiedduring the Crystal Enterprise installation.Figure 7-19 Crystal Enterprise 9 - Log in Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 325
    • After entering the required data, select Log On to proceed. Now we are back at the ePortfolio window in Figure 7-20, but now with user Tivoli authority. Instead of No Folders in the guest users ePortfolio window in Figure 7-18, there is now a link visible with the name IBM Tivoli Storage Manager, Warehouse Enablement Pack in the Tivoli user ePortfolio window in Figure 7-20. Figure 7-20 Crystal Enterprise 9 - Folders326 Implementing Tivoli Data Warehouse 1.2
    • We follow this link by selecting IBM Tivoli Storage Manager, WarehouseEnablement Pack and proceed to the IBM Tivoli Storage Manager 5.2 reportsas shown in Figure 7-21. All reports provided by the IBM Tivoli Storage Manager5.2 warehouse enablement pack are listed there.Figure 7-21 Crystal Enterprise 9 - available reports for ITSM Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 327
    • In order to obtain the reports, select the desired report, for example, How Has Clients use of Server Storage Changed Over Time?, and select Schedule, as shown in Figure 7-22. Figure 7-22 Scheduling Operating System Busiest System report328 Implementing Tivoli Data Warehouse 1.2
    • The schedule report panel starts. In order to run the reports at this time, selectNow under the Run Report option. As this report requires additional parameters,such as time frame, select Parameters under the Customize your Optionsoption, as shown in Figure 7-23.Figure 7-23 Crystal Enterprise 9 - parameters Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 329
    • Figure 7-24 shows the selection of parameters for the report. Select Schedule when ready to run the report. Figure 7-24 Crystal Enterprise 9 - Parameters for the report Because we selected to run this report now, the report is scheduled immediately and the reports history window is opened. The just-scheduled report will run and its initial status is set to Pending. Note: The History window is not updated automatically. Press the Refresh button to view the current state.330 Implementing Tivoli Data Warehouse 1.2
    • To view successful generated reports from the history window, click the link Instance Time in the left column of the table to view the associated report. This report is shown in Figure 7-25.Figure 7-25 How Has Clients use of Server Storage Changed Over Time? Next we show some more examples of reports provided by the IBM Tivoli Storage Manager 5.2 Warehouse Enablement Pack. Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 331
    • Figure 7-26 displays a sample report of a single server called AMORRIS on How Has Clients Use of Server Storage Changed Over Time? It shows how to drill down to a client.Figure 7-26 How Has Clients Use of Server Storage Changed Over Time?332 Implementing Tivoli Data Warehouse 1.2
    • Figure 7-27 displays a sample report of all servers on How Has Clients Use of Server Storage Changed Over Time by Platform?Figure 7-27 How Has Clients Use of Server Storage Changed by Platform? Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 333
    • Figure 7-28 displays a sample report of all servers on How Has My Server Storage Space Utilization Changed Over Time?Figure 7-28 How Has My Server Storage Space Utilization Changed Over Time?334 Implementing Tivoli Data Warehouse 1.2
    • Figure 7-29 displays a sample report of all servers on Which Clients Are Using the Most Server Storage?.Figure 7-29 Which Clients are Using the Most Server Storage? Chapter 7. IBM Tivoli Storage Manager Warehouse Enablement Pack 335
    • 336 Implementing Tivoli Data Warehouse 1.2
    • Part 3Part 3 Appendixes© Copyright IBM Corp. 2004. All rights reserved. 337
    • 338 Implementing Tivoli Data Warehouse 1.2
    • A Appendix A. IBM DB2 UDB administration for other relational DBAs DB2 is the main RDBMS product from IBM. It is a top level RDBMS competing with Oracle, IBM Informix, and Microsoft SQL Server in the market. Each of these products has its advantages and disadvantages; MS SQL Server®, for example, has a huge advantage in integrating enterprise running heterogeneous applications and Data Sources. However, MS SQL Server has the disadvantage of executing only on Windows, while all the competitors are cross platform capable. Oracle® has the advantage of supporting parallel servers using its RAC, but the disadvantage of needing highly specialized professionals at a high cost and demanding considerable time to accomplish even trivial tasks. Informix® is easy to use and to maintain, but still cannot perform online reorganization. IBM DB2 can provide all of these features and even more. It can connect to heterogeneous data sources through the IBM DB2 DataJoiner®; High Availability features using Partitioned Databases or Parallel Features using Sysplex Technology in z/OS, without the overhead presented by Oracle RAC, and can perform data reorganization online with little or no downtime. Also, DB2 is highly scalable, providing data management from hand-held computers with the Everyplace® edition up to multi-terabyte Enterprise Servers with UDB ESE (old EEE), with the same easy-to-use framework.© Copyright IBM Corp. 2004. All rights reserved. 339
    • Common DBA tasks Note: This appendix does not aim to compare the features of the different RDBMSs. It was written to help DBAs working with these RDBMS to perform the same tasks when working with IBM DB2. Because some configurations use pure UNIX, without graphical extensions, we do not show the graphic tools that you can use to perform tasks in an easier way, except when we think this is necessary for clarity. No matter what RDBMS you use, some tasks are common: Installing RDBMS Creating databases Setting up security Allocating space Optimizing performance Installing RDBMS can be accomplished using setup instructions found in the manuals, and usually does not change in general from RDBMS to RDBMS, except in regard to parameters specific to that RDBMS. The differences between the DB2 and other RDBMS methods for accomplishing the tasks listed above are presented in the following sections of this appendix. Our discussion assumes that we are working on a UNIX platform.Creating databases After installing the RDBMS software, the next step is to create an instance or/and database to hold the data structure. The concept of database may vary among RDBMSs. For example, Oracle uses the term DATABASE to refer to the logical grouping of all objects storing the necessary data for an application as well as its internal catalog, and INSTANCE to refers to the memory, processes, and software pieces necessary to manage and access this data. The same naming convention is used by DB2, while Informix, MSSQL, and Sybase ASE use a slightly different convention. Informix, MSSQL, and Sybase all use the term database to refer to the logical grouping of objects that can be accessed by an application. This database is separated from its catalog, which is a database itself. The grouping of architectural objects, such as memory, disk, processes, etc., in each of these RDBMSs is called a server.340 Implementing Tivoli Data Warehouse 1.2
    • In general, an instance/server is configured during the install process, leaving to you the tasks of creating the database to hold data and setting up the protocols for providing access to the instance/server. The process to create a database, without using the graphical tools, is the same for each RDBMS. 1. Connect to the instance 2. Issue the CREATE DATABASE command, providing the location for datafiles, database size, etc.Creating databases in IBM DB2 Connect to the local machine as superuser (root), then set the DB2INSTANCE to the instance created by db2icrt at install time, which was named by you or the default db2inst1. In order to create a database named TIVDW, for example, Issue the following command: db2 CREATE DATABASE TIVDW ON /DB2/tivdwda1/tivdw USING CODESET 1252 TERRITORY US COLLATE USING SYSTEM USER TABLESPACE MANAGED BY SYSTEM USING (/DB2/tivdwda1/tivdw/d1sms) EXTENTSIZE 16 PREFETCHSIZE 16 CATALOG TABLESPACE MANAGED BY SYSTEM USING (/DB2/tivdwda1/tivdw/SYSCATSPACE) EXTENTSIZE 8 PREFETCHSIZE 8 TEMPORARY TABLESPACE MANAGED BY SYSTEM USING (/DB2/tivdwda1/tivdw/TEMPSPACE) EXTENTSIZE 32 PREFETCHSIZE 32; The foregoing command shows many more parameters and attributes than the minimum required to create a database in DB2; this is done for illustration purposes only. The command will create our TIVDW database. After that command is completed, you can start to create the needed tables, Storage Polls, etc. How we could create the same database on a heterogeneous RDBMS? Lets start with Oracle:Creating databases in Oracle Some of the options available to create a database in DB2 are not available in Oracle, and vice versa. The Oracle CREATE DATABASE command is far more complex than in IBM DB2: With Oracle you must specify a large number of parameters to get your database created. To create the same TIVDW Database in Oracle, perform the following steps: 1. Connect to sqlplus, using the SYS or INTERNAL account. 2. Issue the command: CREATE DATABASE tivdw CONTROLFILE REUSE LOGFILE Appendix A. IBM DB2 UDB administration for other relational DBAs 341
    • GROUP 1 (/oracle/tivdw/logs/log1.log, /oracle/tivdw/logsmirror/log1.log) SIZE 50M, GROUP 2 (/oracle/tivdw/logs/log2.log, /oracle/tivdw/logsmirror/log2.log) SIZE 50M MAXLOGFILES 50 MAXLOGHISTORY 100 MAXDATAFILES 100 MAXINSTANCES 2 ARCHIVELOG CHARACTER SET UTF8 NATIONAL CHARACTER SET AL16UTF16 DATAFILE /oracle/tivdw/data/df1.dbf SIZE 50M AUTOEXTEND ON, /oracle/tivdw/logs/df2.dbf SIZE 50M AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED DEFAULT TEMPORARY TABLESPACE temp_ts EXTENT MANAGEMENT LOCAL UNIFORM 32K SIZE 32K UNDO TABLESPACE undo_ts; Notice the following differences between DB2 and Oracle: 1. DB2 does not specify LOG file destination, because its default location is the SQLOGDIR under the path you specified in the create database <dbname> on clause. 2. You can initialize the CATALOG, TEMP, AND USER tablespaces for DB2 by specifying their definitions inside the Create Database command. 3. There are two kinds of tablespaces in DB2, just as in Oracle: System Managed Tablespace and Database Managed Tablespace. See the following sections for a brief description of the differences. For the PREFETCHSIZE and EXTENTSIZE, refer to the section on Creating Tablespaces.Creating databases in Sybase Let us create the same database on Sybase SQL Server or Sybase ASE (Adaptive Server), as it is now called. Every Sybase installation has at least four databases: master, model, tempdb, and sybsystemprocs. Each of these databases has a role in keeping Sybase ASE up and running. The master database is the main database in a Sybase ASE installation, because it contains all the information for databases you create, logins you add to Sybase, SQL Server configuration and options, etc. Mapping to DB2, this is the System Catalog.342 Implementing Tivoli Data Warehouse 1.2
    • To create a database, you must connect to the server, using the isql program, and issue the following tasks: 1. Initialize disk device to be used by your database. This is done by mapping a disk or file to a logical device in the Sybase subsystem, by using the DISK INIT command: DISK INIT NAME = "tvdw_data", PHYSNAME = "/dev/tvdw_data", VDEVNO = 2, SIZE = 25600 DISK INIT NAME = "tvdw_log", PHYSNAME = "/dev/tvdw_log", VDEVNO = 2, SIZE = 25600 These commands would initialize a device called tvdw_data, with 50MB to be used for data storage by our Tivoli Database and a device named tvdw_log, with the same size for transaction log. Information for this device would be inserted in sysdevices table in the master database. 2. After have initialized the device, you could issue the CREATE DATABASE command to create the database for use by Tivoli Data Warehouse, as in the following example: CREATE DATABASE tivdwdb ON tvdw_data = 50, LOG ON tvdw_log=50 Comparing the Sybase syntax to DB2 syntax is fruitless. All of the options in the DB2 syntax are missing in the Sybase syntax, except by database name. Also in DB2, you do not need to pre-initialize a database device. Sybase collation, for example, is server-wide, while DB2 collation is specified at database creation time.Managing space DB2 organizes datafiles in objects called tablespaces, just as Oracle does. But Oracle datafiles are called containers in DB2. Sybase and MSSQL do not have a tablespace concept; Informix uses DBSpace for organizing physical objects.DB2 space management To create a tablespace in DB2, you use the CREATE TABLESPACE command. It has a number of parameters that allows you to control the behavior of this tablespace, how space will be allocated, managed, and accessed. Weve used some of these parameters while creating the initial database. Appendix A. IBM DB2 UDB administration for other relational DBAs 343
    • Consider the syntax: db2 CREATE TABLESPACE TIVTS MANAGED BY DATABASE USING(FILE /DB2/tivdwda1/tivdw/d2dms 1G) EXTENTSIZE 16 PREFETCHSIZE 16 We are creating a tablespace with one container of 1GB (specified as 1000M). One of the options you can use, and which we left out, is PAGESIZE. This parameter can be used when you want to use a non-default pagesize, just in the same way Oracle 9i does with non-default block size caches. To use this non-default PAGESIZE in DB2, you have to create a new BUFFERPOOL configured to use the pagesize you want to, as in the following sample: db2 CREATE BUFFERPOOL TIVBP SIZE 1000 PAGESIZE 16K. Then you would modify our CREATE TABLESPACE example to use this bufferpool and the custom pagesize, in the following way: db2 CREATE TABLESPACE TIVTS PAGESIZE 16K MANAGED BY DATABASE USING(FILE /DB2/tivdwda1/tivdw/d2dms 1G) EXTENTSIZE 16 PREFETCHSIZE 16 BUFFERPOOL TIVBP When creating a tablespace in DB2, here are some considerations: 1. You should specify the space management system for that specific tablespace, except when you want to use SYSTEM MANAGED SPACE, the default. We are using DATABASE MANAGED SPACE because we want to have full control over space allocation. The key difference between these two types of tablespaces is that SMS has its space allocation managed by operating system, while DMS has its space allocation managed by the database manager. When deciding what kind of tablespace to use, keep in mind that SMT cannot have containers added to it after it is created, and that you need to pay attention to OS limitations and allow enough disk space for the tablespace to grow automatically. For example, if your OS has a limit of 2 GB for files, and you need a tablespace of 128 GB, you have to create the tablespace with 64 containers. This is true for the initial CREATE DATABASE command when you specify one of the tablespace creation clauses and for the CREATE TABLESPACE command. 2. EXTENTSIZE is used to set the number of pages DB2 will write before skipping to the next container. This parameter is important for performance because of the way that DB2 stores data, balancing the writing of data between all containers in the tablespace in a cycling manner. 3. PREFETCHSIZE specifies the number of pages that will be read in read-ahead operations. You can use this parameter to reduce I/O in queries. 4. To be able to recover dropped tables, use the DROPPED TABLE RECOVERY option in the CREATE TABLESPACE or in the ALTER TABLESPACE clauses.344 Implementing Tivoli Data Warehouse 1.2
    • After creating the tablespace, you can use the ALTER TABLESPACE clause to modify tablespace options. You can change every option except PAGESIZE and EXTENTSIZE, so it is a good idea to think very carefully about these values. Use ALTER TABLESPACE to add, resize or extend a DMS tablespaces container. For example, the following command will add a container with 1GB: db2 ALTER TABLESPACE TIVTS ADD(FILE /DB2/tivdwda1/tivdw/d3dms 1G) The following command will expand that container by 200 MB: db2 ALTER TABLESPACE TIVTS EXPAND(FILE /DB2/tivdwda1/tivdw/d3dms 200M)Oracle space management To create the same tablespace as discussed above, in Oracle, we could use a variation of the following command: CREATE TABLESPACE TIVTS DATAFILE /oracle/tivdw/data/df1.dbf SIZE 200M MINIMUM EXTENT 500K DEFAULT STORAGE (INITIAL 128K NEXT 128K) LOGGING; Or we could use a locally managed tablespace: CREATE TABLESPACE TIVTS DATAFILE /oracle/tivdw/data/df1.dbf SIZE 200M EXTENT MANAGEMENT LOCAL UNIFORM SIZE 128K; To add a new datafile to an Oracle tablespace, we use the ALTER TABLESPACE command: ALTER TABLESPACE TIVTS ADD DATAFILE /oracle/tivdw/data/df2.dbf SIZE 200M AUTOEXTEND ON NEXT 1M MAXSIZE 1G; And to resize it, we use the ALTER DATABASE command: ALTER DATABASE DATAFILE /oracle/tivdw/data/df2.dbf RESIZE 300M; Appendix A. IBM DB2 UDB administration for other relational DBAs 345
    • Sybase space management Sybase has no tablespace concept. Space is allocated directly to databases, using DISK INIT and ALTER DATABASE commands. For example: You use DISK INIT to map a disk area to a new database device, just as at database create time: DISK INIT NAME = "tvdw_data1", PHYSNAME = "/dev/tvdw_data1", VDEVNO = 2, SIZE = 25600 Then you use the ALTER DATABASE command to increase the database size, using the new device: ALTER DATABASE tivdwdb ON tvdw_data1 = 50Creating objects in the database After you create the database, and allocate space for its objects, it is time to create the objects themselves. Objects include tables, indexes, views and stored procedures. You issue SQL commands against the DB to create all of these objects. The SQL standard establishes these SQL commands.Creating tables in DB2 In its simplest form, you create tables in DB2 using the following command (this will create a table named DEPTO): CREATE TABLE DEPTO (DEPTNO CHAR(3) NOT NULL, DEPTNAME VARCHAR(36) NOT NULL, MGRNO CHAR(6), ADMRDEPT CHAR(3) NOT NULL, PRIMARY KEY(DEPTNO)) IN DEPARTX INDEX IN DEPARTIDX346 Implementing Tivoli Data Warehouse 1.2
    • Creating tables in Oracle To create a table in an Oracle database, issue the following command (this is also going to create a table named DEPTO): CREATE TABLE DEPTO (DEPTNO CHAR(3) NOT NULL, DEPTNAME VARCHAR(36) NOT NULL, MGRNO CHAR(6), ADMRDEPT CHAR(3) NOT NULL, PRIMARY KEY(DEPTNO)) PCTFREE 10 PCTUSED 40 TABLESPACE DEPARTX STORAGE (INITIAL 50K NEXT 50K MINEXTENTS 1 MAXEXTENTS 100 FREELISTS 1)Creating tables in Sybase To create a table in a Sybase database, issue the following statements (this is also going to create a table named DEPTO): CREATE TABLE DEPTO (DEPTNO CHAR(3) NOT NULL, DEPTNAME VARCHAR(36) NOT NULL, MGRNO CHAR(6), ADMRDEPT CHAR(3) NOT NULL, PRIMARY KEY(DEPTNO)) ON DPTO_SEGAdditional table control parameters DB2, Oracle, and Sybase have a number of specialized parameters that control the behavior, storage, and memory management for tables. As an example, in DB2 you can define a SUMMARY table to hold a definition from full select using the following clause: CREATE TABLE tbsummary AS (SELECT DEPTO.*, CURRENT TIMESTAMP AS TIMESTAMP FROM DEPTO) DEFINITION ONLY This will create the table, but will not populate it. This is similar to the CREATE TABLE AS Oracle syntax or SELECT INTO clause in Sybase. However, DB2 also implements a single copy syntax that can be used to directly copy table structure: CREATE TABLE tbLikeSamp LIKE DEPTO Appendix A. IBM DB2 UDB administration for other relational DBAs 347
    • This will create a table called tbLikeSamp with the same structure as the DEPTO Table. As with Oracle syntax, the created table will not have referential attributes, constraints, or indexes. Stored procedures are created using CREATE PROCEDURE command, like this: CREATE PROCEDURE PROC_NAME (IN PRM1 INT, OUT prm2 DOUBLE) RESULT SETS 1 LANGUAGE SQL BEGIN <SQL COMMANDS HERE> DECLARE EXIT HANDLER FOR NOT FOUND; END DB2 procedures can be written in external languages like C, Java, COBOL, or any language that allows writing an ActiveX (on Windows) DLL. Note that you can use an exception handler in the procedure, one feature that is not available in Sybase. The Oracle syntax to create a procedure is again similar: CREATE PROCEDURE PROC_NAME (PRM1 IN INT, prm2 OUT DOUBLE) AS BEGIN <SQL COMMANDS HERE> EXCEPTION WHEN NOT FOUND THEN… ; END Note that Oracle also allows a stored procedure to be written in Java or C languages, giving some flexibility. In Sybase, you issue this command: CREATE PROC PROC_NAME @PRM1 INT = "D%", @PRM2 DOUBLE OUTPUT AS Unfortunately, Sybase has no error handling for procedures, and has no flexibility in choosing the language to write the procedure. If you need more power than those offered by Transact-SQL, then you will have to use an extended procedure, which is generally written in languages like C or Pascal, using the Sybase Open Service API. (This can be really painful!)348 Implementing Tivoli Data Warehouse 1.2
    • B Appendix B. Tivoli Data Warehouse 1.2 reference This appendix lists the reports provided by each warehouse enablement pack and their measurement sources.© Copyright IBM Corp. 2004. All rights reserved. 349
    • Report listingAMY : IBM Tivoli Monitoring for Operating Systems, Version 5.1.1, Warehouse Enablement Pack,Version 1.1.0.3Name AVA Frequency Type CodeHealth of a backup server amy Hourly HealthCheckUsage of a Domain Controller amy Hourly HealthCheckMemory Utilization amy Hourly HealthCheckPaging File Utilization amy Hourly HealthCheckGWA : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: Apache HTTP Server, WarehouseEnablement Pack, Version 1.1.0Name AVA Frequency Type CodeApache Health Check Error Report gwa Hourly HealthCheckApache Health Check Performance Report gwa Hourly HealthCheckINV : Tivoli Configuration Manager, Version 4.2.0, Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeDistribution Failures related to package size inv Daily WorstCaseDistribution Status by distribution id inv Daily SummaryDistribution results by file pack and operation inv Daily SummaryDistribution results by file pack, host, operation inv Daily SummarySuccess rate of distributions by distribution id inv Daily WorstCaseSuccess rate of distributions by file pack name inv Daily WorstCaseSuccess Rate of distributions by time inv Daily HealthCheckElapsed distribution time to target inv Daily WorstCaseReceiving distribution time by target inv Daily WorstCaseDistribution transfer rate by file package in kb/s inv Daily WorstCaseFile pack distributions that have the most failure inv Daily WorstCaseDistributions that have the most failures inv Daily WorstCaseOS that have the most distribution failures inv Daily WorstCaseNetworks that have the most distribution failures inv Daily WorstCase350 Implementing Tivoli Data Warehouse 1.2
    • Operations in verify failure state inv Daily WorstCaseHosts that have the most distribution failures inv Daily WorstCaseCTD : IBM Tivoli Monitoring for Databases, Version 5.1.0: IBM DB2,Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeCTD Hourly Percent Connections Used ctd Hourly HealthCheckCTD Hourly Deadlocks Delta Health Report ctd Hourly HealthCheckCTD Hourly Percent Catalog Cache Hits ctd Hourly HealthCheckCTD Hourly Minimum Buffer Pool Hit Ratio ctd Hourly HealthCheckCTD Hourly Maximum Percentage Used of Primary Log ctd Hourly HealthCheckABA : IBM Tivoli Monitoring for Messaging and Collaboration,Version 5.1.0: Lotus Domino, Warehouse Enablement Pack,Version 1.1.0Name AVA Frequency Type CodeBottom Servers By Availability ABA Daily SummaryCalendar Entry ABA Daily WorstCaseDatabase Access ABA Daily WorstCaseDomino Server Usage: Sessions Dropped ABA Daily WorstCaseMail Average Statistics ABA Daily SummaryMail Dead ABA Daily WorstCaseMail Waiting ABA Daily WorstCaseNAB Search ABA Daily WorstCaseNet Echo ABA Daily WorstCaseReplicate Local ABA Daily WorstCaseRound Trip Mail ABA Daily WorstCaseWeb Access ABA Daily WorstCaseGWI : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0:Internet Information Server, Warehouse Enablement Pack,Version 1.1.0Name AVA Frequency Type Code Appendix B. Tivoli Data Warehouse 1.2 reference 351
    • Internet Information Server Performance Report gwi Hourly HealthCheckCTR : IBM Tivoli Monitoring for Databases, Version 5.1.0: Informix,Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeInformix Health Check - 7 Days ctr Daily HealthCheckInformix Thread Activity - 7 Days ctr Daily HealthCheckInformix Disk Utilization - 7 Days ctr Daily HealthCheckInformix Logical Log - 7 Days ctr Daily HealthCheckGWP : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0:iPlanet Web Server, Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeiPlanet Web Server Performance Report gwp Hourly HealthCheckAMW : IBM Tivoli Monitoring, Version 5.1.0, WarehouseEnablement Pack, Version 1.1.0Name AVA Frequency Type CodeWIN_CPU_STATS amw Daily HealthCheckUNIX_CPU_STATS amw Daily HealthCheckNET_STATS amw Daily HealthCheckBUSIEST_SYS amw Daily WorstCaseAMY : IBM Tivoli Monitoring for Operating Systems, Version 5.1.1,Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeNET_STATS amy Hourly HealthCheckUNIX_CPU_STATS amy Hourly HealthCheckWIN_CPU_STATS amy Hourly HealthCheckBUSIEST_SYS amy Hourly WorstCase352 Implementing Tivoli Data Warehouse 1.2
    • HMI : IBM Tivoli Monitoring for Business Integration 5.1.0 :WebSphere® MQ Integrator, Warehouse Enablement Pack,Version 1.1.0Name AVA Frequency Type CodeHMI Metrics for WebSphere MQI Monitor Nodes hmi Daily SummaryHMI Status Down for WebSphere MQI Brokers hmi Daily WorstCaseHMI Status for WebSphere MQI Brokers hmi Daily SummaryHMI Status for WebSphere MQI Config Managers hmi Daily SummaryHMI Status for WebSphere MQI User Name Servers hmi Daily SummaryCTQ : IBM Tivoli Monitoring for Business Integration, Version 5.1.0: WebSphere MQ, Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeCTQ Maximum Down Status for Queue Managers Daily ctq Daily WorstCaseCTQ Maximum Outstanding Messages for Queues Daily ctq Daily WorstCaseCTQ Maximum Running Status for Channels Daily ctq Daily WorstCaseCTQ Availability Status for Queue Managers Daily ctq Daily SummaryCTQ Message and Handle Summary for Queues Daily ctq Daily SummaryCTQ Availability Status for Channels Daily ctq Daily SummaryCTW : IBM Tivoli Monitoring for Databases, Version 5.1.1:Microsoft SQL Server, Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeDaily Filegroup Space Usage Health Check ctw Daily HealthCheckDaily Database Space Used (Filegroup) Summary ctw Daily SummaryDaily Server Availability Extreme Case ctw Daily WorstCaseDaily Replication Agent Latency Health Check ctw Daily HealthCheckDaily Server CPU Usage Extreme Case ctw Daily WorstCaseDaily Server Error Message Count Summary ctw Daily SummaryDaily Database Usage Health Check ctw Daily HealthCheck Appendix B. Tivoli Data Warehouse 1.2 reference 353
    • ANM : IBM Tivoli Netview, Version 7.1.3, Warehouse EnablementPack, Version 1.1.0Name AVA Frequency Type CodeDaily Status Summary By SmartSet anm Daily SummaryNodes With Longest Outage Time In Routers SmartSet anm Hourly WorstCaseNodes With Most Status Changes In Routers SmartSet anm Daily WorstCaseNodes With The Longest Outage Times anm Hourly WorstCaseNodes With The Most Daily Status Changes anm Daily WorstCaseSummary Of Daily Network Status anm Daily HealthCheckSummary Of Total Outage Time By SmartSet anm Daily SummarySummary Of Total Status Changes By SmartSet anm Daily SummaryTotal Daily Status Changes In Monitored Network anm Daily HealthCheckTotal Daily Status Changes In Routers SmartSet anm Daily HealthCheckCTO : IBM Tivoli Monitoring for Databases, Version 5.1.0: Oracle,Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeBufferCache Hit Ratio (Daily) - Extreme Case cto Daily WorstCaseDeadlocks (Daily) - Health Check cto Daily HealthCheckDispatcher contention (Daily) - Summary cto Daily SummaryOracle RDBMS Availability (Daily) - Extreme Case cto Daily WorstCaseTablespace Usage (Daily) - Extreme Case cto Daily WorstCaseHRM : IBM Tivoli Risk Manager, Version 4.1.0, WarehouseEnablement Pack, Version 1.1.0Name AVA Frequency Type CodeEvents by Destination Host - Last 30 Days hrm Daily WorstCaseService Compromise Events - Last 30 Days hrm Daily HealthCheckEvents by Destination Subnetwork - Last 30 Days hrm Daily SummaryEvents by Destination and Category - Last 30 Days hrm Daily WorstCaseAccess/Authentication Events - Last 30 Days hrm Daily HealthCheckInfection Events - Last 30 Days hrm Daily HealthCheck354 Implementing Tivoli Data Warehouse 1.2
    • Policy/Configuration Events - Last 30 Days hrm Daily HealthCheckABH : IBM Tivoli Monitoring for Applications, Version 5.1.0:mySAP.com, Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodemySAP server CPU ranking by rolling 30 days abh Hourly WorstCasemySAP application area by rolling 30 days abh Hourly WorstCasemySAP application area dialogs by rolling 30 days abh Hourly WorstCasemySAP SLA conformance by rolling 30 days abh Hourly WorstCasemySAP application server logins by rolling 30 days abh Hourly WorstCasemySAP program count by rolling 30 days abh Hourly WorstCasemySAP dialog count by rolling 30 days abh Hourly WorstCasemySAP task type CPU ranking by rolling 30 days abh Hourly WorstCasemySAP application server uptime by rolling 30 days abh Hourly WorstCaseGMS : IBM Tivoli Monitoring for Applications, Version 5.1.0 :Siebel, Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeDaily CPU Usage for Siebel Servers gms Daily WorstCaseWeekly CPU Usage for Siebel Servers gms Weekly WorstCaseHourly CPU Usage for Siebel Servers gms Hourly WorstCaseMonthly CPU Usage for Siebel Servers gms Monthly WorstCaseDaily Memory Usage for Siebel Servers gms Daily WorstCaseDaily Memory Usage Health Check for Siebel Servers gms Daily HealthCheckDaily CPU Usage Health Check for Siebel Servers gms Daily HealthCheckDaily Memory Usage Health Check for Siebel Tasks gms Daily HealthCheckDaily CPU Usage Health Check for Siebel Tasks gms Daily HealthCheckSummary of Daily Status for Connection Broker gms Daily SummarySummary of Daily Status for Siebel Gateways gms Daily SummarySummary of Daily Status for Siebel Servers gms Daily SummarySummary of Daily Memory Usage for Siebel Tasks gms Daily SummarySummary of Daily Memory Usage for Siebel Servers gms Daily SummarySummary of Daily CPU Usage for Siebel Tasks gms Daily Summary Appendix B. Tivoli Data Warehouse 1.2 reference 355
    • Summary of Daily CPU Usage for Siebel Servers gms Daily SummaryCOD : Tivoli License Manager v1.1.0, Warehouse EnablementPack, Version 1.1.0Name AVA Frequency Type CodeProducts Installed by Division Summary Report COD Daily SummaryProducts Installed by Division Extreme Report COD Daily WorstCaseAgents 1.1 Installed by Division Summary Report COD Daily SummaryAgents 1.1 Installed by Division Extreme Report COD Daily WorstCaseAgents 1.0 Installed by Division Summary Report COD Daily SummaryAgents 1.0 Installed by Division Extreme Report COD Daily WorstCaseAgents 1.1.1 Installed by Division Summary Report COD Daily SummaryAgents 1.1.1 Installed by Division Extreme Report COD Daily WorstCaseAWS : IBM Tivoli Workload Scheduler, Version 8.2.0, WarehouseEnablement Pack, Version 1.1.0Name AVA Frequency Type CodeJobs with the highest average duration time aws Daily WorstCaseWorkstation with highest CPU utilization aws Daily WorstCaseRun time statistics for all jobs aws Daily HealthCheckRun states statistics for all jobs aws Daily HealthCheckJobs with highest number of unsuccessful runs aws Daily WorstCaseUnsuccessful runs for a workstation aws Daily WorstCaseIZY : IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0:WebSphere Application Server, Warehouse Enablement Pack,Version 1.1.0Name AVA Frequency Type CodeIZY EJB Performance Health izy Weekly HealthCheckIZY EJB Resource Model Summary izy Weekly SummaryIZY EJBs with the Most Hits izy Weekly WorstCaseIZY JVM Runtime Resource Model Summary izy Weekly SummaryIZY Servlet Performance Health izy Hourly HealthCheck356 Implementing Tivoli Data Warehouse 1.2
    • IZY Servlet Session Resource Model Summary izy Weekly SummaryIZY Servlets with the Highest Response Time izy Monthly WorstCaseIZY Servlets with the Most Hits izy Weekly WorstCaseIZY Transaction Manager Resource Model Summary izy Weekly SummaryIZY Web Application Resource Model Summary izy Weekly SummaryGWL : IBM Tivoli Monitoring for Web Infrastructure: WebLogic,Version 5.1.0, Warehouse Enablement Pack, Version 1.1.0Name AVA Frequency Type CodeWebLogic Server Availability (Daily) - EC gwl Daily WorstCaseWebLogic JDBC Connection Pool Statistics (Daily) gwl Daily SummaryWebLogic EJB Transactions (Daily) - HC gwl Daily HealthCheckWebLogic Servlet Performance (Daily) - EC gwl Daily WorstCaseWebLogic JMS Load (Daily) - EC gwl Daily WorstCaseMeasurement sourcesApp Name Msmt Name Parent Source Code Code390 CICS® D04 Tivoli Decision Support for OS/390 (CICS component) DRL390 CICS DRL Tivoli Decision Support for OS/390 Tivoli390 DB2 D05 Tivoli Decision Support for OS/390 (DB2 component) DRL390 DB2 DRL Tivoli Decision Support for OS/390 Tivoli390 DFSMS D08 Tivoli Decision Support for OS/390 (DFSMS component) DRL390 DFSMS DRL Tivoli Decision Support for OS/390 Tivoli390 IMS D03 Tivoli Decision Support for OS/390 (IMS component) DRL390 IMS DRL Tivoli Decision Support for OS/390 Tivoli390 MQS D06 Tivoli Decision Support for OS/390(MQS component) DRL390 MQS DRL Tivoli Decision Support for OS/390 Tivoli390 MVS™ D01 Tivoli Decision Support for OS/390 (MVS component) DRL390 MVS DRL Tivoli Decision Support for OS/390 Tivoli390 NPM D10 Tivoli Decision Support for OS/390 (NPM component) DRL390 NPM DRL Tivoli Decision Support for OS/390 Tivoli Appendix B. Tivoli Data Warehouse 1.2 reference 357
    • 390 NPMIP D11 Tivoli Decision Support for OS/390 (NPMIP component) DRL390 NPMIP DRL Tivoli Decision Support for OS/390 Tivoli390 OPC D07 Tivoli Decision Support for OS/390 (OPC component) DRL390 OPC DRL Tivoli Decision Support for OS/390 Tivoli390 RACF® D09 Tivoli Decision Support for OS/390 (RACF component) DRL390 RACF DRL Tivoli Decision Support for OS/390 Tivoli390 RMF™ D02 Tivoli Decision Support for OS/390(RMF component) DRL390 RMF DRL Tivoli Decision Support for OS/390 Tivoli390 WAS D12 Tivoli Decision Support for OS/390 (WebSphere component) DRL390 WAS DRL Tivoli Decision Support for OS/390 TivoliApache PAC AMX IBM Tivoli Monitoring TivoliApache PAC GWA IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: AMX Apache HTTP ServerBMC Patrol BP6 BMC PATROL nullCM for ATMs CMA IBM Tivoli Configuration Manager for Automated Teller Tivoli MachinesCM for ATMs Tivoli Tivoli Application nullDB2 PAC AMX IBM Tivoli Monitoring TivoliDB2 PAC CTD IBM Tivoli Monitoring for Databases: DB2 AMXDM 3.7 DMN Distributed Monitoring Classic Edition TivoliDomino PAC ABA IBM Tivoli Messaging and Collaboration, Version 5.1.0: Lotus AMX DominoIIS PAC AMX IBM Tivoli Monitoring TivoliIIS PAC GWI IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: AMX Internet Information ServerITM 4.1 AMW Distributed Monitoring Advanced Edition TivoliITM 5.1.0 AMW IBM - Tivoli Monitoring 5.1 TivoliITM for OS 5.1.1 AMY IBM Tivoli Monitoring for Operating Systems AMXInformix PAC AMX IBM Tivoli Monitoring TivoliInformix PAC CTR IBM Tivoli Monitoring for Databases, Version 5.1.0: Informix AMXInventory-Software DIS IBM Tivoli Software Distribution TivoliDistributionInventory-Software INV IBM Tivoli Inventory TivoliDistributionLicense Manager COD Tivoli License Manager 1.1 TivoliMQ PAC CTQ IBM Tivoli Monitoring for Business Integration, Version 5.1.0 AMX : WebSphere MQ358 Implementing Tivoli Data Warehouse 1.2
    • MQI PAC HMI IBM Tivoli Monitoring for Business Integration 5.1.0 : AMX WebSphere MQ IntegratorMQWorkflow PAC BIW IBM Tivoli Monitoring for Business Integration - MQSeries® AMX Workflow 5.1.0MS SQL CTW IBM Tivoli Monitoring for Databases, Version 5.1.0: Microsoft AMX SQL ServerNetview ANM IBM Tivoli Netview TivoliNetview MODEL1 Tivoli Common Data Model v 1 nullOracle PAC CTO IBM Tivoli Monitoring for Databases, Version 5.1.0: Oracle AMXRisk Manager HRM IBM Tivoli Risk Manager TivoliSiebel PAC AMX Tivoli Monitoring TivoliSiebel PAC GMS IBM Tivoli Monitoring for Applications 5.1.0 : Siebel AMX eBusiness ApplicationsSiebel PAC Tivoli Tivoli Application nullTAPM 2.1 APF Tivoli Application Performance Management TivoliTBSM 1.5 GTM Tivoli Business System Manager TivoliTEC 3.7.1 ECO Tivoli Enterprise Console® TivoliTEC 3.8 ECO Tivoli Enterprise Console TivoliTEDW 1.2 Tivoli Tivoli Application nullTEDW 1.2 MODEL1 Tivoli Common Data Model V1 nullTMTP 5.1 BWM IBM Tivoli Monitoring for Transaction Performance - Web Tivoli Transaction Performance 5.1TSRM BTM IBM Tivoli Storage Resource Manager TivoliTSRM Tivoli Tivoli Application nullTWSA AWT Tivoli Web Site Analyzer 4.2 TivoliTWSM 1.7 BWM Tivoli Web Services Manager 1.7 TivoliTalking Blocks TS2 Talking Blocks nullTalking Blocks SNMP Simple Network Management Protocol nullTalking Blocks SHARED Shared nullTalking Blocks SDESK1 Service Desk nullWS Interchange PAC BIX IBM Tivoli Monitoring for Business Integration - WebSphere AMX InterChange Server 5.2.0WebLogic PAC AMX IBM Tivoli Monitoring TivoliWebLogic PAC GWL IBM Tivoli Monitoring for Web Infrastructure : WebLogic AMXWebSphere PAC IZY IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: AMX WebSphere Application ServeriPlanet PAC AMX IBM Tivoli Monitoring Tivoli Appendix B. Tivoli Data Warehouse 1.2 reference 359
    • iPlanet PAC GWP IBM Tivoli Monitoring for Web Infrastructure, Version 5.1.0: AMX iPlanet Web ServermySAP PAC ABH IBM Tivoli Monitoring for Applications, Version 5.1.0: Tivoli mySAP.commySAP PAC Tivoli Tivoli application null360 Implementing Tivoli Data Warehouse 1.2
    • C Appendix C. Warehouse Enablement Packs properties file This appendix provides a description of the various attributes of the properties file of a warehouse enablement pack.© Copyright IBM Corp. 2004. All rights reserved. 361
    • The twh_install_props.cfg properties file Every warehouse enablement pack must have defined a properties file that indicates how Tivoli Data Warehouse will perform the warehouse pack installation. The properties file is named twh_install_props.cfg and its location in the warehouse enablement pack installation media depends on the version of the Tivoli Data Warehouse product for which the warehouse enablement pack was created. The twh_install_props.cfg file of warehouse enablement packs created for the Tivoli Data Warehouse 1.2 resides in the top level directory of the installation media, specified by the AVA code. For warehouse enablement packs created for the Tivoli Enterprise Data Warehouse 1.1, the twh_install_props.cfg is located in the TDW 1.2 WEPs located in the pkg directory, as shown in Figure C-1. twh_install_props.cfg for TDW 1.2 twh_install_props.cfg for TEDW 1.1 Figure C-1 Location of the twh_install_props.cfg file Here we list some of the characteristics of the properties defined in the twh_install_props.cfg file: Statements take the form PROPERTY=VALUE. The property name must be uppercase. Each property in the file must have a value. The properties file is used by the installer. The installer will request the location of this file when installing the WEP. Ensure that a carriage return terminates each line — including the last line in the file. A property cannot contain white space. That is, do not put white space on the left side of the equal sign (=).362 Implementing Tivoli Data Warehouse 1.2
    • Table C-1 shows the attributes and parameters of the warehouse enablementpack properties file: twh_install_props.cfg.Table C-1 WEP installation properties Property Description AFTER_SCRIPT Optional Perl script that runs at the end of the warehouse pack installation. The script name must be misc/<ava_code>_after.pl, and must be lowercase.. ALLOW_ETL1_MULTI_SOURCE Specifies if the WEP was coded to support multiple data sources. APP_DISPLAY NAME User display name for the WEP. APP_DISPLAY_VERSION User displayed version. APP_REL_VERSION The internal version of the warehouse pack. AVA_CODE Three-character identifier unique to the warehouse pack. CDW_DB_TYPE Vendor support type for CDW. Allowed values are: - DB2UDB (for IBM DB2 UDB Enterprise Edition) - DB2390 (for DB2 UDB for z/OS and OS/390) - DB2UDB,DB2390 (for both) Must be in uppercase. DWC_INIT_STEP_NAME One-time initialization step that must be run manually after installation, and before the first job is scheduled. MART_DB_TYPE Data mart vendor types supported by this warehouse pack. Same options as CDW_DB_TYPE. NLS_LANG_LIST Languages supported by the WEP. Supported values of pt_BR,en,fr,de,it,ja,ko,zh_CN,es,zh_TW or if no internationalization TDW_NO_BUNDLES_REQUIRED. Appendix C. Warehouse Enablement Packs properties file 363
    • Property Description PRE_ETL_SCRIPT Optional Perl script that runs before the TAG file is loaded into the DB2 Data Warehouse Center. PRE_SCRIPT Optional Perl script that runs after the environment has been validated and the files have been copied to the $TWH_TOPDIR directory. REPORTS_INCLUDED If reports are included in the WEP. TRUE or FALSE. SCHEDULABLE_ETL1_STEP_NAME First step in the central data warehouse ETL process. SCHEDULABLE_ETL2_STEP_NAME First step in a data mart ETL process. SOURCE_APP_REL_VERSION The version number of your source application that is required by this warehouse pack. For documentation purposes only. TWH_CORE_PREREQ_VERSION The required version of TDW for this WEP. WEP_ROLE The type of ETL processes implemented by the WEP. ETL1 – only ETL1 processes. ETL2 – only ETL2 processes. E1E2 – both ETL1 and ETL2. WPACK_PREREQ_AVA_NAME Minimum prerequisite WEP names required for this WEP to run. WPACK_PREREQ_VERSION_n Minimum prerequisite WEP version required for this WEP to run.364 Implementing Tivoli Data Warehouse 1.2
    • Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.IBM Redbooks For information on ordering these publications, see “How to get IBM Redbooks” on page 367. Note that some of the documents referenced here may be available in softcopy only. Introduction to Tivoli Enterprise Data Warehouse, SG24-6607 IBM Tivoli Monitoring Version 5.1: Advanced Resource Monitoring, SG24-5519 Tivoli Data Warehouse Report Interfacing, SG24-6084Other publications These publications are also relevant as further information sources: IBM DB2 Universal Database Command Reference Version 7, SC09-2951 IBM Tivoli Monitoring User’s Guide Version 5.1.1, SH19-4569 IBM Tivoli Monitoring - Resource Model Reference, SH19-4570 Tivoli Enterprise Data Warehouse Release Notes, GI11-0857 IBM DB2 Universal Database for Windows Quick Beginnings, GC09-2971 Crystal Enterprise 9 Installation Guide Crystal Enterprise 9 Administrator’s Guide Crystal Enterprise 9 Getting Started Guide Crystal Enterprise 9 ePortfolio User’s Guide© Copyright IBM Corp. 2004. All rights reserved. 365
    • Online resources These Web sites and URLs are also relevant as further information sources: IBM Software Support Web site http://www.ibm.com/software/sysmgmt/products/support Crystal Enterprise Professional version 9 for Tivoli product site http://www.businessobjects.com/products/reporting/default.asp J2SE download site Crystal Enterprise Professional version 9 for Tivoli http://java.sun.com/j2se/1.3/install-solaris-patches.html SunSolve Web site http://sunsolve.sun.com Tivoli Enterprise Data Warehouse http://www.ibm.com/software/sysmgmt/products/support/TivoliEnterprise DataWarehouse.html Java Tutorial provided by Sun Microsystems http://java.sun.com/docs/books/tutorial/jdbc World Wide Web Consortium http://www.w3.org IBM DB2 Universal Database Enterprise Edition support Web site http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/ download.d2 Microsoft Support Web site http://support.microsoft.com/default.aspx?scid=kb Object management group Web site http://www.omg.org DB2 magazine Web site http://www.db2mag.com/db_area/archives/2001/q1/hayes.shtml Index of /html/Something_about_XSLM/About_XSLM http://www.xslm.org/html/Something_about_XSLM/About_XSLM/ Original_Requirements IBM HTTP Server http://www-3.ibm.com/software/webservers/httpservers366 Implementing Tivoli Data Warehouse 1.2
    • HTTP Server Plug-in http://www7.software.ibm.com/vadd-bin/ftpdl?1/vadc/wsdd/pdf/presents/ WS40ST08.pdf DB2 Technical Support – http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/ v7pubs.d2w/en_main – http://www-3.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/ v7fphist.d2w/reportHow to get IBM Redbooks You can search for, view, or download Redbooks, Redpapers, Hints and Tips, draft publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site: ibm.com/redbooksHelp from IBM IBM Support and downloads ibm.com/support IBM Global Services ibm.com/services Related publications 367
    • 368 Implementing Tivoli Data Warehouse 1.2
    • IndexSymbols C CENTR_LOOKUP 61.config 236 Central data warehouse naming convention 39Numerics central data warehouse 4, 36, 381.1 WEPs 46 Centralized logging 19390 systems 45 client administrative 9 Client tier 12AActivating data collection 237 CMC 59adding CDWs 116 Cognos Powerplay 14adding MARTs 119 collecting data 232Addition of warehouse packs 32 compete effectively 6admin server user 81 components requirements 36adverse effects 61 compromising security 58agent 8–9 configuring Crystal Enterprise 87agent configuration 18 configuring the control database 99agent discovery 18 control center server 21agent site 126 control database 38Agent Sites 22 control database configuration 99, 113agents installation 126 control server 36, 38Aggregation time line 231 create the itm_db database 235analytical processing 24 creating agent sites 126, 132Apache 35 creating CDWs 116APS 28, 33 creating MARTs 119Architectural choices 62 Crystal EnterpriseAutomated Process Schedule 28 architecture 11Automated Process Scheduler 33 Limited Edition 10 Professional Edition 10 user authority 59B Crystal Enterprise 9 33backing up the TWH_CDW database 242, 304 Crystal Enterprise configuration 87backing up the TWH_MART database 242, 304 Crystal Enterprise installation 86backing up the TWH_MD database 242, 305 Crystal Enterprise port numbers 56batch report 6 Crystal Enterprise Professional version 9 for TivoliBI 5 11Buffer Pool 54, 115 Crystal Enterprise requirements 33buffer pool 146 Crystal Enterprise Server 22Business Intelligence 5 Crystal Enterprise server 36business intelligence 4, 64 Crystal Enterprise Version 9 Special Edition 11Business Intelligence tools 70 Crystal Management Console 59 Crystal object rights 59 Crystal object security model 59 Crystal Publishing Wizard 105© Copyright IBM Corp. 2004. All rights reserved. 369
    • Crystal Reports 11 DB2 private protocols 51CUST_LOOKUP 61 DB2 Server installation 82 DB2 server installation 76 DB2 setup 78D DB2 Warehouse Managerdata administrative client 9 cleansing 8 agent 9Data collection 67 components 8data collection process 236 metadata 9data manipulation 67–68 DB2 Warehouse Manager installation 128data mart 5, 21, 36 DB2_ENABLE_LDAP settings 77Data mart database DB2-DB2CTLSV 128 naming convention 41 DB2SYSTEM 81data mining 7 demographic data 5data model 5 Development status 274data sources 32 DHTML 58Data tier 13 distributed deployment install 103data warehouse 4 Distributed installation 29 accessing 8 DMS 150 building 8 DMS tablespaces 150 designing 8 DNS 74 governing 8 Domain Name Server 74 integrated 5 Domino 35 maintaining 8 driving forces 6 subject-oriented 5 time-variant 5Data Warehouse Center 8–9 Edatabase heap 148 effect on network 61Database Managed Space 150 endpoint 239datacollector delay 240 ETL development 65datacollector prefix 236 ETL grouping 64datacollector.db_purge_interval 237 ETLs 63datacollector.db_purge_time 237datacollector.delay 237datacollector.max_retry_time 237 F fenced procedures 57datacollector.rim_name 237 fenced user 80datacollector.sleep_time 237 fetched data 146datamart 8 FFDC 19DB2 78 firewall considerations 58DB2 admin 81 First Failure Data Capture 19DB2 data flow 51 Fix Pack for DB2 32DB2 fenced 80 FMID JDB771D 32, 115DB2 Fix Pack 32 fp8_wr21314 84DB2 Fix Pack 8 installation 82, 84 Future data growth 32DB2 instance 79DB2 JDBC Applet Server 128DB2 logs 147 GDB2 on z/OS 32 gateway 233, 239DB2 performance 146 generic ETL1 229370 Implementing Tivoli Data Warehouse 1.2
    • GMT time 240 ITM Middle Layer Repository 228graphical user interface 8–9 ITM V 5.1.1 Generic Warehouse Enablement Pack 241 ITSM 37II/O servers 148IBM Console 10 JIBM DB2 admin 81 Java heap 147IBM DB2 fenced 80 JavaScript 39, 41IBM DB2 instance 79 JDB771D 32, 115IBM DB2 subsystem 41 JDBC code level 81IBM Tivoli Monitoring 37IBM Tivoli Monitoring 5.1.1 232IBM Tivoli Storage Manager 37 L large amounts of data 28, 42IMS 14 LDAP 59, 77increase revenues 6 LDAP authentication 59information access 6 level of DB2 on z/OS 32Information Catalog Manager 9 level token 57information delivery 7 local warehouse agent 50Information Management System 14 location name 41information tokens 77 logbufsz 147–148Informix 33 logfilsiz 147Install methods logical design 36 Distributed installation 28, 43 logprimary 147 Quick start installation 28, 42 logsecond 147Installation LogXML Viewer 19 IBM Tivoli Monitoring 5.1.1 232 single machine 42installation enhancements 18 Minstallation preparation 72 maintenance window 63installation process overview 73 Managed Resource 237installation steps 232 manager 9installing CDWs on z/OS 116 manipulate data 68installing DB2 Warehouse Manager 128 mapping data sources 66installing MARTs on z/OS 119 MDAC 31installing remote agent sites 126 metadata 9installing TDW 93, 103 metricsdata 240installing WEPs 142 metricsdata table 240instance 79 Microsoft Data Engine 33, 86instance owner 57, 235 Microsoft SQL Server 33INSTHOME 81 MON_HEAP_SZ 148integrated 5 monitoring application metadata 229integrated warehouse 5 MSDE 86Intelligence tier 12 MSDE database 33intranet 7 MSDE user account 88iPlanet 35 multicenter support 59–60iPlanet Enterprise Server 87 Multicustomer support 60irs.conf file 74 multicustomer support 59ITM 37 multiple customers support 60 Index 371
    • multiple machines installation 43 Q query 6 quick start install 93N Quick start installation 29naming convention 39, 41netsvc.conf file 74Network Information Service 74 RNIS 74 RDBMS administrator 235nsswitch.conf file 74 RDBMS Interface ModuleNUM_IOCLEANERS 146 See RIMNUM_IOSERVERS 146 Redbooks Web site 367 Contact us xxiii reduce costs 6O remote agents creation 132object rights 59object security 59 remote agents installation 126ODBC 32 remote warehouse agent 50ODBC connections 101, 123 remote warehouse agent site 126ODBC data sources 18 Remote warehouse agents 62OLAP 4, 24 REORG 149OLTP 4 REORGCHK 149online analytical processing 4 Report Interface requirements 33operational data 4 Reporting 70Operational Data Stores 8 reporting 6Oracle 33 requirements for Crystal Enterprise 33organization 67 resolv.conf file 74OS/390 45 resource model 237–238 restart DB2 242, 304 RIM 232P RIM configuration 233PAE 152 RIM host 232page cleaners 146 RIM host machine 235PCI bus 152 RIM object 234, 239performance DB2 146 RIM object connection 235Physical Address Extensions 152 RUNSTATS 149physical database 235physical design 36point of control 8 Spolicy region 237 sa user ID 88port numbers 56 SAP R/3 8prefetchers 146 scalability 8Process Modeller window 259 security considerations 57Processing tier 13 security model 59Product_Code 61 Service Pack 6 33Production status 274 shell script 235production window 63 Single machine installation 42profile 237–238 single machine installation 42profile manager 237 skills required 67Promote the ETL status 272