TFA Collector - what can one do with it

Oracle Trace File Analyzer (TFA) Features
Part of AHF install

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
2
6
Oracle Confidential – Internal

Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
3
6

Why TFA?
Running the necessary diagnostics & collecting the correct output across
multiple cluster nodes is not easy
– Data can be overwritten if not collected in time
– If too much data is collected the results can be very large
– Data collected may not include everything necessary to diagnose the problem
– Many different support tools with different syntaxes
– Majority of issue resolution time is spent obtaining the correct diagnostics
– When problems take a long time to resolve they cost your business money
4Oracle Confidential – Internal

Why TFA?
Provides one interface for
all diagnostic needs
Collects data across the
cluster and consolidates it
in one place
Collects all relevant
diagnostic data at the time
of the problem, with only
what is needed to diagnose
the problem
Reduces time required to
obtain diagnostic data,
which saves your business
money

Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
6
6

• All major Operating Systems are
supported
– Linux (OEL, RedHat, SUSE, Itanium &
zLinux)
– Oracle Solaris (SPARC & x86-64)
– AIX
– HPUX (Itanium & PA-RISC)
– Windows
• All Oracle Database & Grid versions
10.2+ are supported
• You probably already have TFA
installed it is included with:
– Oracle Grid Infrastructure:
• 11.2.0.4+
• 12.1.0.2+
• 12.2.0.1+
– Oracle Database:
• 12.2.0.1+
• Also available from Doc 1513912.2
7
Supported Platforms and Versions

Monitoring By TFA & Automated Collections
8
Automatically
detect event
Collect & package
relevant
diagnostics
Notify
relevant DBA and
or Sys Admin by
email
Upload collection
to Oracle Support
for further help
Significant
problem occurs
1
2
3
4
TFA
DBA(s) / Sys Admin(s)
Oracle Grid Infrastructure
& Database(s)

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 9
TFA
Daemon
Remote
Node
n
TFA
Daemon
tfactl
Initiator Node
( Where command originated)
Cluster
• TFA daemon runs on each cluster
node
• Or single instance when no
Grid Infrastructure is used
• Command line communication is
via tfactl command
• TFA Daemons on all nodes
coordinate:
• Script execution
• Collection of diagnostics
• Trimming of log contents
• Cluster wide collection output is
consolidated on one node
Architecture
Cluster
wide
Collection
Alerts &
Log files
Scripts
TFA
Daemon
Remote
Node
2
Scripts
TFA
Daemon
Remote
Node
1
Alerts &
Log filesScripts
Scripts

Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
10
6

1. Download latest version: Doc 1513912.2
2. Transfer zip to required machine
3. Unzip
4. Execute ahf_setup self extracting install
script as root user
– Will install on all cluster nodes
– Will auto discover relevant Oracle Software &
Exadata Storage Servers
– Will start monitoring all discovered items for
significant events & collect diagnostics when
necessary
Only required if:
• You are using an older versions of Grid or
Database – TFA is included in:
Or
• You need Support Tools bundle (as not
included in Grid / Database install)
11
Installation
./ahf_setup <options>
Oracle Grid
Infrastructure
Oracle Database
11.2.0.4+
19c+12.1.0.2+
12.2.0.1+
19c+

• Enable or disable user access
Configure User Access
tfactl
Grid Home
owner
Database Home
Owners
Everyone
Else
Users & groups authorized to access tfactl by default
tfactl access enable
tfactl access disable
tfactl access add –user <user> [-local]
tfactl access remove –user <user> [-local]
tfactl access removeall [-local]
• Add users
• Remove users or everyone
All commands apply
cluster wide unless
-local used to restrict to
local node
• Reset to default access
tfactl access reset
• List user access
tfactl access lsusers

Set Email Notification Addresses
13
tfactl set notificationAddress=john.doe@oracle.com
Automatically
detect event
Collect &
package relevant
diagnostics
Notify
relevant DBA
and or Sys
Admin by email
Upload
collection to
Oracle Support
for further help
Significant
problem occurs
1
2
3
4
TFA
DBA(s) / Sys Admin(s)
Oracle Grid Infrastructure
& Database(s)
tfactl set notificationAddress=oracle:another.person@oracle.com
• TFA can send email notification
when significant problems are
detected
• To set notification email for any
problem detected:
• To set notification email for specific
ORACLE_HOMEs include the OS
owner:

Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
14
6

Analyze
15
Problem occurs
Collect & package
relevant
diagnostics
Provide summary
of systems,
changes, events,
health & analysis
2
Perform analysis
using support
tools where
required
Request summary
with
tfactl summary
1
Request
collection using
tfactl diagcollect
Request further
analysis using
tfactl
Upload package
to Oracle Support
for help resolving
Resolve problem
Need
more help
?
Need
more info
?Oracle Grid Infrastructure
& Database(s)
TFA DBA(s) / Sys Admin(s)
56
7
89 10
11
No
Yes
No
Yes
Any
Problems
? Yes
3
No
4

• Analyze all important recent log entries: • Search recent log entries:
16
Analyze
tfactl analyze –since 1d tfactl analyze -search “ora-600" -since 8h

• TFA includes all key database support tools
• tfactl provides a single interface to them all
Analyze
17
Most of these Support tools are only available in the
My Oracle Support download, they are not included
in the base Grid or Database install
Tool Description Details
ORAchk Oracle Stack Health Checks on non-engineered
systems
1268927.2
EXAchk Oracle Stack Health Checks on Engineered
Systems
1070954.1
oswatcher Collect and archive OS metrics, useful for
instance / node evictions & performance
Issues
301137.1
procwatcher Automate & capture database performance
diagnostics & session level hangs
459694.1
oratop Near real-time database monitoring 1500864.1
sqlt Capture SQL trace date useful for tuning 215187.1
alertsummary Provides summary of events for one or more
database or ASM alert files from all nodes
ls Lists all files TFA knows about for a given file
name pattern across all nodes
Tool Description
pstack Generate process stack for specified processes across all nodes
grep Search alert or trace files with a given database and file name
pattern, for a search string.
summary High level summary of the configuration
vi Open alert or trace files for viewing a given database and file
name pattern in the vi editor
tail Run a tail on an alert or trace files for a given database and file
name pattern
param Show all database and OS parameters that match a specified
pattern
dbglevel Set and unset multiple CRS trace levels with one command
history Show the shell history for the tfactl shell
changes Report any noted changes in the system setup over a given
time period. This includes database a parameters, OS
parameters, patches applied etc

Oracle Stack Health Checks
• Oracle EXAchk for Engineered Systems
• Oracle ORAchk for everything else
• Proactively scans for the most impactful problems across the layers of your stack
• Simplifies and streamlines how to investigate and analyze which known issues
present a risk to you
• Lightweight tool that runs within your environment without requiring config data to
be sent to Oracle
• High level reports show your system health risks with the ability to drill down into
specific problems and understand their resolutions
• Can be configured to send email notifications when it detects problems
• Collection Manager, a companion Application Express web app, provides a single
dashboard view of collections across your entire enterprise
ORAchk & EXAchk (Support Tools Bundle)
Most of the Support Tools bundle is not installed with
the Database install and PSUs, however ORAchk is

OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics
• Executes standard UNIX utilities (e.g. vmstat, iostat, ps,
etc) on regular intervals
• Built in Analyzer functionality to summarize, graph and
report upon collected metrics
• Output is Required for node reboot and performance
issues
• Simple to install, extremely lightweight
• Runs on ALL platforms (Except Windows)
• MOS Note: 301137.1 – OS Watcher Users Guide

OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics
• Executes standard UNIX utilities (e.g. vmstat, iostat, ps, etc)
on regular intervals
• Built in Analyzer functionality to summarize, graph and
report upon collected metrics
• Output is Required for node reboot and performance issues
• Simple to install, extremely lightweight
• Runs on ALL platforms (Except Windows)

Analyse OS Metrics
tfactl run oswbb
Starting OSW Analyzer V8.1.2
OSWatcher Analyzer Written by Oracle Center of Expertise
Copyright (c) 2017 by Oracle Corporation
Parsing Data. Please Wait...
Scanning file headers for version and platform info...
Parsing file rws1270069_iostat_18.11.24.0900.dat ...
Parsing file rws1270069_iostat_18.11.24.1000.dat ...
...

Analyse OS Metrics
...
Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter GC to Generate All CPU Gif Files
Enter GM to Generate All Memory Gif Files
Enter GD to Generate All Disk Gif Files
Enter GN to Generate All Network Gif Files
Enter L to Specify Alternate Location of Gif Directory
Enter Z to Zoom Graph Time Scale (Does not change analysis dataset)
...

Analyse OS Metrics
...
Enter B to Returns to Baseline Graph Time Scale (Does not change
analysis dataset)
Enter R to Remove Currently Displayed Graphs
Enter X to Export Parsed Data to Flat File
Enter S to Analyze Subset of Data(Changes analysis dataset including
graph time scale)
Enter A to Analyze Data
Enter D to Generate DashBoard
Enter Q to Quit Program
Please Select an Option:1

Analyse OS Metrics
myserver69

Analyse OS Metrics
myserver69
more info 301137.1

Procwatcher (Support Tools Bundle)
Monitor & Examine Database Processes
• Single instance & RAC
• Generates session wait, lock and latch reports as well as call stacks
from any problem process(s)
• Ability to collect stack traces of specific processes using Oracle Tools
and OS Debuggers
• Typically reduces SR resolution for performance related issues
• Runs on ALL major UNIX Platforms
• MOS Note: 459694.1 – Procwatcher Install Guide

oratop (Support Tools Bundle)
• Near real-time Database monitoring
– Single instance & RAC
– Monitoring current database
activities
– Database performance
– Identifying contentions and
bottleneck
– Process & SQL Monitoring
– Real time wait events
– Active Data Guard support

Monitor Database performance
tfactl run oratop -database ogg19c

Monitor Database performance
• Section 1 DATABASE:
Global database
information
• Section 2 INSTANCE:
Database instance
Activity
• Section 3 EVENT: AWR
like “Top 5 Timed
Events“
• Section 4 PROCESS |
SQL: Processes or SQL
mode information
more info 1500864.1

Analyze
• Each tool can be run using tfactl in shell mode
• Start tfactl shell with
• Run a tool with the tool name
1. Where necessary set context with database <dbname>
2. Then run tool
3. Clear context with database
30
tfactl
tfactl > database MyDB
MyDB tfactl > database
tfactl > orachk
MyDB tfactl > oratop

Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
31
6

Collect
32
Problem occurs
Collect & package
relevant
diagnostics
Provide summary
of systems,
changes, events,
health & analysis
2
Perform analysis
using support
tools where
required
Request summary
with
tfactl summary
1
Request
collection using
tfactl diagcollect
Request further
analysis using
tfactl
Upload package
to Oracle Support
for help resolving
Resolve problem
Need
more help
?
Need
more info
?Oracle Grid Infrastructure
& Database(s)
TFA DBA(s) / Sys Admin(s)
56
7
89 10
11
No
Yes
No
Yes
Any
Problems
? Yes
3
No
4

• Trim & collect all important log files updated in
the past 12 hours:
• Collect a problem specific Service Request Data
Collection (SRDC):
33
Collect
tfactl diagcollect
• Collections stored in the repository directory
• Change diagcollect timeframe with –since <n>h|d
• For list of types of srdc collections use tfactl diagcollect -srdc help
tfactl diagcollect -srdc ora600

Storage Usage
$ /u01/app/12.1.0/grid/bin/tfactl print config
.----------------------------------------------------.
| myhost1 |
+--------------------------------------------+-------+
| Configuration Parameter | Value |
+--------------------------------------------+-------+
| TFA version | 3.2 |
| Automatic diagnostic collection | OFF |
| Trimming of files during diagcollection | ON |
| Repository current size (MB) in myhost1 | 44 |
| Repository maximum size (MB) in myhost1 | 10240 |
| Trace level | 1 |
'--------------------------------------------+-------'
.----------------------------------------------------.
| myhost2 |
+--------------------------------------------+-------+
+--------------------------------------------+-------+
| Trace level | 1 |
'--------------------------------------------+-------'.----------------------------------------------------.
| myhost3 |
+--------------------------------------------+-------+
+--------------------------------------------+-------+
| Trace level | 1 |
'--------------------------------------------+-------'
.----------------------------------------------------.
| myhost4 |
+--------------------------------------------+-------+
+--------------------------------------------+-------+
| Trace level | 1 |
'--------------------------------------------+-------'

Collection Subdirectories and Files
# ls -l /u01/app/crsusr/tfa/repository
total 8
drwxr-xr-x 2 root root 4096 Mar 14 13:46 collection_Fri_Mar_14_13_33_06_PDT_2014_node_all
drwxr-xr-x 2 root root 4096 Mar 14 14:27 collection_Fri_Mar_14_14_15_58_PDT_2014_node_all
[root@myhost1 oradb]# ls -l /u01/app/oragrid/tfa/repository/collection_Fri_Mar_14_13_33_06_PDT_2014_node_all
total 22564
-rw-r--r-- 1 root root 4848555 Mar 14 13:40 myhost1.tfa_Fri_Mar_14_13_33_06_PDT_2014.zip
-rw-r--r-- 1 root root 3121 Mar 14 13:40 myhost1.tfa_Fri_Mar_14_13_33_06_PDT_2014.zip.txt
[root@myhost1 oradb]# ls -l /u01/app/oragrid/tfa/repository/collection_Fri_Mar_14_14_15_58_PDT_2014_node_all
total 22932
Two subdirectories
created for collections
Files from first
collection
Files from
second collection

One Command SRDCs
• SRDC for database
performance
problems:
• dbperf
• SRDC for all other
types of internal
database errors:
• internalerror
SRDCs for ORA errors:
• ora600
• ora7445
• ora700
• ora4031
• ora4030
• ora27300
• ora27301
• ora27302
And more coming soon…

One Command SRDCs – Examples of What’s Collected
• Database Performance• ORA4031:
1. IPS Package
2. Patch Listing
3. AWR report
4. Memory information
5. RDA
•
1. ADDM report
2. AWR for good and problem period
3. AWR Compare Period report
4. ASH report for good and problem period
5. OS Watcher
6. IPS Package (if errors during problem period)
7. ORAchk (performance related checks)
tfactl diagcollect –srdc ora4031 tfactl diagcollect –srdc dbperf

Manual Data Gathering vs One Command SRDC
• TFA SRDC• Manual Data Gathering
1. Generate ADDM reviewing Document 1680075.1
2. Identify “good” and “problem” periods and gather AWR
reviewing Document 1903158.1
3. Generate AWR compare report (awrddrpt.sql) using “good”
and “problem” periods
4. Generate ASH report for “good” and “problem” periods
reviewing Document 1903145.1
5. Collect OSWatcher data reviewing Document 301137.1
6. Check alert.log if there are any errors during the “problem”
period
7. Find any trace files generated during the “problem” period
8. Collate and upload all the above files/outputs to SR
1. Run
2. Upload resulting zip file to SR
tfactl diagcollect –srdc dbperf

1. Enter default for event date/time and database name
2. Scans system to identify recent 10 events in the system (ORA600
example shown)
3. Once the relevant event is chosen, proceeds with diagnostic
collection
Confidential – Oracle Internal/Restricted/Highly Restricted 39
One Command SRDC
• Interactive Mode
tfactl diagcollect –srdc <srdc_type>
4. All required files are
identified
5. Trimmed where
applicable
6. Package in a zip ready
to provide to support

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 40
One Command SRDC
• Silent
tfactl diagcollect –srdc <srdc_type> -database <db> -for <time>
1. Parameters(date/time, DB name) are provided
in the command
2. Does not prompt for any more information
3. All required files are identified
4. Trimmed where applicable
5. Package in a zip ready to provide to support

Faster & Easier SR Data Collection
Type of Problem SRDC Types Collection Scope
ORA Errors
• ORA-00600
• ORA-00700
• ORA-04030
• ORA-04031
• ORA-07445
• ORA-27300
• ORA-27301
• ORA-27302
Local only
Other internal database errors • internalerror Local only
Database performance problems • dbperf Cluster wide
Database patching problems
• dbpatchinstall
• dbpatchconflict
Local only
Database install / upgrade problems
• dbinstall
• dbupgrade
Local only
Enterprise Manager tablespace usage metric problems • emtbsmetrics Local only (on EM Agent target)
Enterprise Manager general metrics page or threshold problems • emmetricalert Local only (on EM Agent target & Repository DB)
Enterprise Manager debug log collection
• emdebugon
• emdebugoff
Local only (on EM Agent target & OMS)
New
New
New
New
New
New
New
New
tfactl diagcollect –srdc <srdc_type>

On Demand Collection Functionality
ORA-04031ORA-04030
What to collect?
Where to collect
from?
Which time
period?
How to collect?
ASH Reports
Collect
Everything
ODA Storage
Logs
CRS Logs
DBWLM Logs
ACFS Logs
CRGTOOLS Logs
Installation
related files
ODA Logs
AWR Reports
All files from a
flagged directory
Specific Nodes
Specific
Databases
Data for the past
hours/days
Data from a certain
date / time
Data to a certain
date / time
Data to a certain
date
Copy zip back to
originating node
Trim files
Specific zip
name
Tag to include in
collection name
SRDC
ISP Package
Problem
Collection
Incident
Collection
ORA-00700ORA-00600
Basic DB
Performance
ORA-07445
Internal Errors
Incident Based
Collections
Time Based
Collections
OS Logs
All databases &
all nodes
Past 12 hours
ZDLRA Logs &
Data
RAC DBaaS Logs
Key
Default
Non Default Requires License
Default
(environment
specific)

diagcollect Options
43
Option Description
-all All logs
-database
<all|d1,d2..>
Database logs from
databases specified
-asm ASM logs
-crsclient
Client Logs under
GIBASE/diag/clients
-dbclient
Client logs under DB
ORADBASE/diag/clients
-dbwlm DBWLM logs
-tns TNS logs
-rhp RHP logs
-afd ADF logs
-crs CRS log
-wls WLS logs
-emagent EM Agent logs
-omes OMS logs
-ocm OCM logs
-emplugins EM Plugin logs
-em EM logis
-acfs ACFS, ASM & CRS logs
-install Installation logs
Option Description
-ashhtml Generate & collection ASH HTML Report
-ashtext Generate & collection ASH TEXT Report
-awrhtml AWRHTML logs
-awrtext AWRTEXT logs
-nocores Don’t collect core files
-collectalldirs
all files from a directory marked "Collect
All”
-collectdir <dir> All files in specified directory
-node <all | local|
n1,n2>
comma separated list of host names for
collection
-nocopy
Does not copy back the zip files to
initiating node from all nodes
-notrim Does not trim the files collected
-since <n><h|d>
Only relevant data for the past X hours
or days
-from <time> Only relevant data from <time>
-to <time> Only relevant data to <time>
-for <date> Only relevant data for <date>
-tag <tagname>
Collect files in <tagname> directory in
repository
-z <zipname> Collect files in file called <zipname>
Option Description
-cfgtools CFGTOOLS logs
-os
OS logs such as
/var/log/messages
-ips Incident packaging service
-oraclehome
<oracle_home>
ORACLE_HOME to be used
for the ADRCI binary
-adrbasepath
<adr_basepath>
ADR basepath(s) to be used
for the IPS command
-adrhomepath
<adr_homepath>
ADR home(s) to be used for
the IPS command
-level <corr_level>
ADR correlation level
(basic,typical,all)
-incident
<adr_incident>
ADR incident number
-problem <prob_id> ADR problem number
-problemkey
<prob_key>
ADR problemkey
-manageips
Pause TFA IPS collection for
package manipulation
-resumeips
<collection_id>
Continue TFA IPS collection
after a pause for package
manipulation

Time Based Collections with diagcollect
• Collections can be initiated any time on demand
tfactl diagcollect
• Trim & zip all files updated in the last 4 hours as well as chmos/osw data from across the cluster:
tfactl diagcollect -database mydb -since 1d
• To only collect from specific databases use –database <dbname> :
tfactl diagcollect –since 8h
• To change the time period add –since <hours|days> :
tfactl diagcollect –since 1dor
tfactl diagcollect -database mydb -since 1d –z mydb_collection
• To specify an output file name use –z <name> file will be placed in TFA repository directory :

Time Based Collections with diagcollect
45
tfactl diagcollect –crs -os -node node1,node2 -since 6h
• Trim & zip all CRS files, OS logs and chmos/osw data from specific nodes use –crs –os –node <nodes>:
tfactl diagcollect -asm -node node1 -from Jun/1/2016 -to “Jun/2/2016 21:00:00"
• Trim & zip all ASM logis from a specific node during a specific time period use
–asm –node <nodes> -from <date/time> -to <date/time>
tfactl diagcollect -for “Jun/2/2016"
• To trim & zip all files updated on a specific date across the cluster use –for <date>:
tfactl diagcollect -for “Jun/2/2016 21:00:00"
• To trim & zip all files updated from a specific data & time use –for <date> <time>:

IPS

Incident Based Collections with diagcollect
47
TFA
Listener
TFA
IPS
Trace
Collection
IPS ADR
Collection
Package
File
Remote
Node
n
TFA
IPS
Trace
Collection
IPS ADR
Collection
Package
File
Output
Directory
Master
Node
( Where
command
issued )
Incident Correlation
Key File
TFA
Listener
TFA
IPS
Trace
Collection
IPS ADR
Collection
Package
File
Remote
Node
2
TFA
Listener
TFA
IPS
Trace
Collection
IPS ADR
Collection
Package
File
Remote
Node
1
TFA Cluster
1. Issue the collection command –
this node becomes the master
2. TFA passes the incident
correlation key file to other nodes
in the cluster so they can collect
correlated diagnostic data
3. TFA on each node performs trace
collection & uses IPS to collect the
ADR diagnostic data
4. Everything is put in an IPS
package, retaining ADR directory
structure
5. Generated package files are
returned to output directory on
the originating master node

Use Incident Packaging Service (IPS) to Show Problems
• tfactl can invoke IPS to show
recent problems
• Find the relevant problem or
incident and then generate a
package collection

• Collect Incident packaging service (IPS) –ips
interactively
• Collect IPS in silent mode
• Collect IPS for specific incident id
• Collect IPS for specific problem id
• IPS package collection can be paused while the
package is manipulated then collection resumed
1. Start collection with -manageips
2. Monitor TFA IPS collections with print suspendedips
3. Continue collection with -resumeips
49
Incident Based Collections with diagcollect
tfactl diagcollect -ips -adrbasepath /app/oragrid
-adrhomepath diag/asm/+asm/+ASM1 -since 1d
-node local
-adrhomepath diag/asm/+asm/+ASM1 -since 6h
-node local –incident 1234
-adrhomepath diag/asm/+asm/+ASM1 -since 6h
-node local –problem ora600
tfactl diagcollect –ips –incident 1234
-manageips –node node1
tfactl print suspendedips
tfactl diagcollect
–resumeips 20160603071227myserver5
tfactl diagcollect -ips -since 1d -node node1

Collect Packages
tfactl diagcollect -ips
Select ADR home
Generate package
Rest of collection
Resulting collection zips

SRDC

Incident Based Collections with SRDC
52
• Collects everything requested for
respective SRDC
• Ready for SR upload

• Use srdc <incident type>:
• To specify sid use –sid <oracle sid>
• To specify database use –db <dbname>
• To specify incident date & time use
–inc_date <YYYY-MM-DD> -inc_time <HH:MM:SS>
• To upload directly to the SR use –sr<SR#>
• For dbperf use these parameters to
specify the good & bad performance
periods to compare:
53
Incident Based Collections with SRDC
tfactl srdc ora4030
Incident Type Description
ora4030 For ORA-04030 errors
dbperf For basic db performance problems
tfactl srdc ora4030 -sid orcl –db RDBMS121
-inc_date 2016-06-15 -inc_time 02:48:23
-sr 3-123456789
Parameter Description
perf_base_sd Start date for a good performance period
perf_base_st Start time for a good performance period
perf_base_ed End date for a good performance period
perf_base_et End time for a good performance period
perf_comp_sd Start date for a bad performance period
perf_comp_st Start time for a bad performance period
perf_comp_ed End date for a bad performance period
perf_comp_et End time for a bad performance period
tfactl srdc dbperf –db RDBMS121
–perf_base_sd 2016-06-15 –perf_base_st 01:30:00
–perf_base_ed 2016-06-15 –perf_base_et 02:00:00
–perf_comp_sd 2016-06-16 –perf_comp_st 09:30:00
–perf_comp_ed 2016-06-16 –perf_comp_et 10:00:00
When I try tfactl srdc get
“error SRDC is not deployed”
but tfactl toolstatus shows it is???
Are these correct descriptions

Log File Analysis

TFA Log Analyzer
Search logs for specific text
• Similar to grep but faster
• Only shows matching
messages in desired time-
frame
Analyze OSWatcher log stats
• First, highest, lowest, average
and last 3 readings of each
statistic
• Specify interval to analyze
down to the minute or second
Show most common messages
• Parses:
• Database Logs
• ASM Logs
• CRS Logs
• Alert Logs
• System Message
Log
• OSW Top files
• OSW Slabinfo files
• Identifies common log
messages and displays them in
order of frequency
• Can be filtered by:
• Component
• Error type
• Time

TFA Log Analyzer
Option Description
-search “<pattern>” Search for pattern in system and alert
logs from specified time range
-comp <component> Components to analyzer (default is all)
-type <type> Analyze messages of specified type
(default is ERROR)
-node <node> Comma separated list of host names,
(“local” for local node only, default is all)
-since <n>h|d Analyze duration before current time
-for <time> Analyze specified time using format:
MMM/DD/YYYY HH24:MI:SS
-from <time> –to
<time>
Analyze specified time period using
format: MMM/DD/YYYY HH24:MI:SS
-verbose Verbose output
-o Write the output <file> instead of
screen
tfactl analyze <options> tfactl analyze -search
• Is not case sensitive by default
• To perform case sensitive search:
• Start term with /
• End term with /c
tfactl analyze -search "error" -since 2d
tfactl analyze -search "/ORA-/c” -since 2d

• Restrict search or analysis to a
specific component:
• Analyze message types:
57
TFA Log Analyzer
Component Description
DB Database Alert Logs
ASM ASM Alert Logs
CRS CRS Alert Log
ACFS ACFS Log
OS System Message File
OSW OSW Top Output
OSWSLABINFO OSW Slabinfo Output
tfactl analyze –comp <component>
Type Types of Messages
error
ORA-00600 / ORA-07445
IPS send timeouts
Direct NFS ping timeout / down
ospid timeouts
Instance terminated dur to error
Global Enqueue deadlocks
CRS fatal errors
warning
Disk off line
Corrupted block
CRS none fatal errors
generic Anything else
tfactl analyze –type <type>
tfactl analyze –search “<term>” –comp <component>

tfactl

Admin Commands: User:
59
tfactl Commands
• set: Turn ON/OFF or modify
various TFA features
• start <tool>: Start Support tool
• stop <tool>: Stop Support tool
• restart <tool>: Restart Support
tool
• uninstall: Uninstall TFA
• diagnosetfa: Self diagnose
• start: Start TFA
• stop: Stop TFA
• enable: Enable auto start on
reboot
• disable: Disable auto start
• access: Add, remove or list TFA
Users
• purge: Delete collections from
TFA repository
• directory: Add, remove or modify
directory in TFA
• host: Add or remove host to TFA
• print: Print requested details
• analyze: Summarize events
detected in alert logs
• diagcollect: Collect logs from across
nodes in cluster
• directory: Add, remove or modify
directory in TFA
(only directories that the non root
user has access to)
collection: Manage collections
• toolstatus: Print status of TFA
Support Tools
• run <tool>: Run Support tool
• start <tool>: Start Support tool
• stop <tool>: Stop Support tool
• restart <tool>: Restart Support tool
• run <tool>: Run Support tool
• ls : Search files in TFA
• grep : Grep for input string in logs
• tail : Tail log files
• vi : Search and open files in vi
editor
• param : Prints parameter value
• alertsummary : Prints summary of
important events in database/ASM
alert logs
• analyze : Summarize and search
important timeline files in system.
• changes : Prints system changes
• events : Lists all important events in
system
• ps : Find a process
• pstack : Run pstack on a process
• summary : Prints system summary

Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
60
6

Automatic Database Log Purge
• TFA can automatically purge database logs
– OFF by default
– Except on a Domain Service Cluster (DSC), which it is ON by default
• Turn auto purging on or off:
• Will remove logs older than 30 days
– configurable with:
• Purging runs every 60 minutes
– configurable with:
tfactl set manageLogsAutoPurge=<ON|OFF>
tfactl set manageLogsAutoPurgePolicyAge=<n><d|h>
tfactl set manageLogsAutoPurgeInterval=<minutes>

Manual Database Log Purge
• TFA can manage ADR log and trace files
– Show disk space usage of individual diagnostic destinations
– Purge these file types based on diagnostic location and or age:
• "ALERT“, "INCIDENT“, "TRACE“, "CDUMP“, "HM“, "UTSCDMP“, "LOG“
tfactl managelogs <options>
Runs as the ADR home
owner. So will only be
able to purge files this
owner has permission to
delete
Option Description
–show usage Shows disk space usage per diagnostic directory for both GI and database logs
-show variation –older <n><m|h|d> Use to determine per directory disk space growth.
Shows the disk usage variation for the specified period per directory.
-purge –older <n><m|h|d> Remove all ADR files under the GI_BASE directory, which are older than the time specified
–gi Restrict command to only diagnostic files under the GI_BASE
–database [all | dbname] Restrict command to only diagnostic files under the database directory. Defaults to all,
alternatively specify a database name
-dryrun Use with –purge to estimate how many files will be affected and how much disk space will be
freed by a potential purge command.
May take a while for a
large number of files

• tfactl managelogs –show variation –older
<n><m|h|d>
Use -gi to only
show grid
infrastructure
Use –database to only
show database

• tfactl managelogs –purge –older n<m|h|d>
Use –dryrun
for a “what if”

REST service
• REST support allows invocation & query over HTTPS
• Oracle REST Data Services (ORDS) is included within the install
• Once ORDS is running REST invocations can be made via:
• Post to Result
• Can be downloaded with:
tfactl rest [-status|-start|-stop|-uninstall] [-dir ] [-port ] [-user ] [-debug [-level ]]
https://host:port/ords/api
https://myhost:9090/ords/tfactl/diagcollect {
"collectionId" : "20180111011121slc12ekf",
"zipName" : "TFA_DEF_ZIP_20180111011121",
"tagName" : "TFA_DEF_TAG_20180111011121"
}
https://myhost:9090/ords/tfactl/download/20180111011121slc12ekf
The tfactl rest command
can only be run by root
user
The standalone ORDS setup feature utilizes file based user authentication and is provided solely for use in test and development environments.
For production use, the included ords.war should be deployed and configured.

REST service
Option Description
-status Prints the current status
-start Starts Oracle Trace File Analyzer REST Services if not already running
-stop Stops Oracle Trace File Analayzer REST services if running
-uninstall Removes the Oracle Trace File Analyzer REST configuration
-dir
The directory to use to store the Oracle Trace File Analyzer REST configuration details.
Defaults to the users home directory
-port
The port to run ORDS on
Defaults to 9090
-user
The user to start ORDS as
Defaults to the GRID owner
-debug Enables debug
-level
The level of debug to use, where available levels are:
•1 - FATAL
•2 - ERROR
•3 - WARNING
•4 - INFO (Default)
•5 - DEBUG
•6 - TRACE

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted 67
REST Service Extensions
• New run commands added to REST API:
Shows summary of events from alert logshttps://host:port/ords/tfactl/run/alertsummary
Shows major events from the cluster event loghttps://host:port/ords/tfactl/run/calog
Shows system changes including DB parameters, OS parameters & patcheshttps://host:port/ords/tfactl/run/changes
Reports warnings and errors seen in the logshttps://host:port/ords/tfactl/run/events
Reports history of commands for the tfactl shellhttps://host:port/ords/tfactl/run/history

• Option 1
– Applying standard RUs will
automatically update TFA
• Option 2
– To update with latest AHF
1. Download latest version: Doc 1513912.2
2. Transfer zip to required machine
3. Unzip
4. Execute AHF self extracting install script as root user
68
Maintain
Upgrade to the latest version whenever possible to include bug fixes, new features & optimizations

Program Agenda
Why TFA?
Overview
Install & Configure
Analyze
Collect
Maintain
1
2
3
4
5
69
6

TFA Collector - what can one do with it

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to TFA Collector - what can one do with it

Similar to TFA Collector - what can one do with it (20)

More from Sandesh Rao

More from Sandesh Rao (20)

Recently uploaded

Recently uploaded (20)

TFA Collector - what can one do with it