The document discusses Oracle's AHF Insights, a tool designed for proactive database management that provides a comprehensive diagnostic overview and effective issue resolution guidance. It highlights the importance of maintaining consistent software versions, troubleshooting various issues such as node evictions, and the benefits of regular software updates. New features and user experience improvements of AHF Insights are also outlined, emphasizing better diagnostic reporting and visual aids for problem resolution.

1. Beyond Metrics – Oracle AHF Insights for
Proactive Database Management
Sandesh Rao
VP AIOps and Machine Learning , Autonomous Database
@sandeshr
https://www.linkedin.com/in/raosandesh/
https://www.slideshare.net/SandeshRao4

2. Confidential - Oracle Restricted
Confidential - Oracle Restricted
AHF Stack (on a database system)

3. Confidential - Oracle Restricted
Confidential - Oracle Restricted
▪ Purpose
▪ Provides bird's eye view of the system from diagnostic perspective.
▪ Offers insights for effective issue resolution with guidance & co-relation.
▪ Unifies the AHF stack under a single user interface
▪ Usage
▪ Reactive, Proactive.
▪ Target Users
▪ Customers, Operations, Support, Development
▪ Collected by Default in AHF Collections from AHF 24.2
▪ To generate AHF Insights, run :
AHF Insights

4. AHF Insights Overview - Demo

5. Confidential - Oracle Restricted
Confidential - Oracle Restricted
AHF Insights (over command-line)

6. Confidential - Oracle Restricted
Confidential - Oracle Restricted
AHF Insights as part of AHF collection

7. Confidential - Oracle Restricted
Confidential - Oracle Restricted
AHF Insights as part of AHF collection

8. Confidential - Oracle Restricted
Confidential - Oracle Restricted
AHF Insights Overview

9. Need configurations details while
troubleshooting an issue

10. Confidential - Oracle Restricted
Confidential - Oracle Restricted
▪ BareMetal system (DomU has access to query storage server and fabric switch details)
▪ DomU has limited access
Configuration Details

11. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Cluster Details

12. Confidential - Oracle Restricted
Confidential - Oracle Restricted
ASM Details

13. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Database Details

14. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Database Server Configuration

15. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Database Server Configuration

16. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Database Parameters

17. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Kernel Parameters

18. Troubleshoot :
Issue due to inconsistent RPMs

19. Confidential - Oracle Restricted
Confidential - Oracle Restricted
▪ Example Scenario - Inconsistent glibc Package Versions
▪ Issue - Node Eviction and Clusterware Instability
▪ Example
▪ In a four-node Oracle RAC cluster, nodes 1 and 2 have glibc version 2.17-307 installed, while nodes 3 and 4
have glibc version 2.17-307.el7.1 installed. This discrepancy can cause several problems.
▪ Impact
▪ Node Eviction - Due to the different versions of glibc, nodes 3 and 4 might face eviction as the clusterware
detects inconsistencies in the environment.
▪ Clusterware Instability - The inconsistency in glibc can cause instability in Oracle Clusterware, leading to
startup failures and communication errors.
Issue due to inconsistent RPMs

20. Confidential - Oracle Restricted
Confidential - Oracle Restricted
RPMS & Inconsistencies

21. Troubleshoot :
Issue due to software version lower than
MAA Software Recommendations

22. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Recommended Software
All software should be updated regularly. Maintaining software at current or recent releases provides the following
benefits:
▪ Better software security
▪ More stable maintenance releases
▪ Continued compatibility with newer related software
▪ Better support and faster resolution of issues
▪ Ability to receive fixes for newly discovered issues.

23. Troubleshoot :
Issue due to recent changes on the system

24. Confidential - Oracle Restricted
Confidential - Oracle Restricted
▪ Issue due to Application of new Patch
▪ Issue due to Changes on ASM / Database parameter
▪ Issue due to New OS package installed
▪ Issue due to New Oracle Software installed
Issue due to recent changes on the system

25. Confidential - Oracle Restricted
Confidential - Oracle Restricted
System Changes

26. Troubleshoot :
Space Usage Issues

27. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Space Usage Issues

28. Troubleshoot :
Issue due to Best Practice Violations

29. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Best Practice Violations

30. Troubleshoot :
Major Events happening across the cluster

31. Confidential - Oracle Restricted
Confidential - Oracle Restricted
▪ For troubleshooting one needs to know :
▪ What type of system does user have ?
▪ What’s going on around the time of issue ?
▪ Can I get a full picture across all nodes ?
▪ Can I zoom into specific timeframe ?
▪ Can I look at the data from various perspectives ?
Customer Complains of “Grid failure - CRS-8503 []” in SR

32. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Customer’s System around the time of Issue

33. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Major Events around the time of issue

34. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Major Events around the time of issue

35. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Major Events around the time of issue

36. Troubleshoot :
Operating System Issues

37. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Customer’s System undergoes Node eviction

38. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Customer’s System undergoes Node eviction

39. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Customer’s System undergoes Node eviction
High Memory Pressure
Increase in RSS consumption
by ‘extract’ process

40. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Customer’s System undergoes Node eviction
50GB RSS hogged by extract process

41. Troubleshoot :
Database Anomalies

42. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Database Anomalies as observed by Cluster Health Advisor

43. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Database Anomalies as observed by Cluster Health Advisor

44. Troubleshoot :
Node Eviction Problems

45. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Node eviction due to Huge Page over-allocation

46. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Node eviction due to Huge Page over-allocation

47. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Node eviction due to Huge Page over-allocation

48. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Node eviction due to Huge Page over-allocation

49. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Node eviction due to Huge Page over-allocation

50. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Node eviction due to Huge Page over-allocation

51. Confidential - Oracle Restricted
Confidential - Oracle Restricted
Node eviction due to Huge Page over-allocation

52. Confidential - Oracle Restricted
Confidential - Oracle Restricted
✓ Insights in Diagnostic Collections by Default
✓ All Manual collection and subset of SRDCs
✓ Ability to schedule automatic AHF Insights generation
✓ Support Insights for Longer Time Ranges
✓Intelligent OS Resolution reduction
✓ Database Anomalies Advisor (New Section)
✓ Space Usage (New Section)
✓ Support for Single Instance Systems
✓ Augment Storage Cell information from Exawatcher Data
✓ Data Guard support in Insights (New Section)
✓ Detected Problem in Insights (New Section)
✓ Insights Co-relation Engine
✓ CHA Bayesian Network
• Performance Reports in Insights (New Section)
AHF Insights – New Features
✓ Problem Summary - Guided Resolutions
✓Node Evictions & Performance Issues Are Easier to
Resolve
✓Memory exhaustion due to
✓HugePages are over allocated
✓Database or Grid Infrastructure process increasing
memory usage
✓New Database started
✓Multipath disk failures
✓Hangs and performance issues caused by
✓Archiver stuck
✓Latch contention due to misconfigured target_pdbs
parameter

53. Confidential - Oracle Restricted
Confidential - Oracle Restricted
• Improvised Operating system reporting
• More space to explore the findings with a full widescreen view.
• View the co-related event information in a subplot within the Summary Timeline Gantt Chart
• View metrics associated with the problem finding in a visual format
• Easy to jump to appropriate problem section by interacting with charts
• Easier to Diagnose Problems with Disks or OS Processes
• Improvised Timeline
• Additional Timeline Views (Database Faceted, Component Faceted)
• Timeline Includes Patch Information
• Timeline chart and table now dynamically adjust time ranges.
• Performance improvement
• Report’s browser load times
• Insights Report generation time along with Diagnostic Collections
• Deep linking into individual sections of Insights report
• Copy Diagnostic Information from AHF Insights as Plain Text from more sections
• Insights Accessibility Improvements
AHF Insights – User Experience Improvements