SlideShare a Scribd company logo
Top 10 DB2 SupportTop 10 DB2 SupportTop 10 DB2 SupportTop 10 DB2 Support
Nightmares & How toNightmares & How toNightmares & How toNightmares & How to
Avoid ThemAvoid ThemAvoid ThemAvoid Them
#9#9#9#9
Part 9 – In the event of an emergency call
The situation
Image of a junior DBA
A customer was running HADR for disaster
recovery. They had no cluster software used
for monitoring or failover. HADR was being
monitored on a regular basis using a shell
script.
Then what happened…?
Disk failure on the primary site! Some tablespaces
were put into “Rollforward Pending” state.
Transactions accessing data in these tablespaces
failed
Image of a junior DBA
The last run of the HADR state monitoring
script indicated a Peer State so it was decided
to issue TAKEOVER command on the DR site to
switch roles. When the application started,
some transactions failed with the same error as
on the primary site.
List tablespaces command showed a number of
tables in Rollforward Pending state. To get out
of the pending state, ROLLFORWARD command
was issued with the list of affected tablespaces.
The rollforward was trying to retrieve a log,
which was a few thousand logs older than the
current one. After a few tries the
ROLLFORWARD was given up.
The database was restored from the latest
backup image
• We went through the db2diag.log and the notification logs. We could see that there
were physical errors reported in some of the tablespaces on the DR site around 100 days
prior to the incident. This was reported in the db2diag.log and the affected tablespaces
were “excluded from the rollforward set”
• Based on other entries in the db2diag file, we were able to confirm that the log file
requested for rollforward on the DR site was used at the time the physical errors
occurred there.
• HADR continued to apply logs for the other tablespaces and was reporting to be in “Peer”
State. In reality, some of the tablespaces were being ignored.
Triton Analysis
Regular monitoring on the log files is essential to identify and resolve the
issue on the DR site in advance of this incident.
Make sure you know who to call before disaster strikes!
Triton Consulting +44 870 2411 550
www.triton.co.uk
The moral of the story
www.triton.co.uk

More Related Content

What's hot

Congratsyourthedbatoo
CongratsyourthedbatooCongratsyourthedbatoo
Congratsyourthedbatoo
Dave Stokes
 
Hadoop installation by santosh nage
Hadoop installation by santosh nageHadoop installation by santosh nage
Hadoop installation by santosh nage
Santosh Nage
 
Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)
Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)
Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)
Adam Kawa
 
Drobo products ppt
Drobo products pptDrobo products ppt
Drobo products ppt
Clayton Desouza
 
Raid data recovery Tips
Raid data recovery TipsRaid data recovery Tips
Raid data recovery Tips
Hone Software
 
Hadoop single node installation on ubuntu 14
Hadoop single node installation on ubuntu 14Hadoop single node installation on ubuntu 14
Hadoop single node installation on ubuntu 14
jijukjoseph
 
Raid 5
Raid 5Raid 5
Hadoop Cluster With High Availability
Hadoop Cluster With High AvailabilityHadoop Cluster With High Availability
Hadoop Cluster With High Availability
Edureka!
 
Apache Hadoop YARN, NameNode HA, HDFS Federation
Apache Hadoop YARN, NameNode HA, HDFS FederationApache Hadoop YARN, NameNode HA, HDFS Federation
Apache Hadoop YARN, NameNode HA, HDFS Federation
Adam Kawa
 
2 db2 instance creation
2 db2 instance creation2 db2 instance creation
2 db2 instance creation
Ravikumar Nandigam
 
Improving Hadoop Performance via Linux
Improving Hadoop Performance via LinuxImproving Hadoop Performance via Linux
Improving Hadoop Performance via Linux
Alex Moundalexis
 
Introduction to hadoop high availability
Introduction to hadoop high availability Introduction to hadoop high availability
Introduction to hadoop high availability
Omid Vahdaty
 
Setting up repositories
Setting up repositoriesSetting up repositories
Setting up repositories
Iryna Kuchma
 
leiame
leiameleiame
leiame
sindiconet
 

What's hot (14)

Congratsyourthedbatoo
CongratsyourthedbatooCongratsyourthedbatoo
Congratsyourthedbatoo
 
Hadoop installation by santosh nage
Hadoop installation by santosh nageHadoop installation by santosh nage
Hadoop installation by santosh nage
 
Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)
Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)
Hadoop Adventures At Spotify (Strata Conference + Hadoop World 2013)
 
Drobo products ppt
Drobo products pptDrobo products ppt
Drobo products ppt
 
Raid data recovery Tips
Raid data recovery TipsRaid data recovery Tips
Raid data recovery Tips
 
Hadoop single node installation on ubuntu 14
Hadoop single node installation on ubuntu 14Hadoop single node installation on ubuntu 14
Hadoop single node installation on ubuntu 14
 
Raid 5
Raid 5Raid 5
Raid 5
 
Hadoop Cluster With High Availability
Hadoop Cluster With High AvailabilityHadoop Cluster With High Availability
Hadoop Cluster With High Availability
 
Apache Hadoop YARN, NameNode HA, HDFS Federation
Apache Hadoop YARN, NameNode HA, HDFS FederationApache Hadoop YARN, NameNode HA, HDFS Federation
Apache Hadoop YARN, NameNode HA, HDFS Federation
 
2 db2 instance creation
2 db2 instance creation2 db2 instance creation
2 db2 instance creation
 
Improving Hadoop Performance via Linux
Improving Hadoop Performance via LinuxImproving Hadoop Performance via Linux
Improving Hadoop Performance via Linux
 
Introduction to hadoop high availability
Introduction to hadoop high availability Introduction to hadoop high availability
Introduction to hadoop high availability
 
Setting up repositories
Setting up repositoriesSetting up repositories
Setting up repositories
 
leiame
leiameleiame
leiame
 

Viewers also liked

Linux Disaster Recovery Solutions
Linux Disaster Recovery SolutionsLinux Disaster Recovery Solutions
Linux Disaster Recovery Solutions
Gratien D'haese
 
DB2 High Availability für IBM Connections, Sametime oder Traveler
DB2 High Availability für IBM Connections, Sametime oder TravelerDB2 High Availability für IBM Connections, Sametime oder Traveler
DB2 High Availability für IBM Connections, Sametime oder Traveler
Nico Meisenzahl
 
A05
A05A05
High availability solutions bakostech
High availability solutions bakostechHigh availability solutions bakostech
High availability solutions bakostech
Viktoria Bakos
 
Design patterns and plan for developing high available azure applications
Design patterns and plan for developing high available azure applicationsDesign patterns and plan for developing high available azure applications
Design patterns and plan for developing high available azure applications
Himanshu Sahu
 
High Availability Options for Modern Oracle Infrastructures
High Availability Options for Modern Oracle InfrastructuresHigh Availability Options for Modern Oracle Infrastructures
High Availability Options for Modern Oracle Infrastructures
Simon Haslam
 
Oracle Maximum Availability Architecture
Oracle Maximum Availability ArchitectureOracle Maximum Availability Architecture
Oracle Maximum Availability Architecture
MarketingArrowECS_CZ
 
SQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability Groups
SQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability GroupsSQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability Groups
SQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability Groups
turgaysahtiyan
 
D02 Evolution of the HADR tool
D02 Evolution of the HADR toolD02 Evolution of the HADR tool
D02 Evolution of the HADR tool
Jeyabarathi (JB) Chakrapani
 
HA & DR System Design - Concepts and Solution
HA & DR System Design - Concepts and SolutionHA & DR System Design - Concepts and Solution
HA & DR System Design - Concepts and Solution
Continuity and Resilience
 
DB2 V 10 HADR Multiple Standby
DB2 V 10 HADR Multiple StandbyDB2 V 10 HADR Multiple Standby
DB2 V 10 HADR Multiple Standby
Dale McInnis
 
High Availability and Disaster Recovery
High Availability and Disaster RecoveryHigh Availability and Disaster Recovery
High Availability and Disaster Recovery
Akelios
 
Oracle database high availability solutions
Oracle database high availability solutionsOracle database high availability solutions
Oracle database high availability solutions
Kirill Loifman
 
Linux Disaster Recovery Best Practices with rear
Linux Disaster Recovery Best Practices with rearLinux Disaster Recovery Best Practices with rear
Linux Disaster Recovery Best Practices with rear
Gratien D'haese
 
A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...
A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...
A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...
Symantec
 
High Availability (HA) Explained
High Availability (HA) ExplainedHigh Availability (HA) Explained
High Availability (HA) Explained
Maciej Lasyk
 
SharePoint Backup And Disaster Recovery with Joel Oleson
SharePoint Backup And Disaster Recovery with Joel OlesonSharePoint Backup And Disaster Recovery with Joel Oleson
SharePoint Backup And Disaster Recovery with Joel Oleson
Joel Oleson
 
Architecting for High Availability
Architecting for High AvailabilityArchitecting for High Availability
Architecting for High Availability
Amazon Web Services
 
Best Practices in Disaster Recovery Planning and Testing
Best Practices in Disaster Recovery Planning and TestingBest Practices in Disaster Recovery Planning and Testing
Best Practices in Disaster Recovery Planning and Testing
Axcient
 
AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...
AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...
AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...
Amazon Web Services
 

Viewers also liked (20)

Linux Disaster Recovery Solutions
Linux Disaster Recovery SolutionsLinux Disaster Recovery Solutions
Linux Disaster Recovery Solutions
 
DB2 High Availability für IBM Connections, Sametime oder Traveler
DB2 High Availability für IBM Connections, Sametime oder TravelerDB2 High Availability für IBM Connections, Sametime oder Traveler
DB2 High Availability für IBM Connections, Sametime oder Traveler
 
A05
A05A05
A05
 
High availability solutions bakostech
High availability solutions bakostechHigh availability solutions bakostech
High availability solutions bakostech
 
Design patterns and plan for developing high available azure applications
Design patterns and plan for developing high available azure applicationsDesign patterns and plan for developing high available azure applications
Design patterns and plan for developing high available azure applications
 
High Availability Options for Modern Oracle Infrastructures
High Availability Options for Modern Oracle InfrastructuresHigh Availability Options for Modern Oracle Infrastructures
High Availability Options for Modern Oracle Infrastructures
 
Oracle Maximum Availability Architecture
Oracle Maximum Availability ArchitectureOracle Maximum Availability Architecture
Oracle Maximum Availability Architecture
 
SQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability Groups
SQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability GroupsSQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability Groups
SQLSaturday Bulgaria : HA & DR with SQL Server AlwaysOn Availability Groups
 
D02 Evolution of the HADR tool
D02 Evolution of the HADR toolD02 Evolution of the HADR tool
D02 Evolution of the HADR tool
 
HA & DR System Design - Concepts and Solution
HA & DR System Design - Concepts and SolutionHA & DR System Design - Concepts and Solution
HA & DR System Design - Concepts and Solution
 
DB2 V 10 HADR Multiple Standby
DB2 V 10 HADR Multiple StandbyDB2 V 10 HADR Multiple Standby
DB2 V 10 HADR Multiple Standby
 
High Availability and Disaster Recovery
High Availability and Disaster RecoveryHigh Availability and Disaster Recovery
High Availability and Disaster Recovery
 
Oracle database high availability solutions
Oracle database high availability solutionsOracle database high availability solutions
Oracle database high availability solutions
 
Linux Disaster Recovery Best Practices with rear
Linux Disaster Recovery Best Practices with rearLinux Disaster Recovery Best Practices with rear
Linux Disaster Recovery Best Practices with rear
 
A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...
A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...
A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...
 
High Availability (HA) Explained
High Availability (HA) ExplainedHigh Availability (HA) Explained
High Availability (HA) Explained
 
SharePoint Backup And Disaster Recovery with Joel Oleson
SharePoint Backup And Disaster Recovery with Joel OlesonSharePoint Backup And Disaster Recovery with Joel Oleson
SharePoint Backup And Disaster Recovery with Joel Oleson
 
Architecting for High Availability
Architecting for High AvailabilityArchitecting for High Availability
Architecting for High Availability
 
Best Practices in Disaster Recovery Planning and Testing
Best Practices in Disaster Recovery Planning and TestingBest Practices in Disaster Recovery Planning and Testing
Best Practices in Disaster Recovery Planning and Testing
 
AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...
AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...
AWS re:Invent 2016: Disaster Recovery and Business Continuity for Systemicall...
 

Similar to Top 10 DB2 Support Nightmares #9

A DBA’s guide to using TSA
A DBA’s guide to using TSAA DBA’s guide to using TSA
A DBA’s guide to using TSA
Frederik Engelen
 
Top 10 DB2 Support Nightmares #7
Top 10 DB2 Support Nightmares  #7Top 10 DB2 Support Nightmares  #7
Top 10 DB2 Support Nightmares #7
Carol Davis-Mann
 
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
DataStax
 
URL to HTML
URL to HTMLURL to HTML
URL to HTML
Francois Marier
 
Ad disasters & how to prevent them
Ad disasters & how to prevent themAd disasters & how to prevent them
Ad disasters & how to prevent them
Concentrated Technology
 
Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)
Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)
Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)
VMware Tanzu
 
Oracle utilities in 11g R2
Oracle utilities in 11g R2Oracle utilities in 11g R2
Oracle utilities in 11g R2
Guenadi JILEVSKI
 
Dead Lock In Operating Systems
Dead Lock In Operating SystemsDead Lock In Operating Systems
Dead Lock In Operating Systems
totallooser
 
Oracle no sql release 3 4 overview
Oracle no sql release 3 4 overviewOracle no sql release 3 4 overview
Oracle no sql release 3 4 overview
Anand Chandak
 
TA-002-P.pdf
TA-002-P.pdfTA-002-P.pdf
TA-002-P.pdf
ssuserea9ab8
 
Oracle sharding : Installation & Configuration
Oracle sharding : Installation & ConfigurationOracle sharding : Installation & Configuration
Oracle sharding : Installation & Configuration
suresh gandhi
 
PHDAYS: DGAs and Threat Intelligence
PHDAYS: DGAs and Threat IntelligencePHDAYS: DGAs and Threat Intelligence
PHDAYS: DGAs and Threat Intelligence
John Bambenek
 
URL to HTML
URL to HTMLURL to HTML
URL to HTML
Francois Marier
 
DB2UDB_the_Basics Day 6
DB2UDB_the_Basics Day 6DB2UDB_the_Basics Day 6
DB2UDB_the_Basics Day 6
Pranav Prakash
 
Php melb cqrs-ddd-predaddy
Php melb cqrs-ddd-predaddyPhp melb cqrs-ddd-predaddy
Php melb cqrs-ddd-predaddy
Douglas Reith
 
CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012
CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012
CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012
Amazon Web Services
 
Hoya for Code Review
Hoya for Code ReviewHoya for Code Review
Hoya for Code Review
Steve Loughran
 
db2dart and inspect
db2dart and inspectdb2dart and inspect
db2dart and inspect
dbawork
 
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
James Chen
 
2013 11-19-hoya-status
2013 11-19-hoya-status2013 11-19-hoya-status
2013 11-19-hoya-status
Steve Loughran
 

Similar to Top 10 DB2 Support Nightmares #9 (20)

A DBA’s guide to using TSA
A DBA’s guide to using TSAA DBA’s guide to using TSA
A DBA’s guide to using TSA
 
Top 10 DB2 Support Nightmares #7
Top 10 DB2 Support Nightmares  #7Top 10 DB2 Support Nightmares  #7
Top 10 DB2 Support Nightmares #7
 
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
Troubleshooting Cassandra (J.B. Langston, DataStax) | C* Summit 2016
 
URL to HTML
URL to HTMLURL to HTML
URL to HTML
 
Ad disasters & how to prevent them
Ad disasters & how to prevent themAd disasters & how to prevent them
Ad disasters & how to prevent them
 
Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)
Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)
Cloud Foundry Summit 2015: Building a Robust Cloud Foundry (HA, Security and DR)
 
Oracle utilities in 11g R2
Oracle utilities in 11g R2Oracle utilities in 11g R2
Oracle utilities in 11g R2
 
Dead Lock In Operating Systems
Dead Lock In Operating SystemsDead Lock In Operating Systems
Dead Lock In Operating Systems
 
Oracle no sql release 3 4 overview
Oracle no sql release 3 4 overviewOracle no sql release 3 4 overview
Oracle no sql release 3 4 overview
 
TA-002-P.pdf
TA-002-P.pdfTA-002-P.pdf
TA-002-P.pdf
 
Oracle sharding : Installation & Configuration
Oracle sharding : Installation & ConfigurationOracle sharding : Installation & Configuration
Oracle sharding : Installation & Configuration
 
PHDAYS: DGAs and Threat Intelligence
PHDAYS: DGAs and Threat IntelligencePHDAYS: DGAs and Threat Intelligence
PHDAYS: DGAs and Threat Intelligence
 
URL to HTML
URL to HTMLURL to HTML
URL to HTML
 
DB2UDB_the_Basics Day 6
DB2UDB_the_Basics Day 6DB2UDB_the_Basics Day 6
DB2UDB_the_Basics Day 6
 
Php melb cqrs-ddd-predaddy
Php melb cqrs-ddd-predaddyPhp melb cqrs-ddd-predaddy
Php melb cqrs-ddd-predaddy
 
CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012
CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012
CPN208 Failures at Scale & How to Ride Through Them - AWS re: Invent 2012
 
Hoya for Code Review
Hoya for Code ReviewHoya for Code Review
Hoya for Code Review
 
db2dart and inspect
db2dart and inspectdb2dart and inspect
db2dart and inspect
 
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
 
2013 11-19-hoya-status
2013 11-19-hoya-status2013 11-19-hoya-status
2013 11-19-hoya-status
 

More from Laura Hood

Top 10 db2 support nightmares #6
Top 10 db2 support nightmares #6Top 10 db2 support nightmares #6
Top 10 db2 support nightmares #6
Laura Hood
 
Consultancy on Demand - Infographic
Consultancy on Demand - InfographicConsultancy on Demand - Infographic
Consultancy on Demand - Infographic
Laura Hood
 
A Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and BeyondA Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
Laura Hood
 
Top 10 DB2 Support Nightmares #1
Top 10 DB2 Support Nightmares  #1Top 10 DB2 Support Nightmares  #1
Top 10 DB2 Support Nightmares #1
Laura Hood
 
Db2 10 memory management uk db2 user group june 2013 [read-only]
Db2 10 memory management   uk db2 user group june 2013 [read-only]Db2 10 memory management   uk db2 user group june 2013 [read-only]
Db2 10 memory management uk db2 user group june 2013 [read-only]
Laura Hood
 
DB2 10 Security Enhancements
DB2 10 Security EnhancementsDB2 10 Security Enhancements
DB2 10 Security Enhancements
Laura Hood
 
DbB 10 Webcast #3 The Secrets Of Scalability
DbB 10 Webcast #3   The Secrets Of ScalabilityDbB 10 Webcast #3   The Secrets Of Scalability
DbB 10 Webcast #3 The Secrets Of Scalability
Laura Hood
 
DB2 10 Webcast #2 - Justifying The Upgrade
DB2 10 Webcast #2  - Justifying The UpgradeDB2 10 Webcast #2  - Justifying The Upgrade
DB2 10 Webcast #2 - Justifying The Upgrade
Laura Hood
 
DB2 10 Webcast #1 - Overview And Migration Planning
DB2 10 Webcast #1 - Overview And Migration PlanningDB2 10 Webcast #1 - Overview And Migration Planning
DB2 10 Webcast #1 - Overview And Migration Planning
Laura Hood
 
Time Travelling With DB2 10 For zOS
Time Travelling With DB2 10 For zOSTime Travelling With DB2 10 For zOS
Time Travelling With DB2 10 For zOS
Laura Hood
 
DB2DART - DB2Night Show October 2011
DB2DART - DB2Night Show October 2011DB2DART - DB2Night Show October 2011
DB2DART - DB2Night Show October 2011
Laura Hood
 
DB2 z/OS & Java - What\'s New?
DB2 z/OS & Java - What\'s New?DB2 z/OS & Java - What\'s New?
DB2 z/OS & Java - What\'s New?
Laura Hood
 
Temporal And Other DB2 10 For Z Os Highlights
Temporal And Other DB2 10 For Z Os HighlightsTemporal And Other DB2 10 For Z Os Highlights
Temporal And Other DB2 10 For Z Os Highlights
Laura Hood
 
DB210 Smarter Database IBM Tech Forum 2011
DB210 Smarter Database   IBM Tech Forum 2011DB210 Smarter Database   IBM Tech Forum 2011
DB210 Smarter Database IBM Tech Forum 2011
Laura Hood
 
UKGSE DB2 pureScale
UKGSE DB2 pureScaleUKGSE DB2 pureScale
UKGSE DB2 pureScale
Laura Hood
 
UKCMG DB2 pureScale
UKCMG DB2 pureScaleUKCMG DB2 pureScale
UKCMG DB2 pureScale
Laura Hood
 
Episode 4 DB2 pureScale Performance Webinar Oct 2010
Episode 4 DB2 pureScale Performance Webinar Oct 2010Episode 4 DB2 pureScale Performance Webinar Oct 2010
Episode 4 DB2 pureScale Performance Webinar Oct 2010
Laura Hood
 
Episode 3 DB2 pureScale Availability And Recovery [Read Only] [Compatibility...
Episode 3  DB2 pureScale Availability And Recovery [Read Only] [Compatibility...Episode 3  DB2 pureScale Availability And Recovery [Read Only] [Compatibility...
Episode 3 DB2 pureScale Availability And Recovery [Read Only] [Compatibility...
Laura Hood
 
Episode 2 Installation Triton Slides
Episode 2 Installation Triton SlidesEpisode 2 Installation Triton Slides
Episode 2 Installation Triton Slides
Laura Hood
 
Episode 2 DB2 pureScale Installation, Instance Management & Monitoring
Episode 2 DB2 pureScale Installation, Instance Management & MonitoringEpisode 2 DB2 pureScale Installation, Instance Management & Monitoring
Episode 2 DB2 pureScale Installation, Instance Management & Monitoring
Laura Hood
 

More from Laura Hood (20)

Top 10 db2 support nightmares #6
Top 10 db2 support nightmares #6Top 10 db2 support nightmares #6
Top 10 db2 support nightmares #6
 
Consultancy on Demand - Infographic
Consultancy on Demand - InfographicConsultancy on Demand - Infographic
Consultancy on Demand - Infographic
 
A Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and BeyondA Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
A Time Traveller's Guide to DB2: Technology Themes for 2014 and Beyond
 
Top 10 DB2 Support Nightmares #1
Top 10 DB2 Support Nightmares  #1Top 10 DB2 Support Nightmares  #1
Top 10 DB2 Support Nightmares #1
 
Db2 10 memory management uk db2 user group june 2013 [read-only]
Db2 10 memory management   uk db2 user group june 2013 [read-only]Db2 10 memory management   uk db2 user group june 2013 [read-only]
Db2 10 memory management uk db2 user group june 2013 [read-only]
 
DB2 10 Security Enhancements
DB2 10 Security EnhancementsDB2 10 Security Enhancements
DB2 10 Security Enhancements
 
DbB 10 Webcast #3 The Secrets Of Scalability
DbB 10 Webcast #3   The Secrets Of ScalabilityDbB 10 Webcast #3   The Secrets Of Scalability
DbB 10 Webcast #3 The Secrets Of Scalability
 
DB2 10 Webcast #2 - Justifying The Upgrade
DB2 10 Webcast #2  - Justifying The UpgradeDB2 10 Webcast #2  - Justifying The Upgrade
DB2 10 Webcast #2 - Justifying The Upgrade
 
DB2 10 Webcast #1 - Overview And Migration Planning
DB2 10 Webcast #1 - Overview And Migration PlanningDB2 10 Webcast #1 - Overview And Migration Planning
DB2 10 Webcast #1 - Overview And Migration Planning
 
Time Travelling With DB2 10 For zOS
Time Travelling With DB2 10 For zOSTime Travelling With DB2 10 For zOS
Time Travelling With DB2 10 For zOS
 
DB2DART - DB2Night Show October 2011
DB2DART - DB2Night Show October 2011DB2DART - DB2Night Show October 2011
DB2DART - DB2Night Show October 2011
 
DB2 z/OS & Java - What\'s New?
DB2 z/OS & Java - What\'s New?DB2 z/OS & Java - What\'s New?
DB2 z/OS & Java - What\'s New?
 
Temporal And Other DB2 10 For Z Os Highlights
Temporal And Other DB2 10 For Z Os HighlightsTemporal And Other DB2 10 For Z Os Highlights
Temporal And Other DB2 10 For Z Os Highlights
 
DB210 Smarter Database IBM Tech Forum 2011
DB210 Smarter Database   IBM Tech Forum 2011DB210 Smarter Database   IBM Tech Forum 2011
DB210 Smarter Database IBM Tech Forum 2011
 
UKGSE DB2 pureScale
UKGSE DB2 pureScaleUKGSE DB2 pureScale
UKGSE DB2 pureScale
 
UKCMG DB2 pureScale
UKCMG DB2 pureScaleUKCMG DB2 pureScale
UKCMG DB2 pureScale
 
Episode 4 DB2 pureScale Performance Webinar Oct 2010
Episode 4 DB2 pureScale Performance Webinar Oct 2010Episode 4 DB2 pureScale Performance Webinar Oct 2010
Episode 4 DB2 pureScale Performance Webinar Oct 2010
 
Episode 3 DB2 pureScale Availability And Recovery [Read Only] [Compatibility...
Episode 3  DB2 pureScale Availability And Recovery [Read Only] [Compatibility...Episode 3  DB2 pureScale Availability And Recovery [Read Only] [Compatibility...
Episode 3 DB2 pureScale Availability And Recovery [Read Only] [Compatibility...
 
Episode 2 Installation Triton Slides
Episode 2 Installation Triton SlidesEpisode 2 Installation Triton Slides
Episode 2 Installation Triton Slides
 
Episode 2 DB2 pureScale Installation, Instance Management & Monitoring
Episode 2 DB2 pureScale Installation, Instance Management & MonitoringEpisode 2 DB2 pureScale Installation, Instance Management & Monitoring
Episode 2 DB2 pureScale Installation, Instance Management & Monitoring
 

Recently uploaded

Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Zilliz
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Pixlogix Infotech
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
TIPNGVN2
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neo4j
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
mikeeftimakis1
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Adtran
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 

Recently uploaded (20)

Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
 
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024GraphSummit Singapore | The Art of the  Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 daysPushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 

Top 10 DB2 Support Nightmares #9

  • 1. Top 10 DB2 SupportTop 10 DB2 SupportTop 10 DB2 SupportTop 10 DB2 Support Nightmares & How toNightmares & How toNightmares & How toNightmares & How to Avoid ThemAvoid ThemAvoid ThemAvoid Them #9#9#9#9
  • 2. Part 9 – In the event of an emergency call The situation Image of a junior DBA A customer was running HADR for disaster recovery. They had no cluster software used for monitoring or failover. HADR was being monitored on a regular basis using a shell script.
  • 3. Then what happened…? Disk failure on the primary site! Some tablespaces were put into “Rollforward Pending” state. Transactions accessing data in these tablespaces failed
  • 4. Image of a junior DBA The last run of the HADR state monitoring script indicated a Peer State so it was decided to issue TAKEOVER command on the DR site to switch roles. When the application started, some transactions failed with the same error as on the primary site. List tablespaces command showed a number of tables in Rollforward Pending state. To get out of the pending state, ROLLFORWARD command was issued with the list of affected tablespaces. The rollforward was trying to retrieve a log, which was a few thousand logs older than the current one. After a few tries the ROLLFORWARD was given up. The database was restored from the latest backup image
  • 5. • We went through the db2diag.log and the notification logs. We could see that there were physical errors reported in some of the tablespaces on the DR site around 100 days prior to the incident. This was reported in the db2diag.log and the affected tablespaces were “excluded from the rollforward set” • Based on other entries in the db2diag file, we were able to confirm that the log file requested for rollforward on the DR site was used at the time the physical errors occurred there. • HADR continued to apply logs for the other tablespaces and was reporting to be in “Peer” State. In reality, some of the tablespaces were being ignored. Triton Analysis
  • 6. Regular monitoring on the log files is essential to identify and resolve the issue on the DR site in advance of this incident. Make sure you know who to call before disaster strikes! Triton Consulting +44 870 2411 550 www.triton.co.uk The moral of the story