The document discusses database recovery techniques. It covers the purpose of database recovery which is to restore the database to its last consistent state prior to failure. It describes different types of failures and how transaction logs are used to store before and after images of data to enable rollback and redo during recovery. It also discusses checkpointing to minimize recovery processing and different recovery schemes for deferred and immediate data updating.
What is Database Backup? The 3 Important Recovery Techniques from transaction...Raj vardhan
What is Database Backup?
What is Database recovery techniques
Why recovery is needed? (What causes a Transaction to fail?)
The 3 Important Recovery Techniques from transaction failures:
The figure below illustrates the use of Shadow paging techniques:
What is Database Backup? The 3 Important Recovery Techniques from transaction...Raj vardhan
What is Database Backup?
What is Database recovery techniques
Why recovery is needed? (What causes a Transaction to fail?)
The 3 Important Recovery Techniques from transaction failures:
The figure below illustrates the use of Shadow paging techniques:
This presentation discusses the following topics:
What is Recovery ?
Database Recovery techniques
System log
Working of Commit and Roll back
Recovery techniques
Backup techniques
Introduction to transaction processing concepts and theoryZainab Almugbel
Modified version of Chapter 21 of the book Fundamentals_of_Database_Systems,_6th_Edition with review questions
as part of database management system course
Adbms 34 transaction processing and recoveryVaibhav Khanna
A transaction is an atomic unit of work that is either completed in its entirety or not done at all.
– For recovery purposes, the system needs to keep track of when the transaction starts, terminates, and commits or aborts.
Transaction states:
– Active state
– Partially committed state
– Committed state
– Failed state
– Terminated State
Transaction Processing; Concurrency control; ACID properties; Schedule and Discoverability; Serialization; Concurrency control and Recovery; Two Phase locking; Deadlock Shadow Paging
This presentation discusses the following topics:
What is Recovery ?
Database Recovery techniques
System log
Working of Commit and Roll back
Recovery techniques
Backup techniques
Introduction to transaction processing concepts and theoryZainab Almugbel
Modified version of Chapter 21 of the book Fundamentals_of_Database_Systems,_6th_Edition with review questions
as part of database management system course
Adbms 34 transaction processing and recoveryVaibhav Khanna
A transaction is an atomic unit of work that is either completed in its entirety or not done at all.
– For recovery purposes, the system needs to keep track of when the transaction starts, terminates, and commits or aborts.
Transaction states:
– Active state
– Partially committed state
– Committed state
– Failed state
– Terminated State
Transaction Processing; Concurrency control; ACID properties; Schedule and Discoverability; Serialization; Concurrency control and Recovery; Two Phase locking; Deadlock Shadow Paging
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
Data Centers - Striving Within A Narrow Range - Research Report - MCG - May 2...pchutichetpong
M Capital Group (“MCG”) expects to see demand and the changing evolution of supply, facilitated through institutional investment rotation out of offices and into work from home (“WFH”), while the ever-expanding need for data storage as global internet usage expands, with experts predicting 5.3 billion users by 2023. These market factors will be underpinned by technological changes, such as progressing cloud services and edge sites, allowing the industry to see strong expected annual growth of 13% over the next 4 years.
Whilst competitive headwinds remain, represented through the recent second bankruptcy filing of Sungard, which blames “COVID-19 and other macroeconomic trends including delayed customer spending decisions, insourcing and reductions in IT spending, energy inflation and reduction in demand for certain services”, the industry has seen key adjustments, where MCG believes that engineering cost management and technological innovation will be paramount to success.
MCG reports that the more favorable market conditions expected over the next few years, helped by the winding down of pandemic restrictions and a hybrid working environment will be driving market momentum forward. The continuous injection of capital by alternative investment firms, as well as the growing infrastructural investment from cloud service providers and social media companies, whose revenues are expected to grow over 3.6x larger by value in 2026, will likely help propel center provision and innovation. These factors paint a promising picture for the industry players that offset rising input costs and adapt to new technologies.
According to M Capital Group: “Specifically, the long-term cost-saving opportunities available from the rise of remote managing will likely aid value growth for the industry. Through margin optimization and further availability of capital for reinvestment, strong players will maintain their competitive foothold, while weaker players exit the market to balance supply and demand.”
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
2. Outline
1.Purpose of Database Recovery
2.Types of Failure
3. Transaction Log
4. Data Updates
5.Data Caching
6.Transaction Roll-back (Undo) and Roll-
Forward(Redo)
7.Checkpointing
8.Recovery schemes
9.Recovery in Multidatabase System
Outline
2
3. 1 Purpose of Database Recovery
–To bring the database into the last consistent state,
which existed prior to the failure.
–To preserve transaction properties (Atomicity,
Consistency, Isolation and Durability).
• Example:
–If the system crashes before a fund transfer transaction
completes its execution, then either one or both
accounts may have incorrect value. Thus, the database
must be restored to the state before the transaction
modified any of the accounts.
Database Recovery
3
4. 2 Types of Failure
– The database may become unavailable for use due to
• Transaction failure: Transactions may fail because of
incorrect input, deadlock, incorrect synchronization.
• System failure: System may fail because of addressing
error, application error, operating system fault, RAM
failure, etc.
• Media failure: Disk head crash, power disruption,
etc.
• Concurrency control enforcements(e.g. deadlock)
• Local errors or exceptions
Database Recovery(Cont.…)
4
5. Database Recovery
3 Transaction Log
– For recovery from any type of failure data values prior to
modification (BFIM - BeFore Image) and the new value after
modification (AFIM –AFter Image) are required.
– These values and other information is stored in a sequential file
calledTransaction log. A sample log is given below.
Database Recovery(Cont.…)
5
6. Database Recovery
4 Data Update
– Immediate Update: As soon as a data item is modified in cache, the
disk copy is updated.
-Allows updates of an uncommitted transaction to be made to the
buffer, or the disk itself, before the transaction commits
– Deferred Update: All modified data items in the cache is written either
after a transaction ends its execution or after a fixed number of
transactions have completed their execution.
o performs updates to buffer/disk only at the time of transaction
commit.
- Simplifies some aspects of recovery
- But has overhead of storing local copy
– Shadow update: The modified version of a data item does not
overwrite its disk copy but is written at a separate disk location.
– In-place update:The disk version of the data item is overwritten by
the cache version.
Database Recovery(Cont.…)
6
7. Database Recovery
5 Data Caching
–Data items to be modified are first stored into database
cache by the Cache Manager (CM).
–After modification, they are flushed (written) to the
disk.
–The flushing is controlled by dirty and Pin-Unpin
bits.
• Dirty bits=1: Indicates that the data item is
modified.
• Pin-Unpin bits: a page in cache is pinned (bit
value=1) if it can not be written back to disk as yet.
Database Recovery(Cont.…)
7
8. Database Recovery
Steal/No-Steal and Force/No-Force recovery protocol
– Possible ways for flushing database cache to database disk:
1. Steal: Cache can be flushed before transaction commits.
o It avoids the need for a very large buffer space to store updated
pages in memory
2. No-Steal: Cache cannot be flushed before transaction commit.
3. Force: pages updated by a transaction are immediately written to
disk before the transaction commits.
- REDO will never be needed during recovery.
4. No-Force: An updated page of a committed transaction may still be
in the buffer when another transaction needs to update it
-This eliminate the I/O cost to read that page again from disk
– These give rise to four different ways for handling recovery:
• Steal/Force (Undo/No-redo)
• Steal/No-Force (Undo/Redo)
• No-Steal/No-Force (No-undo/Redo)
• No-Steal/Force (No-undo/No-redo)
Database Recovery(Cont.…)
8
9. Database Recovery
6 Transaction Roll-back (Undo) and Roll-Forward (Redo)
• To maintain atomicity, a transaction’s operations are redone or
undone.
– Undo: Restore all BFIMs on to disk (Remove all AFIMs).
– Redo: Restore all AFIMs on to disk.
– Database recovery is achieved either by performing only Undos or only
Redos or by a combination of the two.
• Undo and Redo ofTransactions
– undo(Ti) restores the value of all data items updated byTi to their
old values, going backwards from the last log record forTi
o each time a data item X is restored to its old valueV a special log record
<Ti , X,V> is written out.
o when undo of a transaction is complete, a log record <Ti abort> is
written out.
– redo(Ti) sets the value of all data items updated byTi to the new
values, going forward from the first log record forT
Database Recovery(Cont.…)
9
10. • When recovering after failure:
o TransactionTi needs to be undone if the log
▪ Contains the record <Ti start>,
▪ but does not contain either the record <Ti commit> or <Ti
abort>.
o TransactionTi needs to be redone if the log
▪ Ccontains the records <Ti start>
▪ and contains the record <Ti commit> or <Ti abort>
• Note that If transaction Ti was undone earlier and the <Ti abort> record
written to the log, and then a failure occurs, on recovery from failure Ti is
redone.
o Such a redo redoes all the original actions including the steps that
restored old values
▪ Known as repeating history.
▪ Seems wasteful, but simplifies recovery greatly. 10
Undo and Redo on Recovering from Failure
11. Below we show the log as it appears at three instances of time.
Recovery actions in each case above are:
(a) undo (T0): B is restored to 2000 and A to 1000, and log records
<T0, B, 2000>, <T0,A, 1000>, <T0, abort> are written out
(b) redo (T0) and undo (T1): A and B are set to 950 and 2050 and C is
restored to 700. Log records <T1, C, 700>, <T1, abort> are
written out.
(c) redo (T0) and redo (T1):A and B are set to 950 and 2050
respectively.Then C is set to 600
Immediate DB Modification Recovery Example
12. The read and write operations of three transactions
Database Recovery(Cont.…)
12
14. Write-Ahead Logging
• When in-place update (immediate or deferred) is used then log is
necessary for recovery and it must be available to recovery
manager. This is achieved by Write-Ahead Logging (WAL)
protocol.
– WAL states that
• For Undo: Before a data item’sAFIM is flushed to the
database disk (overwriting the BFIM) its BFIM must be
written to the log and the log must be saved on a stable store
(log disk).
• For Redo: Before a transaction executes its commit
operation, all itsAFIMs must be written to the log and the
log must be saved on a stable store.
Database Recovery(Cont.…)
14
15. Database Recovery
7 Checkpointing
• Redoing/undoing all transactions recorded in the log can be very slow
o Processing the entire log is time-consuming if the system has run for a long time.
o We might unnecessarily redo transactions which have already output their updates to the
database.
– Randomly or under some criteria, the database flushes its buffer to database disk to
minimize the task of recovery. The following steps defines a checkpoint operation:
1. Suspend execution of transactions temporarily.
2. Force write modified buffer data to disk.
3. Write a [checkpoint] record to the log, save the log to disk.
4. Resume normal transaction execution.
• During recovery redo or undo is required to transactions appearing after
[checkpoint] record.
• During recovery we need to consider only the most recent transaction Ti that
started before the checkpoint, and transactions that started after Ti.
• Transactions that committed or aborted before the checkpoint already have all
their updates output to stable storage.
Database Recovery(Cont.…)
15
16. o T1 can be ignored (updates already output to disk due to
checkpoint)
oT2 and T3 redone.
oT4 undone
Tc
Tf
T1
T2
T3
T4
Checkpoint System failure
Example of Checkpoints
18. 8 Recovery Scheme
• For Deferred Update (No Undo/Redo)
–The data update goes as follows:
• A set of transactions record their updates in the log.
• At commit point underWAL scheme, these updates
are saved on database disk.
• After reboot from a failure the log is used to redo all
the transactions affected by this failure.
• No undo is required because noAFIM is flushed to
the disk before a transaction commits.
Database Recovery(Cont.…)
18
20. Database Recovery
Deferred Update with concurrent users
• This environment requires some concurrency control mechanism
to guarantee isolation property of transactions. In the system
recovery, transactions which were recorded in the log after the last
checkpoint were redone. The recovery manager may scan some
of the transactions recorded before the checkpoint to get the
AFIMs.
Database Recovery(Cont.…)
20
22. Deferred Update with concurrent users
• Two tables are required for implementing this
protocol:
–Active table: All active transactions are
entered in this table.
–Commit table: Transactions to be committed
are entered in this table.
• During recovery, all transactions of the commit
table are redone and all transactions of active
tables are ignored since none of their AFIMs
reached the database.
Database Recovery(Cont.…)
22
23. RecoveryTechniques Based on Immediate Update
• Undo/No-redo Algorithm
–In this algorithm AFIMs of a transaction are
flushed to the database disk under WAL
before it commits.
–For this reason the recovery manager
undoes all transactions during recovery.
–No transaction is redone.
Database Recovery(Cont.…)
23
24. RecoveryTechniques Based on Immediate Update
– Undo/Redo Algorithm
• Recovery schemes of this category apply undo and also
redo for recovery.
• In a single-user environment no concurrency control is
required but a log is maintained underWAL.
• Note that at any time there will be one transaction in the
system and it will be either in the commit table or in the
active table.
• The recovery manager performs:
–Undo of a transaction if it is in the active table.
–Redo of a transaction if it is in the commit table.
Database Recovery(Cont.…)
24
25. Database Recovery
Shadow Paging
• This recovery scheme does not require the use of a log in a
single user environment.
• The AFIM does not overwrite its BFIM but recorded at
another place on the disk.
• Thus, at any time a data item has AFIM and BFIM (Shadow
copy of the data item) at two different places on the disk.
X Y
Database
X' Y'
X and Y: Shadow copies of data items
X' and Y': Current copies of data items
Database Recovery(Cont.…)
25
26. Shadow Paging
• To manage access of data items by concurrent transactions,
two directories (current and shadow) are used.
–The directory arrangement is illustrated below. Here a
page is a data item.
An example of shadow paging
Database Recovery(Cont.…)
26
27. Database Recovery
• A multidatabase system is a special distributed database system where
one node may be running relational database system under UNIX,
another may be running object-oriented system under Windows and so
on.
• Databases may even be stored on different types of DBMSs; for
example, some DBMSs may be relational, whereas others are object-
oriented, hierarchical, or network DBMSs.
• A transaction may run in a distributed fashion at multiple nodes.
• In this execution scenario, the transaction commits only when all these
multiple nodes agree to commit individually the part of the transaction
they were executing.
• This commit scheme is referred to as “two-phase commit” (2PC).
– If any one of these nodes fails or cannot commit the part of the
transaction, then the transaction is aborted.
• Each node recovers the transaction under its own recovery protocol.
Recovery in multidatabase system
27
28. • To maintain the atomicity of multidatabase transaction(MDT) , it is
necessary to have a two level recovery mechanism.
• A global recovery manager or coordinator is needed to maintain
information needed for recovery
• The coordinator usually follows a Two-phase commit protocol which
can be explained as follows:
• Phase1: when all participating databases signal the coordinator that the
part of the MDT involving each has concluded, the coordinator sends a
“prepare for commit” message to each participant to get ready for
committing the transaction
– Each participating DB receiving that message will force write log
records to disk and send “ready to commit” or “OK” signal to the
coordinator .
– If the coordinator does not receive reply from a DB within certain
time out interval, it assumes a “not ok” response
Recovery in multidatabase system(cont.…)
28
29. • Phase 2: If all participating databases reply ok, the transaction
is successful and the coordinator sends a commit signal to
the participating DBs
– Because all the local effects of the transaction and
information needed for local recovery is recorded in the
logs of participating DBs, recovery from failure is now
possible.
– Each participating DB completes transaction commit by
writing a [commit] for the transaction in the log
– If one or more of the participating DB or the coordinator
have a “not OK” response, the T has failed and the
coordinator sends a message to rollback or Undo the local
effect of the transaction to each participating DB.
Recovery in multidatabase system(cont…)
29
30. • Consider the log records shown on the next slide
by transactions T1, T2, T3 and T4 with initial values
of B=15, C=50, D=40 and E=25. Using deferred
update, show the final values of B, C, D and E after
recovery from failure if the crash occurred after the
indicated point.
o Which transactions are rolled back?
oWhich operations in the log are redone and
which (if any) are undone?
Exercise
30
32. • All the techniques we have discussed apply to
noncatastrophic failures.
• The system log(or the shadow directory) is maintained on
the disk and is not lost as a result of the failure.
• The recovery manager of a DBMS must also be equipped
to handle more catastrophic failures such as disk crashes.
• The main technique used to handle such crashes is a
database backup.
32
Database Recovery from Catastrophic Failures
Read More about Recovery from Catastrophic Failures?