This document discusses the file and buffer manager components of a database system. It describes how the file manager provides abstractions like device, allocation, and address independence. It also explains how the buffer manager caches data from external storage in main memory using techniques like logging, locking, and checkpointing to ensure data integrity and concurrency control during transaction processing. Key aspects of the buffer manager covered include its interface, replacement policies, and how it supports recovery through logging and maintaining page-level write-ahead logs.

1. 
Aug. 2 Aug. 3 Aug. 4 Aug. 5 Aug. 6
9:00 Intro &
terminology
TP mons
& ORBs
Logging &
res. Mgr.
Files &
Buffer Mgr.
Structured
files
11:00 Reliability Locking
theory
Res. Mgr. &
Trans. Mgr.
COM+ Access paths
13:30 Fault
tolerance
Locking
techniques
CICS & TP
& Internet
CORBA/
EJB + TP
Groupware
15:30 Transaction
models
Queueing Advanced
Trans. Mgr.
Replication Performance
& TPC
18:00 Reception Workflow Cyberbricks Party FREE
Files and Buffer Manager
Chapter 15

2. 
Abstractions Provided by the File
Manager
Device independence: The file manager turns the large
variety of external storage devices, such as disks (with
their different numbers of cylinders, tracks, arms, and
read/write heads), ram-disks, tapes, and so on, into
simple abstract data types.
Allocation independence: The file manager does its
own space management for storing the data objects
presented by the client. It may store the same objects in
more than one place (replication).

3. 
Abstractions Provided by the File
Manager
Address independence: Whereas objects in main
memory are always accessed through their addresses,
the file manager provides mechanisms for associative
access. Thus, for example, the client can request
access to all records with a specified value in some
field of the record. Support for associative access
comes in many flavors, from simple mechanisms
yielding fast retrieval via the primary key up to the
expressive power of the SQL select statement.

4. 
External Storage vs. Main Memory
Capacity: Main memory is usually limited to a size that
is some orders of magnitude smaller than what large
databases need.
Economics: External storage holds large volumes of
data at reasonable cost.
Durability: Main memory is volatile. External storage
devices such as magnetic or optical disks are
inherently durable and therefore are appropriate for
storing persistent objects. After a crash, recovery
starts with what is found in durable storage.

5. 
External Storage vs. Main Memory
Speed: External storage devices are some orders of
magnitude slower than main memory. As a result, it is
more costly, both in terms of latency and in terms of
pathlength, to get data from external storage to the CPU
than to load data from main memory.
Functionality: Data cannot be processed directly on
external storage: they can neither be compared nor
modified “out there.”

6. 
The Storage Pyramid
main
memory
online
external
storage
near line
(archive)
storage
typical capacity
Electronic RAM
and bulk
storage
Magnetic
/ optical
disks
Automated archives
(e.g. optical disk
jukeboxes, tape
robots, etc.)
current data
stale
data

7. 
Interfacing to External Memory:
Read-Write Mapping
ExternalStorage
FileA
FileB
FileC
FileD
MainMemory
readobjectw readobjectyreadobjectx
/writeobjectx

8. 
Interfacing to External Memory:
File Mapping
ExternalStorage
FileA
FileB
FileC
FileD
MainMemory FileC
mapFileC
unmapFileC

9. 
Interfacing to External Memory:
Single-Level Storage
External Storage
File A
File B
File C
File D
Main Memory
File A File B File C File D
Virtual
memory
Explicit mapping

10. 
Locality and Cacheing
The movement of data through the pyramid is guided by
the principle of locality:
Locality of active data: Data that have recently been
referenced will very likely be referenced again.
Locality of passive data: Data that have not been
referenced recently will most likely not be referenced in
the future.

11. 
Levels of Abstraction in a File and
Database Manager
main
memory
online
external
memory
nearline
external
memory
DBMS
Application
database
access
modules
databaseb
uffer mgr.
logging
recovery
media
and file
manager
archive
manager
Transaction
programs
Tuple
management,
associative
access
Buffer management
File management
Archive management
manages
manages
manages
tuple
oriented
access
block
oriented
access
device
oriented
access
setoriented
access
Application
sort, join,...
read, write

12. 
Operations of the Basic File System
STATUS create(filename, allocparmp)
STATUS delete(filename)
STATUS open(filename, ACCESSMODE, FILEID);
STATUS close(FILEID)
STATUS extend(FILEID, allocparmp)
STATUS read(FILEID, BLOCKID, BLOCKP)
STATUS readc(FILEID, BLOCKID, blockcount,
BLOCKP)
STATUS write(FILEID, BLOCKID, BLOCKP)
STATUS writec(FILEID, BLOCKID, blockcount,
BLOCKP)

13. 
Mapping Files To Disk
File A
File B
File C
File D
File E
Disk
1
The denote the
address
mapping
between
disk and
files.

14. 
Issues in Managing Disk Space
Initial allocation: When a file is created, how
many contiguous slots should be allocated to
it?
Incremental expansion: If an existing file grows
beyond the number of slots currently allocated,
how many additional contiguous blocks should
be assigned to that file?
Reorganization: When and how should the
free space on the disk be reorganized?

15. 
Extent-Based Allocation
D i s k - A D i s k - B D i s k - A D i s k - Ad i s k - i d
e x t e n t i n d e x
a c c u m - l e n g t h
1 4 1 8 7 3 2 1 4
1 0 0 3 5 0 6 0 0 8 5 0
p r i m a r y 1 . s e c o n d a r y 2 . s e c o n d a r y 3 . s e c o n d a r y
f i l e d i r e c t o r y
A Be x t e n t d i r e c t o r y

16. 
Buddy Systems
Slots (shadedextents are free)
Free
extents
Type
0
1
2
3
0110
0010
1100
1011
Buddytypes
0
0
0
0
0
0
0
1
0
0
1
0
0
0
1
1
0
1
0
0
0
1
0
1
0
1
1
0
0
1
1
1
1
0
0
0
1
0
0
1
1
0
1
0
1
0
1
1
1
1
0
0
1
1
0
1
1
1
1
0
1
1
1
1
0
1
2
3

17. 
Simple Mapping of Relations To Disks
filesystem&operatingsystem
databasesystem
realdisks
Filesystem
segments
relations
extents

18. 
A Usual Way of Mapping of Relations
To Disks
operating system database system
real disks logical disks OS-files
tabelspaces
segments
relations
extents

19. 
Principles of the Database Buffer
process of access
module
buffer
storage
area
buffer is accessible
fromthecaller's
process
(sharedmemory)
buffer
manager
interface
readdirect
directory
bufferfix(P, ...)
Giveme
pageP
findframein
buffer
determine
FILEIDand
blocknumber
return
frame
address F
1
23
4
5

20. 
Design Options for the Buffer
Manager
Buffer per file: Each file has its own private buffer pool..
Buffer per page size: In systems with different page
(and block) sizes, there is usually at least one buffer for
each page size.
Buffer per file type: There are files like indices, which
are accessed in a significantly different way from other
files. Therefore, some systems dedicate buffers to files
depending on the access pattern and try to manage
each of them in a way that is optimal for the respective
file organization.

21. 
Logic of the Buffer Manager
Search in buffer: Check if the requested page is
in the buffer. If found, return the address F of
this frame to the caller.
Find free frame: If the page is not in the buffer,
find a frame that holds no valid page.
Determine replacement victim: If no such frame
exists, determine a page that can be removed
from the buffer (in order to reuse its frame).

22. 
Logic of the Buffer Manager
Write modified page: If replacement page has
been changed, write it.
Establish frame address: Denote the start
address of the frame as F.
Determine block address: Translate the
requested PAGEID P into a FILEID and a block
number. Read the block into the frame
selected.
Return: Return the frame address F to the
caller.

23. 
Synchronization in the Buffer
process A process B process A process B
bufferfix (P,...) bufferfix (P,...)
give copy
to caller
give copy
to caller
change P to P' change P to P"
rewrite page rewrite page
?
a) Access module in
process Arequests
access to page P;
gets private copy.
b) Access module in
process Brequests
access to page P;
gets private copy.
c) Both processes try to rewrite
an updated version of the page,
but these versions are different.
Only the version written last
will be on disk; this is the "lost
update" anomaly.

24. 
What the Buffer Manager Does for
Synchronization
Sharing: Pages are made addressable to all
processes that run the database code.
Semaphore protection: Each requestor gets
the address of a semaphor protecting the page.
Durable storage: The access modules inform
the buffer manager if their page access has
resulted in an update of the page; the actual
write operation, however, is issued by the buffer
manager, probably at a time when the update
transaction is long gone.

25. 
The Interface to the Buffer Manager
typedef struct
{PAGEID pageid; /* id of page in file */
PAGEPTR pageaddr; /* base addr. in buffer */
int index; /* record within page
*/
semaphore * pagesem; /* pointer to the sem. */
Boolean modified; /* caller modif. page
*/
Boolean invalid; /* destroyed page */
} BUFFER_ACC_CB, *BUFFER_ACC_CBP;
/* control block for buffer access */

26. 
The Need for Fix and Unfix
transaction X
bufferfix(P,...)
page P
not
in buffer
read block
containing
page P
return base
address in
buffer
transaction Y transaction Y
bufferfix(Q,...)
page Q not
in buffer;
replace P
read block
containing
page Q
bufferfix(Q,...)
return base
address in
buffer
a) Transaction X
requests access
to page P; gets
base address in buffer.
b) Transaction Y
requests access
to page Q; buffer
mgr. decides to
replace page P
c) Transaction Y
gets the base
address of Q in
the buffer - is
same as P's.

27. 
The Fix-Use-Unfix Protocol I
FIX: The client requests access to a page
using the bufferfix interface.
USE: The client uses the page and the pointer
to the frame containing the page will remain
valid.
UNFIX: The client explicitly waives further
usage of the frame pointer; that is, it tells the
buffer manager that it no longer wants to use
that page.

28. 
The Fix-Use-Unfix Protocol II
page P
page Q
page R
fix page P
use
use
unfix page P
fix page R
use
use
use
unfix page R
fix page Quse
use
use
unfix page Q

29. 
Structure of the Buffer Manager
bufferpool
frames
hash table
buffer
control
block
buffer
control
block
buffer
control
block
buffer
control
block
buffer
control
block
buffer
control
block
p
a
g
e
s
frame_index first_bcb
next_in_hclass
mru_pagelru_page
buffer
access
control
block
to and from client
index of frame holding the page (address pointer in
case of buffer access control block)
chain of buffer control blocks in same hash class
LRU - chain

30. 
Logging and Recovery from the
Buffer Manager's Perspective I
Transaction Buffer Database Remark
running
running
running
running
committed
committed
committed
committed
A BA B OK; old state in DB
BA A B OK; old state in DB
BA A B database corrupted
BA BA conflicting view on TA
A BA B
BA A B
BA A B
BA BA
OK; Read-only TA
DB not in new state
database corrupted
OK; new state in DB

31. 
Logging and Recovery from the
Buffer Manager's Perspective II
state of
transaction
TA
aborted
aborted
committed
committed
state of
page A in
database
old
new
new
result of recovery
using operation log
wrong tuple might be deleted
old
inverse operation succeeds
operation succeeds
duplicate of tuple is inserted

32. 
The Log and Page LSNs
b u f f e r
m a n a g e r
l o g
m a n a g e r l o g r e c o r d f o r
p a g e A
l o g r e c o r d f o r
p a g e A
l o g r e c o r d f o r
p a g e A
w r i t e t o d i s k
p a g e A L S N 1
w r i t e t o d i s k
p a g e A L S N 3
l o g r e c o r d f o r
p a g e A
t i m e
L S N 1 L S N 2 L S N 3 L S N 4

33. 
Different Buffer Management Policies
Steal policy: When the buffer manager needs
space, it can decide to replace dirty pages.
No-Steal policy: Pages can be replaced only if
they are clean.
Force policy: At end of transaction, all
modified pages are forced to disk in a series of
synchronous write operations.
No-Force policy: No modified page is forced
during commit. REDO log records are written to
the log.

34. 
The Problem of Hotspot Pages
bufferpool
durable storage
log
TA1
TA2
TA3
TA4 TA5
TA6
TA7
TA8
force
The dotted arrows indicate an update of the page by the respective
transaction.The arrows at 45 degrees indicate the forced writing of the page
during commit processing.The downward arrows indicate the writing of log
records for the respective transaction.
update
operations
page A

35. 
The Basic Checkpoint Algorithm
Quiesce: Delay all incoming update DML calls
until all fixes with exclusive semaphores have
been released.
Flush the buffer: Write all modified pages.
Log the checkpoint: Write a record to the log,
saying that a checkpoint has been generated.
Resume normal operation: The bufferfix requests
for updates that have been delayed in order to
take the checkpoint can now be processed again.

36. 
The Case for Indirect Checkpointing
checkpoints
c1 c2 c3
1
2
3
4
5
6
7
8
9
10
frame-
numbers
log
When taking a checkpoint, the PAGEIDs of the pages
currently in buffer are written to the log.

37. 
The Indirect Checkpointing Algorithm
Record TOC: Log the list of PAGEIDs.
Compare with prev. ckpt: See if any modified
pages have not been replaced since last ckpt.
Force lazy pages: Schedule the writing of those
pages during the next checkpoint interval.
Low-water mark: Find the LSN of the oldest
still-volatile update; write it to the log.
Write “Checkpoint done” record
Resume normal operation

38. 
Further Possibilities for Optimization
Pre-flushing can be performed by an
asynchronous process that scans the buffer for
"old" modified pages. Writing is done under
semaphore protection.
Pre-fetching can, among other things, be used
to make restart more efficient. If page reads are
logged one can use the recent checkpoint plus
the log to prime the bufferpool, i.e. it will look
almost exactly like at the moment of the crash.

39. 
Further Possibilities for Optimization
Transaction scheduling and buffer management
can take hints from the query optimizer:
This relation will be scanned sequentially.
This is a sequential scan of the leaves of a B-
tree.
This is the traversal of a B-tree, starting at the
root.
This is a nested-loop join, where the inner
relation is scanned in physically sequential
order.