Wrong confirmation ID

Email
Favorite
Favorited ×
Download
Embed
Private Content
Copy and paste this code into your blog or website Copy Customize… Close
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

+ Follow

Sedna XML Database: Memory Management

by Ivan Shcheklein on Apr 07, 2010

1,598 views

Database memory management: 64-bit address space on 32-bit architecture.

More…

Database memory management: 64-bit address space on 32-bit architecture.

Less

1 Embed 10

http://www.slideshare.net

Statistics

Favorites: 0
Downloads: 55
Comments: 0
Embed Views: 10
Views on SlideShare: 1,588
Total Views: 1,598

I am going to talk a bit about memory management in databases in general and then I will focus on Sedna-specific details. It is convenient to introduce a ‘Database Address Space’ concept when talking about memory management. Basicly all database objects – I mean low level objects like individual records – must be somehow uniquely identified. Each low level object is given a address – a unique integral number used to locate the object. The most basic design is to store DB as a single file. In this case we can use an offset of the object in the database file as it’s address. DAS is the set of all valid database objects addresses. Huge datasets are stored in databases – need large enough DAS. We can not work with data on HDD directly. We have to upload data in the main memory first. The common design is to allocate a dedicated chunk of memory for this purpose (so-called ‘buffer memory’). A data block is loaded from HDD in the buffer. DBMS works with the data. When DBMS is running low on buffer memory it starts swapping blocks back to disk. Generally buffer memory is locked so that OS can not swap it. DBMS has a better knowledge of the data-access pattern and hence can manage swap process more efficiently. (Swapping is not nessesary tied to low memory condition. For instance a background flusher can be implemented.) DAS pointer is passed to the buffer manager when asking for data. The manager either find the requested data in buffers or not. In the latter case the data location on disk is computed and IO occurs. If DAS pointers are followed infrequently the performance of the pointer dereference operation isn’t critical. However if DAS pointers are followed frequently and the referenced data is nearly always availible from buffers we have to optimize the DAS pointer dereference operation. This is the Sedna case – we are using internal XML representation with a large amount of interconnecting pointers.

Sedna XML Database: Memory Management — Presentation Transcript

Sedna: Memory Management Nickolay Zavaritsky [email_address] Software Developer Sedna Team
Background
- Huge database address space (64 bits enough?)
- Dedicated memory region – buffer memory
- DBMS decides when to swap particular buffer, not the OS!
- Internal structures in DB are interconnected with database address space pointers
- XML specifics – DAS pointers everywhere!
Challenge
- XML nodes are interconnected with DAS pointers
- DAS pointers followed very often
- Traditionally pointer swizzling is used
- Swizzling is not for free
- Custom method developed – fast DAS pointers dereference without pointer swizzling .
Dereferencing DAS pointer
- Extract addr – this is the in-memory address of where the data can be
- Go to block header and check layer
- Example follows…
Example
- Pointer – ( layer : 7 , addr : 1020 )
- Block – ( layer : 7 , addr : 10 00 )
- The block can be mapped in VM at 1000 only!
- Have to check the block header (at 1000 ) and ensure that layer is 7 .
- The final result is 1020 : the in-memory address of the object we are interested in.
Performance Estimation
- Experiment details:
- Loading 113Mb XML (xmark, generated)
- Huge buffer memory – no swapping
- Avg. 60M DAS pointers dereferenced
- Machine: 1.5GHz P4, 2GB RAM, Red Hat Linux 9.0
Following DAS pointer is nearly as fast as following a plain C pointer! (1% overhead) 62.1 sec 63.5 sec Sedna with light DAS pointers Normal Sedna
Conclusion
- 64- bit address space on 32-bit architecture
- Fast pointer dereferencing (no pointer swizzling)
- Control over buffer management strategy
Questions?

Sedna XML Database: Memory Management

by Ivan Shcheklein on Apr 07, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 10

Statistics

Sedna XML Database: Memory Management — Presentation Transcript