• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
[Www.pkbulk.blogspot.com]file and indexing
 

[Www.pkbulk.blogspot.com]file and indexing

on

  • 257 views

Introduction to database lecture : This lecture is all about File and indexing and more......

Introduction to database lecture : This lecture is all about File and indexing and more......

Statistics

Views

Total Views
257
Views on SlideShare
144
Embed Views
113

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 113

http://pkbulk.blogspot.com 104
http://www.pkbulk.blogspot.com 9

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • 4
  • 5
  • 6
  • 3
  • 7
  • 12
  • 13

[Www.pkbulk.blogspot.com]file and indexing [Www.pkbulk.blogspot.com]file and indexing Presentation Transcript

  • 1Data Storage and Indexing
  • 2Outline• Data Storage– storing data: disks– buffer manager– representing data– external sorting• Indexing– types of indexes– B+ trees– hash tables (maybe)
  • 3Physical Addresses• Each block and each record have a physicaladdress that consists of:– The host– The disk– The cylinder number– The track number– The block within the track– For records: an offset in the block• sometimes this is in the block’s header
  • 4Logical Addresses• Logical address: a string of bytes (10-16)• More flexible: can blocks/records around• But need translation table:Logical address Physical addressL1 P1L2 P2L3 P3
  • 5Main Memory Address• When the block is read in main memory, itreceives a main memory address• Buffer manager has another translation tableMemory address Logical addressM1 L1M2 L2M3 L3
  • 6The I/O Model of Computation• In main memory algorithms– we care about CPU time• In databases time is dominated by I/O cost• Assumption: cost is given only by I/O• Consequence: need to redesign certainalgorithms• Will illustrate here with sorting
  • 7Sorting• Illustrates the difference in algorithm designwhen your data is not in main memory:– Problem: sort 1Gb of data with 1Mb of RAM.• Arises in many places in database systems:– Data requested in sorted order (ORDER BY)– Needed for grouping operations– First step in sort-merge join algorithm– Duplicate removal– Bulk loading of B+-tree indexes.
  • 82-Way Merge-sort:Requires 3 Buffers• Pass 1: Read a page, sort it, write it.– only one buffer page is used• Pass 2, 3, …, etc.:– three buffer pages used.Main memory buffersINPUT 1INPUT 2OUTPUTDiskDisk
  • 9Two-Way External Merge Sort• Each pass we read + writeeach page in file.• N pages in the file => thenumber of passes• So total cost is:• Improvement: start withlarger runs• Sort 1GB with 1MB memoryin 10 passes = +log2 1N ( )2 12N Nlog +Input file1-page runs2-page runs4-page runs8-page runsPASS 0PASS 1PASS 2PASS 393,4 6,2 9,4 8,7 5,6 3,1 23,4 5,62,6 4,9 7,8 1,3 22,34,64,78,91,35,6 22,34,46,78,91,23,561,22,33,44,56,67,8
  • 10Can We Do Better ?• We have more main memory• Should use it to improve performance
  • 11Cost Model for Our Analysis• B: Block size• M: Size of main memory• N: Number of records in the file• R: Size of one record
  • 12Indexes
  • 13Outline• Types of indexes• B+ trees• Hash tables (maybe)
  • 14Indexes• An index on a file speeds up selections on thesearch key field(s)• Search key = any subset of the fields of a relation– Search key is not the same as key (minimal set of fieldsthat uniquely identify a record in a relation).• Entries in an index: (k, r), where:– k = the key– r = the record OR record id OR record ids
  • 15Index Classification• Clustered/unclustered– Clustered = records sorted in the key order– Unclustered = no• Dense/sparse– Dense = each record has an entry in the index– Sparse = only some records have• Primary/secondary– Primary = on the primary key– Secondary = on any key– Some books interpret these differently• B+ tree / Hash table / …
  • 16Clustered Index• File is sorted on the index attribute• Dense index: sequence of (key,pointer) pairs10203040506070801020304050607080
  • 17Clustered Index• Sparse index: one key per data block10305070901101301501020304050607080
  • 18Clustered Index with DuplicateKeys• Dense index: point to the first record withthat key10203040506070801010102020203040
  • 19Clustered Index with DuplicateKeys• Sparse index: pointer to lowest search key ineach block:• Search for 20101020301010102020203040
  • 20Clustered Index with DuplicateKeys• Better: pointer to lowest new search key ineach block:• Search for 2010203040506070801010102030304050
  • 21Unclustered Indexes• To index other attributes than primary key• Always dense (why ?)10102020203030302030302010201030
  • 22Summary Clustered vs.Unclustered IndexData entries(Index File)(Data file)Data RecordsData entriesData RecordsCLUSTERED UNCLUSTERED
  • 23Composite Search Keys• Composite Search Keys: Searchon a combination of fields.– Equality query: Every fieldvalue is equal to a constantvalue. E.g. wrt <sal,age>index:• age=20 and sal =75– Range query: Some fieldvalue is not a constant. E.g.:• age =20; or age=20 andsal > 10sue 13 75bobcaljoe 121020801112name age sal<sal, age><age, sal> <age><sal>12,2012,1011,8013,7520,1210,1275,1380,111112121310207580Data recordssorted by nameData entries in indexsorted by <sal,age>Data entriessorted by <sal>Examples of composite keyindexes using lexicographic order.
  • 24B+ Trees• Search trees• Idea in B Trees:– make 1 node = 1 block• Idea in B+ Trees:– Make leaves into a linked list (range queries areeasier)
  • 25• Parameter d = the degree• Each node has >= d and <= 2d keys (except root)• Each leaf has >=d and <= 2d keys:B+ Trees Basics30 120 240Keys k < 30Keys 30<=k<120 Keys 120<=k<240 Keys 240<=k40 50 6040 50 60Next leaf
  • 26B+ Tree Example8020 60 100 120 14010 15 18 20 30 40 50 60 65 80 85 9010 15 18 20 30 40 50 60 65 80 85 90d = 2
  • 27B+ Tree Design• How large d ?• Example:– Key size = 4 bytes– Pointer size = 8 bytes– Block size = 4096 byes• 2d x 4 + (2d+1) x 8 <= 4096• d = 170
  • 28Searching a B+ Tree• Exact key values:– Start at the root– Proceed down, to the leaf• Range queries:– As above– Then sequential traversalSelect nameFrom peopleWhere age = 25Select nameFrom peopleWhere 20 <= ageand age <= 30
  • 29B+ Trees in Practice• Typical order: 100. Typical fill-factor: 67%.– average fanout = 133• Typical capacities:– Height 4: 1334= 312,900,700 records– Height 3: 1333= 2,352,637 records• Can often hold top levels in buffer pool:– Level 1 = 1 page = 8 Kbytes– Level 2 = 133 pages = 1 Mbyte– Level 3 = 17,689 pages = 133 MBytes
  • 30Insertion in a B+ TreeInsert (K, P)• Find leaf where K belongs, insert• If no overflow (2d keys or less), halt• If overflow (2d+1 keys), split node, insert in parent:• If leaf, keep K3 too in right node• When root splits, new root has 1 key onlyK1 K2 K3 K4 K5P0 P1 P2 P3 P4 p5K1 K2P0 P1 P2K4 K5P3 P4 p5(K3, ) to parent
  • 31Insertion in a B+ Tree8020 60 100 120 14010 15 18 20 30 40 50 60 65 80 85 9010 15 18 20 30 40 50 60 65 80 85 90Insert K=19
  • 32Insertion in a B+ Tree8020 60 100 120 14010 15 18 19 20 30 40 50 60 65 80 85 9010 15 18 20 30 40 50 60 65 80 85 9019After insertion
  • 33Insertion in a B+ Tree8020 60 100 120 14010 15 18 19 20 30 40 50 60 65 80 85 9010 15 18 20 30 40 50 60 65 80 85 9019Now insert 25
  • 34Insertion in a B+ Tree8020 60 100 120 14010 15 18 19 20 2530 4050 60 65 80 85 9010 15 18 20 25 30 40 60 65 80 85 9019After insertion50
  • 35Insertion in a B+ Tree8020 60 100 120 14010 15 18 19 20 2530 4050 60 65 80 85 9010 15 18 20 25 30 40 60 65 80 85 9019But now have to split !50
  • 36Insertion in a B+ Tree8020 30 60 100 120 14010 15 18 19 20 2560 65 80 85 9010 15 18 20 25 30 40 60 65 80 85 9019After the split5030 4050
  • 37Deletion from a B+ Tree8020 30 60 100 120 14010 15 18 19 20 2560 65 80 85 9010 15 18 20 25 30 40 60 65 80 85 9019Delete 305030 4050
  • 38Deletion from a B+ Tree8020 30 60 100 120 14010 15 18 19 20 2560 65 80 85 9010 15 18 20 25 40 60 65 80 85 9019After deleting 305040 50May change to40, or not
  • 39Deletion from a B+ Tree8020 30 60 100 120 14010 15 18 19 20 2560 65 80 85 9010 15 18 20 25 40 60 65 80 85 9019Now delete 255040 50
  • 40Deletion from a B+ Tree8020 30 60 100 120 14010 15 18 19 20 60 65 80 85 9010 15 18 20 40 60 65 80 85 9019After deleting 25Need to rebalanceRotate5040 50
  • 41Deletion from a B+ Tree8019 30 60 100 120 14010 15 18 19 2060 65 80 85 9010 15 18 20 40 60 65 80 85 9019Now delete 405040 50
  • 42Deletion from a B+ Tree8019 30 60 100 120 14010 15 18 19 2060 65 80 85 9010 15 18 20 60 65 80 85 9019After deleting 40Rotation not possibleNeed to merge nodes5050
  • 43Deletion from a B+ Tree8019 60 100 120 14010 15 18 19 205060 65 80 85 9010 15 18 20 60 65 80 85 9019Final tree50
  • 44Hash Tables• Secondary storage hash tables are much likemain memory ones• Recall basics:– There are n buckets– A hash function f(k) maps a key k to {0, 1, …, n-1}– Store in bucket f(k) a pointer to record with key k• Secondary storage: bucket = block, useoverflow blocks when needed
  • 45• Assume 1 bucket (block) stores 2 keys +pointers• h(e)=0• h(b)=h(f)=1• h(g)=2• h(a)=h(c)=3Hash Table Exampleebfgac0123
  • 46• Search for a:• Compute h(a)=3• Read bucket 3• 1 disk accessSearching in a Hash Tableebfgac0123
  • 47• Place in right bucket, if space• E.g. h(d)=2Insertion in Hash Tableebfgdac0123
  • 48• Create overflow block, if no space• E.g. h(k)=1• More over-flow blocksmay be neededInsertion in Hash Tableebfgdac0123k
  • 49Hash Table Performance• Excellent, if no overflow blocks• Degrades considerably when number ofkeys exceeds the number of buckets (I.e.many overflow blocks).
  • 50Extensible Hash Table• Allows has table to grow, to avoidperformance degradation• Assume a hash function h that returnsnumbers in {0, …, 2k– 1}• Start with n = 2i<< 2k, only look at first imost significant bits
  • 51Extensible Hash Table• E.g. i=1, n=2, k=4• Note: we only look at the first bit (0 or 1)0(010)1(011)i=1 1101
  • 52Insertion in Extensible HashTable• Insert 11100(010)1(011)1(110)i=1 1101
  • 53Insertion in Extensible HashTable• Now insert 1010• Need to extend table, split blocks• i becomes 20(010)1(011)1(110), 1(010)i=1 1101
  • 54Insertion in Extensible HashTable• Now insert 11100(010)10(11)10(10)i=2 120001101111(10) 2
  • 55Insertion in Extensible HashTable• Now insert 0000, then 0101• Need to split block0(010)0(000), 0(101)10(11)10(10)i=2 120001101111(10) 2
  • 56Insertion in Extensible HashTable• After splitting the block00(10)00(00)10(11)10(10)i=2220001101111(10) 201(01) 2
  • 57Performance Extensible HashTable• No overflow blocks: access always one read• BUT:– Extensions can be costly and disruptive– After an extension table may no longer fit inmemory
  • 58Linear Hash Table• Idea: extend only one entry at a time• Problem: n= no longer a power of 2• Let i be such that 2i<= n < 2i+1• After computing h(k), use last i bits:– If last i bits represent a number > n, change msbfrom 1 to 0 (get a number <= n)
  • 59Linear Hash Table Example• N=3(01)00(11)00(10)10i=2000110(01)11 BIT FLIP
  • 60Linear Hash Table Example• Insert 1000: overflow blocks…(01)00(11)00(10)10i=2000110(01)11(10)00
  • 61Linear Hash Tables• Extension: independent on overflow blocks• Extend n:=n+1 when average number ofrecords per block exceeds (say) 80%
  • 62Linear Hash Table Extension• From n=3 to n=4• Only need to touchone block (which one ?)(01)00(11)00(10)10i=2000110(01)11(01)11(01)11i=2000110(10)10(01)00(11)0011
  • 63Linear Hash Table Extension• From n=3 to n=4 finished• Extension from n=4to n=5 (new bit)• Need to touch everysingle block (why ?)(01)11i=2000110(10)10(01)00(11)0011