Another Way to Attack the  BLOB: <ul><li>Server-side Access via PL/SQL and Perl </li></ul>
Why Server-side? <ul><li>Your choice of tools to handle queries and generate reports </li></ul><ul><li>Complete programmat...
Syllabus <ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via P...
<ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </l...
MARC? <ul><li>MARC is an acronym for MAchine Readable Cataloging </li></ul>
MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an i...
MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an i...
MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an i...
MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an i...
A MARC record’s three pieces: <ul><li>Leader </li></ul><ul><li>Directory </li></ul><ul><li>Data </li></ul>
01551nam  22003738a 4500 001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130...
01551nam  22003738a 4500 001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130...
01551nam  22003738a 4500001001300000003000600013005001700019008004100036 0100017000770350018000940400018001120430012001300...
01551 nam  22 00373 8a 4500 001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200...
Dissection of MARC record directory 01551nam  22003738a 4500 001001300000 003000600013 005001700019 008004100036 010001700...
Where in the record does a field’s data start? 01551nam  22 00373 8a 4500 001001300000 003000600013 005001700019 008004100...
01551nam  22003738a 4500001001300000003000600013005001700019008004100036 0100017000770350018000940400018001120430012001300...
01551nam  22003738a 4500001001300000003000600013005001700019008004100036 0100017000770350018000940400018001120430012001300...
Programmer’s MARC format review <ul><li>Get the record length from the 1 st  5 columns. </li></ul>
Programmer’s MARC format review <ul><li>Get the record length from the 1 st  5 columns. </li></ul><ul><li>Get the data bas...
Programmer’s MARC format review <ul><li>Get the record length from the 1 st  5 columns. </li></ul><ul><li>Get the data bas...
Programmer’s MARC format review <ul><li>Get the record length from the 1 st  5 columns. </li></ul><ul><li>Get the data bas...
Programmer’s MARC format review <ul><li>Get the record length from the 1 st  5 columns. </li></ul><ul><li>Get the data bas...
Programmer’s MARC format review <ul><li>Get the record length from the 1 st  5 columns. </li></ul><ul><li>Get the data bas...
<ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </l...
The BLOB Plan of Attack <ul><li>Voyager’s BLOB data is stored the same way for the Auth, Bib, and Mfhd data tables. </li><...
The BLOB Plan of Attack table _ data  (where “table” is auth, bib, or mfhd) table_id record_segment seqnum A MARC record i...
The BLOB Plan of Attack table   data  (where “table” is auth, bib, or mfhd) table_id record_segment seqnum Each table_id i...
The BLOB Plan of Attack An example of a record contained completely in one row. This record is ready to be processed after...
The BLOB Plan of Attack This longer record is spread across 3 rows. Assemble the MARC record by concatenating MARC data in...
<ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </l...
PL/SQL Example The example code retrieves a few MARC records, and displays them on the screen in human-readable format, al...
Use a cursor to retrieve data PL/SQL Example Also declare necessary variables in this section
PL/SQL Example Open the cursor and start looping through the rows
PL/SQL Example Get a row from the cursor into the program variables
PL/SQL Example Assemble the marc record. The typical record fits into one row, thus seqnum = 1 and we skip the loop.
PL/SQL Example For a longer, multi-segment record (from an earlier example), we 1 st  have seqnum=3 & put it into marc. Th...
Why go “backwards” in assembling a MARC record? If we predicate the segment-to-marc-record assembly on when the auth_id ch...
PL/SQL Example Now that we have a MARC record, let’s get the record length and data base-address. We set our pointer to th...
PL/SQL Example As we loop through the directory, we read the tag id, its length, and its offset in the data part. The actu...
PL/SQL Example In the last line here, the subfield indicators (hex 1f = dec 31) are replaced by the vertical bar character...
PL/SQL Example Along with the subfield indicator character substitution, we add some space formatting to further increase ...
PL/SQL Example
PL/SQL Example Now we can output the tag’s data. Output is broken into 80 character chunks to get around the 255 character...
PL/SQL Example We’re done with this tag, so we move on to the next tag in the directory. At the end, close loops and clean...
<ul><li>Demo… </li></ul><ul><li>example.pls </li></ul>PL/SQL Example
<ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </l...
Additional tools required for Perl to talk to Oracle: <ul><li>DBI, the generic DataBase Interface software. </li></ul><ul>...
Getting  and installing DBI and DBD <ul><li>Point your browser to: </li></ul><ul><li>http://www.cpan.org/authors/id/TIMB/ ...
Getting and  installing  DBI and DBD <ul><li>gunzip each file. </li></ul><ul><li>un-tar each file. </li></ul><ul><li>READ ...
<ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </l...
Perl Example The following real-world example lets you retrieve an arbitrary range of MARC records from your choice of Aut...
Perl Example Must pull in DBI stuff Handle program arguments and show how to  use it if  necessary
Perl Example Here we create the database connection and assign its context to a database handle. We need to specify what t...
Perl Example We saw this query in the PL/SQL example. Here we build the query statement, inserting the program arguments w...
Perl Example Execute the statement and receive a return code. Create the query context and assign it to a statement handle.
Perl Example This is how we get rows from the result set of the query, via the statement handle. The three columns in the ...
Perl Example Output last record here Raw output: On record transition, output the MARC record we just built, reset the ID ...
Perl Example Formatted (not raw) output: On record transition, store the accumulated MARC record and start building a new ...
Perl Example Release the resources associated with the statement handle and the database handle.
Perl Example Executing this part for formatted, readable output MARC data contains no CR-LFs; instead it uses the hex 1d c...
Perl Example Executing this part for formatted, readable output Start looping through the array of MARC records.
Perl Example Executing this part for formatted, readable output We get and output the leader, and then get the record leng...
Perl Example Executing this part for formatted, readable output Loop through the directory
Perl Example Executing this part for formatted, readable output Get the tag id, its length, and its offset. Then read the ...
Perl Example Executing this part for formatted, readable output Now do some formatting for readability. We substitute the ...
Perl Example Executing this part for formatted, readable output Output the tag’s parameters, and the data. Then go to the ...
Perl Example Executing this part for formatted, readable output End of program stuff. Close loops and show count of record...
Demo… example.pl Perl Example
Perl <ul><li>PROBLEM : if you’re reading the entire table, you can still run into problems with too much data at one time....
Perl Large Table Solution in a Nutshell <ul><li>This example uses the BIB_DATA table </li></ul><ul><li>in your setup secti...
This outer loop goes through the entire table: while beginning_bib_id < max_bib_id call chunkthrudb set  beginning_bib_id ...
sub chunkthrudb select bib_id, record_segment, seqnum from bib_data where bib_id >= beginning_bib_id and bib_id < ending_b...
sub processrec process the MARC record as needed end sub Perl Large Table Solution in a Nutshell
Page 8 of the handout has a diagram illustrating this process. Perl Large Table Solution in a Nutshell
Questions? Email: zimmer@wmich.edu Phone: 616.387.3885 Thanks for listening.
Upcoming SlideShare
Loading in …5
×

Another Way to Attack the BLOB: Server-side Access via PL/SQL and Perl

1,293 views

Published on

Different ways to retrieve raw MARC data
Presented at VUGM 2002

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,293
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 1
  • Another Way to Attack the BLOB: Server-side Access via PL/SQL and Perl

    1. 1. Another Way to Attack the BLOB: <ul><li>Server-side Access via PL/SQL and Perl </li></ul>
    2. 2. Why Server-side? <ul><li>Your choice of tools to handle queries and generate reports </li></ul><ul><li>Complete programmatic control </li></ul><ul><li>Easier to write complex reports </li></ul><ul><li>No (well, fewer) limitations </li></ul><ul><li>Easier to restrict database access to the masses </li></ul>
    3. 3. Syllabus <ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </li></ul><ul><li>Required tools for Perl: getting DBD & DBI </li></ul><ul><li>Data Retrieval via Perl </li></ul>
    4. 4. <ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </li></ul><ul><li>Required tools for Perl: getting DBD & DBI </li></ul><ul><li>Data Retrieval via Perl </li></ul>
    5. 5. MARC? <ul><li>MARC is an acronym for MAchine Readable Cataloging </li></ul>
    6. 6. MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an item’s data. </li></ul>
    7. 7. MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an item’s data. </li></ul><ul><li>It’s machine readable, but not so easy for us humans to read. </li></ul>
    8. 8. MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an item’s data. </li></ul><ul><li>It’s machine readable, but not so easy for us humans to read. </li></ul><ul><li>With a bit of practice, a raw MARC record can be parsed by hand. </li></ul>
    9. 9. MARC <ul><li>MARC is an acronym for MAchine Readable Cataloging. </li></ul><ul><li>It’s a standard format for storing an item’s data. </li></ul><ul><li>It’s machine readable, but not so easy for us humans to read. </li></ul><ul><li>With a bit of practice, a raw MARC record can be parsed by hand. </li></ul><ul><li>However, doing so is about as exciting and satisfying as trying to thread a needle one-handed. </li></ul>
    10. 10. A MARC record’s three pieces: <ul><li>Leader </li></ul><ul><li>Directory </li></ul><ul><li>Data </li></ul>
    11. 11. 01551nam 22003738a 4500 001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 074000900197082001600206086001700222099001700239100001800256245011000274 260011200384300003800496490005400534500016500588500007500753500003400828 500003900862504005200901650004600953650005000999650004901049710002901098 830005001127ocm10726696 OCoLC19961223115432.0840406s1996 dcuab b f000 0 eng a 84600065 a(GPO)97054409 dGPOdDLCdMvI an-us-az awdoc,sudci3114100999573400aQE611.5.U6bF84 1996 a06 ////////////// skipping part of record here //////////// turalzArizonazMohave County.2 aGeological Survey (U.S.) 0aGeologic al Survey professional paper ;v1266. Partial view of a MARC record this is the leader
    12. 12. 01551nam 22003738a 4500 001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 074000900197082001600206086001700222099001700239100001800256245011000274 260011200384300003800496490005400534500016500588500007500753500003400828 500003900862504005200901650004600953650005000999650004901049710002901098 830005001127 ocm10726696 OCoLC19961223115432.0840406s1996 dcuab b f000 0 eng a 84600065 a(GPO)97054409 dGPOdDLCdMvI an-us-az awdoc,sudci3114100999573400aQE611.5.U6bF84 1996 a06 ////////////// skipping part of record here //////////// turalzArizonazMohave County.2 aGeological Survey (U.S.) 0aGeologic al Survey professional paper ;v1266. Partial view of a MARC record this is the directory
    13. 13. 01551nam 22003738a 4500001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 074000900197082001600206086001700222099001700239100001800256245011000274 260011200384300003800496490005400534500016500588500007500753500003400828 500003900862504005200901650004600953650005000999650004901049710002901098 830005001127 ocm10726696 OCoLC19961223115432.0840406s1996 dcuab b f000 0 eng a 84600065 a(GPO)97054409 dGPOdDLCdMvI an-us-az awdoc,sudci3114100999573400aQE611.5.U6bF84 1996 a06 ////////////// skipping part of record here //////////// turalzArizonazMohave County.2 aGeological Survey (U.S.) 0aGeologic al Survey professional paper ;v1266. Partial view of a MARC record this is the data
    14. 14. 01551 nam 22 00373 8a 4500 001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 074000900197082001600206086001700222099001700239100001800256245011000274 260011200384300003800496490005400534500016500588500007500753500003400828 500003900862504005200901650004600953650005000999650004901049710002901098 830005001127ocm10726696 OCoLC19961223115432.0840406s1996 dcuab b f000 0 eng a 84600065 a(GPO)97054409 dGPOdDLCdMvI an-us-az awdoc,sudci3114100999573400aQE611.5.U6bF84 1996 a06 ////////////// skipping part of record here //////////// turalzArizonazMohave County.2 aGeological Survey (U.S.) 0aGeologic al Survey professional paper ;v1266. Dissection of MARC record leader record length data starts at this offset, the base address (pertinent details)
    15. 15. Dissection of MARC record directory 01551nam 22003738a 4500 001001300000 003000600013 005001700019 008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 01551nam 22003738a 4500 header 001 0013 00000 003 0006 00013 005 0017 00019 008 0041 00036 010 0017 00077 035 0018 00094 040 0018 00112 etc. tag len offset how to parse it Each 12-character “triplet” is associated with one field.
    16. 16. Where in the record does a field’s data start? 01551nam 22 00373 8a 4500 001001300000 003000600013 005001700019 008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 01551nam 22003738a 4500 header 001 0013 00000 003 0006 00013 005 0017 00019 008 0041 00036 010 0017 00077 035 0018 00094 040 0018 00112 etc. tag len offset Where a field’s data starts is determined by adding its offset to the base address . Data for the first field, tag 001, begins at position 373, tag 003 begins at 386, tag 005 begins at 392, etc.
    17. 17. 01551nam 22003738a 4500001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 074000900197082001600206086001700222099001700239100001800256245011000274 260011200384300003800496490005400534500016500588500007500753500003400828 500003900862504005200901650004600953650005000999650004901049710002901098 830005001127 ocm10726696 OCoLC19961223115432.0840406s1996 dcuab b f000 0 eng a 84600065 a(GPO)97054409 dGPOdDLCdMvI an-us-az awdoc,sudci3114100999573400aQE611.5.U6bF84 1996 a06 ////////////// skipping part of record here //////////// turalzArizonazMohave County.2 aGeological Survey (U.S.) 0aGeologic al Survey professional paper ;v1266. Partial view of a raw MARC record, data section The “box characters” below are the MARC format binary separation characters.
    18. 18. 01551nam 22003738a 4500001001300000003000600013005001700019008004100036 010001700077035001800094040001800112043001200130049003000142050002500172 074000900197082001600206086001700222099001700239100001800256245011000274 260011200384300003800496490005400534500016500588500007500753500003400828 500003900862504005200901650004600953650005000999650004901049710002901098 830005001127 <TAG> ocm10726696 <TAG> OCoLC <TAG> 19961223115432.0 <TAG> 840406s 1996 dcuab b f000 0 eng <TAG> <SUB> a 84600065 <TAG> <SUB> a( GPO)97054409 <TAG> <SUB> dGPO <SUB> dDLC <SUB> dMvI <TAG> <SUB> an-us-az <TAG> <SUB> awdoc,sudc <SUB> i31141009995734 <TAG> 00 <SUB> aQE611.5.U6 <SUB> bF84 199 6 <TAG> <SUB> a06 ////////////// skipping part of record here //////////// tural <SUB> zArizona <SUB> zYavapai County. <TAG> 0 <SUB> aGeology, Structural <S UB> zArizona <SUB> zMohave County. <TAG> 2 <SUB> aGeological Survey (U.S.) <TAG > 0 <SUB> aGeological Survey professional paper ; <SUB> v1266. <TAG> <EOR> Partial view of a raw MARC record, data section The MARC format uses the following characters: <TAG> hex 1e tag delimiter <SUB> hex 1f subfield delimiter <EOR> hex 1d end of record indicator
    19. 19. Programmer’s MARC format review <ul><li>Get the record length from the 1 st 5 columns. </li></ul>
    20. 20. Programmer’s MARC format review <ul><li>Get the record length from the 1 st 5 columns. </li></ul><ul><li>Get the data base-address from columns 13-17. </li></ul>
    21. 21. Programmer’s MARC format review <ul><li>Get the record length from the 1 st 5 columns. </li></ul><ul><li>Get the data base-address from columns 13-17. </li></ul><ul><li>Parse through the directory for the desired field by looking at the 1 st 3 columns of each tag’s 12-character “triplet”. Get the tag’s length (next 4 columns) and offset (last 5 columns of the “triplet”). </li></ul>
    22. 22. Programmer’s MARC format review <ul><li>Get the record length from the 1 st 5 columns. </li></ul><ul><li>Get the data base-address from columns 13-17. </li></ul><ul><li>Parse through the directory for the desired field by looking at the 1 st 3 columns of each tag’s 12-character “triplet”. Get the tag’s length (next 4 columns) and offset (last 5 columns of the “triplet”). </li></ul><ul><li>Read the tag’s data by: Adding the tag’s offset to the record’s base address. Starting at that position, read the tag’s data for tag length columns. </li></ul>
    23. 23. Programmer’s MARC format review <ul><li>Get the record length from the 1 st 5 columns. </li></ul><ul><li>Get the data base-address from columns 13-17. </li></ul><ul><li>Parse through the directory for the desired field by looking at the 1 st 3 columns of each tag’s 12-character “triplet”. Get the tag’s length (next 4 columns) and offset (last 5 columns of the “triplet”). </li></ul><ul><li>Read the tag’s data by: Adding the tag’s offset to the record’s base address. Starting at that position, read the tag’s data for tag length columns. </li></ul><ul><li>Make sure the position you’re reading from is not beyond the end of the record. </li></ul>
    24. 24. Programmer’s MARC format review <ul><li>Get the record length from the 1 st 5 columns. </li></ul><ul><li>Get the data base-address from columns 13-17. </li></ul><ul><li>Parse through the directory for the desired field by looking at the 1 st 3 columns of each tag’s 12-character “triplet”. Get tag’s length (next 4 columns) and offset (last 5 columns of the “triplet”). </li></ul><ul><li>Read the tag’s data by: Adding the tag’s offset to the record’s base address. Starting at that position, read the tag’s data for tag length columns. </li></ul><ul><li>Make sure the position you’re reading from is not beyond the end of the record. </li></ul>Beware of the common “off by 1” error. Depending on the language you’re using, you could be off by 1 in either direction regarding your position within the record.
    25. 25. <ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </li></ul><ul><li>Required tools for Perl: getting DBD & DBI </li></ul><ul><li>Data Retrieval via Perl </li></ul>
    26. 26. The BLOB Plan of Attack <ul><li>Voyager’s BLOB data is stored the same way for the Auth, Bib, and Mfhd data tables. </li></ul>table _ data (where “table” is auth, bib, or mfhd) table_id record_segment seqnum
    27. 27. The BLOB Plan of Attack table _ data (where “table” is auth, bib, or mfhd) table_id record_segment seqnum A MARC record is typically stored entirely in one row in the table. Longer records which are longer than the record_segment size have to be stored in more than one row.
    28. 28. The BLOB Plan of Attack table data (where “table” is auth, bib, or mfhd) table_id record_segment seqnum Each table_id is unique to an item’s record. However, if more than one row makes up a record, we will have duplicate table_ids. In that case, we’ll have seqnum = 1, 2, 3, etc., for that record.
    29. 29. The BLOB Plan of Attack An example of a record contained completely in one row. This record is ready to be processed after extraction from the record_segment. 1 MARC data 635406 seqnum record_segment auth_id
    30. 30. The BLOB Plan of Attack This longer record is spread across 3 rows. Assemble the MARC record by concatenating MARC data in seqnum order: MARC-record = record_segment<-seqnum1 + record_segment<-seqnum2 + record_segment<-seqnum3 This record is then ready to be processed. 3 MARC data 635406 2 MARC data 635406 1 MARC data 635406 seqnum record_segment auth_id
    31. 31. <ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </li></ul><ul><li>Required tools for Perl: getting DBD & DBI </li></ul><ul><li>Data Retrieval via Perl </li></ul>
    32. 32. PL/SQL Example The example code retrieves a few MARC records, and displays them on the screen in human-readable format, along with some diagnostics. (The code examined in the following slides starts on Page 2 of the handout.)
    33. 33. Use a cursor to retrieve data PL/SQL Example Also declare necessary variables in this section
    34. 34. PL/SQL Example Open the cursor and start looping through the rows
    35. 35. PL/SQL Example Get a row from the cursor into the program variables
    36. 36. PL/SQL Example Assemble the marc record. The typical record fits into one row, thus seqnum = 1 and we skip the loop.
    37. 37. PL/SQL Example For a longer, multi-segment record (from an earlier example), we 1 st have seqnum=3 & put it into marc. Then we have seqnum=2 and PREPEND that to marc. Last we exit the loop since now seqnum=1 and the last statement here takes care of that.
    38. 38. Why go “backwards” in assembling a MARC record? If we predicate the segment-to-marc-record assembly on when the auth_id changes in our loop structure, once it changes we've gone too far and can't go back to get the last segment to completely assemble the now previous record. It’s simpler to predicate looping on seqnum in reverse order because there will always be a seqnum of 1. If there are multiple segments, we'll always end with a seqnum of 1 and still be on the same auth_id and can go on processing the record. This reasoning is not for PL/SQL only, although that is “where” the idea came from. PL/SQL Example
    39. 39. PL/SQL Example Now that we have a MARC record, let’s get the record length and data base-address. We set our pointer to the start of the directory and start looping through the directory.
    40. 40. PL/SQL Example As we loop through the directory, we read the tag id, its length, and its offset in the data part. The actual tag address where we get the data is the data base-address plus the offset.
    41. 41. PL/SQL Example In the last line here, the subfield indicators (hex 1f = dec 31) are replaced by the vertical bar character “ | ” for better readability.
    42. 42. PL/SQL Example Along with the subfield indicator character substitution, we add some space formatting to further increase readability. Thus, instead of 0aPetroleumxDrilling fluids we get 0|a Petroleum |x Drilling fluids for tag data.
    43. 43. PL/SQL Example
    44. 44. PL/SQL Example Now we can output the tag’s data. Output is broken into 80 character chunks to get around the 255 character limit of dbms_output and for better readability.
    45. 45. PL/SQL Example We’re done with this tag, so we move on to the next tag in the directory. At the end, close loops and clean up. End looping for directory traversal End looping for cursor Don’t forget that this ending character is required for your PL/SQL code to run!
    46. 46. <ul><li>Demo… </li></ul><ul><li>example.pls </li></ul>PL/SQL Example
    47. 47. <ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </li></ul><ul><li>Required tools for Perl: getting DBD & DBI </li></ul><ul><li>Data Retrieval via Perl </li></ul>
    48. 48. Additional tools required for Perl to talk to Oracle: <ul><li>DBI, the generic DataBase Interface software. </li></ul><ul><li>DBD, the specific DataBase Driver, for Oracle in our case. </li></ul>
    49. 49. Getting and installing DBI and DBD <ul><li>Point your browser to: </li></ul><ul><li>http://www.cpan.org/authors/id/TIMB/ </li></ul><ul><li>Complete the above URL with </li></ul><ul><li>“ DBD-Oracle-1.12.tar.gz ” to get DBD software </li></ul><ul><li>“ DBI-1.20.tar.gz ” to get DBI software </li></ul>
    50. 50. Getting and installing DBI and DBD <ul><li>gunzip each file. </li></ul><ul><li>un-tar each file. </li></ul><ul><li>READ the instructions! </li></ul><ul><li>Installation takes 4 or 5 steps and requires you to be root. </li></ul><ul><li>If you don’t have root access, or if you’re uncomfortable doing any of this, seek out your SysAdmin for assistance. </li></ul>
    51. 51. <ul><li>Brief MARC record review </li></ul><ul><li>The BLOB Plan of Attack </li></ul><ul><li>Data Retrieval via PL/SQL </li></ul><ul><li>Required tools for Perl: getting DBD & DBI </li></ul><ul><li>Data Retrieval via Perl </li></ul>
    52. 52. Perl Example The following real-world example lets you retrieve an arbitrary range of MARC records from your choice of Auth, Bib, or Mfhd. Output goes to <stdout>, and can be raw MARC data, or formatted for human readability. (The code examined in the following slides starts on Page 5 of the handout.)
    53. 53. Perl Example Must pull in DBI stuff Handle program arguments and show how to use it if necessary
    54. 54. Perl Example Here we create the database connection and assign its context to a database handle. We need to specify what type of data (Oracle), the name of the machine to which we’re connecting, the SID, and the username and password.
    55. 55. Perl Example We saw this query in the PL/SQL example. Here we build the query statement, inserting the program arguments where needed. This allows this query to work with any MARC table type and an arbitrary table_id range.
    56. 56. Perl Example Execute the statement and receive a return code. Create the query context and assign it to a statement handle.
    57. 57. Perl Example This is how we get rows from the result set of the query, via the statement handle. The three columns in the row fall into the list of three variables.
    58. 58. Perl Example Output last record here Raw output: On record transition, output the MARC record we just built, reset the ID variable, and store the MARC data for the record we just started reading. If on the same record, keep on storing MARC data.
    59. 59. Perl Example Formatted (not raw) output: On record transition, store the accumulated MARC record and start building a new one, else just prepend to the present marc record. Store last record here (We’re effectively building a MARC file in memory, a virtual file, in the $marcstuff variable.)
    60. 60. Perl Example Release the resources associated with the statement handle and the database handle.
    61. 61. Perl Example Executing this part for formatted, readable output MARC data contains no CR-LFs; instead it uses the hex 1d character to delimit the end of a MARC record. Create the array of MARC records here.
    62. 62. Perl Example Executing this part for formatted, readable output Start looping through the array of MARC records.
    63. 63. Perl Example Executing this part for formatted, readable output We get and output the leader, and then get the record length and the data base-address. Then we position ourselves at the start of the directory.
    64. 64. Perl Example Executing this part for formatted, readable output Loop through the directory
    65. 65. Perl Example Executing this part for formatted, readable output Get the tag id, its length, and its offset. Then read the tag’s data. The actual tag address where we get the data is the data base-address plus the offset.
    66. 66. Perl Example Executing this part for formatted, readable output Now do some formatting for readability. We substitute the vertical bar character “ | ” for the subfield delimiter, and remove the other delimiters.
    67. 67. Perl Example Executing this part for formatted, readable output Output the tag’s parameters, and the data. Then go to the next tag in the directory.
    68. 68. Perl Example Executing this part for formatted, readable output End of program stuff. Close loops and show count of records output.
    69. 69. Demo… example.pl Perl Example
    70. 70. Perl <ul><li>PROBLEM : if you’re reading the entire table, you can still run into problems with too much data at one time. </li></ul><ul><li>SOLUTION : process your data in small chunks. </li></ul><ul><li>Dividing the table into chunks of about 50,000 rows has worked very well for us. </li></ul><ul><li>The following method has proven useful: </li></ul>
    71. 71. Perl Large Table Solution in a Nutshell <ul><li>This example uses the BIB_DATA table </li></ul><ul><li>in your setup section, set a db_increment variable to 50,000 </li></ul><ul><li>set max_bib_id to highest bib_id from table </li></ul><ul><li>set beginning_bib_id to 0, ending_bib_id to db_increment </li></ul>
    72. 72. This outer loop goes through the entire table: while beginning_bib_id < max_bib_id call chunkthrudb set beginning_bib_id to (ending_bib_id + 1) increment ending_bib_id by db_increment end while Perl Large Table Solution in a Nutshell
    73. 73. sub chunkthrudb select bib_id, record_segment, seqnum from bib_data where bib_id >= beginning_bib_id and bib_id < ending_bib_id order by bib_id asc, seqnum desc build the MARC record and call processrec end sub This inner loop goes through db_increment-sized chunks: Perl Large Table Solution in a Nutshell
    74. 74. sub processrec process the MARC record as needed end sub Perl Large Table Solution in a Nutshell
    75. 75. Page 8 of the handout has a diagram illustrating this process. Perl Large Table Solution in a Nutshell
    76. 76. Questions? Email: zimmer@wmich.edu Phone: 616.387.3885 Thanks for listening.

    ×