Using the PERL DBI with an ILS Presented at ELUNA 2008

1. Perl DBI Scripting with the ILS Roy Zimmer Western Michigan University

2. What do I need for database interactions with Perl?

3. What do I need for database interactions with Perl? DBI – DataBase Interface (always)

4. What do I need for database interactions with Perl? DBI – DataBase Interface (always) DBD – DataBase Driver (one for each type of database you must access)

5. What do I need for database interactions with Perl? DBI – DataBase Interface (always) DBD – DataBase Driver (one for each type of database you must access) a database? (some DBDs let you access non-database data, such as CSV files)

6. What do I need for database interactions with Perl? DBI – DataBase Interface (always) DBD – DataBase Driver (one for each type of database you must access) a database? (some DBDs let you access non-database data, such as CSV files) some Perl proficiency

7. What do I need for database interactions with Perl? DBI – DataBase Interface (always) DBD – DataBase Driver (one for each type of database you must access) a database? (some DBDs let you access non-database data, such as CSV files) some Perl proficiency But we only care about using our Voyager database, so we’ll stick with that.

8. What does this mean to you?

9. What does this mean to you? DBI – DataBase Interface (already on your Voyager box)

10. What does this mean to you? DBI – DataBase Interface (already on your Voyager box) DBD – DataBase Driver (for , already on your Voyager box)

11. What does this mean to you? DBI – DataBase Interface (already on your Voyager box) DBD – DataBase Driver (for , already on your Voyager box) a database? (already on your Voyager box)

12. What does this mean to you? DBI – DataBase Interface (already on your Voyager box) DBD – DataBase Driver (for , already on your Voyager box) a database? (already on your Voyager box) some Perl proficiency ( the only thing you supply! )

13. A simple program Always needed for database access nameletters.pl

14. A simple program Get access information from a setup file and connect to a database. File format (the only record): library.box.university.edu / username / password / VGER nameletters.pl

15. A simple program Connect to the database and get a handle. nameletters.pl

16. A simple program… for each letter, how many patrons’ last names start with that letter Create the query sprintf is your friend A sample query here is: select count (*) from wmichdb.patron where last_name like ‘S%’ nameletters.pl

17. A simple program… for each letter, how many patrons’ last names start with that letter Prepare the query, associating it with a database, giving it a handle. nameletters.pl

18. A simple program… for each letter, how many patrons’ last names start with that letter Run the query, get a return code. nameletters.pl

19. A simple program… for each letter, how many patrons’ last names start with that letter Get the query result… nameletters.pl

20. A simple program… for each letter, how many patrons’ last names start with that letter …and print it. nameletters.pl

21. A simple program… for each letter, how many patrons’ last names start with that letter A: 003016 B: 007113 C: 005041 D: 003792 E: 001322 F: 002605 G: 003603 H: 005388 I: 000368 J: 001970 K: 004371 L: 003763 M: 007039 N: 001622 O: 001299 P: 003792 Q: 000121 R: 003770 S: 008217 T: 002528 U: 000248 V: 001791 W: 004487 X: 000018 Y: 000562 Z: 000709 Output nameletters.pl

22. Nested query example… getting barcodes of retirees who’ve borrowed something within the past year retirees.pl

23. Nested query example… getting barcodes of retirees who’ve borrowed something within the past year Outer query Get patron information that meets the criteria, and connecting information for barcodes. (sprintf actually not needed here) retirees.pl

24. Nested query example… getting barcodes of retirees who’ve borrowed something within the past year Inner query Remember the outer query? Note the positional correspondence of the query parameters with the while statement receiving a query row. retirees.pl

25. Nested query example… getting barcodes of retirees who’ve borrowed something within the past year Inner query The inner query has a different query string name… retirees.pl

26. Nested query example… getting barcodes of retirees who’ve borrowed something within the past year Inner query …as do the other variables for this query. retirees.pl

27. Nested query example… getting barcodes of retirees who’ve borrowed something within the past year Inner query Doe, John R doe1 pgroup: EMERIT/RET 21141002502810 08/16/1999 Other 99952803601000 11/25/2001 Expired Doe, Josephine doe2 pgroup: RETIREDSTF 21141001297289 08/16/1999 Other 21141002300660 08/16/1999 Other 11130755000000 09/04/2002 Expired retirees.pl Sample output:

28. Termination Notes $sth->finish finish a query $dbh->disconnect disconnect from a database finish seems like a good practice, but it’s not really needed. Yet… disconnect isn’t really needed (for our purposes, read-only). However, without using the finish statement, you could run into error situations. leading us to…

29. CPAN the Comprehensive Perl Archive Network Great resource for *all* kinds of Perl modules Documentation for modules - comprehensively so for DBI and DBD - about 100 pages for DBI and 50 pages for DBD

30. A bit more documentation… $sth->fetchrow_array returns a single value, array, or list There are a number of other ways to get query data, but this seems to work best. You can also get lots of details about your database environment via DBI calls. See the DBI documentation for more information.

31. A quick little DBI utility dbddrivers.pl

32. A quick little DBI utility Available Perl DBD drivers on this system DBM ExampleP File Gofer Oracle Proxy Sponge Output dbddrivers.pl

33. What if you need to look at every bib record you’ve got?

34. What if you need to look at every bib record you’ve got? And you need to access the marc record (synonymous with )? Here’s one approach… THE BLOB

35. connect to the database query bib_data table to get the maximum bib ID set increment to 50,000 set ending bib ID to increment set beginning bib ID to 0 Traversing every record and dealing with THE BLOB along the way

36. connect to the database query bib_data table to get the maximum bib ID set increment to 50,000 set ending bib ID to increment set beginning bib ID to 0 while beginning bid ID < maximum bib ID chunkthroughdb() provide feedback of progress beginning bib ID = ending bib ID + 1 add increment to ending bib ID end provide final feedback Traversing every record and dealing with THE BLOB along the way

37. sub chunkthrudb query bib_data for blob data based on bib IDs >= beginning bib ID and bib IDs < ending bib ID for each record from the query assemble each bib ID's data into a marc record do the required processing for this record end end Traversing every record and dealing with THE BLOB along the way

38. Traversing every record and dealing with THE BLOB along the way Discussion of this approach Alternative method: read the whole database at once! - might be impossible - might not be feasible - probably not efficient The program presented here…seems to be the better method.

39. Traversing every record and dealing with THE BLOB along the way Discussion of this approach If nothing else, “chunking” your way through your database is more efficient We have close to 1.6 million bib records. Based on the program we’re about to see, traversing our database… using 50,000 record chunks takes about 50 minutes without “chunking” it takes about 76 minutes

40. Traversing every record and dealing with THE BLOB along the way findbadleader.pl

41. Traversing every record and dealing with THE BLOB along the way Get our boundary condition findbadleader.pl

42. Traversing every record and dealing with THE BLOB along the way 0-50,000 – our first chunk of records findbadleader.pl

43. Traversing every record and dealing with THE BLOB along the way the loop findbadleader.pl

44. Traversing every record and dealing with THE BLOB along the way set up the next chunk the loop findbadleader.pl

45. Traversing every record and dealing with THE BLOB along the way - chunking subroutine (chunkthrudb) “ seqnum desc” is key to getting the blob data for each record This gets one chunk’s worth of MARC records (blob data) findbadleader.pl

46. Traversing every record and dealing with THE BLOB along the way - chunking subroutine (chunkthrudb ) Creates the MARC record for each bib retrieved. For larger records that don’t fit in one table row, assemble the record in reverse order (from the query). findbadleader.pl

47. Traversing every record and dealing with THE BLOB along the way - chunking subroutine (chunkthrudb ) Process each record. In this case, we’re looking for records with bad leaders (not ending in “4500”). Sample output: findbadleader.pl

48. Being in a is a good thing… bind sprintf is a friend, but bind is a better friend

49. Being in a is a good thing… bind sprintf is a friend, but bind is a better friend usual query method

50. Being in a is a good thing… bind sprintf is a friend, but bind is a better friend usual query method query with bind values method

51. testbind.pl Illustrating and contrasting the query with bind method create, prepare, and execute a query

52. testbind.pl Illustrating and contrasting the query with bind method prepare and execute a query

53. testbind.pl Illustrating and contrasting the query with bind method prepare and execute a query, improved prepare moved outside of the loop

54. testbind.pl Illustrating and contrasting the query with bind method Using testbind.pl for about 80,000 patron records, these are the results: 10 Prepare and execute a query, improved (with bind) 11 Prepare and execute a query (with bind) 20 Create, prepare and execute a query Runtime (seconds) Method

55. Running reports, queries via Perl on your PC ActiveState sells products and also has free versions. I recommend their free Perl. Get the MSI version. It comes with DBI and DBD for Oracle, among many other modules (and lots of documentation). Still (!) requires external Oracle client software (at least as of version 5.10.0, build 1003)

56. The files listed below are available at http://homepages.wmich.edu/~zimmer/files/eugm2008 dbi.ppt this presentation nameletters.pl for every letter, find number of patrons whose last names begin with that letter retirees.pl get barcodes of retirees who’ve borrowed something in the last year dbddrivers.pl get list of DBD drivers on your system findbadleader.pl find bib records with bad leaders (not ending in 4500) testbind.pl demonstrates varying query method efficiencies Resources

57. CPAN http://cpan.org DBI http://search.cpan.org/~timb/DBI-1.605/DBI.pm DBD Oracle http://search.cpan.org/~pythian/DBD-Oracle-1.21/Oracle.pm Active State Perl http://activestate.com/downloads/index.mhtml There is also a book on Perl and DBI. I’d recommend using it along with the most current documentation from the CPAN sites above. Resources

58. Thank you for listening. Questions? [email_address] Picture © 2008 by Roy Zimmer