Automating a Vendor File Load Process with Perl and Shell Scripting

  • 1,254 views
Uploaded on

How we automate the file loading process …

How we automate the file loading process
Presented at ELUNA 2008

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,254
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
28
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Automating a Vendor File Load Process with Perl and Shell Scripting Roy Zimmer Western Michigan University
  • 2. We needed to get Promptcat approval files, from OCLC’s ftp site.
  • 3. We needed to get Promptcat approval files, from OCLC’s ftp site. Historically, we’ve done file retrieval and processing via shell scripting, with some supporting Perl software.
  • 4. We needed to get Promptcat approval files, from OCLC’s ftp site. Historically, we’ve done file retrieval and processing via shell scripting, with some supporting Perl software. In this case, we started out mostly manual, with some programmatic support.
  • 5. We needed to get Promptcat approval files, from OCLC’s ftp site. Historically, we’ve done file retrieval and processing via shell scripting, with some supporting Perl software. In this case, we started out mostly manual, with some programmatic support. We kind of snuck up on the final method of retrieval and processing.
  • 6. The ftp site has quite a few files, including a number of different types: LBL, RPT, APPR, FIRM…
  • 7. Let’s use a representative sample for this presentation…
  • 8. Out of this large number of files, only one or a few will be of interest. For example, take the files for May 7.
  • 9. Out of this large number of files, only one or a few will be of interest. In this case, the files for May 7. How do we pick them out?
  • 10. This is where Perl comes to the rescue. With Perl, you can do many things.
  • 11. Code details, main program, ftp stuff ftppcatappr.pl Required when using ftp within Perl
  • 12. Site password is stored here. Code details, main program, ftp stuff ftppcatappr.pl Required when using ftp within Perl
  • 13. Site password is stored here. Code details, main program, ftp stuff ftppcatappr.pl Required when using ftp within Perl - Site URL - Username - directory where files are - transfer mode
  • 14. Code details, main program, ftp stuff ftppcatappr.pl Self-explanatory Required when using ftp within Perl Site password is stored here. - Site URL - Username - directory where files are - transfer mode
  • 15. Site password is stored in a file. Code details, main program, ftp stuff ftppcatappr.pl
  • 16. Site password is stored in a file. Setting up for FTP Code details, main program, ftp stuff ftppcatappr.pl
  • 17. Retrieve ftp site file listing into a variable as an array of directory entries. Code details, main program, ftp stuff ftppcatappr.pl
  • 18. Set each line up to be split on the space character and then do so. Code details, main program, ftp stuff Retrieve ftp site file listing into a variable as an array of directory entries. ftppcatappr.pl
  • 19. Set each line up to be split on the space character and then do so. The last piece in each line will be the filename. Split this into pieces based on the period. Code details, main program, ftp stuff Retrieve ftp site file listing into a variable as an array of directory entries. ftppcatappr.pl
  • 20. Set each line up to be split on the space character and then do so. The last piece in each line will be the filename. Split this into pieces based on the period. Look for the one(s) that correspond(s) with yesterday’s date and keep those. Code details, main program, ftp stuff Retrieve ftp site file listing into a variable as an array of directory entries. ftppcatappr.pl
  • 21. Want the files to be processed in order Code details, main program, ftp stuff ftppcatappr.pl
  • 22. Code details, main program, processing each file ftppcatappr.pl Get the files
  • 23. Code details, main program, processing each file Records will need some editing… (Thanks to Birong Ho, our systems librarian, for originally supplying this editing code.) Get the records ftppcatappr.pl
  • 24. Records will need some editing… “grab” the fields of interest Get the records Code details, main program, processing each file ftppcatappr.pl
  • 25. Records will need some editing… Some fields are deleted… Code details, main program, processing each file ftppcatappr.pl
  • 26. Records will need some editing… …and others are edited. More edits than this are performed; the basic syntax is the same for each of them. Code details, main program, processing each file ftppcatappr.pl
  • 27. File will need some splitting… Split each file up based on the invoice number found in field 980 |f Code details, main program, processing each file
  • 28. File will need some splitting… Split each file up based on the invoice number found in field 980 |f The next program takes care of this… I did say we snuck up on this, didn’t I? Code details, main program, processing each file
  • 29. File will need some splitting…split each file up based on the invoice number found in field 980 |f Rather than using the familiar LF, the MARC format uses a different EOL character. Code details, helper program, processing each file oclc980.pl
  • 30. File will need some splitting…split each file up based on the invoice number found in field 980 |f This section reads each MARC record, looking for the 980 field. Code details, helper program, processing each file oclc980.pl
  • 31. File will need some splitting…split each file up based on the invoice number found in field 980 |f Get the subfields into an array. Code details, helper program, processing each file oclc980.pl
  • 32. File will need some splitting…split each file up based on the invoice number found in field 980 |f Get the subfields into an array. Code details, helper program, processing each file Look for subfield f and read it to get the invoice number. oclc980.pl
  • 33. File will need some splitting…split each file up based on the invoice number found in field 980 |f Get the subfields into an array. Code details, helper program, processing each file Look for subfield f and read it to get the invoice number. Determine if it’s a new or “existing” invoice number. This also lets us count records for each invoice. oclc980.pl
  • 34. File will need some splitting…split each file up based on the invoice number found in field 980 |f Get the subfields into an array. Code details, helper program, processing each file Look for subfield f and read it to get the invoice number. Determine if it’s a new or “existing” invoice number. This also lets us count records for each invoice. Use append mode to open, write a record, and close the file for each invoice number. oclc980.pl
  • 35. There are usually several files after splitting the file being processed. Each one must be further processed and loaded into Voyager. This is controlled via a small shell script. Code details, helper program, processing each invoice file
  • 36. There are usually several files after splitting the file being processed. Each one must be further processed and loaded into Voyager. This is controlled via a small shell script. It calls another shell script for preprocessing and bulk loading of each of the invoice files. Code details, helper program, processing each invoice file (Thanks to Keith Kelley, director of systems, for creating this script.) importall.sh
  • 37. Code details, helper program, importing each invoice file $1 is the default first parameter to the script. Let’s use a more descriptive variable. prodimport.script
  • 38. Code details, helper program, importing each invoice file $1 is the default first parameter to the script. Let’s use a more descriptive variable. Let’s also drop the filename extension, so that we can “reuse” the filename. prodimport.script
  • 39. Code details, helper program, importing each invoice file $1 is the default first parameter to the script. Let’s use a more descriptive variable. Let’s also drop the filename extension, so that we can “reuse” the filename. Get ready for the prebulk processing. prodimport.script
  • 40. Code details, helper program, importing each invoice file prodimport.script 1 st use of file referenced by $1, so that use is OK
  • 41. Code details, helper program, importing each invoice file Start with some final edits… prodimport.script
  • 42. Code details, helper program, importing each invoice file Start with some final edits… We’ll use marcedit.pl to replace the contents of field 981 |a, as illustrated: There are 59 such edits possible. prodimport.script
  • 43. Code details, helper program, importing each invoice file Start with some final edits… Prep for bulkimport, too prodimport.script
  • 44. Code details, helper program, importing each invoice file Start with some final edits… Prep for bulkimport, too Self-explanatory prodimport.script
  • 45. Code details, helper program, importing each invoice file Prebulk output is bulkimport input. Perform the bulkimport prodimport.script
  • 46. Code details, helper program, importing each invoice file Prebulk output is bulkimport input. Perform the bulkimport The final step for each file is to do some cleanup and moving files to the loaded directory. prodimport.script
  • 47. Password maintenance The ftp site requires us to change our password every 90 days. We wanted all this to run hands-off, so that had to be automated, also. The password gets changed every two months.
  • 48. Password maintenance, getpromptcatpw.ksh
  • 49. Password maintenance, getpromptcatpw.ksh
  • 50. Password maintenance, pwgen.pl Want an 8-character password
  • 51. Password maintenance, pwgen.pl Want an 8-character password Password length defaults to 10
  • 52. Password maintenance, pwgen.pl Want an 8-character password Password length defaults to 10 Password consists of these characters
  • 53. Password maintenance, pwgen.pl Want an 8-character password Password length defaults to 10 Password consists of these characters Seed the random number generator
  • 54. Password maintenance, pwgen.pl Want an 8-character password Password length defaults to 10 Password consists of these characters Seed the random number generator Generate the password
  • 55. Review Run ftppcatappr.pl Login to OCLC ftp site for promptcat Find desired files and retrieve them Do process each file remove unwanted 6xx, 938, 948 fields edit some 856 fields run oclc980.pl do process each record in the current file look at the 980 |f (contains the invoice number) if it contains invoice NNN, (create and) put this record in file NNN.marc, etc. end do run importall.sh do process each file created by oclc980.pl run prodimport.script use marcedit.pl to process 981 |a replacements (59 possible edits) prebulk bulk import wait 1.5 minutes before continuing end do move all interim .marc, .preimp, and .imp files to /loaded End do Move all RCD* files to /loaded
  • 56. Review Run ftppcatappr.pl Login to OCLC ftp site for promptcat Find desired files and retrieve them Do process each file remove unwanted 6xx, 938, 948 fields edit some 856 fields run oclc980.pl do process each record in the current file look at the 980 |f (contains the invoice number) if it contains invoice NNN, (create and) put this record in file NNN.marc, etc. end do run importall.sh do process each file created by oclc980.pl run prodimport.script use marcedit.pl to process 981 |a replacements (59 possible edits) prebulk bulk import wait 1.5 minutes before continuing end do move all interim .marc, .preimp, and .imp files to /loaded End do Move all RCD* files to /loaded
  • 57. The files listed below are available at http://homepages.wmich.edu/~zimmer/files/eugm2008 fileload.ppt this presentation ftppcatappr.pl gets the files and controls the processing oclc980.pl splits retrieved files based on invoice number pwgen.pl generates a password importall.sh ensures that each “split file” for a particular retrieved file is processed prodimport.ksh does the actual processing of each file getpromptpw.ksh handles all the details of a password change Resources except for marcedit.pl enables batch editing of MARC files which is at http://homepages.wmich.edu/~zimmer/marc_index.html
  • 58. CPAN http://cpan.org FTP http://search.cpan.org/~gbarr/libnet-1.22/Net/FTP.pm I’m not sure if the FTP module is supplied on Voyager boxes or not. If you don’t have it, go to the above URL. It also has good documentation on this module. Resources
  • 59. Thank you for listening. Roy Zimmer [email_address] Picture © 2008 by Roy Zimmer