Cummingsdceluna2012

From Voyager to your website: Using Linux Shell scripts and Oracle SQL*Plus to generate web pages.
ExLibris ELUNA 2012 Salt Lake City, UT 1 WRLC & George Washington University
Michael Cummings
The George Washington University
James O. Austin
Washington Research Library Consortium
From Voyager to your website
Using Linux Shell scripts and Oracle SQL*Plus to generate web pages
ELUNA 2012 Salt Lake City, UT

• About WRLC and GWU
• WRLC services / reports
• ExLibris MS Access reports
• Oracle SQL*Plus
• GWU New Books List
• Steps to creating the List
• Setup at Your Library
Install SQL*Plus on Linux
Get the newbooks scripts
Code Walk Through
• Questions?
Agenda
Agenda

Georgetown
University
American
University
Catholic
University
Gallaudet
University
About WRLC and GWU
Voyager 8
9.6 million bibliographic records
10.5 million holdings
5.8 million items

“Canned Reports”
WRLC provides technical services, including reports

Microsoft Access reports from ExLibris are often utilized
Reports.mdb

Linux
Alternative to MS Access: Oracle SQL*Plus (for Linux)

George Washington University Libraries New Books List one example

George Washington University Libraries New Books List Updated weekly

George Washington University Libraries New Books List Call Number Guide

George Washington University Libraries New Books List RSS Menu

George Washington University Libraries New Books List RSS feed

George Washington University New Titles, Media another example

George Washington University New Titles, Media

Steps to creating the New Book Shelf web page Pick and Scan

<1> Connect to the web server
via SSH.
Oracle SQL*Plus
Shell / Perl
you could
use a cron
task instead
Shell scripts and SQL*Plus
<2> Run the newbookshelf shell script.
Steps to creating the New Book Shelf web page

Setup at your library

• These instructions have been validated using
Ubuntu 10.04 LTS in both i686 and x64 desktop
and server editions.
• In order to download files from Oracle, you will
need to register for a free account at
Oracle.com
• We presume in these instructions that you are
using the BASH shell.
• The person performing the setup needs root
user sudo rights, or root account.
Assumptions

Install SQL*Plus on Linux Get files from Oracle
• To obtain the required installable packages, navigate to
http://www.oracle.com/technetwork/indexes/downloads/index.html
Instant Client Package - Basic
Instant Client Package - SQL*Plus
• Scroll down to the Database category and choose Instant Client.
• Choose the appropriate Operating System.
• Check the box indicating you accept the licensing agreement.
• Download the two client packages listed below in .rpm format.
We recommend 10.2.0.4
• Transfer the two files to the Linux server if you downloaded them
somewhere else first.

Install SQL*Plus on Linux Install files from Oracle
The next steps are completed in a terminal window.
Convert the .rpm packages to .deb to install them in Ubuntu.
Do this by using an application called 'alien'.
If it is not already installed, you can do so by typing:
sudo apt-get install alien
Next, navigate to where you saved the .rpm packages and type:
sudo alien -k oracle-instantclient*.rpm
The .deb packages are ready to be installed. Do so by typing:
sudo dpkg -i oracle-instantclient*basic*.deb
sudo dpkg -i oracle-instantclient*sqlplus*.deb

The library files are installed but we must create an environment variables
called LD_LIBRARY_PATH and SQLPATH in order for SQL*Plus to locate
them. We'll do this by typing:
gedit ~/.bashrc and pasting into it:
export LD_LIBRARY_PATH=/usr/lib/oracle/10.2.0.4/client64/lib
export SQLPATH=/usr/lib/oracle/10.2.0.4/client64/lib
Note: The path to ../10.2.0.4/client64/lib will be different depending on which version of SQL*Plus you
installed and which architecture your platform runs on. Change '10.2.0.4' and 'client64' part of the
path accordingly. If you are installing on a server platform use vi or nano instead of gedit.
We need to fix one more dependency, so do: sudo apt-get install libaio1
You may now go ahead and delete the .rpm files downloaded and subsequent
.deb that were created.
Install files from Oracle

Oracle SQL*Plus is now installed and configured. Create a blank
tnsnames.ora file in your home directory by typing:
touch ~/tnsnames.ora
Create an environment variable so SQL*Plus can reference this tnsnames.ora
file we created.
Type: gedit ~/.bashrc and paste into it: export TNS_ADMIN=~
Logout of your account and then log back in to apply the changes to your
shell.
Note: use sqlplus or sqlplus64 as appropriate and change your USERNAME, PASSWORD, HOSTNAME, PORT and
SID as applicable.
You can test your installation by typing on one line:
sqlplus
USERNAME/PASSWORD@'(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(
HOST=HOSTNAME)(PORT=1521)))(CONNECT_DATA=(SID=VGER)))‘

You can put the appropriate information in your tnsnames.ora file using
the following as a template:
IDENTIFIER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL= TCP)(Host=HOSTNAME)(Port= 1521))
(CONNECT_DATA = (SID = VGER))
)
You will then be able to connect with the following command:
sqlplus USERNAME/PASSWORD@IDENTIFIER

Download the files from https://github.com/cummingsm/VoyagerELUNA2012
Get the newbooks scripts, install instructions, this slide file

pageHeader.template
lcguideHeaderpt1.out
lcguideFactory.out
authorsectionHeader.template
callnumbersectionFactory.out
authorsectionFactory.out
titlesectionHeader.template
titlesectionFactory.out
pagefooterFactory.out
Sections of newlibrarybooks.html
The completed web page is a
combination of several sections, some
are boilerplate templates, others are
rows of information output produced by
the script having parsed and
reformatted the SQL results.
• Page header
• The LC Call Number and RSS guide
• List sorted by Call Number
• List sorted by Author
• List sorted by Title
• Page footer
We also generate separate RSS pages.

newbookshelf.sh
This Shell script calls the other shell and Perl scripts,
and the SQLplus query. (Snippet shown on next slide)
sqlplusQuery.sh
Reads several parameters from the configuration file config.cnf
(Oracle username, password, port, host, etc) and generates
tempQuery.sh. It saves a SQL statement in newbook.sql.
Then it runs tempQuery.sh which saves output in
a tab-delimited file named tbl_newbooklocation.tab
config.cnf
This is the local configuration setting file.
Example: ORACLEPORT=1521
You must edit it.
Code walk-through pg 1 of 17

log=/tmp/newlibrarybooks.log
echo "newbookshelf.sh " > $log
date >> $log
# run the SQLPlus query
echo "calling sqlplusQuery.sh" >> $log
sh sqlplusQuery.sh;
echo -n ':::::::::: Processing the new libray books query output :';
#
echo 'Step 1: calling doPrelim.sh to save previous file' >> $log
sh doPrelim.sh
echo -n ':'
echo 'Step 2: calling sortCallNumbers.sh (and sortLC.pl) ' >> $log
sh sortCallNumbers.sh;
.
. s n i p
.
echo 'DONE!'
echo 'Copy newlibrarybooks.html file to web site directory; copy /rss
directory and files to a directory below the one containing
newlibrarybooks.html.'
newbookshelf.shCode walk-through pg 2 of 17
User runs
this script.
newbookshelf.sh calls the other scripts and creates the log file.

#
ORACLEUSER =`grep ORACLEUSER config.cnf | cut -f2 -d'='`
ORACLEPWD =`grep ORACLEPWD config.cnf | cut -f2 -d'='`
ORACLEHOST =`grep ORACLEHOST config.cnf | cut -f2 -d'='`
ORACLEPORT =`grep ORACLEPORT config.cnf | cut -f2 -d'='`
ORACLESID =`grep ORACLESID config.cnf | cut -f2 -d'='`
ORACLEPATH =`grep ORACLEPATH config.cnf | cut -f2 -d'='`
myTEMPLOC =`grep TEMPLOC config.cnf | cut -f2 -d'='`
#
LD_LIBRARY_PATH=$ORACLEPATH
SQLPATH=$ORACLEPATH
#
# Generate the complete sql file by merging templates with values from the config file
#
cat sqlplus_newbooktop.template > newbook.sql
echo "item.temp_location='$myTEMPLOC'" >> newbook.sql
cat sqlplus_newbookbottom.template >> newbook.sql
sqlplusQuery.shCode walk-through pg 3 of 17
Read config
file. Make
SQL from
template.
sqlplusQuery.sh generates the SQL command and query statement

# Output commands,parameters to shell script
cmdstring_exec=$SQLPATH/sqlplus
cmdstring_acct=$ORACLEUSER/$ORACLEPWD@
cmdstring_desc="'(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)
(HOST=$ORACLEHOST)(PORT=$ORACLEPORT)))(CONNECT_DATA=(SID=$ORACLESID)))'"
cmdstring_file="< newbook.sql"
echo 'LD_LIBRARY_PATH='$ORACLEPATH > tempQuery.sh
echo 'SQL_PATH='$ORACLEPATH >> tempQuery.sh
echo 'export LD_LIBRARY_PATH' >> tempQuery.sh
echo 'export SQLPATH' >> tempQuery.sh
echo $cmdstring_exec $cmdstring_acct$cmdstring_desc $cmdstring_file >>
tempQuery.sh
#
# run the script
#
sh tempQuery.sh
sqlplusQuery.sh
Generate
a command
line; run it
sqlplusQuery.sh executes the query

set ECHO OFF
set COLSEP ' '
set LINESIZE 3000
set PAGESIZE 500
set WRAP OFF
set HEADING OFF
set FEEDBACK OFF
set SCAN OFF
set ESCAPE
column callnum Format A30
column callnumgroup Format A1
column title Format A255
column bib Format 99999999
SPOOL tbl_newbooklocation.tab
http://ss64.com/ora/syntax-sqlplus-set.html
SQL*PLUS set statements
Optional
formatting
newbook.sql

select
bib_vw.call_no as callnum,
bib_vw.call_no as callnumgroup,
bib_item.bib_id as bib,
bib_text.title,
bib_text.author || ' ' as author_td,
publisher || ' ' || pub_place || ' ' || publisher_date as pub
from
bib_vw,
bib_item,
bib_text,
item
SQL Query
Author
always has
the string
 

SQL Query
where
item.temp_location='$myTEMPLOC'
and
bib_vw.bib_id=bib_text.bib_id
and
bib_text.bib_id=bib_item.bib_id
and
bib_item.item_id=item.item_id
and
bib_vw.call_no NOT LIKE 'GW%'
and
…etc
bib_vw.call_no NOT LIKE '9%' and bib_vw.call_no
is not NULL
order by bib_vw.call_no
/
SPOOL OFF
We filter
certain
callnums
Temp loc
from the
config file

sortCallNumbers.sh
Removes the query statement from the query output and then
passes the output to a Perl script (sortLC.pl) for better
sorting by LC Class call number. Result is sortedfile.tab
callNumberFactory.sh
This script and others with “Factory” in the name produce a
file with snippets of HTML code.
Using awk, read sortedfile.tab and generate the HTML section
listing the titles sorted by call number.
doPrelim.sh
Removes problem characters from tbl_newbooklocation.tab
Saves a backup copy of the previous html and sql output files

callNumberFactory.sh generates callnumberFactory.out
callnumberFactory.out contains rows for an HTML table.
The lines will look like this in a browser:
callNumberFactory.shCode walk-through pg 9 of 17

# Initialize flag to indicate if a row was printed.
# This flag determines whether to print a shaded row or not
rowprint=0;
#
max=`wc -l sortedfile.tab | cut -c1-3`
for ((i=1; i<=$max; i++))
do
#
# Grab one line of data from the file and save it temporarily in a file
#
awk NR==$i sortedfile.tab> htmlfactory.tmp
# Get contents of fields from the temporary file into table data elements
td1=`cat htmlfactory.tmp | cut -f1`
tab is the
default
delimiter
callNumberFactory.sh reads the fields in the SQL output file

# Set the background color of each row.
if [ "$rowprint" == "0" ]; then
rowcolor=FFFFFF; rowprint=1;
else
rowcolor=E4DFE6; rowprint=0;
fi
# Generate HTML tags around the retrieved data
NEWROW="<TR BGCOLOR=#"$rowcolor" valign=TOP><TD grp="$td2">
        "
CALLNUMBER="<A HREF="http://"$myOPAC"/cgi-bin/Pwebrecon.cgi?BBID=
"$td3"">"$td1"</A></TD>";
TITLE="<TD>"$td4"</TD>";
AUTHOR="<TD>"$td5" </TD>";
PUBLISHER="<TD>"$td6"</TD></TR>";
#
# Append each generated HTML line to the output file
echo $NEWROW $CALLNUMBER $TITLE $AUTHOR $PUBLISHER >> callnumberFactory.out
done
callNumberFactory.sh outputs HTML with field values

callNumberSectionFactory.sh
callNumberSectionFactory.sh prints headings and inserts relevant lines
from callnumberFactory.out after the heading.
callNumberSectionFactory.sh generates callnumbersectionFactory.out
Now the output will look like this:

lcguideFactory.sh
lcguideFactory.sh generates lcguideFactory.out
The RSS pages generated by this script are the .xml files in the rss directory
The RSS icon displayed on the page is feed.png
rsspageFactory.sh and makeRSS.pl generate multiple .xml files

authorSectionFactory.sh & titleSectionFactory.sh
authorSectionFactory.sh generates authorsectionFactory.out
titleSectionFactory.sh generates titlesectionFactory.out

webPageMergeFactory.sh
WEB=webpagemergeFactory.out
echo ''> $WEB
# Static HTML text page header with date stamp
cat pageHeader.template >> $WEB
cat lcguideHeaderpt1.template >> $WEB
date >> $WEB
cat lcguideHeaderpt2.template >> $WEB
cat lcguideFactory.out >> $WEB
# call number section
cat callnumbersectionFactory.out >> $WEB
# Heading for the author section
cat authorsectionHeader.template >> $WEB
webPageMergeFactory.sh generates webpagemergeFactory.out
Continues
next page…

# Rows of titles by author
cat authorsectionFactory.out >> $WEB
#
# Heading for the title section
cat titlesectionHeader.template >> $WEB
#
# Rows of titles
cat titlesectionFactory.out >> $WEB
#
# Static HTML text footer
cat pagefooterFactory.out >> $WEB
#
cp $WEB $HOME/public_html/newlibrarybooks.html
#
webPageMergeFactory.sh
pageHeader.template
lcguideFactory.out
authorsectionHeader.template
callnumbersectionFactory.out
authorsectionFactory.out
titlesectionHeader.template
titlesectionFactory.out
pagefooterFactory.out
Sections of newlibrarybooks.html

The XML files for RSS pages are saved in the directory named “rss”.
There is one file per LC Class heading.
RSS files
newlibrarybooksB.xml
newlibrarybooksC.xml
newlibrarybooksD.xml
newlibrarybooksH.xml
.
.
.
newlibrarybooksR.xml
newlibrarybooksT.xml
/rss

The last steps are publishing the HTML and RSS files
1. Copy $HOME/public_html/newlibrarybooks.html to the directory
where your web server will display the page. (Copy rssnb.png too)
2. Copy the directory rss, and the .xml files it contains, to a directory
named rss located under the directory where you have placed
newlibrarybooks.html
• At GWU we perform the above steps with commands placed at
the end of the webpagemergeFactory.sh file
Publish the files

You may want to customize further
• Add steps to the end of webPageMergeFactory.sh
that copy the files to your web directory
• Customize the page header template with your
own library branding
• You may trigger the process automatically
with a cron task.
• Edit the SQL template for your own needs.
• If you prefer CSS formatting, convert the script from
HTML <table> to CSS <div> statements

QUESTIONS
In this presentation you learned how to:
• install SQL*Plus on Linux
• download GWU’s new books scripts
• edit GWU’s scripts
We hope you can adapt these scripts
for use at your own library.
Questions?

Cummingsdceluna2012

More Related Content

What's hot

Similar to Cummingsdceluna2012

More from Michael Cummings

Recently uploaded

Cummingsdceluna2012