• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
EG Reports - Delicious Data
 

EG Reports - Delicious Data

on

  • 345 views

Delicious Data: Automated Techniques for Complex Reports: Get data into the hands of those that need it most by automating SQL reports, scheduling data extracts using the Evergreen reporter, and ...

Delicious Data: Automated Techniques for Complex Reports: Get data into the hands of those that need it most by automating SQL reports, scheduling data extracts using the Evergreen reporter, and extending the reporter with new source definitions. Ben Shum from Bibliomation and Jeff Godin from the Traverse Area District Library will show you how you can meet your advanced or complex reporting needs, both with and without direct database access. Join us in our efforts to eliminate manual, time-consuming reporting workflows!
Ben Shum (Biblio), Jeff Godin (TADL)

Statistics

Views

Total Views
345
Views on SlideShare
344
Embed Views
1

Actions

Likes
0
Downloads
4
Comments
0

1 Embed 1

http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

EG Reports - Delicious Data EG Reports - Delicious Data Presentation Transcript

  • Delicious Data:Automated Techniques for Complex Reports Presented by Jeff Godin and Benjamin Shum
  • How NOT to run reports...
  • Planning the framework
  • Planning the framework SQL Query Report
  • Planning the framework SQL Query Export Results Report to CSV
  • Planning the framework SQL Query Export Results Report to CSV Convert CSV to XLS
  • Planning the framework SQL Query Export Results Report to CSV Convert Email XLS CSV to XLS to Library
  • Planning the framework SQL Query Export Results Report to CSV Schedule as Cron Job Convert Email XLS CSV to XLS to Library
  • Some StrategiesSchedule as Use our experience Cron Job setting up crontab
  • SQL Query Use our existing Report SQL reportsExport Results CSV is a standard to CSV format to work with
  • ConvertCSV to XLS Find a converterEmail XLS Find an email clientto Library
  • Implementation• Can find online at http://evergreen-ils.org/dokuwiki/doku.php? id=scratchpad:automated_sql_reports
  • Future of the work• Additional formatting• Dealing with reports with zero results• Adding new reports• Implementing more refinements
  • No database access?• Create scheduled reports• Completion notification via e-mail• Script pulls report data• Import and process
  • Don’t do this unless you have to
  • Reports as Exports• Create template that outputs raw data• Schedule report• Enable e-mail notification to special address
  • Act on the e-mail• fetchmail -> procmail -> perl• Report data is automatically downloaded• Within a minute or so
  • act-on-mail.pl#!/usr/bin/perl# called by procmail, and receives the mail message on stdinuse strict; use warnings;my $url;while (<>) { if ($_ =~ /https://reporter/) { $url = $_; last; }}system("/home/stats/bin/fetch-report.pl", "-u", $url);
  • fetch-report.pl logic• Fetch URL like: https://host/reporter/[...]/report-data.html• Automatically log in if needed (keep a cookie)• Download report-data.csv and save to output dir
  • imports.ini example 1[general]dir=/home/stats/reportsstate_dir=/home/stats/state[group holdratio]name=Hold Ratiofile1=holds_outstanding.csvfile2=holdable_copies.csvfile3=bib_info.csvcommand=psql -f /home/stats/sql/holds_init.sql#post-command=/home/stats/bin/mail-holdratio.pl
  • imports.ini example 2[general]dir=/home/stats/reportsstate_dir=/home/stats/state[group items_for_export]name=Items for Exportfile1=items_for_export.csvcommand=psql -f /home/stats/sql/items_for_export.sqlpost-command=/home/stats/bin/export_bibs.pl -f /home/stats/output/bre_ids.csv
  • Import script overview• Runs from cron• Have all input files been updated since the last time I ran this import?• Run import• Update timestamp• Run post-import command
  • The import process• drop table• create table• import data• process to output
  • The post-import• Send the report output somewhere• E-mail notify that “report’s ready at someurl”• Trigger something like a bib export of the records that were output
  • One more thing...• Don’t forget to delete your scheduled report output
  • Custom reportingviews exposed in the IDL
  • SQL views in the IDL• Exposed via reporting interface• Good for on-demand reporting• Don’t break the IDL • xmllint is your friend • pick a class id unlikely to collide
  • <class id="rlcd" controller="open-ils.cstore open-ils.pcrud open-ils.reporter-store" oils_obj:fieldmapper="reporter::last_copy_deleted" oils_persist:readonly="true" reporter:core="true" reporter:label="Last Copy Delete Time" >
  • <oils_persist:source_definition>SELECT b.id, MAX(dcp.edit_date) AS last_delete_dateFROM biblio.record_entry b JOIN asset.call_number cn ON (cn.record = b.id) JOIN asset.copy dcp ON (cn.id = dcp.call_number)WHERE NOT b.deletedGROUP BY b.idHAVING SUM( CASE WHEN NOT dcp.deleted THEN 1ELSE 0 END) = 0
  • <!-- continued --> <fields oils_persist:primary="id"oils_persist:sequence="biblio.record_entry"> <field reporter:label="Record ID" name="id"reporter:datatype="id"/> <field reporter:label="Delete Date/Time"name="last_delete_date" reporter:datatype="timestamp"/> </fields> <links> <link field="id" reltype="has_a" key="id" map=""class="bre"/> </links>
  • <!-- continued --> <permacrud xmlns="http://open-ils.org/spec/opensrf/IDL/permacrud/v1"> <actions> <retrieve/> </actions> </permacrud> </class>
  • The Future• General third party reporting frameworks?• Library Dashboards?• Your Ideas?
  • Photo CreditsSurrounded by papers by Flickr user suratlozowickhttp://www.flickr.com/photos/suratlozowick/4522926028/
  • Thanks!