Integrating Perl and Databases Making simple things easy


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Why? Why are you here? Why are so many developers and companies using Perl? Why Perl?
  • I’m sure other speakers are giving their reasons. These are some of mine: If I had top give one reason: Fewer lines of code. Higher level language => less code => less errors (Mythical Man Month) => Consider “A picture is worth a thousand words” - rich in powerful idioms (compact but not so terse as to be unreadable later - like APL) vs “Can’t see the wood for the trees” - in other languages like C. “ No limits” - often hard to do in other languages where much more care/effort id required Exception handling - simplifies logic - less code - less errors Rapid prototyping Rapid response to change - simply less code to maintain Many free reusable modules Perl is an approriate language for a very wide range of tasks. Learning Perl often avoids the need to learn one or more other tools.
  • => most data and database related software does one or more of these. Commercial tools for these tend to lack the flexibility of Perl (in breadth and depth). You can find yourself wasting time working around the limitations of blunt commercial tools.
  • Very few lines of code. Just about every line directly related to the task. Not the best code in the world but it gets the job done. (The split could be replaced with two lines of code to do the right thing with quoted fields and embedded commas, for example.)
  • Here we use perl’s built-in formatting mechanism to define the top of page headers and the layout and formatting of the body rows. Again: very few lines of code and just about every line directly related to the task.
  • Here we read each line from the costs text file, skip lines that don't start with digits, extract the product and cost, apply the margin (or a default), apply the note (or a default), and write the output. Again: very few lines of code and just about every line directly related to the task.
  • The company grows and now stocks thousands of products. To save time the derived price for each product can be saved in a simple DBM' database (support for which is built into perl) and made available to other perl programs. Just two additional lines of code is all that's needed. (A similar pair of lines is all that's needed to read the data.)
  • Here we add just 8 lines to store the prices in the database. The code not only updates prices for products already in the database but also inserts new records for any products not in the database.
  • Here we have changed just 4 lines to fetch the costs from the database.
  • Once upon a time: Perl4: oraperl, ingperl, uniperl, fooperl Ted Lemon: “Wouldn’t it be nice if” - Sept 29th 1992 Applications portable between databases Skills portable between databases! A vision of the future - Talk, talk, talk- the dbperl-interest mailing list Buzz Moschetti, Kevin Stock, Kurt Andersen, Ted Lemon June 93: Kevin and Kurt changed jobs and left the list. 30 people on the list. I took the lead at that point. Jan 1994: Dbperl spec is progressing and Perl 5 appears on the horizon June 1994: Perl 5 alpha releases becoming stable. Almost 200 people on list. Prototype pure-perl ‘testbed’ released for experimentation. Sept 1994 (Two years and 600 messages - 2MB - later...): Renamed to DBI and DBD. First release of the DBI (with a pure-perl driver) The future is now: continuous development interface stabilizing and more fully documented
  • Similar in concept to an ODBC Driver Manager DBI provided methods, functions and tools etc help ensure conformant implementations and efficient implementations and more automatic support for new features
  • Porting All Unix variants - (not heard of any it won’t build on) Windows 95, Windows NT, VMS Builds out-of-the-box Drivers ODBC, Oracle, Informix, DB2, Ingres, Solid, … enhancements to the DBI (inc. docs) will make it easier ODBC/JDBC Standard ways of specifying (data types, meta-data, options etc) and ways of doing The DBI won’t re-invent these wheels - but may make them easier to steer! (by offering higher-level interfaces to them). First is data types and meta-data Specifying the unspecified Documenting features once they’ve stabilized such as placeholders and bind variables
  • No particular order or time-scale! Generic standards-based SQL92 Parser (in C) supplied with the DBI but not tied closely to it. Available for use by any module. SQL interfaces could appear on all sorts of modules. Outputs a parse tree as a perl recursive data structure. A DBD::Proxy to ‘forward’ method calls and arguments across a network to any normal driver on a remote machine and return results A DBD::Pipe to act as a virtual base class to simplify the development of drivers for data sources that don’t offer a programming interface/API. Data types: date/time/interval - returned as a ref with ““ overloaded. Define interface for non-blocking: $dbh->{NonBlock}, $h->done Initially Oracle and ODBC drivers. Only if database API supports it. Make any changes required to support Threading in DB (if any) Individual drivers may need to declare themselves ‘thread-safe’ If needed the DBI may block multi-threads from entering same driver. Binding output parameters, typically for stored procedures. Binding Arrays - especially Oracle
  • Native API’s will generally offer the most power, e.g., Oracle OCI. But take a lot of effort to learn and use. Knowledge is rarely portable to other APIs Generic application language interfaces: ODBC and JDBC . Still lots of effort to learn Knowledge is portable to other APIs Power is dependant on quality of ODBC driver Pre-compilers for C (where SQL is embedded in C and ‘precompiled’) Simpler than a raw C API but typically less power. Hard to debug. Command line access - Just type it in! Generate SQL from scripts etc. Not robust - hard to check for and respond to errors Script Languages TCL (oratcl, sybtcl) generally much slower. Binary data hard. Win32::ODBC - a bottom-up approach - thin layer over raw ODBC calls. DBI is higher level but DBD::ODBC doesn’t yet have the same breadth of ODBC specific functionality. Recommended for existing users and people who need that degree of control. DBI+DBD::ODBC can run over twice as fast as Win32::ODBC. Sybase - and other database specific perl extensions Can gain some performance advantage if well written but loose out on the DBI extras: portable apps & knowledge, tools, subclasses etc.
  • It delivers what it promises It works! It’s here, there and everywhere Portable and widely ported It’s fast, flexible and well proven Thousands of users It’s free, with source on the same terms as Perl itself Commercial support is available From or It has a large user base and a strong future
  • So What? What’s in it for you? Simply: Greater success, faster and for less effort.
  • To show that the value of Integrating Perl and Databases isn’t all hot air and hype I offer you three Success stories from commercial users. (The British Telecom CMI service is described in detail in the Perl Conference handouts.)
  • Integrating Perl and Databases Making simple things easy

    1. 1. Integrating Perl and Databases Making simple things easy and difficult things possible Tim Bunce August 1997
    2. 2. Why?
    3. 3. Why Perl? <ul><li>Fewer lines of code </li></ul><ul><ul><li>faster development / fewer errors / easier maintenance </li></ul></ul><ul><li>“ A picture is worth a thousand words” vs “Can’t see the wood for the trees” </li></ul><ul><li>Freedom of expression - “There’s more than one way to do it” </li></ul><ul><li>The tool builders tool factory </li></ul>
    4. 4. Why Perl for Databases? <ul><li>Ideally suited to data processing : </li></ul><ul><ul><li>parsing , cleansing / filtering / merging and formatting </li></ul></ul><ul><li>Applications </li></ul><ul><ul><li>Data Migration </li></ul></ul><ul><ul><li>Data acquisition and reporting </li></ul></ul><ul><ul><li>Web Page creation </li></ul></ul><ul><li>Building sharper tools faster </li></ul>
    5. 5. Your Mission... should you choose to accept it...
    6. 6. The Task <ul><li>Read a tabular text report file of product codes and costs </li></ul><ul><li>CODE DESCRIPTION COST OTHER NOTES </li></ul><ul><li>---- ----------- ------ ------------------- </li></ul><ul><li>1201 WIDGET 37A 500.00 LEFT HANDED ONLY </li></ul><ul><li>1202 BRACKET X7J 790.00 REDUCED FROM 800.00 </li></ul><ul><li>1203 FLANGE 9Q7 449.00 LIMITED STOCK </li></ul><ul><li>Read a CSV file of product codes, price markup and comments </li></ul><ul><li>1201,0.90,Special Offer for limited period only </li></ul><ul><li>1202,1.30, </li></ul><ul><li>Output a new tabular report containing the new product prices and comments word wrapped in a narrow field </li></ul>
    7. 7. Reading the CSV file <ul><li>open MARGIN, “<margin.csv” or die “open margin.csv: $!”; </li></ul><ul><li>while (<MARGIN>) { </li></ul><ul><li>chop; </li></ul><ul><li>($prod, $margin, $notes) = split /,/; </li></ul><ul><li>$prod_margin{$prod} = $margin; </li></ul><ul><li>$prod_notes{$prod} = $notes; </li></ul><ul><li>} </li></ul>
    8. 8. Defining the Output Format <ul><li>format STDOUT_TOP = </li></ul><ul><li>Code Price Notes </li></ul><ul><li>---- ------- -------- </li></ul><ul><li>. </li></ul><ul><li>format STDOUT = </li></ul><ul><li>@>>> ^###.## ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< </li></ul><ul><li>$prod, $cost, $note </li></ul><ul><li>~~ ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< </li></ul><ul><li>$note </li></ul><ul><li>. </li></ul>
    9. 9. Writing the New Report <ul><li>open COST, &quot;<cost.txt&quot; or die &quot;open cost.txt: $!&quot;; </li></ul><ul><li>while ($line = <COST>) { </li></ul><ul><li>next unless $line =~ m/^d/; </li></ul><ul><li>($prod, $cost) = unpack 'A4 @20 A7', $line; </li></ul><ul><li>$cost *= $prod_margin{$prod} || 1.10; </li></ul><ul><li>$note = $prod_notes{$prod} || 'n/a'; </li></ul><ul><li>write; # using format with $prod, $cost and $note </li></ul><ul><li>} </li></ul>
    10. 10. Mission Accomplished! <ul><li>Code Price Notes </li></ul><ul><li>---- ------- -------- </li></ul><ul><li>1201 450.00 Special offer for limited </li></ul><ul><li>period only </li></ul><ul><li>1202 1027.00 n/a </li></ul><ul><li>1203 493.90 n/a </li></ul>
    11. 11. Change is the only Constant Your new mission...
    12. 12. “Cache the data for other uses!” <ul><li>dbmopen(%price_cache, &quot;price_cache&quot;, 0644) or die …; </li></ul><ul><li>while ($line = <COST>) { </li></ul><ul><li>. . . </li></ul><ul><li>write; # using format with $prod, $cost and $note </li></ul><ul><li> $price_cache{$prod} = &quot;$cost $note&quot;; </li></ul><ul><li>} </li></ul>
    13. 13. “Update the Prices in a Database!” <ul><li>use DBI; </li></ul><ul><li>$db = DBI->connect('dbi:ODBC:PRICE', 'user', 'password', </li></ul><ul><li>{ RaiseError => 1 }); </li></ul><ul><li>$upd = $db->prepare('UPDATE prices SET price=? WHERE prod=?'); </li></ul><ul><li>$ins = $db->prepare('INSERT INTO prices(prod,price) VALUES (?,?)'); </li></ul><ul><li>while ($line = <COST>) { </li></ul><ul><li>. . . </li></ul><ul><li>$rows = $upd->execute($price, $prod); </li></ul><ul><li>$ins->execute($prod,$price) if $rows == 0; </li></ul><ul><li>} </li></ul><ul><li>$db->commit; </li></ul>
    14. 14. “ Now get Cost Data from a Database!” <ul><li>Original Code </li></ul><ul><li>open COST, &quot;<cost.txt&quot; or die &quot;open cost.txt: $!&quot;; </li></ul><ul><li>while ($line = <COST>) { </li></ul><ul><li>next unless $line =~ m/^d/; </li></ul><ul><li>($prod, $cost) = unpack 'A4 @20 A7', $line; </li></ul><ul><li>. . . </li></ul><ul><li>New Code </li></ul><ul><li>$costs = $cost_db->prepare('SELECT prod,cost FROM costs'); </li></ul><ul><li>$costs->execute; </li></ul><ul><li>while ( ($prod, $cost) = $costs->fetchrow_array ) { </li></ul><ul><li>. . . </li></ul>
    15. 15. What Is the Perl DBI? <ul><li>The standard Database Interface for Perl </li></ul><ul><li>“A perl module and specification that defines a consistent database interface independent of the actual database being used” </li></ul>
    16. 16. Why the Perl DBI? <ul><li>Once upon a time… </li></ul><ul><ul><li>One language, many database interfaces </li></ul></ul><ul><li>A vision of the future... “DBperl” </li></ul><ul><ul><li>Talk, talk, talk - the dbperl-interest mailing list </li></ul></ul><ul><li>Perl 5 - A new way </li></ul><ul><ul><li>Modules and Objects. The DBI is born. </li></ul></ul><ul><li>The future is now… </li></ul><ul><ul><li>ODBC, Oracle, Informix, Ingres, mSQL, mysql, DB2, Solid, Sybase , Postgress , Quickbase , Empress , Fulcrum, ... </li></ul></ul>
    17. 17. Making simple things easy and difficult things possible <ul><li>Goals </li></ul><ul><ul><li>Be simple to use for simple applications </li></ul></ul><ul><ul><li>Have sufficient flexibility to accommodate unusual functionality and non-SQL databases </li></ul></ul><ul><ul><li>Conform to applicable standards (ODBC etc.) </li></ul></ul><ul><ul><li>Enable the creation of database-independent Perl scripts without being limited to the lowest functionality </li></ul></ul><ul><ul><li>Be free. </li></ul></ul><ul><li>A ‘higher-level’ interface than ODBC/JDBC </li></ul>
    18. 18. Getting Technical... for the techies...
    19. 19. Under the Hood <ul><li>DBI defines and implements an interface </li></ul><ul><li>Driver modules do much of the real work </li></ul><ul><li>DBI provides default methods, functions, tools etc for drivers </li></ul><ul><li>Not limited to the lowest common denominator - mechanism provided for driver specific extensions </li></ul><ul><li>Designed and built for speed </li></ul><ul><li>Valuable detailed call tracing/debugging built-in </li></ul>
    20. 20. A Picture is Worth... DBI Module Perl Application DBD:: Other DBD::Informix DBD::Oracle Oracle Server Informix Server Other Server
    21. 21. State of the Nation <ul><li>DBI builds and runs on many systems </li></ul><ul><ul><li>Unix, Windows 95, Windows NT, VMS etc </li></ul></ul><ul><li>Many drivers exist </li></ul><ul><ul><li>more in development </li></ul></ul><ul><li>Merging the ODBC/JDBC ‘inheritance’ </li></ul><ul><ul><li>data types, meta-data etc </li></ul></ul><ul><li>Specifying the unspecified </li></ul><ul><ul><li>Documenting the undocumented </li></ul></ul>
    22. 22. What Next? <ul><li>SQL for Non-SQL data sources </li></ul><ul><li>Networking for Non-Networked data sources </li></ul><ul><li>Template driver for command line interfaces </li></ul><ul><li>Data types, especially date/time/interval </li></ul><ul><li>Non-blocking (method calls return at once) </li></ul><ul><li>Multi-threading </li></ul><ul><li>Binding In/Out Variables </li></ul><ul><li>Binding Arrays </li></ul>
    23. 23. Comparing DBI and Alternatives (Attempting the Impossible!) “ Power” “ Effort” Command Line Native API ODBC/ JDBC and drivers DBI and Drivers Precompilers Win32::ODBC TCL (oratcl, sybtcl) Please note: This chart is correct only for some values of correct ! Sybase::DBlib
    24. 24. So why use the Perl DBI? <ul><li>Because... </li></ul><ul><ul><li>It delivers what it promises </li></ul></ul><ul><ul><li>It’s here, there and everywhere </li></ul></ul><ul><ul><li>It’s fast, flexible and well proven </li></ul></ul><ul><ul><li>It’s free, with source </li></ul></ul><ul><ul><li>Commercial support is available </li></ul></ul><ul><ul><li>It has a large user base and a strong future </li></ul></ul>
    25. 25. So What?
    26. 26. Success Stories in Industry <ul><li>Large Scale Data Migration </li></ul><ul><ul><ul><li>by Michael Peppler </li></ul></ul></ul><ul><li>Database Interfaced Web Pages </li></ul><ul><ul><ul><li>by Frank San Filippo </li></ul></ul></ul><ul><li>The British Telecom Call Management Information Service </li></ul><ul><ul><ul><li>by Tim Bunce </li></ul></ul></ul>
    27. 27. Too Good to Be True? <ul><li>Just three examples of many </li></ul><ul><li>Thousands of Perl DBI developers around the world </li></ul><ul><li>Millions of users of DBI powered web sites </li></ul><ul><li>Oracle and Apache web servers with built-in Perl and DBI </li></ul><ul><li>Commercial support available from The Perl Clinic </li></ul>
    28. 28. And finally... <ul><li>“ We have replaced all Netscape JavaScript code with Perl DBI/DBD to Informix” - Del Simmons, Cox Interactive Media. </li></ul><ul><li>“ I have found performance and stability to be exceptional” - Stuart Huges, PECC. </li></ul><ul><li>“ Perl and DBI saved our butt. The tool we came closest to purchasing was SQR but it just doesn’t have the flexibility and power of Perl/DBI.” - Jared Still, RxNet. </li></ul><ul><li>“ I have been able to recommend the Perl/DBI route to a number of our clients and all of them have been pleased with the results” - Paul Rubenis, Ulysses Telemedia Networks </li></ul>
    29. 29. The end. Till next year...