• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content




slides for YAPC::Asia 2012

slides for YAPC::Asia 2012



Total Views
Views on SlideShare
Embed Views



4 Embeds 73

http://yapcasia.org 58
https://twitter.com 11
http://leapf.org 3
https://si0.twimg.com 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    DBD::SQLite DBD::SQLite Presentation Transcript

    • DBD::SQLiteRecipes, Issues, and Plans Kenichi Ishigaki (@charsbar) YAPC::Asia 2012 Sep 28, 2012
    • Thank youfor coming.
    • Today I‘ll showyou 10 recipes and 10 issues.
    • Recipe 1: Bulk InsertRecipe 2: In-memory Database And BackupRecipe 3: Alter TableRecipe 4: Attach DatabaseRecipe 5: SQLite HooksRecipe 6: UnicodeRecipe 7: Get More InformationRecipe 8: TroubleshootingRecipe 9: Full Text SearchRecipe 10: Custom Extensions
    • Issue 1: Transaction ModeIssue 2: Refactoring of "execute"Issue 3: OptimizationIssue 4: Compiler IssuesIssue 5: iOSIssue 6: Better ErrorsIssue 7: Type InformationIssue 8: Async IOIssue 9: CookbookIssue 10: SQLite 4
    • I don’t thinkeverything shouldbe covered today.
    • Questions andsuggestions are all welcome.
    • Recipe 1Bulk Insert
    • This is fairly SLOW.my $dbh = DBI->connect(...);my $stmt = "INSERT INTO foo VALUES(?)";for (@data) { $dbh->do($stmt, undef, $_);}
    • Much faster.my $sth = $dbh->prepare($stmt);for (@data) { $sth->execute($_);}
    • A bit more faster.$dbh->begin_work;for (@data) { $sth->execute($_);}$dbh->commit;
    • A bit more faster (maybe).my $ct = 0;$dbh->begin_work;for (@data) { $sth->execute($_); # this number is arbitrary unless (++$ct % 1000) { $dbh->commit; $dbh->begin_work; }}$dbh->commit;
    • A convenient way.$dbh->{AutoCommit} = 0;for (@data) { $sth->execute($_); unless (++$ct % 1000) { $dbh->commit; }}$dbh->commit;$dbh->{AutoCommit} = 1;
    • Much faster (but dangerous; SQLite only)$dbh->do("PRAGMA synchronous = OFF");for (@data) { ...}$dbh->do("PRAGMA synchronous = FULL");
    • Much faster (since 1.37)# INSERT INTO foo VALUES (?),(?),...my $placeholders = join ",", ((?)) x 500;my $stmt = "INSERT INTO foo VALUES $placeholders";my $sth = $dbh->prepare($stmt);while (@data) { if (@data > 500) { $sth->execute(splice @data, 0, 500); } else { ... }}
    • Summary Use "prepare". Dont insert one by one. Dont insert everything at once. Do The Right Thing when ACIDity matters. Multi-row insert may help. Beware SQLITE_MAX_VARIABLE_NUMBER (default: 999)
    • Recipe 2In-memory Database And Backup
    • An in-memory database is quite fast.# volatile, in-memory databasemy $dbh = DBI->connect(dbi:SQLite::memory:...);# cf. volatile database based on a temporary file# my $dbh = DBI->connect(dbi:SQLite:...);
    • Its also handy for tests.{ my $dbh = DBI->connect(dbi:SQLite::memory:...); ... # do whatever you need}# everything is gone (Cleaning-up files may be a bit tricky)
    • What to doif the testsshould fail?
    • Back it up as necessary.if (!Test::More->builder->is_passing) { $dbh->sqlite_backup_to_file("failed.db");}
    • Load into an in- memory database.my $dbh = DBI->connect(dbi:SQLite::memory:‘...);$dbh->sqlite_backup_from_file("backup.db");
    • CAVEATS Loading time may matter. Everything is REPLACED (naturally). Just copying a file may suffice. Shared (read) lock helps. http://www.sqlite.org/backup.html
    • Recipe 3Alter Table
    • SQLite only supports "ADD COLUMN".my $dbh = DBI->connect(...);$dbh->do("ALTER TABLE foo ADD COLUMN new"); # and renaming a table
    • Use a temporarytable if you want more.
    • Remove a column (col2)$dbh->begin_work;eval { $dbh->do("CREATE TEMP TABLE t (col1, col3)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col3)"); $dbh->do("INSERT INTO foo SELECT col1, col3 FROM t"); $dbh->do("DROP TABLE t");};!$@ ? $dbh->commit : $dbh->rollback;
    • Rename a column (col3 -> col4)$dbh->begin_work;eval { $dbh->do("CREATE TEMP TABLE t (col1, col4)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col4)"); $dbh->do("INSERT INTO foo SELECT col1, col4 FROM t"); $dbh->do("DROP TABLE t");};!$@ ? $dbh->commit : $dbh->rollback;
    • NOTE ACIDity matters here. A temporary table is reasonably fast. Want some wrapper? http://sqlite.org/faq.html#q11
    • Recipe 4Attach Database
    • One big file with all tables =Easier to copy/move.
    • One big file with all tables =Easier to be locked.
    • Splitting a database intosmaller databases may help.
    • ATTACH databases if you need to join.$dbh->do("ATTACH DATABASE sub.db AS sub");
    • You can ATTACHin-memory databases. $dbh->do("ATTACH DATABASE :memory: AS mem");
    • CAVEATS ATTACH before you begin a transaction. Beware SQLITE_MAX_ATTACHED (default: 10) Split a database if concurrency really matters. Or use better server/client databases.
    • Recipe 5SQLite Hooks
    • May be a problem.my $select = "SELECT id FROM foo WHERE status = 0 LIMIT 1";my $update = "UPDATE foo SET status = 1 WHERE id = ?";my ($id) = $dbh->selectrow_array($select);$dbh->do($update, undef, $id);
    • Update first to get a write lock.my $update = q{ UPDATE foo SET status = 1 WHERE id = ( SELECT id FROM foo WHERE status = 0 LIMIT 1 )};
    • What should we doto find an updated row? # $dbh->last_insert_id doesnt work
    • "update_hook" may help.my ($action, $database, $table, $rowid);$dbh->sqlite_update_hook(sub { ($action, $database, $table, $rowid) = @_; ... # you cant do anything that modifies # the database connection (incl. "prepare")});$dbh->do($update);
    • Retreive the row with the id.my $stmt = "SELECT * FROM foo WHERE ROWID = ?";my $row = $dbh->selectrow_arrayref( $stmt, undef, $rowid);
    • NOTE Also sqlite_(commit|rollback)_hook One hook per connection Optional "UPDATE ... LIMIT" clause? SQLITE_ENABLE_UPDATE_DELETE_LIMIT Amalgamated source doesnt support this (because part of the source needs to be regenerated).
    • Recipe 6Unicode
    • You can use unicodecolumn/table names.$dbh->do(‘ create table テーブル (カラム,カラム,カラム));
    • Basic rules decode what you get from a database encode what you put into a database
    • sqlite_unicode => 1 what you get will be decoded what you put will be ...
    • Summary Dont need to care usually. Be careful when you use _info() methods. More work may be needed.
    • Recipe 7Get More Information
    • Supported *_info() methods my $sth = $dbh->table_info(...); my $sth = $dbh->column_info(...); my $sth = $dbh->primary_key_info(...); my $sth = $dbh->foreign_key_info(...); (since 1.38_01)
    • table_info() to get table names (accepts wild cards)
    • column_info() to get column names (accepts wild cards) to see if a column is nullable to get type definition
    • primary_key_info() to get primary key names
    • foreign_key_info() to get foreign key names or referred tables
    • SQLite Pragmata to get information PRAGMA database_list PRAGMA table_info(table) PRAGMA foreign_key_list(table)
    • To get info of attached databases. PRAGMA db.table_info(table) PRAGMA db.foreign_key_list(table)
    • System tablesSELECT * FROM sqlite_master;SELECT * FROM sqlite_temp_master; # SQLs stored in system tables are not always what you used to create tables
    • Recipe 8Troubleshooting
    • sqlite_trace()$dbh->sqlite_trace(sub { my ($stmt) = @_; say $stmt;});
    • bound params are embedded.$dbh->do(q{ INSERT INTO foo VALUES (?)}, undef, 1);# INSERT INTO foo VALUES (1)
    • sqlite_profile()$dbh->sqlite_profile(sub { my ($stmt, $time) = @_; say "$stmt (elapsed: $time ms)";});
    • bound params are not embedded.$dbh->do(q{ INSERT INTO foo VALUES (?)}, undef, 1);# INSERT INTO foo VALUES (?)(elapsed: 0 ms)
    • EXPLAINEXPLAIN QUERY PLANFor interactive analysisand troubleshooting only.
    • Check "SCAN TABLE" without using indices.my $stmt = "SELECT * FROM foo WHERE id = ?";my $sth = $dbh->prepare("EXPLAIN QUERY PLAN $stmt");$sth->execute(1);while(my $plan = $sth->fetchrow_hashref) { my $detail = $plan->{detail}; if ($detail =~ /SCAN TABLE/ && $detail !~ /INDEX/){ ... }}
    • In case you want to know memory usage etc... my $status = DBD::SQLite::sqlite_status(); my $status = $dbh->sqlite_db_status(); my $status = $sth->sqlite_st_status();
    • Recipe 9Full Text Search
    • SQLite supportsfull text search. http://sqlite.org/fts3.html
    • "perl" tokenizer is supported$dbh->do( CREATE VIRTUAL TABLE foo USING fts3 ( content, tokenize=perl "main::tokenizer" ));
    • my $mecab = Text::MeCab->new;sub tokenizer { return sub { my $node = $mecab->parse($_[0]); my ($index, $pos) = (0, 0); return sub { my $token = $node->surface or return; my $length = $node->length; my $start = $pos; my $end = $pos += $length; $node = $node->next; return ($token, $length, $start, $end, $index++); };}}http://d.hatena.ne.jp/charsbar/20100828/1282937592
    • Search::Tokenizer  word  word_locale  word_unicode  unaccent
    • Want more?
    • Recipe 10Custom Extensions
    • Before you use extensions...$dbh->sqlite_enable_load_extension(1);
    • Use load_extension() function to load$dbh->do(q{ SELECT load_extension(./ex.dll)});
    • Limitation of load_extension() Cant register other functions from the extension.
    • Use sqlite_load_extension()$dbh->sqlite_load_extension(./ex.dll);
    • How to write an extension#include "sqlite3ext.h"SQLITE_EXTENSION_INIT1;int sqlite3_extension_init( sqlite3 *db, char **error, const sqlite3_api_routines *api){ SQLITE_EXTENSION_INIT2(api); /* do whatever you like */ return SQLITE_OK;}
    • Use XS for more portability.#include <EXTERN.h>#include <perl.h>#include <XSUB.h>#include "sqlite3ext.h“SQLITE_EXTENSION_INIT1;int sqlite3_extension_init(...) { SQLITE_EXTENSION_INIT2(api); ...}MODULE = DBD::SQLite::Extension PACKAGE= DBD::SQLite::ExtensionPROTOTYPES: DISABLE
    • Prepare Makefile.PLuse strict;use warnings;use ExtUtils::MakeMaker;# Get installed header filesget_sqlite_header("sqlite3.h") or exit;get_sqlite_header("sqlite3ext.h") or exit;WriteMakefile( NAME => DBD::SQLite::Extension, CONFIGURE_REQUIRES => { File::ShareDir => 1.0, # to get headers DBD::SQLite => 1.38_01, }, FUNCLIST => [sqlite3_extension_init], DL_FUNCS => {DBD::SQLite::Extension => []},);
    • It Works.my $dll ="./blib/arch/auto/DBD/SQLite/Extension/Extension.dll";my $dbh = DBI->connect(dbi:SQLite::memory:);$dbh->sqlite_enable_load_extension(1);$dbh->sqlite_load_extension($dll);
    • NOTE Needs more tricks to make complex extensions (malloc/free conflicts, DBI macros etc) Use always the same headers as used in DBD::SQLite "Using SQLite" (O’Reilly) helps.
    • Questions so far?
    • Issue 1Transaction Mode
    • A controversial changein DBD::SQLite 1.38_01 The default transaction mode becomes "IMMEDIATE".
    • Deferred Transaction SQLite default Maximum concurrency May cause a deadlock (multiple clients in transactions wait for others releasing their locks)
    • Immediate Transaction DBD::SQLite’s current default Immediately reserve a write lock No deadlock (if you can begin a transaction, youll most probably be able to commit either).
    • Why changed? by request (#56444, in April 2010) (other reports on locking issues: #42205, #46289 (both in 2009)) to avoid unexpected errors/deadlocks while testing. ditto for smaller web applications. After all, who uses DBD::SQLite most?
    • Use "Deferred" whenyou know a database is almost read-only.
    • To defer a transaction$dbh->{sqlite_use_immediate_transaction} = 0;$dbh->do("BEGIN");
    • Issue 2Refactoring of "execute"
    •  Was suggested to move parameter binding to sqlite_bind_ph Broke tests / Would change behaviors Not sure if its worth trying begin_work
    • Issue 3Optimization
    •  Asked to use the same optimization as perl. Reverted as it broke an R-Tree test (floating point issue).
    • Issue 4Compiler Issues
    • Anyone using... ? AIX 6.1 Sun C compiler
    • Issue 5 iOS
    •  A reporter says some databases are said to be "encrypted"...
    • Issue 6Better Errors
    •  sqlite3_extended_errcode sqlite3_extended_result_codes
    • Issue 7Type Information
    •  $dbh->type_info(_all) is not implemented yet. $sth->{TYPE} returns an array ref of strings. (Not conformed to the DBI spec.) Dynamic typing / Type affinity
    • Issue 8Async IO
    •  Nice to have. Not amalgamated. Would need significant refactoring. Separate distribution?
    • Issue 9Cookbook
    •  Almost forgotten. Maybe these slides help. Suggestions?
    • Issue 10SQLite 4
    • SQLite4 is an alternative, not a replacement, for SQLite3.- http://sqlite.org/src4/doc/trunk/www/design.wiki
    • Still in its earliest stage. No downloadable packages. Not enough features yet.
    • Wanna try? fossil clone http://www.sqlite.org/src4 sqlite4.fossil mkdir sqlite4 && cd sqlite4 fossil open ../sqlite4.fossil you probably need to tweak Makefile (or apply proper compile options)
    • DBD::SQLite4? Will probably be needed eventually Should share the same sqlite_ prefix? Able to share the common part?
    • Questions?
    • Thank you