DBD::SQLite
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

DBD::SQLite

  • 1,784 views
Uploaded on

slides for YAPC::Asia 2012

slides for YAPC::Asia 2012

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,784
On Slideshare
1,711
From Embeds
73
Number of Embeds
4

Actions

Shares
Downloads
23
Comments
0
Likes
2

Embeds 73

http://yapcasia.org 58
https://twitter.com 11
http://leapf.org 3
https://si0.twimg.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. DBD::SQLiteRecipes, Issues, and Plans Kenichi Ishigaki (@charsbar) YAPC::Asia 2012 Sep 28, 2012
  • 2. Thank youfor coming.
  • 3. Today I‘ll showyou 10 recipes and 10 issues.
  • 4. Recipe 1: Bulk InsertRecipe 2: In-memory Database And BackupRecipe 3: Alter TableRecipe 4: Attach DatabaseRecipe 5: SQLite HooksRecipe 6: UnicodeRecipe 7: Get More InformationRecipe 8: TroubleshootingRecipe 9: Full Text SearchRecipe 10: Custom Extensions
  • 5. Issue 1: Transaction ModeIssue 2: Refactoring of "execute"Issue 3: OptimizationIssue 4: Compiler IssuesIssue 5: iOSIssue 6: Better ErrorsIssue 7: Type InformationIssue 8: Async IOIssue 9: CookbookIssue 10: SQLite 4
  • 6. I don’t thinkeverything shouldbe covered today.
  • 7. Questions andsuggestions are all welcome.
  • 8. Recipe 1Bulk Insert
  • 9. This is fairly SLOW.my $dbh = DBI->connect(...);my $stmt = "INSERT INTO foo VALUES(?)";for (@data) { $dbh->do($stmt, undef, $_);}
  • 10. Much faster.my $sth = $dbh->prepare($stmt);for (@data) { $sth->execute($_);}
  • 11. A bit more faster.$dbh->begin_work;for (@data) { $sth->execute($_);}$dbh->commit;
  • 12. A bit more faster (maybe).my $ct = 0;$dbh->begin_work;for (@data) { $sth->execute($_); # this number is arbitrary unless (++$ct % 1000) { $dbh->commit; $dbh->begin_work; }}$dbh->commit;
  • 13. A convenient way.$dbh->{AutoCommit} = 0;for (@data) { $sth->execute($_); unless (++$ct % 1000) { $dbh->commit; }}$dbh->commit;$dbh->{AutoCommit} = 1;
  • 14. Much faster (but dangerous; SQLite only)$dbh->do("PRAGMA synchronous = OFF");for (@data) { ...}$dbh->do("PRAGMA synchronous = FULL");
  • 15. Much faster (since 1.37)# INSERT INTO foo VALUES (?),(?),...my $placeholders = join ",", ((?)) x 500;my $stmt = "INSERT INTO foo VALUES $placeholders";my $sth = $dbh->prepare($stmt);while (@data) { if (@data > 500) { $sth->execute(splice @data, 0, 500); } else { ... }}
  • 16. Summary Use "prepare". Dont insert one by one. Dont insert everything at once. Do The Right Thing when ACIDity matters. Multi-row insert may help. Beware SQLITE_MAX_VARIABLE_NUMBER (default: 999)
  • 17. Recipe 2In-memory Database And Backup
  • 18. An in-memory database is quite fast.# volatile, in-memory databasemy $dbh = DBI->connect(dbi:SQLite::memory:...);# cf. volatile database based on a temporary file# my $dbh = DBI->connect(dbi:SQLite:...);
  • 19. Its also handy for tests.{ my $dbh = DBI->connect(dbi:SQLite::memory:...); ... # do whatever you need}# everything is gone (Cleaning-up files may be a bit tricky)
  • 20. What to doif the testsshould fail?
  • 21. Back it up as necessary.if (!Test::More->builder->is_passing) { $dbh->sqlite_backup_to_file("failed.db");}
  • 22. Load into an in- memory database.my $dbh = DBI->connect(dbi:SQLite::memory:‘...);$dbh->sqlite_backup_from_file("backup.db");
  • 23. CAVEATS Loading time may matter. Everything is REPLACED (naturally). Just copying a file may suffice. Shared (read) lock helps. http://www.sqlite.org/backup.html
  • 24. Recipe 3Alter Table
  • 25. SQLite only supports "ADD COLUMN".my $dbh = DBI->connect(...);$dbh->do("ALTER TABLE foo ADD COLUMN new"); # and renaming a table
  • 26. Use a temporarytable if you want more.
  • 27. Remove a column (col2)$dbh->begin_work;eval { $dbh->do("CREATE TEMP TABLE t (col1, col3)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col3)"); $dbh->do("INSERT INTO foo SELECT col1, col3 FROM t"); $dbh->do("DROP TABLE t");};!$@ ? $dbh->commit : $dbh->rollback;
  • 28. Rename a column (col3 -> col4)$dbh->begin_work;eval { $dbh->do("CREATE TEMP TABLE t (col1, col4)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col4)"); $dbh->do("INSERT INTO foo SELECT col1, col4 FROM t"); $dbh->do("DROP TABLE t");};!$@ ? $dbh->commit : $dbh->rollback;
  • 29. NOTE ACIDity matters here. A temporary table is reasonably fast. Want some wrapper? http://sqlite.org/faq.html#q11
  • 30. Recipe 4Attach Database
  • 31. One big file with all tables =Easier to copy/move.
  • 32. One big file with all tables =Easier to be locked.
  • 33. Splitting a database intosmaller databases may help.
  • 34. ATTACH databases if you need to join.$dbh->do("ATTACH DATABASE sub.db AS sub");
  • 35. You can ATTACHin-memory databases. $dbh->do("ATTACH DATABASE :memory: AS mem");
  • 36. CAVEATS ATTACH before you begin a transaction. Beware SQLITE_MAX_ATTACHED (default: 10) Split a database if concurrency really matters. Or use better server/client databases.
  • 37. Recipe 5SQLite Hooks
  • 38. May be a problem.my $select = "SELECT id FROM foo WHERE status = 0 LIMIT 1";my $update = "UPDATE foo SET status = 1 WHERE id = ?";my ($id) = $dbh->selectrow_array($select);$dbh->do($update, undef, $id);
  • 39. Update first to get a write lock.my $update = q{ UPDATE foo SET status = 1 WHERE id = ( SELECT id FROM foo WHERE status = 0 LIMIT 1 )};
  • 40. What should we doto find an updated row? # $dbh->last_insert_id doesnt work
  • 41. "update_hook" may help.my ($action, $database, $table, $rowid);$dbh->sqlite_update_hook(sub { ($action, $database, $table, $rowid) = @_; ... # you cant do anything that modifies # the database connection (incl. "prepare")});$dbh->do($update);
  • 42. Retreive the row with the id.my $stmt = "SELECT * FROM foo WHERE ROWID = ?";my $row = $dbh->selectrow_arrayref( $stmt, undef, $rowid);
  • 43. NOTE Also sqlite_(commit|rollback)_hook One hook per connection Optional "UPDATE ... LIMIT" clause? SQLITE_ENABLE_UPDATE_DELETE_LIMIT Amalgamated source doesnt support this (because part of the source needs to be regenerated).
  • 44. Recipe 6Unicode
  • 45. You can use unicodecolumn/table names.$dbh->do(‘ create table テーブル (カラム,カラム,カラム));
  • 46. Basic rules decode what you get from a database encode what you put into a database
  • 47. sqlite_unicode => 1 what you get will be decoded what you put will be ...
  • 48. Summary Dont need to care usually. Be careful when you use _info() methods. More work may be needed.
  • 49. Recipe 7Get More Information
  • 50. Supported *_info() methods my $sth = $dbh->table_info(...); my $sth = $dbh->column_info(...); my $sth = $dbh->primary_key_info(...); my $sth = $dbh->foreign_key_info(...); (since 1.38_01)
  • 51. table_info() to get table names (accepts wild cards)
  • 52. column_info() to get column names (accepts wild cards) to see if a column is nullable to get type definition
  • 53. primary_key_info() to get primary key names
  • 54. foreign_key_info() to get foreign key names or referred tables
  • 55. SQLite Pragmata to get information PRAGMA database_list PRAGMA table_info(table) PRAGMA foreign_key_list(table)
  • 56. To get info of attached databases. PRAGMA db.table_info(table) PRAGMA db.foreign_key_list(table)
  • 57. System tablesSELECT * FROM sqlite_master;SELECT * FROM sqlite_temp_master; # SQLs stored in system tables are not always what you used to create tables
  • 58. Recipe 8Troubleshooting
  • 59. sqlite_trace()$dbh->sqlite_trace(sub { my ($stmt) = @_; say $stmt;});
  • 60. bound params are embedded.$dbh->do(q{ INSERT INTO foo VALUES (?)}, undef, 1);# INSERT INTO foo VALUES (1)
  • 61. sqlite_profile()$dbh->sqlite_profile(sub { my ($stmt, $time) = @_; say "$stmt (elapsed: $time ms)";});
  • 62. bound params are not embedded.$dbh->do(q{ INSERT INTO foo VALUES (?)}, undef, 1);# INSERT INTO foo VALUES (?)(elapsed: 0 ms)
  • 63. EXPLAINEXPLAIN QUERY PLANFor interactive analysisand troubleshooting only.
  • 64. Check "SCAN TABLE" without using indices.my $stmt = "SELECT * FROM foo WHERE id = ?";my $sth = $dbh->prepare("EXPLAIN QUERY PLAN $stmt");$sth->execute(1);while(my $plan = $sth->fetchrow_hashref) { my $detail = $plan->{detail}; if ($detail =~ /SCAN TABLE/ && $detail !~ /INDEX/){ ... }}
  • 65. In case you want to know memory usage etc... my $status = DBD::SQLite::sqlite_status(); my $status = $dbh->sqlite_db_status(); my $status = $sth->sqlite_st_status();
  • 66. Recipe 9Full Text Search
  • 67. SQLite supportsfull text search. http://sqlite.org/fts3.html
  • 68. "perl" tokenizer is supported$dbh->do( CREATE VIRTUAL TABLE foo USING fts3 ( content, tokenize=perl "main::tokenizer" ));
  • 69. my $mecab = Text::MeCab->new;sub tokenizer { return sub { my $node = $mecab->parse($_[0]); my ($index, $pos) = (0, 0); return sub { my $token = $node->surface or return; my $length = $node->length; my $start = $pos; my $end = $pos += $length; $node = $node->next; return ($token, $length, $start, $end, $index++); };}}http://d.hatena.ne.jp/charsbar/20100828/1282937592
  • 70. Search::Tokenizer  word  word_locale  word_unicode  unaccent
  • 71. Want more?
  • 72. Recipe 10Custom Extensions
  • 73. Before you use extensions...$dbh->sqlite_enable_load_extension(1);
  • 74. Use load_extension() function to load$dbh->do(q{ SELECT load_extension(./ex.dll)});
  • 75. Limitation of load_extension() Cant register other functions from the extension.
  • 76. Use sqlite_load_extension()$dbh->sqlite_load_extension(./ex.dll);
  • 77. How to write an extension#include "sqlite3ext.h"SQLITE_EXTENSION_INIT1;int sqlite3_extension_init( sqlite3 *db, char **error, const sqlite3_api_routines *api){ SQLITE_EXTENSION_INIT2(api); /* do whatever you like */ return SQLITE_OK;}
  • 78. Use XS for more portability.#include <EXTERN.h>#include <perl.h>#include <XSUB.h>#include "sqlite3ext.h“SQLITE_EXTENSION_INIT1;int sqlite3_extension_init(...) { SQLITE_EXTENSION_INIT2(api); ...}MODULE = DBD::SQLite::Extension PACKAGE= DBD::SQLite::ExtensionPROTOTYPES: DISABLE
  • 79. Prepare Makefile.PLuse strict;use warnings;use ExtUtils::MakeMaker;# Get installed header filesget_sqlite_header("sqlite3.h") or exit;get_sqlite_header("sqlite3ext.h") or exit;WriteMakefile( NAME => DBD::SQLite::Extension, CONFIGURE_REQUIRES => { File::ShareDir => 1.0, # to get headers DBD::SQLite => 1.38_01, }, FUNCLIST => [sqlite3_extension_init], DL_FUNCS => {DBD::SQLite::Extension => []},);
  • 80. It Works.my $dll ="./blib/arch/auto/DBD/SQLite/Extension/Extension.dll";my $dbh = DBI->connect(dbi:SQLite::memory:);$dbh->sqlite_enable_load_extension(1);$dbh->sqlite_load_extension($dll);
  • 81. NOTE Needs more tricks to make complex extensions (malloc/free conflicts, DBI macros etc) Use always the same headers as used in DBD::SQLite "Using SQLite" (O’Reilly) helps.
  • 82. Questions so far?
  • 83. Issue 1Transaction Mode
  • 84. A controversial changein DBD::SQLite 1.38_01 The default transaction mode becomes "IMMEDIATE".
  • 85. Deferred Transaction SQLite default Maximum concurrency May cause a deadlock (multiple clients in transactions wait for others releasing their locks)
  • 86. Immediate Transaction DBD::SQLite’s current default Immediately reserve a write lock No deadlock (if you can begin a transaction, youll most probably be able to commit either).
  • 87. Why changed? by request (#56444, in April 2010) (other reports on locking issues: #42205, #46289 (both in 2009)) to avoid unexpected errors/deadlocks while testing. ditto for smaller web applications. After all, who uses DBD::SQLite most?
  • 88. Use "Deferred" whenyou know a database is almost read-only.
  • 89. To defer a transaction$dbh->{sqlite_use_immediate_transaction} = 0;$dbh->do("BEGIN");
  • 90. Issue 2Refactoring of "execute"
  • 91.  Was suggested to move parameter binding to sqlite_bind_ph Broke tests / Would change behaviors Not sure if its worth trying begin_work
  • 92. Issue 3Optimization
  • 93.  Asked to use the same optimization as perl. Reverted as it broke an R-Tree test (floating point issue).
  • 94. Issue 4Compiler Issues
  • 95. Anyone using... ? AIX 6.1 Sun C compiler
  • 96. Issue 5 iOS
  • 97.  A reporter says some databases are said to be "encrypted"...
  • 98. Issue 6Better Errors
  • 99.  sqlite3_extended_errcode sqlite3_extended_result_codes
  • 100. Issue 7Type Information
  • 101.  $dbh->type_info(_all) is not implemented yet. $sth->{TYPE} returns an array ref of strings. (Not conformed to the DBI spec.) Dynamic typing / Type affinity
  • 102. Issue 8Async IO
  • 103.  Nice to have. Not amalgamated. Would need significant refactoring. Separate distribution?
  • 104. Issue 9Cookbook
  • 105.  Almost forgotten. Maybe these slides help. Suggestions?
  • 106. Issue 10SQLite 4
  • 107. SQLite4 is an alternative, not a replacement, for SQLite3.- http://sqlite.org/src4/doc/trunk/www/design.wiki
  • 108. Still in its earliest stage. No downloadable packages. Not enough features yet.
  • 109. Wanna try? fossil clone http://www.sqlite.org/src4 sqlite4.fossil mkdir sqlite4 && cd sqlite4 fossil open ../sqlite4.fossil you probably need to tweak Makefile (or apply proper compile options)
  • 110. DBD::SQLite4? Will probably be needed eventually Should share the same sqlite_ prefix? Able to share the common part?
  • 111. Questions?
  • 112. Thank you