• Like
DBD::SQLite
Upcoming SlideShare
Loading in...5
×
Uploaded on

slides for YAPC::Asia 2012

slides for YAPC::Asia 2012

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,270
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
25
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. DBD::SQLiteRecipes, Issues, and Plans Kenichi Ishigaki (@charsbar) YAPC::Asia 2012 Sep 28, 2012
  • 2. Thank youfor coming.
  • 3. Today I‘ll showyou 10 recipes and 10 issues.
  • 4. Recipe 1: Bulk InsertRecipe 2: In-memory Database And BackupRecipe 3: Alter TableRecipe 4: Attach DatabaseRecipe 5: SQLite HooksRecipe 6: UnicodeRecipe 7: Get More InformationRecipe 8: TroubleshootingRecipe 9: Full Text SearchRecipe 10: Custom Extensions
  • 5. Issue 1: Transaction ModeIssue 2: Refactoring of "execute"Issue 3: OptimizationIssue 4: Compiler IssuesIssue 5: iOSIssue 6: Better ErrorsIssue 7: Type InformationIssue 8: Async IOIssue 9: CookbookIssue 10: SQLite 4
  • 6. I don’t thinkeverything shouldbe covered today.
  • 7. Questions andsuggestions are all welcome.
  • 8. Recipe 1Bulk Insert
  • 9. This is fairly SLOW.my $dbh = DBI->connect(...);my $stmt = "INSERT INTO foo VALUES(?)";for (@data) { $dbh->do($stmt, undef, $_);}
  • 10. Much faster.my $sth = $dbh->prepare($stmt);for (@data) { $sth->execute($_);}
  • 11. A bit more faster.$dbh->begin_work;for (@data) { $sth->execute($_);}$dbh->commit;
  • 12. A bit more faster (maybe).my $ct = 0;$dbh->begin_work;for (@data) { $sth->execute($_); # this number is arbitrary unless (++$ct % 1000) { $dbh->commit; $dbh->begin_work; }}$dbh->commit;
  • 13. A convenient way.$dbh->{AutoCommit} = 0;for (@data) { $sth->execute($_); unless (++$ct % 1000) { $dbh->commit; }}$dbh->commit;$dbh->{AutoCommit} = 1;
  • 14. Much faster (but dangerous; SQLite only)$dbh->do("PRAGMA synchronous = OFF");for (@data) { ...}$dbh->do("PRAGMA synchronous = FULL");
  • 15. Much faster (since 1.37)# INSERT INTO foo VALUES (?),(?),...my $placeholders = join ",", ((?)) x 500;my $stmt = "INSERT INTO foo VALUES $placeholders";my $sth = $dbh->prepare($stmt);while (@data) { if (@data > 500) { $sth->execute(splice @data, 0, 500); } else { ... }}
  • 16. Summary Use "prepare". Dont insert one by one. Dont insert everything at once. Do The Right Thing when ACIDity matters. Multi-row insert may help. Beware SQLITE_MAX_VARIABLE_NUMBER (default: 999)
  • 17. Recipe 2In-memory Database And Backup
  • 18. An in-memory database is quite fast.# volatile, in-memory databasemy $dbh = DBI->connect(dbi:SQLite::memory:...);# cf. volatile database based on a temporary file# my $dbh = DBI->connect(dbi:SQLite:...);
  • 19. Its also handy for tests.{ my $dbh = DBI->connect(dbi:SQLite::memory:...); ... # do whatever you need}# everything is gone (Cleaning-up files may be a bit tricky)
  • 20. What to doif the testsshould fail?
  • 21. Back it up as necessary.if (!Test::More->builder->is_passing) { $dbh->sqlite_backup_to_file("failed.db");}
  • 22. Load into an in- memory database.my $dbh = DBI->connect(dbi:SQLite::memory:‘...);$dbh->sqlite_backup_from_file("backup.db");
  • 23. CAVEATS Loading time may matter. Everything is REPLACED (naturally). Just copying a file may suffice. Shared (read) lock helps. http://www.sqlite.org/backup.html
  • 24. Recipe 3Alter Table
  • 25. SQLite only supports "ADD COLUMN".my $dbh = DBI->connect(...);$dbh->do("ALTER TABLE foo ADD COLUMN new"); # and renaming a table
  • 26. Use a temporarytable if you want more.
  • 27. Remove a column (col2)$dbh->begin_work;eval { $dbh->do("CREATE TEMP TABLE t (col1, col3)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col3)"); $dbh->do("INSERT INTO foo SELECT col1, col3 FROM t"); $dbh->do("DROP TABLE t");};!$@ ? $dbh->commit : $dbh->rollback;
  • 28. Rename a column (col3 -> col4)$dbh->begin_work;eval { $dbh->do("CREATE TEMP TABLE t (col1, col4)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col4)"); $dbh->do("INSERT INTO foo SELECT col1, col4 FROM t"); $dbh->do("DROP TABLE t");};!$@ ? $dbh->commit : $dbh->rollback;
  • 29. NOTE ACIDity matters here. A temporary table is reasonably fast. Want some wrapper? http://sqlite.org/faq.html#q11
  • 30. Recipe 4Attach Database
  • 31. One big file with all tables =Easier to copy/move.
  • 32. One big file with all tables =Easier to be locked.
  • 33. Splitting a database intosmaller databases may help.
  • 34. ATTACH databases if you need to join.$dbh->do("ATTACH DATABASE sub.db AS sub");
  • 35. You can ATTACHin-memory databases. $dbh->do("ATTACH DATABASE :memory: AS mem");
  • 36. CAVEATS ATTACH before you begin a transaction. Beware SQLITE_MAX_ATTACHED (default: 10) Split a database if concurrency really matters. Or use better server/client databases.
  • 37. Recipe 5SQLite Hooks
  • 38. May be a problem.my $select = "SELECT id FROM foo WHERE status = 0 LIMIT 1";my $update = "UPDATE foo SET status = 1 WHERE id = ?";my ($id) = $dbh->selectrow_array($select);$dbh->do($update, undef, $id);
  • 39. Update first to get a write lock.my $update = q{ UPDATE foo SET status = 1 WHERE id = ( SELECT id FROM foo WHERE status = 0 LIMIT 1 )};
  • 40. What should we doto find an updated row? # $dbh->last_insert_id doesnt work
  • 41. "update_hook" may help.my ($action, $database, $table, $rowid);$dbh->sqlite_update_hook(sub { ($action, $database, $table, $rowid) = @_; ... # you cant do anything that modifies # the database connection (incl. "prepare")});$dbh->do($update);
  • 42. Retreive the row with the id.my $stmt = "SELECT * FROM foo WHERE ROWID = ?";my $row = $dbh->selectrow_arrayref( $stmt, undef, $rowid);
  • 43. NOTE Also sqlite_(commit|rollback)_hook One hook per connection Optional "UPDATE ... LIMIT" clause? SQLITE_ENABLE_UPDATE_DELETE_LIMIT Amalgamated source doesnt support this (because part of the source needs to be regenerated).
  • 44. Recipe 6Unicode
  • 45. You can use unicodecolumn/table names.$dbh->do(‘ create table テーブル (カラム,カラム,カラム));
  • 46. Basic rules decode what you get from a database encode what you put into a database
  • 47. sqlite_unicode => 1 what you get will be decoded what you put will be ...
  • 48. Summary Dont need to care usually. Be careful when you use _info() methods. More work may be needed.
  • 49. Recipe 7Get More Information
  • 50. Supported *_info() methods my $sth = $dbh->table_info(...); my $sth = $dbh->column_info(...); my $sth = $dbh->primary_key_info(...); my $sth = $dbh->foreign_key_info(...); (since 1.38_01)
  • 51. table_info() to get table names (accepts wild cards)
  • 52. column_info() to get column names (accepts wild cards) to see if a column is nullable to get type definition
  • 53. primary_key_info() to get primary key names
  • 54. foreign_key_info() to get foreign key names or referred tables
  • 55. SQLite Pragmata to get information PRAGMA database_list PRAGMA table_info(table) PRAGMA foreign_key_list(table)
  • 56. To get info of attached databases. PRAGMA db.table_info(table) PRAGMA db.foreign_key_list(table)
  • 57. System tablesSELECT * FROM sqlite_master;SELECT * FROM sqlite_temp_master; # SQLs stored in system tables are not always what you used to create tables
  • 58. Recipe 8Troubleshooting
  • 59. sqlite_trace()$dbh->sqlite_trace(sub { my ($stmt) = @_; say $stmt;});
  • 60. bound params are embedded.$dbh->do(q{ INSERT INTO foo VALUES (?)}, undef, 1);# INSERT INTO foo VALUES (1)
  • 61. sqlite_profile()$dbh->sqlite_profile(sub { my ($stmt, $time) = @_; say "$stmt (elapsed: $time ms)";});
  • 62. bound params are not embedded.$dbh->do(q{ INSERT INTO foo VALUES (?)}, undef, 1);# INSERT INTO foo VALUES (?)(elapsed: 0 ms)
  • 63. EXPLAINEXPLAIN QUERY PLANFor interactive analysisand troubleshooting only.
  • 64. Check "SCAN TABLE" without using indices.my $stmt = "SELECT * FROM foo WHERE id = ?";my $sth = $dbh->prepare("EXPLAIN QUERY PLAN $stmt");$sth->execute(1);while(my $plan = $sth->fetchrow_hashref) { my $detail = $plan->{detail}; if ($detail =~ /SCAN TABLE/ && $detail !~ /INDEX/){ ... }}
  • 65. In case you want to know memory usage etc... my $status = DBD::SQLite::sqlite_status(); my $status = $dbh->sqlite_db_status(); my $status = $sth->sqlite_st_status();
  • 66. Recipe 9Full Text Search
  • 67. SQLite supportsfull text search. http://sqlite.org/fts3.html
  • 68. "perl" tokenizer is supported$dbh->do( CREATE VIRTUAL TABLE foo USING fts3 ( content, tokenize=perl "main::tokenizer" ));
  • 69. my $mecab = Text::MeCab->new;sub tokenizer { return sub { my $node = $mecab->parse($_[0]); my ($index, $pos) = (0, 0); return sub { my $token = $node->surface or return; my $length = $node->length; my $start = $pos; my $end = $pos += $length; $node = $node->next; return ($token, $length, $start, $end, $index++); };}}http://d.hatena.ne.jp/charsbar/20100828/1282937592
  • 70. Search::Tokenizer  word  word_locale  word_unicode  unaccent
  • 71. Want more?
  • 72. Recipe 10Custom Extensions
  • 73. Before you use extensions...$dbh->sqlite_enable_load_extension(1);
  • 74. Use load_extension() function to load$dbh->do(q{ SELECT load_extension(./ex.dll)});
  • 75. Limitation of load_extension() Cant register other functions from the extension.
  • 76. Use sqlite_load_extension()$dbh->sqlite_load_extension(./ex.dll);
  • 77. How to write an extension#include "sqlite3ext.h"SQLITE_EXTENSION_INIT1;int sqlite3_extension_init( sqlite3 *db, char **error, const sqlite3_api_routines *api){ SQLITE_EXTENSION_INIT2(api); /* do whatever you like */ return SQLITE_OK;}
  • 78. Use XS for more portability.#include <EXTERN.h>#include <perl.h>#include <XSUB.h>#include "sqlite3ext.h“SQLITE_EXTENSION_INIT1;int sqlite3_extension_init(...) { SQLITE_EXTENSION_INIT2(api); ...}MODULE = DBD::SQLite::Extension PACKAGE= DBD::SQLite::ExtensionPROTOTYPES: DISABLE
  • 79. Prepare Makefile.PLuse strict;use warnings;use ExtUtils::MakeMaker;# Get installed header filesget_sqlite_header("sqlite3.h") or exit;get_sqlite_header("sqlite3ext.h") or exit;WriteMakefile( NAME => DBD::SQLite::Extension, CONFIGURE_REQUIRES => { File::ShareDir => 1.0, # to get headers DBD::SQLite => 1.38_01, }, FUNCLIST => [sqlite3_extension_init], DL_FUNCS => {DBD::SQLite::Extension => []},);
  • 80. It Works.my $dll ="./blib/arch/auto/DBD/SQLite/Extension/Extension.dll";my $dbh = DBI->connect(dbi:SQLite::memory:);$dbh->sqlite_enable_load_extension(1);$dbh->sqlite_load_extension($dll);
  • 81. NOTE Needs more tricks to make complex extensions (malloc/free conflicts, DBI macros etc) Use always the same headers as used in DBD::SQLite "Using SQLite" (O’Reilly) helps.
  • 82. Questions so far?
  • 83. Issue 1Transaction Mode
  • 84. A controversial changein DBD::SQLite 1.38_01 The default transaction mode becomes "IMMEDIATE".
  • 85. Deferred Transaction SQLite default Maximum concurrency May cause a deadlock (multiple clients in transactions wait for others releasing their locks)
  • 86. Immediate Transaction DBD::SQLite’s current default Immediately reserve a write lock No deadlock (if you can begin a transaction, youll most probably be able to commit either).
  • 87. Why changed? by request (#56444, in April 2010) (other reports on locking issues: #42205, #46289 (both in 2009)) to avoid unexpected errors/deadlocks while testing. ditto for smaller web applications. After all, who uses DBD::SQLite most?
  • 88. Use "Deferred" whenyou know a database is almost read-only.
  • 89. To defer a transaction$dbh->{sqlite_use_immediate_transaction} = 0;$dbh->do("BEGIN");
  • 90. Issue 2Refactoring of "execute"
  • 91.  Was suggested to move parameter binding to sqlite_bind_ph Broke tests / Would change behaviors Not sure if its worth trying begin_work
  • 92. Issue 3Optimization
  • 93.  Asked to use the same optimization as perl. Reverted as it broke an R-Tree test (floating point issue).
  • 94. Issue 4Compiler Issues
  • 95. Anyone using... ? AIX 6.1 Sun C compiler
  • 96. Issue 5 iOS
  • 97.  A reporter says some databases are said to be "encrypted"...
  • 98. Issue 6Better Errors
  • 99.  sqlite3_extended_errcode sqlite3_extended_result_codes
  • 100. Issue 7Type Information
  • 101.  $dbh->type_info(_all) is not implemented yet. $sth->{TYPE} returns an array ref of strings. (Not conformed to the DBI spec.) Dynamic typing / Type affinity
  • 102. Issue 8Async IO
  • 103.  Nice to have. Not amalgamated. Would need significant refactoring. Separate distribution?
  • 104. Issue 9Cookbook
  • 105.  Almost forgotten. Maybe these slides help. Suggestions?
  • 106. Issue 10SQLite 4
  • 107. SQLite4 is an alternative, not a replacement, for SQLite3.- http://sqlite.org/src4/doc/trunk/www/design.wiki
  • 108. Still in its earliest stage. No downloadable packages. Not enough features yet.
  • 109. Wanna try? fossil clone http://www.sqlite.org/src4 sqlite4.fossil mkdir sqlite4 && cd sqlite4 fossil open ../sqlite4.fossil you probably need to tweak Makefile (or apply proper compile options)
  • 110. DBD::SQLite4? Will probably be needed eventually Should share the same sqlite_ prefix? Able to share the common part?
  • 111. Questions?
  • 112. Thank you