DBD::SQLite

DBD::SQLite
Recipes, Issues,
and Plans
Kenichi Ishigaki
(@charsbar)
YAPC::Asia 2012
Sep 28, 2012

Today I‘ll show
you 10 recipes
and 10 issues.

Recipe 1: Bulk Insert
Recipe 2: In-memory Database And Backup
Recipe 3: Alter Table
Recipe 4: Attach Database
Recipe 5: SQLite Hooks
Recipe 6: Unicode
Recipe 7: Get More Information
Recipe 8: Troubleshooting
Recipe 9: Full Text Search
Recipe 10: Custom Extensions

Issue 1: Transaction Mode
Issue 2: Refactoring of "execute"
Issue 3: Optimization
Issue 4: Compiler Issues
Issue 5: iOS
Issue 6: Better Errors
Issue 7: Type Information
Issue 8: Async IO
Issue 9: Cookbook
Issue 10: SQLite 4

I don’t think
everything should
be covered today.

Questions and
suggestions are
all welcome.

This is fairly SLOW.

my $dbh = DBI->connect(...);
my $stmt = "INSERT INTO foo VALUES(?)";
for (@data) {
$dbh->do($stmt, undef, $_);
}

Much faster.
my $sth = $dbh->prepare($stmt);
for (@data) {
$sth->execute($_);
}

A bit more faster.
$dbh->begin_work;
for (@data) {
$sth->execute($_);
}
$dbh->commit;

A bit more faster (maybe).
my $ct = 0;
$dbh->begin_work;
for (@data) {
$sth->execute($_);
# this number is arbitrary
unless (++$ct % 1000) {
$dbh->commit;
$dbh->begin_work;
}
}
$dbh->commit;

A convenient way.
$dbh->{AutoCommit} = 0;
for (@data) {
$sth->execute($_);
unless (++$ct % 1000) {
$dbh->commit;
}
}
$dbh->commit;
$dbh->{AutoCommit} = 1;

Much faster
(but dangerous; SQLite only)

$dbh->do("PRAGMA synchronous = OFF");
for (@data) {
...
}
$dbh->do("PRAGMA synchronous = FULL");

Much faster (since 1.37)
# INSERT INTO foo VALUES (?),(?),...
my $placeholders = join ",", ('(?)') x 500;
my $stmt = "INSERT INTO foo VALUES $placeholders";
my $sth = $dbh->prepare($stmt);
while (@data) {
if (@data > 500) {
$sth->execute(splice @data, 0, 500);
} else {
...
}
}

Summary
 Use "prepare".
 Don't insert one by one.
 Don't insert everything at once.
 Do The Right Thing when ACIDity matters.
 Multi-row insert may help.
 Beware SQLITE_MAX_VARIABLE_NUMBER
(default: 999)

Recipe 2
In-memory Database
And Backup

An in-memory database
is quite fast.

# volatile, in-memory database
my $dbh = DBI->connect('dbi:SQLite::memory:'...);

# cf. volatile database based on a temporary file
# my $dbh = DBI->connect('dbi:SQLite:'...);

It's also handy
for tests.
{
my $dbh = DBI->connect('dbi:SQLite::memory:'...);
... # do whatever you need
}
# everything is gone

(Cleaning-up files may be a bit tricky)

What to do
if the tests
should fail?

Back it up as
necessary.
if (!Test::More->builder->is_passing) {
$dbh->sqlite_backup_to_file("failed.db");
}

Load into an in-
memory database.

my $dbh = DBI->connect('dbi:SQLite::memory:‘...);
$dbh->sqlite_backup_from_file("backup.db");

CAVEATS
 Loading time may matter.
 Everything is REPLACED (naturally).
 Just copying a file may suffice.
 Shared (read) lock helps.
 http://www.sqlite.org/backup.html

SQLite only supports
"ADD COLUMN".

my $dbh = DBI->connect(...);
$dbh->do("ALTER TABLE foo ADD COLUMN new");

# and renaming a table

Use a temporary
table if you want
more.

Remove a column (col2)
$dbh->begin_work;
eval {
$dbh->do("CREATE TEMP TABLE t (col1, col3)");
$dbh->do("INSERT INTO t SELECT col1, col3 FROM foo");
$dbh->do("DROP TABLE foo");
$dbh->do("CREATE TABLE foo (col1, col3)");
$dbh->do("INSERT INTO foo SELECT col1, col3 FROM t");
$dbh->do("DROP TABLE t");
};
!$@ ? $dbh->commit : $dbh->rollback;

Rename a column
(col3 -> col4)
$dbh->begin_work;
eval {
$dbh->do("CREATE TEMP TABLE t (col1, col4)");
$dbh->do("INSERT INTO t SELECT col1, col3 FROM foo");
$dbh->do("DROP TABLE foo");
$dbh->do("CREATE TABLE foo (col1, col4)");
$dbh->do("INSERT INTO foo SELECT col1, col4 FROM t");
$dbh->do("DROP TABLE t");
};
!$@ ? $dbh->commit : $dbh->rollback;

NOTE
 ACIDity matters here.
 A temporary table is reasonably fast.
 Want some wrapper?
 http://sqlite.org/faq.html#q11

One big file with
all tables
=
Easier to copy/move.

One big file with
all tables
=
Easier to be locked.

Splitting a
database into
smaller databases
may help.

ATTACH databases
if you need to join.

$dbh->do("ATTACH DATABASE 'sub.db' AS sub");

You can ATTACH
in-memory databases.
$dbh->do("ATTACH DATABASE ':memory:' AS mem");

CAVEATS
 ATTACH before you begin a transaction.
 Beware SQLITE_MAX_ATTACHED
(default: 10)
 Split a database if concurrency really
matters.
 Or use better server/client databases.

May be a problem.
my $select =
"SELECT id FROM foo WHERE status = 0 LIMIT 1";
my $update = "UPDATE foo SET status = 1 WHERE id = ?";
my ($id) = $dbh->selectrow_array($select);
$dbh->do($update, undef, $id);

Update first to
get a write lock.
my $update = q{
UPDATE foo SET status = 1
WHERE id = (
SELECT id FROM foo
WHERE status = 0 LIMIT 1
)
};

What should we do
to find an updated
row?
# $dbh->last_insert_id doesn't work

"update_hook" may help.

my ($action, $database, $table, $rowid);
$dbh->sqlite_update_hook(sub {
($action, $database, $table, $rowid) = @_;
...
# you can't do anything that modifies
# the database connection (incl. "prepare")
});
$dbh->do($update);

Retreive the row
with the id.
my $stmt = "SELECT * FROM foo WHERE ROWID = ?";
my $row = $dbh->selectrow_arrayref(
$stmt, undef, $rowid
);

NOTE
 Also sqlite_(commit|rollback)_hook
 One hook per connection
 Optional "UPDATE ... LIMIT" clause?
 SQLITE_ENABLE_UPDATE_DELETE_LIMIT
 Amalgamated source doesn't support this
 (because part of the source needs to be
regenerated).

You can use unicode
column/table names.

$dbh->do(‘
create table テーブル
(カラム,カラム,カラム)
');

Basic rules
 decode what you get from a
database
 encode what you put into a
database

sqlite_unicode => 1

 what you get will be
decoded
 what you put will be ...

Summary
 Don't need to care usually.
 Be careful when you use
_info() methods.
 More work may be needed.

Supported *_info() methods

 my $sth = $dbh->table_info(...);
 my $sth = $dbh->column_info(...);
 my $sth = $dbh->primary_key_info(...);
 my $sth = $dbh->foreign_key_info(...);
(since 1.38_01)

table_info()
 to get table names
(accepts wild cards)

column_info()
 to get column names
(accepts wild cards)
 to see if a column is nullable
 to get type definition

primary_key_info()

 to get primary key names

foreign_key_info()

 to get foreign key names
 or referred tables

SQLite Pragmata to
get information

 PRAGMA database_list
 PRAGMA table_info('table')
 PRAGMA foreign_key_list('table')

To get info of
attached databases.

 PRAGMA db.table_info('table')
 PRAGMA db.foreign_key_list('table')

System tables

SELECT * FROM sqlite_master;
SELECT * FROM sqlite_temp_master;

# SQLs stored in system tables are not
always what you used to create tables

sqlite_trace()

$dbh->sqlite_trace(sub {
my ($stmt) = @_;
say $stmt;
});

bound params are
embedded.
$dbh->do(q{
INSERT INTO foo VALUES (?)
}, undef, 1);

# INSERT INTO foo VALUES ('1')

sqlite_profile()

$dbh->sqlite_profile(sub {
my ($stmt, $time) = @_;
say "$stmt (elapsed: $time ms)";
});

bound params are
not embedded.
$dbh->do(q{
INSERT INTO foo VALUES (?)
}, undef, 1);

# INSERT INTO foo VALUES (?)
(elapsed: 0 ms)

EXPLAIN
EXPLAIN QUERY PLAN

For interactive analysis
and troubleshooting only.

Check "SCAN TABLE"
without using indices.
my $stmt = "SELECT * FROM foo WHERE id = ?";
my $sth = $dbh->prepare("EXPLAIN QUERY PLAN $stmt");
$sth->execute(1);
while(my $plan = $sth->fetchrow_hashref) {
my $detail = $plan->{detail};
if ($detail =~ /SCAN TABLE/ && $detail !~ /INDEX/)
{
...
}
}

In case you want to know
memory usage etc...

 my $status = DBD::SQLite::sqlite_status();
 my $status = $dbh->sqlite_db_status();
 my $status = $sth->sqlite_st_status();

SQLite supports
full text search.

 http://sqlite.org/fts3.html

"perl" tokenizer is
supported
$dbh->do('
CREATE VIRTUAL TABLE foo
USING fts3 (
content,
tokenize=perl "main::tokenizer"
)
');

my $mecab = Text::MeCab->new;

sub tokenizer { return sub {
my $node = $mecab->parse($_[0]);
my ($index, $pos) = (0, 0);
return sub {
my $token = $node->surface or return;
my $length = $node->length;
my $start = $pos;
my $end = $pos += $length;
$node = $node->next;
return ($token, $length, $start, $end, $index++);
};
}}

http://d.hatena.ne.jp/charsbar/20100828/1282937592

Search::Tokenizer
 word
 word_locale
 word_unicode
 unaccent

Before you use
extensions...

$dbh->sqlite_enable_load_extension(1);

Use load_extension()
function to load

$dbh->do(q{
SELECT load_extension('./ex.dll')
});

Limitation of
load_extension()
 Can't register other functions
from the extension.

Use sqlite_load_extension()

$dbh->sqlite_load_extension('./ex.dll');

How to write an extension
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1;

int sqlite3_extension_init(
sqlite3 *db,
char **error,
const sqlite3_api_routines *api
){
SQLITE_EXTENSION_INIT2(api);
/* do whatever you like */
return SQLITE_OK;
}

Use XS for more portability.
#include <EXTERN.h>
#include <perl.h>
#include <XSUB.h>
#include "sqlite3ext.h“
SQLITE_EXTENSION_INIT1;
int sqlite3_extension_init(...) {
SQLITE_EXTENSION_INIT2(api);
...
}

MODULE = DBD::SQLite::Extension PACKAGE
= DBD::SQLite::Extension
PROTOTYPES: DISABLE

Prepare Makefile.PL
use strict;
use warnings;
use ExtUtils::MakeMaker;
# Get installed header files
get_sqlite_header("sqlite3.h") or exit;
get_sqlite_header("sqlite3ext.h") or exit;
WriteMakefile(
NAME => 'DBD::SQLite::Extension',
CONFIGURE_REQUIRES => {
'File::ShareDir' => '1.0', # to get headers
'DBD::SQLite' => '1.38_01',
},
FUNCLIST => ['sqlite3_extension_init'],
DL_FUNCS => {'DBD::SQLite::Extension' => []},
);

It Works.
my $dll =
"./blib/arch/auto/DBD/SQLite/Extension/Extension.dll";
my $dbh = DBI->connect('dbi:SQLite::memory:');
$dbh->sqlite_enable_load_extension(1);
$dbh->sqlite_load_extension($dll);

NOTE
 Needs more tricks to make complex
extensions (malloc/free conflicts,
DBI macros etc)
 Use always the same headers as used
in DBD::SQLite
 "Using SQLite" (O’Reilly) helps.

A controversial change
in DBD::SQLite 1.38_01

The default
transaction mode
becomes
"IMMEDIATE".

Deferred Transaction

 SQLite default
 Maximum concurrency
 May cause a deadlock (multiple
clients in transactions wait for
others releasing their locks)

Immediate Transaction

 DBD::SQLite’s current default
 Immediately reserve a write lock
 No deadlock (if you can begin a
transaction, you'll most probably be
able to commit either).

Why changed?
 by request (#56444, in April 2010)
(other reports on locking issues:
#42205, #46289 (both in 2009))
 to avoid unexpected errors/deadlocks
while testing.
 ditto for smaller web applications.
 After all, who uses DBD::SQLite most?

Use "Deferred" when
you know a database
is almost read-only.

To defer a transaction

$dbh->{sqlite_use_immediate_transaction} = 0;
$dbh->do("BEGIN");

Issue 2
Refactoring of
"execute"

 Was suggested to move
parameter binding to
sqlite_bind_ph
 Broke tests / Would change
behaviors
 Not sure if it's worth trying
 begin_work

 Asked to use the same
optimization as perl.
 Reverted as it broke an
R-Tree test (floating
point issue).

Anyone using... ?

 AIX 6.1
 Sun C compiler

 A reporter says
some databases
are said to be
"encrypted"...

 sqlite3_extended_errcode
 sqlite3_extended_result_codes

 $dbh->type_info(_all) is not
implemented yet.
 $sth->{TYPE} returns an array
ref of strings. (Not conformed
to the DBI spec.)
 Dynamic typing / Type affinity

 Nice to have.
 Not amalgamated.
 Would need significant
refactoring.
 Separate distribution?

 Almost forgotten.
 Maybe these
slides help.
 Suggestions?

SQLite4 is an alternative,
not a replacement, for
SQLite3.

- http://sqlite.org/src4/doc/trunk/www/design.wiki

Still in its
earliest stage.

 No downloadable packages.
 Not enough features yet.

Wanna try?
 fossil clone http://www.sqlite.org/src4
sqlite4.fossil
 mkdir sqlite4 && cd sqlite4
 fossil open ../sqlite4.fossil
 you probably need to tweak Makefile (or
apply proper compile options)

DBD::SQLite4?

 Will probably be needed eventually
 Should share the same sqlite_
prefix?
 Able to share the common part?

DBD::SQLite

More Related Content

What's hot

Viewers also liked

Similar to DBD::SQLite

More from charsbar

DBD::SQLite