SlideShare a Scribd company logo
DBD::SQLite
Recipes, Issues,
   and Plans
           Kenichi Ishigaki
              (@charsbar)
          YAPC::Asia 2012
             Sep 28, 2012
Thank you
for coming.
Today I‘ll show
you 10 recipes
 and 10 issues.
Recipe 1: Bulk Insert
Recipe 2: In-memory Database And Backup
Recipe 3: Alter Table
Recipe 4: Attach Database
Recipe 5: SQLite Hooks
Recipe 6: Unicode
Recipe 7: Get More Information
Recipe 8: Troubleshooting
Recipe 9: Full Text Search
Recipe 10: Custom Extensions
Issue 1: Transaction Mode
Issue 2: Refactoring of "execute"
Issue 3: Optimization
Issue 4: Compiler Issues
Issue 5: iOS
Issue 6: Better Errors
Issue 7: Type Information
Issue 8: Async IO
Issue 9: Cookbook
Issue 10: SQLite 4
I don’t think
everything should
be covered today.
Questions and
suggestions are
  all welcome.
Recipe 1
Bulk Insert
This is fairly SLOW.

my $dbh = DBI->connect(...);
my $stmt = "INSERT INTO foo VALUES(?)";
for (@data) {
  $dbh->do($stmt, undef, $_);
}
Much faster.
my $sth = $dbh->prepare($stmt);
for (@data) {
  $sth->execute($_);
}
A bit more faster.
$dbh->begin_work;
for (@data) {
  $sth->execute($_);
}
$dbh->commit;
A bit more faster (maybe).
my $ct = 0;
$dbh->begin_work;
for (@data) {
  $sth->execute($_);
  # this number is arbitrary
  unless (++$ct % 1000) {
    $dbh->commit;
    $dbh->begin_work;
  }
}
$dbh->commit;
A convenient way.
$dbh->{AutoCommit} = 0;
for (@data) {
  $sth->execute($_);
  unless (++$ct % 1000) {
    $dbh->commit;
  }
}
$dbh->commit;
$dbh->{AutoCommit} = 1;
Much faster
  (but dangerous; SQLite only)

$dbh->do("PRAGMA synchronous = OFF");
for (@data) {
  ...
}
$dbh->do("PRAGMA synchronous = FULL");
Much faster (since 1.37)
# INSERT INTO foo VALUES (?),(?),...
my $placeholders = join ",", ('(?)') x 500;
my $stmt = "INSERT INTO foo VALUES $placeholders";
my $sth = $dbh->prepare($stmt);
while (@data) {
  if (@data > 500) {
    $sth->execute(splice @data, 0, 500);
  } else {
    ...
  }
}
Summary
   Use "prepare".
   Don't insert one by one.
   Don't insert everything at once.
   Do The Right Thing when ACIDity matters.
   Multi-row insert may help.
   Beware SQLITE_MAX_VARIABLE_NUMBER
    (default: 999)
Recipe 2
In-memory Database
    And Backup
An in-memory database
       is quite fast.

# volatile, in-memory database
my $dbh = DBI->connect('dbi:SQLite::memory:'...);

# cf. volatile database based on a temporary file
# my $dbh = DBI->connect('dbi:SQLite:'...);
It's also handy
           for tests.
{
    my $dbh = DBI->connect('dbi:SQLite::memory:'...);
    ... # do whatever you need
}
# everything is gone




      (Cleaning-up files may be a bit tricky)
What to do
if the tests
should fail?
Back it up as
        necessary.
if (!Test::More->builder->is_passing) {
  $dbh->sqlite_backup_to_file("failed.db");
}
Load into an in-
 memory database.

my $dbh = DBI->connect('dbi:SQLite::memory:‘...);
$dbh->sqlite_backup_from_file("backup.db");
CAVEATS
   Loading time may matter.
   Everything is REPLACED (naturally).
   Just copying a file may suffice.
   Shared (read) lock helps.
   http://www.sqlite.org/backup.html
Recipe 3
Alter Table
SQLite only supports
    "ADD COLUMN".

my $dbh = DBI->connect(...);
$dbh->do("ALTER TABLE foo ADD COLUMN new");




     # and renaming a table
Use a temporary
table if you want
      more.
Remove a column (col2)
$dbh->begin_work;
eval {
  $dbh->do("CREATE TEMP TABLE t (col1, col3)");
  $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo");
  $dbh->do("DROP TABLE foo");
  $dbh->do("CREATE TABLE foo (col1, col3)");
  $dbh->do("INSERT INTO foo SELECT col1, col3 FROM t");
  $dbh->do("DROP TABLE t");
};
!$@ ? $dbh->commit : $dbh->rollback;
Rename a column
         (col3 -> col4)
$dbh->begin_work;
eval {
  $dbh->do("CREATE TEMP TABLE t (col1, col4)");
  $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo");
  $dbh->do("DROP TABLE foo");
  $dbh->do("CREATE TABLE foo (col1, col4)");
  $dbh->do("INSERT INTO foo SELECT col1, col4 FROM t");
  $dbh->do("DROP TABLE t");
};
!$@ ? $dbh->commit : $dbh->rollback;
NOTE
   ACIDity matters here.
   A temporary table is reasonably fast.
   Want some wrapper?
   http://sqlite.org/faq.html#q11
Recipe 4
Attach Database
One big file with
    all tables
         =
Easier to copy/move.
One big file with
   all tables
        =
Easier to be locked.
Splitting a
  database into
smaller databases
    may help.
ATTACH databases
     if you need to join.

$dbh->do("ATTACH DATABASE 'sub.db' AS sub");
You can ATTACH
in-memory databases.
 $dbh->do("ATTACH DATABASE ':memory:' AS mem");
CAVEATS
 ATTACH before you begin a transaction.
 Beware SQLITE_MAX_ATTACHED
  (default: 10)
 Split a database if concurrency really
  matters.
 Or use better server/client databases.
Recipe 5
SQLite Hooks
May be a problem.
my $select =
        "SELECT id FROM foo WHERE status = 0 LIMIT 1";
my $update = "UPDATE foo SET status = 1 WHERE id = ?";
my ($id) = $dbh->selectrow_array($select);
$dbh->do($update, undef, $id);
Update first to
 get a write lock.
my $update = q{
  UPDATE foo SET status = 1
    WHERE id = (
      SELECT id FROM foo
      WHERE status = 0 LIMIT 1
    )
};
What should we do
to find an updated
       row?
 # $dbh->last_insert_id doesn't work
"update_hook" may help.

my ($action, $database, $table, $rowid);
$dbh->sqlite_update_hook(sub {
  ($action, $database, $table, $rowid) = @_;
  ...
  # you can't do anything that modifies
  # the database connection (incl. "prepare")
});
$dbh->do($update);
Retreive the row
   with the id.
my $stmt = "SELECT * FROM foo WHERE ROWID = ?";
my $row = $dbh->selectrow_arrayref(
  $stmt, undef, $rowid
);
NOTE
   Also sqlite_(commit|rollback)_hook
   One hook per connection
   Optional "UPDATE ... LIMIT" clause?
   SQLITE_ENABLE_UPDATE_DELETE_LIMIT
   Amalgamated source doesn't support this
   (because part of the source needs to be
    regenerated).
Recipe 6
Unicode
You can use unicode
column/table names.

$dbh->do(‘
  create table テーブル
    (カラム,カラム,カラム)
');
Basic rules
 decode what you get from a
  database
 encode what you put into a
  database
sqlite_unicode => 1

 what you get will be
  decoded
 what you put will be ...
Summary
 Don't need to care usually.
 Be careful when you use
  _info() methods.
 More work may be needed.
Recipe 7
Get More Information
Supported *_info() methods


   my $sth = $dbh->table_info(...);
   my $sth = $dbh->column_info(...);
   my $sth = $dbh->primary_key_info(...);
   my $sth = $dbh->foreign_key_info(...);
    (since 1.38_01)
table_info()
 to get table names
  (accepts wild cards)
column_info()
 to get column names
  (accepts wild cards)
 to see if a column is nullable
 to get type definition
primary_key_info()

 to get primary key names
foreign_key_info()

 to get foreign key names
 or referred tables
SQLite Pragmata to
    get information

 PRAGMA database_list
 PRAGMA table_info('table')
 PRAGMA foreign_key_list('table')
To get info of
 attached databases.

 PRAGMA db.table_info('table')
 PRAGMA db.foreign_key_list('table')
System tables

SELECT * FROM sqlite_master;
SELECT * FROM sqlite_temp_master;



 # SQLs stored in system tables are not
  always what you used to create tables
Recipe 8
Troubleshooting
sqlite_trace()

$dbh->sqlite_trace(sub {
  my ($stmt) = @_;
  say $stmt;
});
bound params are
   embedded.
$dbh->do(q{
  INSERT INTO foo VALUES (?)
}, undef, 1);

# INSERT INTO foo VALUES ('1')
sqlite_profile()

$dbh->sqlite_profile(sub {
  my ($stmt, $time) = @_;
  say "$stmt (elapsed: $time ms)";
});
bound params are
 not embedded.
$dbh->do(q{
  INSERT INTO foo VALUES (?)
}, undef, 1);

# INSERT INTO foo VALUES (?)
(elapsed: 0 ms)
EXPLAIN
EXPLAIN QUERY PLAN

For interactive analysis
and troubleshooting only.
Check "SCAN TABLE"
      without using indices.
my $stmt = "SELECT * FROM foo WHERE id = ?";
my $sth = $dbh->prepare("EXPLAIN QUERY PLAN $stmt");
$sth->execute(1);
while(my $plan = $sth->fetchrow_hashref) {
  my $detail = $plan->{detail};
  if ($detail =~ /SCAN TABLE/ && $detail !~ /INDEX/)
{
    ...
  }
}
In case you want to know
   memory usage etc...

 my $status = DBD::SQLite::sqlite_status();
 my $status = $dbh->sqlite_db_status();
 my $status = $sth->sqlite_st_status();
Recipe 9
Full Text Search
SQLite supports
full text search.

 http://sqlite.org/fts3.html
"perl" tokenizer is
       supported
$dbh->do('
  CREATE VIRTUAL TABLE foo
    USING fts3 (
      content,
      tokenize=perl "main::tokenizer"
    )
');
my $mecab = Text::MeCab->new;

sub tokenizer { return sub {
  my $node = $mecab->parse($_[0]);
  my ($index, $pos) = (0, 0);
  return sub {
    my $token = $node->surface or return;
    my $length = $node->length;
    my $start = $pos;
    my $end    = $pos += $length;
    $node = $node->next;
    return ($token, $length, $start, $end, $index++);
  };
}}



http://d.hatena.ne.jp/charsbar/20100828/1282937592
Search::Tokenizer
     word
     word_locale
     word_unicode
     unaccent
Want more?
Recipe 10
Custom Extensions
Before you use
  extensions...

$dbh->sqlite_enable_load_extension(1);
Use load_extension()
  function to load

$dbh->do(q{
  SELECT load_extension('./ex.dll')
});
Limitation of
  load_extension()
 Can't register other functions
  from the extension.
Use sqlite_load_extension()



$dbh->sqlite_load_extension('./ex.dll');
How to write an extension
#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1;

int sqlite3_extension_init(
  sqlite3 *db,
  char **error,
  const sqlite3_api_routines *api
){
  SQLITE_EXTENSION_INIT2(api);
 /* do whatever you like */
 return SQLITE_OK;
}
Use XS for more portability.
#include <EXTERN.h>
#include <perl.h>
#include <XSUB.h>
#include "sqlite3ext.h“
SQLITE_EXTENSION_INIT1;
int sqlite3_extension_init(...) {
  SQLITE_EXTENSION_INIT2(api);
  ...
}

MODULE = DBD::SQLite::Extension PACKAGE
= DBD::SQLite::Extension
PROTOTYPES: DISABLE
Prepare Makefile.PL
use strict;
use warnings;
use ExtUtils::MakeMaker;
# Get installed header files
get_sqlite_header("sqlite3.h") or exit;
get_sqlite_header("sqlite3ext.h") or exit;
WriteMakefile(
  NAME => 'DBD::SQLite::Extension',
  CONFIGURE_REQUIRES => {
    'File::ShareDir' => '1.0', # to get headers
    'DBD::SQLite' => '1.38_01',
  },
  FUNCLIST => ['sqlite3_extension_init'],
  DL_FUNCS => {'DBD::SQLite::Extension' => []},
);
It Works.
my $dll =
"./blib/arch/auto/DBD/SQLite/Extension/Extension.dll";
my $dbh = DBI->connect('dbi:SQLite::memory:');
$dbh->sqlite_enable_load_extension(1);
$dbh->sqlite_load_extension($dll);
NOTE
 Needs more tricks to make complex
  extensions (malloc/free conflicts,
  DBI macros etc)
 Use always the same headers as used
  in DBD::SQLite
 "Using SQLite" (O’Reilly) helps.
Questions so far?
Issue 1
Transaction Mode
A controversial change
in DBD::SQLite 1.38_01

    The default
 transaction mode
     becomes
  "IMMEDIATE".
Deferred Transaction

 SQLite default
 Maximum concurrency
 May cause a deadlock (multiple
  clients in transactions wait for
  others releasing their locks)
Immediate Transaction

 DBD::SQLite’s current default
 Immediately reserve a write lock
 No deadlock (if you can begin a
  transaction, you'll most probably be
  able to commit either).
Why changed?
 by request (#56444, in April 2010)
  (other reports on locking issues:
  #42205, #46289 (both in 2009))
 to avoid unexpected errors/deadlocks
  while testing.
 ditto for smaller web applications.
 After all, who uses DBD::SQLite most?
Use "Deferred" when
you know a database
 is almost read-only.
To defer a transaction

$dbh->{sqlite_use_immediate_transaction} = 0;
$dbh->do("BEGIN");
Issue 2
Refactoring of
  "execute"
 Was suggested to move
  parameter binding to
  sqlite_bind_ph
 Broke tests / Would change
  behaviors
 Not sure if it's worth trying
 begin_work
Issue 3
Optimization
 Asked to use the same
  optimization as perl.
 Reverted as it broke an
  R-Tree test (floating
  point issue).
Issue 4
Compiler Issues
Anyone using... ?

 AIX 6.1
 Sun C compiler
Issue 5
 iOS
 A reporter says
  some databases
  are said to be
  "encrypted"...
Issue 6
Better Errors
 sqlite3_extended_errcode
 sqlite3_extended_result_codes
Issue 7
Type Information
 $dbh->type_info(_all) is not
  implemented yet.
 $sth->{TYPE} returns an array
  ref of strings. (Not conformed
  to the DBI spec.)
 Dynamic typing / Type affinity
Issue 8
Async IO
 Nice to have.
 Not amalgamated.
 Would need significant
  refactoring.
 Separate distribution?
Issue 9
Cookbook
 Almost forgotten.
 Maybe these
  slides help.
 Suggestions?
Issue 10
SQLite 4
SQLite4 is an alternative,
 not a replacement, for
        SQLite3.

- http://sqlite.org/src4/doc/trunk/www/design.wiki
Still in its
  earliest stage.

 No downloadable packages.
 Not enough features yet.
Wanna try?
 fossil clone http://www.sqlite.org/src4
  sqlite4.fossil
 mkdir sqlite4 && cd sqlite4
 fossil open ../sqlite4.fossil
 you probably need to tweak Makefile (or
  apply proper compile options)
DBD::SQLite4?

 Will probably be needed eventually
 Should share the same sqlite_
  prefix?
 Able to share the common part?
Questions?
Thank you

More Related Content

What's hot

Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
Advanced Apache Spark Meetup Project Tungsten Nov 12 2015Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
Chris Fregly
 
Oracle SQL Basics
Oracle SQL BasicsOracle SQL Basics
Oracle SQL Basics
Dhananjay Goel
 
sql function(ppt)
sql function(ppt)sql function(ppt)
sql function(ppt)
Ankit Dubey
 
Presentation slides of Sequence Query Language (SQL)
Presentation slides of Sequence Query Language (SQL)Presentation slides of Sequence Query Language (SQL)
Presentation slides of Sequence Query Language (SQL)
Punjab University
 
1. SQL Basics - Introduction
1. SQL Basics - Introduction1. SQL Basics - Introduction
1. SQL Basics - Introduction
Varun A M
 
NoSQL databases
NoSQL databasesNoSQL databases
NoSQL databases
Harri Kauhanen
 
Sql tutorial
Sql tutorialSql tutorial
Sql tutorial
Rumman Ansari
 
Lecture 9 - DSA - Python Data Structures
Lecture 9 - DSA - Python Data StructuresLecture 9 - DSA - Python Data Structures
Lecture 9 - DSA - Python Data Structures
Haitham El-Ghareeb
 
Oracle PL/SQL online training | PL/SQL online Training
Oracle PL/SQL online training | PL/SQL online TrainingOracle PL/SQL online training | PL/SQL online Training
Oracle PL/SQL online training | PL/SQL online Training
suresh
 
Banking Database
Banking DatabaseBanking Database
Banking Database
Ashwin Dinoriya
 
introdution to SQL and SQL functions
introdution to SQL and SQL functionsintrodution to SQL and SQL functions
introdution to SQL and SQL functions
farwa waqar
 
Sql Tutorials
Sql TutorialsSql Tutorials
Sql Tutorials
Priyabrat Kar
 
PL/SQL Fundamentals I
PL/SQL Fundamentals IPL/SQL Fundamentals I
PL/SQL Fundamentals I
Nick Buytaert
 
CS8391 Data Structures Part B Questions Anna University
CS8391 Data Structures Part B Questions Anna UniversityCS8391 Data Structures Part B Questions Anna University
CS8391 Data Structures Part B Questions Anna University
P. Subathra Kishore, KAMARAJ College of Engineering and Technology, Madurai
 
SQLite - Overview
SQLite - OverviewSQLite - Overview
SQLite - Overview
Emanuele Bartolesi
 
Guía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez H
Guía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez HGuía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez H
Guía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez H
José Ricardo Tillero Giménez
 
MYSQL
MYSQLMYSQL
SQL Tutorial - Basic Commands
SQL Tutorial - Basic CommandsSQL Tutorial - Basic Commands
SQL Tutorial - Basic Commands
1keydata
 
12 SQL
12 SQL12 SQL
Sql queires
Sql queiresSql queires
Sql queires
MohitKumar1985
 

What's hot (20)

Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
Advanced Apache Spark Meetup Project Tungsten Nov 12 2015Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
Advanced Apache Spark Meetup Project Tungsten Nov 12 2015
 
Oracle SQL Basics
Oracle SQL BasicsOracle SQL Basics
Oracle SQL Basics
 
sql function(ppt)
sql function(ppt)sql function(ppt)
sql function(ppt)
 
Presentation slides of Sequence Query Language (SQL)
Presentation slides of Sequence Query Language (SQL)Presentation slides of Sequence Query Language (SQL)
Presentation slides of Sequence Query Language (SQL)
 
1. SQL Basics - Introduction
1. SQL Basics - Introduction1. SQL Basics - Introduction
1. SQL Basics - Introduction
 
NoSQL databases
NoSQL databasesNoSQL databases
NoSQL databases
 
Sql tutorial
Sql tutorialSql tutorial
Sql tutorial
 
Lecture 9 - DSA - Python Data Structures
Lecture 9 - DSA - Python Data StructuresLecture 9 - DSA - Python Data Structures
Lecture 9 - DSA - Python Data Structures
 
Oracle PL/SQL online training | PL/SQL online Training
Oracle PL/SQL online training | PL/SQL online TrainingOracle PL/SQL online training | PL/SQL online Training
Oracle PL/SQL online training | PL/SQL online Training
 
Banking Database
Banking DatabaseBanking Database
Banking Database
 
introdution to SQL and SQL functions
introdution to SQL and SQL functionsintrodution to SQL and SQL functions
introdution to SQL and SQL functions
 
Sql Tutorials
Sql TutorialsSql Tutorials
Sql Tutorials
 
PL/SQL Fundamentals I
PL/SQL Fundamentals IPL/SQL Fundamentals I
PL/SQL Fundamentals I
 
CS8391 Data Structures Part B Questions Anna University
CS8391 Data Structures Part B Questions Anna UniversityCS8391 Data Structures Part B Questions Anna University
CS8391 Data Structures Part B Questions Anna University
 
SQLite - Overview
SQLite - OverviewSQLite - Overview
SQLite - Overview
 
Guía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez H
Guía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez HGuía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez H
Guía 01. Ejercicios. Creación de Base de Datos en MySQL - José J Sánchez H
 
MYSQL
MYSQLMYSQL
MYSQL
 
SQL Tutorial - Basic Commands
SQL Tutorial - Basic CommandsSQL Tutorial - Basic Commands
SQL Tutorial - Basic Commands
 
12 SQL
12 SQL12 SQL
12 SQL
 
Sql queires
Sql queiresSql queires
Sql queires
 

Viewers also liked

SQLite in Adobe AIR
SQLite in Adobe AIRSQLite in Adobe AIR
SQLite in Adobe AIR
Peter Elst
 
Databases for Beginners SQLite
Databases for Beginners SQLiteDatabases for Beginners SQLite
Databases for Beginners SQLite
Christopher Wimble
 
Sqlite tutorial
Sqlite tutorialSqlite tutorial
Sqlite tutorial
Diogenes Silva Sousa
 
SQLite Techniques
SQLite TechniquesSQLite Techniques
SQLite Techniques
joaopmaia
 
Sqlite
SqliteSqlite
Sqlite
Raghu nath
 
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQueryRemedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Tatsuhiko Miyagawa
 
SQLite 3
SQLite 3SQLite 3
SQLite 3
Scott MacVicar
 
สร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย Photoshop
สร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย Photoshopสร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย Photoshop
สร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย PhotoshopSiwaphon
 
รู้จักโปรแกรม Adobe Photoshop CS6
รู้จักโปรแกรม Adobe Photoshop CS6รู้จักโปรแกรม Adobe Photoshop CS6
รู้จักโปรแกรม Adobe Photoshop CS6
Khon Kaen University
 
การใช้งานเบื้องต้น Adobe photoshop cs5
การใช้งานเบื้องต้น Adobe photoshop cs5การใช้งานเบื้องต้น Adobe photoshop cs5
การใช้งานเบื้องต้น Adobe photoshop cs5
Rattapadol Gunhakoon
 
ความรู้ทั่วไปในงานออกแบบ
ความรู้ทั่วไปในงานออกแบบความรู้ทั่วไปในงานออกแบบ
ความรู้ทั่วไปในงานออกแบบ
Keerati Santisak
 
How to Become a Data Scientist
How to Become a Data ScientistHow to Become a Data Scientist
How to Become a Data Scientist
ryanorban
 
Intro to Data Science for Enterprise Big Data
Intro to Data Science for Enterprise Big DataIntro to Data Science for Enterprise Big Data
Intro to Data Science for Enterprise Big Data
Paco Nathan
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data Scientist
Daniel Tunkelang
 
Titan: Big Graph Data with Cassandra
Titan: Big Graph Data with CassandraTitan: Big Graph Data with Cassandra
Titan: Big Graph Data with Cassandra
Matthias Broecheler
 
A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)
Prof. Dr. Diego Kuonen
 
Myths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsMyths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data Scientists
David Pittman
 
Titan: The Rise of Big Graph Data
Titan: The Rise of Big Graph DataTitan: The Rise of Big Graph Data
Titan: The Rise of Big Graph Data
Marko Rodriguez
 
Introduction to R for Data Mining
Introduction to R for Data MiningIntroduction to R for Data Mining
Introduction to R for Data Mining
Revolution Analytics
 

Viewers also liked (20)

SQLite in Adobe AIR
SQLite in Adobe AIRSQLite in Adobe AIR
SQLite in Adobe AIR
 
Databases for Beginners SQLite
Databases for Beginners SQLiteDatabases for Beginners SQLite
Databases for Beginners SQLite
 
Sqlite tutorial
Sqlite tutorialSqlite tutorial
Sqlite tutorial
 
SQLite Techniques
SQLite TechniquesSQLite Techniques
SQLite Techniques
 
Sqlite
SqliteSqlite
Sqlite
 
การสร้างภาพพื้นหลัง
การสร้างภาพพื้นหลังการสร้างภาพพื้นหลัง
การสร้างภาพพื้นหลัง
 
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQueryRemedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
 
SQLite 3
SQLite 3SQLite 3
SQLite 3
 
สร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย Photoshop
สร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย Photoshopสร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย Photoshop
สร้างพื้นหลังกราฟฟิกอย่างง่ายด้วย Photoshop
 
รู้จักโปรแกรม Adobe Photoshop CS6
รู้จักโปรแกรม Adobe Photoshop CS6รู้จักโปรแกรม Adobe Photoshop CS6
รู้จักโปรแกรม Adobe Photoshop CS6
 
การใช้งานเบื้องต้น Adobe photoshop cs5
การใช้งานเบื้องต้น Adobe photoshop cs5การใช้งานเบื้องต้น Adobe photoshop cs5
การใช้งานเบื้องต้น Adobe photoshop cs5
 
ความรู้ทั่วไปในงานออกแบบ
ความรู้ทั่วไปในงานออกแบบความรู้ทั่วไปในงานออกแบบ
ความรู้ทั่วไปในงานออกแบบ
 
How to Become a Data Scientist
How to Become a Data ScientistHow to Become a Data Scientist
How to Become a Data Scientist
 
Intro to Data Science for Enterprise Big Data
Intro to Data Science for Enterprise Big DataIntro to Data Science for Enterprise Big Data
Intro to Data Science for Enterprise Big Data
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data Scientist
 
Titan: Big Graph Data with Cassandra
Titan: Big Graph Data with CassandraTitan: Big Graph Data with Cassandra
Titan: Big Graph Data with Cassandra
 
A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)
 
Myths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsMyths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data Scientists
 
Titan: The Rise of Big Graph Data
Titan: The Rise of Big Graph DataTitan: The Rise of Big Graph Data
Titan: The Rise of Big Graph Data
 
Introduction to R for Data Mining
Introduction to R for Data MiningIntroduction to R for Data Mining
Introduction to R for Data Mining
 

Similar to DBD::SQLite

Sqlite perl
Sqlite perlSqlite perl
Sqlite perl
Ashoka Vanjare
 
Working with databases in Perl
Working with databases in PerlWorking with databases in Perl
Working with databases in Perl
Laurent Dami
 
Database Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering College
Database Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering CollegeDatabase Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering College
Database Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering College
Dhivyaa C.R
 
UNIT V (5).pptx
UNIT V (5).pptxUNIT V (5).pptx
UNIT V (5).pptx
DrDhivyaaCRAssistant
 
Working with databases
Working with databasesWorking with databases
PHP Data Objects
PHP Data ObjectsPHP Data Objects
PHP Data Objects
Wez Furlong
 
Drupal II: The SQL
Drupal II: The SQLDrupal II: The SQL
Drupal II: The SQL
ddiers
 
FYBSC IT Web Programming Unit V Advanced PHP and MySQL
FYBSC IT Web Programming Unit V  Advanced PHP and MySQLFYBSC IT Web Programming Unit V  Advanced PHP and MySQL
FYBSC IT Web Programming Unit V Advanced PHP and MySQL
Arti Parab Academics
 
Drupal7 dbtng
Drupal7  dbtngDrupal7  dbtng
Drupal7 dbtng
Nicolas Leroy
 
Drupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary EditionDrupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary Edition
ddiers
 
DrupalCamp Foz - Novas APIs Drupal 7
DrupalCamp Foz - Novas APIs Drupal 7DrupalCamp Foz - Novas APIs Drupal 7
DrupalCamp Foz - Novas APIs Drupal 7
chuvainc
 
The History of PHPersistence
The History of PHPersistenceThe History of PHPersistence
The History of PHPersistence
Hugo Hamon
 
Php MySql For Beginners
Php MySql For BeginnersPhp MySql For Beginners
Php MySql For Beginners
Priti Solanki
 
Quebec pdo
Quebec pdoQuebec pdo
Quebec pdo
Rengga Aditya
 
Ex[1].3 php db connectivity
Ex[1].3 php db connectivityEx[1].3 php db connectivity
Ex[1].3 php db connectivity
Mouli Chandira
 
Php tips-and-tricks4128
Php tips-and-tricks4128Php tips-and-tricks4128
Php tips-and-tricks4128
PrinceGuru MS
 
Php 101: PDO
Php 101: PDOPhp 101: PDO
Php 101: PDO
Jeremy Kendall
 
Quebec pdo
Quebec pdoQuebec pdo
Quebec pdo
Valentine Dianov
 
PHP and Mysql
PHP and MysqlPHP and Mysql
PHP and Mysql
Sankhadeep Roy
 
Talkaboutlithium
TalkaboutlithiumTalkaboutlithium
Talkaboutlithium
noppoman722
 

Similar to DBD::SQLite (20)

Sqlite perl
Sqlite perlSqlite perl
Sqlite perl
 
Working with databases in Perl
Working with databases in PerlWorking with databases in Perl
Working with databases in Perl
 
Database Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering College
Database Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering CollegeDatabase Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering College
Database Connectivity MYSQL by Dr.C.R.Dhivyaa Kongu Engineering College
 
UNIT V (5).pptx
UNIT V (5).pptxUNIT V (5).pptx
UNIT V (5).pptx
 
Working with databases
Working with databasesWorking with databases
Working with databases
 
PHP Data Objects
PHP Data ObjectsPHP Data Objects
PHP Data Objects
 
Drupal II: The SQL
Drupal II: The SQLDrupal II: The SQL
Drupal II: The SQL
 
FYBSC IT Web Programming Unit V Advanced PHP and MySQL
FYBSC IT Web Programming Unit V  Advanced PHP and MySQLFYBSC IT Web Programming Unit V  Advanced PHP and MySQL
FYBSC IT Web Programming Unit V Advanced PHP and MySQL
 
Drupal7 dbtng
Drupal7  dbtngDrupal7  dbtng
Drupal7 dbtng
 
Drupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary EditionDrupal - dbtng 25th Anniversary Edition
Drupal - dbtng 25th Anniversary Edition
 
DrupalCamp Foz - Novas APIs Drupal 7
DrupalCamp Foz - Novas APIs Drupal 7DrupalCamp Foz - Novas APIs Drupal 7
DrupalCamp Foz - Novas APIs Drupal 7
 
The History of PHPersistence
The History of PHPersistenceThe History of PHPersistence
The History of PHPersistence
 
Php MySql For Beginners
Php MySql For BeginnersPhp MySql For Beginners
Php MySql For Beginners
 
Quebec pdo
Quebec pdoQuebec pdo
Quebec pdo
 
Ex[1].3 php db connectivity
Ex[1].3 php db connectivityEx[1].3 php db connectivity
Ex[1].3 php db connectivity
 
Php tips-and-tricks4128
Php tips-and-tricks4128Php tips-and-tricks4128
Php tips-and-tricks4128
 
Php 101: PDO
Php 101: PDOPhp 101: PDO
Php 101: PDO
 
Quebec pdo
Quebec pdoQuebec pdo
Quebec pdo
 
PHP and Mysql
PHP and MysqlPHP and Mysql
PHP and Mysql
 
Talkaboutlithium
TalkaboutlithiumTalkaboutlithium
Talkaboutlithium
 

More from charsbar

Common boolean class_for_perl5
Common boolean class_for_perl5Common boolean class_for_perl5
Common boolean class_for_perl5
charsbar
 
2018年夏のPerl5
2018年夏のPerl52018年夏のPerl5
2018年夏のPerl5
charsbar
 
萬國之津梁
萬國之津梁萬國之津梁
萬國之津梁
charsbar
 
Better detection of what modules are used by some Perl 5 code
Better detection of what modules are used by some Perl 5 codeBetter detection of what modules are used by some Perl 5 code
Better detection of what modules are used by some Perl 5 code
charsbar
 
2017年夏のPerl
2017年夏のPerl2017年夏のPerl
2017年夏のPerl
charsbar
 
2017年春のPerl
2017年春のPerl2017年春のPerl
2017年春のPerl
charsbar
 
Json(::PP) is a-changing
Json(::PP) is a-changingJson(::PP) is a-changing
Json(::PP) is a-changing
charsbar
 
2016年のPerl (Long version)
2016年のPerl (Long version)2016年のPerl (Long version)
2016年のPerl (Long version)
charsbar
 
JSON, JSON::PP, and more
JSON, JSON::PP, and moreJSON, JSON::PP, and more
JSON, JSON::PP, and more
charsbar
 
perl language update
perl language updateperl language update
perl language update
charsbar
 
CPANの依存モジュールをもう少し正しく検出したい
CPANの依存モジュールをもう少し正しく検出したいCPANの依存モジュールをもう少し正しく検出したい
CPANの依存モジュールをもう少し正しく検出したい
charsbar
 
2013年のCPANモジュール作成事情
2013年のCPANモジュール作成事情2013年のCPANモジュール作成事情
2013年のCPANモジュール作成事情
charsbar
 
What you need to remember when you upload to CPAN
What you need to remember when you upload to CPANWhat you need to remember when you upload to CPAN
What you need to remember when you upload to CPAN
charsbar
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLite
charsbar
 
typemap in Perl/XS
typemap in Perl/XS  typemap in Perl/XS
typemap in Perl/XS
charsbar
 
Analyze CPAN, Analyze Community
Analyze CPAN, Analyze CommunityAnalyze CPAN, Analyze Community
Analyze CPAN, Analyze Community
charsbar
 
Annual Report 2012
Annual Report 2012Annual Report 2012
Annual Report 2012
charsbar
 
CPANTS: Kwalitative website and its tools
CPANTS: Kwalitative website and its toolsCPANTS: Kwalitative website and its tools
CPANTS: Kwalitative website and its tools
charsbar
 
CPANTS 2012
CPANTS 2012CPANTS 2012
CPANTS 2012
charsbar
 
Revisiting ppm
Revisiting ppmRevisiting ppm
Revisiting ppm
charsbar
 

More from charsbar (20)

Common boolean class_for_perl5
Common boolean class_for_perl5Common boolean class_for_perl5
Common boolean class_for_perl5
 
2018年夏のPerl5
2018年夏のPerl52018年夏のPerl5
2018年夏のPerl5
 
萬國之津梁
萬國之津梁萬國之津梁
萬國之津梁
 
Better detection of what modules are used by some Perl 5 code
Better detection of what modules are used by some Perl 5 codeBetter detection of what modules are used by some Perl 5 code
Better detection of what modules are used by some Perl 5 code
 
2017年夏のPerl
2017年夏のPerl2017年夏のPerl
2017年夏のPerl
 
2017年春のPerl
2017年春のPerl2017年春のPerl
2017年春のPerl
 
Json(::PP) is a-changing
Json(::PP) is a-changingJson(::PP) is a-changing
Json(::PP) is a-changing
 
2016年のPerl (Long version)
2016年のPerl (Long version)2016年のPerl (Long version)
2016年のPerl (Long version)
 
JSON, JSON::PP, and more
JSON, JSON::PP, and moreJSON, JSON::PP, and more
JSON, JSON::PP, and more
 
perl language update
perl language updateperl language update
perl language update
 
CPANの依存モジュールをもう少し正しく検出したい
CPANの依存モジュールをもう少し正しく検出したいCPANの依存モジュールをもう少し正しく検出したい
CPANの依存モジュールをもう少し正しく検出したい
 
2013年のCPANモジュール作成事情
2013年のCPANモジュール作成事情2013年のCPANモジュール作成事情
2013年のCPANモジュール作成事情
 
What you need to remember when you upload to CPAN
What you need to remember when you upload to CPANWhat you need to remember when you upload to CPAN
What you need to remember when you upload to CPAN
 
On UnQLite
On UnQLiteOn UnQLite
On UnQLite
 
typemap in Perl/XS
typemap in Perl/XS  typemap in Perl/XS
typemap in Perl/XS
 
Analyze CPAN, Analyze Community
Analyze CPAN, Analyze CommunityAnalyze CPAN, Analyze Community
Analyze CPAN, Analyze Community
 
Annual Report 2012
Annual Report 2012Annual Report 2012
Annual Report 2012
 
CPANTS: Kwalitative website and its tools
CPANTS: Kwalitative website and its toolsCPANTS: Kwalitative website and its tools
CPANTS: Kwalitative website and its tools
 
CPANTS 2012
CPANTS 2012CPANTS 2012
CPANTS 2012
 
Revisiting ppm
Revisiting ppmRevisiting ppm
Revisiting ppm
 

DBD::SQLite

  • 1. DBD::SQLite Recipes, Issues, and Plans Kenichi Ishigaki (@charsbar) YAPC::Asia 2012 Sep 28, 2012
  • 3. Today I‘ll show you 10 recipes and 10 issues.
  • 4. Recipe 1: Bulk Insert Recipe 2: In-memory Database And Backup Recipe 3: Alter Table Recipe 4: Attach Database Recipe 5: SQLite Hooks Recipe 6: Unicode Recipe 7: Get More Information Recipe 8: Troubleshooting Recipe 9: Full Text Search Recipe 10: Custom Extensions
  • 5. Issue 1: Transaction Mode Issue 2: Refactoring of "execute" Issue 3: Optimization Issue 4: Compiler Issues Issue 5: iOS Issue 6: Better Errors Issue 7: Type Information Issue 8: Async IO Issue 9: Cookbook Issue 10: SQLite 4
  • 6. I don’t think everything should be covered today.
  • 9. This is fairly SLOW. my $dbh = DBI->connect(...); my $stmt = "INSERT INTO foo VALUES(?)"; for (@data) { $dbh->do($stmt, undef, $_); }
  • 10. Much faster. my $sth = $dbh->prepare($stmt); for (@data) { $sth->execute($_); }
  • 11. A bit more faster. $dbh->begin_work; for (@data) { $sth->execute($_); } $dbh->commit;
  • 12. A bit more faster (maybe). my $ct = 0; $dbh->begin_work; for (@data) { $sth->execute($_); # this number is arbitrary unless (++$ct % 1000) { $dbh->commit; $dbh->begin_work; } } $dbh->commit;
  • 13. A convenient way. $dbh->{AutoCommit} = 0; for (@data) { $sth->execute($_); unless (++$ct % 1000) { $dbh->commit; } } $dbh->commit; $dbh->{AutoCommit} = 1;
  • 14. Much faster (but dangerous; SQLite only) $dbh->do("PRAGMA synchronous = OFF"); for (@data) { ... } $dbh->do("PRAGMA synchronous = FULL");
  • 15. Much faster (since 1.37) # INSERT INTO foo VALUES (?),(?),... my $placeholders = join ",", ('(?)') x 500; my $stmt = "INSERT INTO foo VALUES $placeholders"; my $sth = $dbh->prepare($stmt); while (@data) { if (@data > 500) { $sth->execute(splice @data, 0, 500); } else { ... } }
  • 16. Summary  Use "prepare".  Don't insert one by one.  Don't insert everything at once.  Do The Right Thing when ACIDity matters.  Multi-row insert may help.  Beware SQLITE_MAX_VARIABLE_NUMBER (default: 999)
  • 18. An in-memory database is quite fast. # volatile, in-memory database my $dbh = DBI->connect('dbi:SQLite::memory:'...); # cf. volatile database based on a temporary file # my $dbh = DBI->connect('dbi:SQLite:'...);
  • 19. It's also handy for tests. { my $dbh = DBI->connect('dbi:SQLite::memory:'...); ... # do whatever you need } # everything is gone (Cleaning-up files may be a bit tricky)
  • 20. What to do if the tests should fail?
  • 21. Back it up as necessary. if (!Test::More->builder->is_passing) { $dbh->sqlite_backup_to_file("failed.db"); }
  • 22. Load into an in- memory database. my $dbh = DBI->connect('dbi:SQLite::memory:‘...); $dbh->sqlite_backup_from_file("backup.db");
  • 23. CAVEATS  Loading time may matter.  Everything is REPLACED (naturally).  Just copying a file may suffice.  Shared (read) lock helps.  http://www.sqlite.org/backup.html
  • 25. SQLite only supports "ADD COLUMN". my $dbh = DBI->connect(...); $dbh->do("ALTER TABLE foo ADD COLUMN new"); # and renaming a table
  • 26. Use a temporary table if you want more.
  • 27. Remove a column (col2) $dbh->begin_work; eval { $dbh->do("CREATE TEMP TABLE t (col1, col3)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col3)"); $dbh->do("INSERT INTO foo SELECT col1, col3 FROM t"); $dbh->do("DROP TABLE t"); }; !$@ ? $dbh->commit : $dbh->rollback;
  • 28. Rename a column (col3 -> col4) $dbh->begin_work; eval { $dbh->do("CREATE TEMP TABLE t (col1, col4)"); $dbh->do("INSERT INTO t SELECT col1, col3 FROM foo"); $dbh->do("DROP TABLE foo"); $dbh->do("CREATE TABLE foo (col1, col4)"); $dbh->do("INSERT INTO foo SELECT col1, col4 FROM t"); $dbh->do("DROP TABLE t"); }; !$@ ? $dbh->commit : $dbh->rollback;
  • 29. NOTE  ACIDity matters here.  A temporary table is reasonably fast.  Want some wrapper?  http://sqlite.org/faq.html#q11
  • 31. One big file with all tables = Easier to copy/move.
  • 32. One big file with all tables = Easier to be locked.
  • 33. Splitting a database into smaller databases may help.
  • 34. ATTACH databases if you need to join. $dbh->do("ATTACH DATABASE 'sub.db' AS sub");
  • 35. You can ATTACH in-memory databases. $dbh->do("ATTACH DATABASE ':memory:' AS mem");
  • 36. CAVEATS  ATTACH before you begin a transaction.  Beware SQLITE_MAX_ATTACHED (default: 10)  Split a database if concurrency really matters.  Or use better server/client databases.
  • 38. May be a problem. my $select = "SELECT id FROM foo WHERE status = 0 LIMIT 1"; my $update = "UPDATE foo SET status = 1 WHERE id = ?"; my ($id) = $dbh->selectrow_array($select); $dbh->do($update, undef, $id);
  • 39. Update first to get a write lock. my $update = q{ UPDATE foo SET status = 1 WHERE id = ( SELECT id FROM foo WHERE status = 0 LIMIT 1 ) };
  • 40. What should we do to find an updated row? # $dbh->last_insert_id doesn't work
  • 41. "update_hook" may help. my ($action, $database, $table, $rowid); $dbh->sqlite_update_hook(sub { ($action, $database, $table, $rowid) = @_; ... # you can't do anything that modifies # the database connection (incl. "prepare") }); $dbh->do($update);
  • 42. Retreive the row with the id. my $stmt = "SELECT * FROM foo WHERE ROWID = ?"; my $row = $dbh->selectrow_arrayref( $stmt, undef, $rowid );
  • 43. NOTE  Also sqlite_(commit|rollback)_hook  One hook per connection  Optional "UPDATE ... LIMIT" clause?  SQLITE_ENABLE_UPDATE_DELETE_LIMIT  Amalgamated source doesn't support this  (because part of the source needs to be regenerated).
  • 45. You can use unicode column/table names. $dbh->do(‘ create table テーブル (カラム,カラム,カラム) ');
  • 46. Basic rules  decode what you get from a database  encode what you put into a database
  • 47. sqlite_unicode => 1  what you get will be decoded  what you put will be ...
  • 48. Summary  Don't need to care usually.  Be careful when you use _info() methods.  More work may be needed.
  • 49. Recipe 7 Get More Information
  • 50. Supported *_info() methods  my $sth = $dbh->table_info(...);  my $sth = $dbh->column_info(...);  my $sth = $dbh->primary_key_info(...);  my $sth = $dbh->foreign_key_info(...); (since 1.38_01)
  • 51. table_info()  to get table names (accepts wild cards)
  • 52. column_info()  to get column names (accepts wild cards)  to see if a column is nullable  to get type definition
  • 53. primary_key_info()  to get primary key names
  • 54. foreign_key_info()  to get foreign key names  or referred tables
  • 55. SQLite Pragmata to get information  PRAGMA database_list  PRAGMA table_info('table')  PRAGMA foreign_key_list('table')
  • 56. To get info of attached databases.  PRAGMA db.table_info('table')  PRAGMA db.foreign_key_list('table')
  • 57. System tables SELECT * FROM sqlite_master; SELECT * FROM sqlite_temp_master; # SQLs stored in system tables are not always what you used to create tables
  • 59. sqlite_trace() $dbh->sqlite_trace(sub { my ($stmt) = @_; say $stmt; });
  • 60. bound params are embedded. $dbh->do(q{ INSERT INTO foo VALUES (?) }, undef, 1); # INSERT INTO foo VALUES ('1')
  • 61. sqlite_profile() $dbh->sqlite_profile(sub { my ($stmt, $time) = @_; say "$stmt (elapsed: $time ms)"; });
  • 62. bound params are not embedded. $dbh->do(q{ INSERT INTO foo VALUES (?) }, undef, 1); # INSERT INTO foo VALUES (?) (elapsed: 0 ms)
  • 63. EXPLAIN EXPLAIN QUERY PLAN For interactive analysis and troubleshooting only.
  • 64. Check "SCAN TABLE" without using indices. my $stmt = "SELECT * FROM foo WHERE id = ?"; my $sth = $dbh->prepare("EXPLAIN QUERY PLAN $stmt"); $sth->execute(1); while(my $plan = $sth->fetchrow_hashref) { my $detail = $plan->{detail}; if ($detail =~ /SCAN TABLE/ && $detail !~ /INDEX/) { ... } }
  • 65. In case you want to know memory usage etc...  my $status = DBD::SQLite::sqlite_status();  my $status = $dbh->sqlite_db_status();  my $status = $sth->sqlite_st_status();
  • 67. SQLite supports full text search.  http://sqlite.org/fts3.html
  • 68. "perl" tokenizer is supported $dbh->do(' CREATE VIRTUAL TABLE foo USING fts3 ( content, tokenize=perl "main::tokenizer" ) ');
  • 69. my $mecab = Text::MeCab->new; sub tokenizer { return sub { my $node = $mecab->parse($_[0]); my ($index, $pos) = (0, 0); return sub { my $token = $node->surface or return; my $length = $node->length; my $start = $pos; my $end = $pos += $length; $node = $node->next; return ($token, $length, $start, $end, $index++); }; }} http://d.hatena.ne.jp/charsbar/20100828/1282937592
  • 70. Search::Tokenizer  word  word_locale  word_unicode  unaccent
  • 73. Before you use extensions... $dbh->sqlite_enable_load_extension(1);
  • 74. Use load_extension() function to load $dbh->do(q{ SELECT load_extension('./ex.dll') });
  • 75. Limitation of load_extension()  Can't register other functions from the extension.
  • 77. How to write an extension #include "sqlite3ext.h" SQLITE_EXTENSION_INIT1; int sqlite3_extension_init( sqlite3 *db, char **error, const sqlite3_api_routines *api ){ SQLITE_EXTENSION_INIT2(api); /* do whatever you like */ return SQLITE_OK; }
  • 78. Use XS for more portability. #include <EXTERN.h> #include <perl.h> #include <XSUB.h> #include "sqlite3ext.h“ SQLITE_EXTENSION_INIT1; int sqlite3_extension_init(...) { SQLITE_EXTENSION_INIT2(api); ... } MODULE = DBD::SQLite::Extension PACKAGE = DBD::SQLite::Extension PROTOTYPES: DISABLE
  • 79. Prepare Makefile.PL use strict; use warnings; use ExtUtils::MakeMaker; # Get installed header files get_sqlite_header("sqlite3.h") or exit; get_sqlite_header("sqlite3ext.h") or exit; WriteMakefile( NAME => 'DBD::SQLite::Extension', CONFIGURE_REQUIRES => { 'File::ShareDir' => '1.0', # to get headers 'DBD::SQLite' => '1.38_01', }, FUNCLIST => ['sqlite3_extension_init'], DL_FUNCS => {'DBD::SQLite::Extension' => []}, );
  • 80. It Works. my $dll = "./blib/arch/auto/DBD/SQLite/Extension/Extension.dll"; my $dbh = DBI->connect('dbi:SQLite::memory:'); $dbh->sqlite_enable_load_extension(1); $dbh->sqlite_load_extension($dll);
  • 81. NOTE  Needs more tricks to make complex extensions (malloc/free conflicts, DBI macros etc)  Use always the same headers as used in DBD::SQLite  "Using SQLite" (O’Reilly) helps.
  • 84. A controversial change in DBD::SQLite 1.38_01 The default transaction mode becomes "IMMEDIATE".
  • 85. Deferred Transaction  SQLite default  Maximum concurrency  May cause a deadlock (multiple clients in transactions wait for others releasing their locks)
  • 86. Immediate Transaction  DBD::SQLite’s current default  Immediately reserve a write lock  No deadlock (if you can begin a transaction, you'll most probably be able to commit either).
  • 87. Why changed?  by request (#56444, in April 2010) (other reports on locking issues: #42205, #46289 (both in 2009))  to avoid unexpected errors/deadlocks while testing.  ditto for smaller web applications.  After all, who uses DBD::SQLite most?
  • 88. Use "Deferred" when you know a database is almost read-only.
  • 89. To defer a transaction $dbh->{sqlite_use_immediate_transaction} = 0; $dbh->do("BEGIN");
  • 91.  Was suggested to move parameter binding to sqlite_bind_ph  Broke tests / Would change behaviors  Not sure if it's worth trying  begin_work
  • 93.  Asked to use the same optimization as perl.  Reverted as it broke an R-Tree test (floating point issue).
  • 95. Anyone using... ?  AIX 6.1  Sun C compiler
  • 97.  A reporter says some databases are said to be "encrypted"...
  • 101.  $dbh->type_info(_all) is not implemented yet.  $sth->{TYPE} returns an array ref of strings. (Not conformed to the DBI spec.)  Dynamic typing / Type affinity
  • 103.  Nice to have.  Not amalgamated.  Would need significant refactoring.  Separate distribution?
  • 105.  Almost forgotten.  Maybe these slides help.  Suggestions?
  • 107. SQLite4 is an alternative, not a replacement, for SQLite3. - http://sqlite.org/src4/doc/trunk/www/design.wiki
  • 108. Still in its earliest stage.  No downloadable packages.  Not enough features yet.
  • 109. Wanna try?  fossil clone http://www.sqlite.org/src4 sqlite4.fossil  mkdir sqlite4 && cd sqlite4  fossil open ../sqlite4.fossil  you probably need to tweak Makefile (or apply proper compile options)
  • 110. DBD::SQLite4?  Will probably be needed eventually  Should share the same sqlite_ prefix?  Able to share the common part?