DBIx-DataModel v2.0 in detail

DBIx::DataModel 2.0 in detail YAPC::EU::2011, Riga [email_address] Département Office

Agenda
Introduction : Object-Relational Mappings & UML
DBIx::DataModel 2.0 Architecture
Modelling the schema
Selecting data
Row objects and statement objects
Joins
Inserts and updates
Customization
Strengths and limitations of DBIx::DataModel
today on CPAN : v 1.99_05

Object-Relational Mappers

Perl and databases Database DBD driver DBI Object-Relational Mapper Perl program

ORM principle r1 r2 ... c1 c2 c3 ... c3 c4 +c1: String +c2: String +c3: class2 r1 : class1 RDBMS r2 : class1 RAM table1 table2

Impedance mismatch
SELECT c1, c2 FROM table1
 missing c3 , so cannot navigate to class2
is it a valid instance of class1 ?
SELECT * FROM table1 LEFT JOIN table2 ON …
 is it a valid instance of class1 ?
 what to do with the c4 column ?
SELECT c1, c2, length(c2) AS l_c2 FROM table1
 no predeclared method in class1 for accessing l_c2
c1 c2 c3 c3 c4 +c1: String +c2: String +c3: class2 r1 : class1 RDBMS RAM table1 table2

The Unified Modeling Language (UML)

Example : CPAN model Author Distribution Module 1 * 1 * * multiplicity role class association dependent _distribs * prereq _modules contains ► assoc. name composition depends_on ►

Modelling a schema

Architecture Schema Source Table Join Statement My::DB My::DB::Table_n My::DB::AutoJoin:: row statement row DBIx::DataModel classes application classes objects schema  quite similar to DBI architecture (dbh, sth)

All definitions in one single file
use DBIx::DataModel;
DBIx::DataModel-> Schema ("My::DB")
-> Table (qw/Author author author_id /)
->Table(qw/Distribution distribution distrib_id/)
->Table(qw/Module module module_id /)
-> Association ([qw/Author author 1 /],
[qw/Distribution distribs 0..* /])
-> Composition ([qw/Distribution distrib 1 /],
[qw/Module modules 1..* /]);
creates package My::DB creates package My::DB::Author adds methods into both packages

Multiplicities
"$min..$max"
"*" means "0..POSIX::INT_MAX"
"1" means "1..1"
$min == 0 ? joins are "LEFT OUTER JOIN"
: joins are "INNER JOIN"
$max > 1 ? default result is a list
: default result is a single object

Meta-Architecture Schema Table Join My::DB My::DB::Table_n My::DB::Auto_join Meta::Source Meta::Table Meta::Join meta::table meta::join meta::schema Meta::Schema Meta::Path Meta::Association Meta::Type meta::assoc meta::path meta::type

Data retrieval

Fetching one single record
# fetch from primary key
# by default, retrieves all columns ('*')
my $author = My::DB-> table ('Author')-> fetch ( 'DAMI' );
# reach columns through the hashref API
while (my ($k, $v) = each %$author) {
print "$k : $vn";
}

Multi-schema mode
# create a schema
my $dbh = DBI->connect(… ):
my $schema = My::DB->new(dbh => $dbh);
# fetch data
my $author = $schema-> table ('Author')-> fetch ( 'DAMI' );

Fetching a list of records
# select multiple records
my $recent_distribs = My::DB->table('Distribution')
-> select (
-columns => [qw/ distrib_name d_release /],
-where => { d_release => {'>' => $some_date}},
-order_by => [qw/ -d_release +distrib_name /],
);
foreach my $distrib (@$recent_distribs) {
...
}

Select API : overview
my $result = $source-> select (
-columns => @columns, ,
-where => %where,
-group_by => @groupings,
-having => %criteria,
-order_by => @order,
-for => 'read only',
-post_SQL => sub { … },
-pre_exec => sub { … },
-post_exec => sub { … },
-post_bless => sub { … },
-page_size => …, -page_index => …,
-limit => …, -offset => …,
-column_types => %types,
-result_as => 'rows' || 'sth' || 'sql'
|| 'statement' || 'hashref'
);

Arguments to select()

SQL::Abstract::More : named parameters
my $result = $source-> select (
-columns => @columns, ,
-where => %where,
-order_by => @order,
);
 SQL::Abstract->new ->select($table, @columns, %where, @order)

SQL::Abstract::More : extensions
-columns => [qw/col1 |alias1 max(col2) |alias2 /]
SELECT col1 AS alias1, max(col2) AS alias2
-columns => [ -DISTINCT => qw/col1 col2 col3/]
SELECT DISTINCT col1, col2, col3
-order_by => [qw/col1 + col2 – col3/]
SELECT … ORDER BY col1, col2 ASC, col3 DESC
-for => "update" || "read only"
SELECT … FOR UPDATE

Grouping
-group_by => [qw/col1 col2 …/]
-having => { col1 => {"<" => val1} , col2 => ... }
SELECT … GROUP BY col1, col2 HAVING col1 < ? AND col2 …
 separate call to SQL::Abstract and re-injection into the SQL

Paging
-page_size => $num_rows, -page_index => $page_index
# or
-limit => $num_rows, -offset => $row_index
 either new call to $sth->execute(), or use scrollable cursors (DBIx.:DataModel::Statement::JDBC)
starts at 1 starts at 0

Callbacks
-post_SQL => sub { … },
-pre_exec => sub { … },
-post_exec => sub { … },
-post_bless => sub { … },
hooks to various states within the statement lifecycle (see later)
sometimes useful for DB-specific features

Polymorphic result
-result_as =>
'rows' (default) : arrayref of row objects
'firstrow' : a single row object (or undef)
'hashref' : hashref keyed by primary keys
[hashref => @cols] : cascaded hashref
'flat_arrayref' : flattened values from each row
'statement' : a statement object (iterator)
'fast_statement' : statement reusing same memory
'sth' : DBI statement handle
'sql' : ($sql, @bind_values)
'subquery' : ["($sql)", @bind]
  don't need method variants : select_hashref(), select_arrayref(), etc.

Row objects

A row object …
is just a hashref
keys are column names
values are column values
nothing else
actually, when in multi-schema mode, there is an additional __schema field
is blessed into the table class
has a metadm method (accessor to the metaclass)
has a schema method (accessor to the schema)
has methods for navigating to related tables
can be dumped as is
to Dumper / YAML / JSON / XML
to Perl debugger

Columns …
basically, are plain scalars, not objects
but can be "inflated/deflated" through a Type()
programmer chooses the column list, at each select()
-columns => @columns # arrayref
-columns => " col1, col2 " # string
-columns => " * " # default
objects have variable size !
if missing keys : runtime error
when following joins
when updating and deleting

Navigation to associated tables
Method names come from association declarations
Exactly like a select ()
automatically chooses –result_as => 'rows' || 'firstrow' from multiplicity information
# ->Association([qw/Author author 1 /],
# [qw/Distribution distribs 0..* /])
my $author = $distrib-> author ();
my $other_distribs = $author-> distribs (
-columns => [qw/. . ./],
-where => { . . . },
-order_by => [qw/. . ./],
);

Statement objects

Statement: an encapsulated query statement meta::source My::Source 1 1 * * schema dbh 0..1 * row next() / all() My::Schema meta::schema in single-schema mode in multi-schema mode singleton()

Statement lifecycle new sqlized prepared executed schema + source data row(s) new() sqlize() prepare() execute() bind() refine() bind() bind() bind() execute() next() / all() blessed column types applied -post_bless -pre_exec -post_exec -post_SQL

When to explicitly use a statement
as iterator
my $statement = $source->select(..., -result_as => 'statement');
while (my $row = $statement-> next ) {
. . .
}
for paging
$statement-> goto_page (123);
for loop efficiency
my $statement = My::Table-> join (qw/role1 role2/);
$statement- > prepare (-columns => ...,
-where => ...);
my $list = My::Table-> select (...);
foreach my $obj (@$list) {
my $related_rows = $statement-> execute ($obj)-> all ;
...
}

Fast statement
like a regular statement
but reuses the same memory location for each row
see DBI::bind_col()
my $statement = $source->select(
. . . ,
-result_as => ' fast_statement '
);
while (my $row = $statement->next) {
. . .
# DO THIS : print $row->{col1}, $row->{col2}
# BUT DON'T DO THIS : push @results, $row;
}

Database joins

Basic join
$rows = My::DB-> join (qw/ Author distribs modules /)
-> select (-where => ...);
Author Distrib Module My::DB::AutoJoin::… DBIDM::Source::Join new class created on the fly

Left / inner joins
->Association([qw/Author author 1 /],
[qw/Distribution distribs 0..* /])
# default : LEFT OUTER JOIN
->Composition([qw/Distribution distrib 1 /],
[qw/Module modules 1..* /]);
# default : INNER JOIN
# but defaults can be overridden
My::DB->join([qw/Author <=> distribs/)-> . . .
My::DB->join([qw/Distribution => modules /)-> . . .

Join from an instance
$rows = $author-> join (qw/ distribs modules /)->select(
-columns => [qw/distrib_name module_name/],
-where => { d_release => {'<' => $date} },
);
SELECT distrib_name, module_name FROM distribution INNER JOIN module ON distribution.distrib_id = module.distrib_id WHERE distrib.author_id = $author->{author_id} AND d_release < $date

Insert / Update

Insert
@ids = MyDB::Author-> insert (
{ firstname => 'Larry', lastname => 'Wall' },
{ firstname => 'Damian', lastname => 'Conway' },
);
INSERT INTO author(firstname, lastname) VALUES (?, ?)

Bulk insert
@ids = MyDB::Author-> insert (
[qw/firstname lastname/],
[qw/Larry Wall /],
[qw/Damian Conway /],
);

Insert into / cascaded insert
@id_trees = $author-> insert_into_distribs (
{distrib_name => 'DBIx-DataModel',
modules => [
{module_name => 'DBIx::DataModel', ..},
{module_name => 'DBIx::DataModel::Statement', ..},
]},
{distrib_name => 'Pod-POM-Web',
…
},
-returning => {},
);

Update
$obj->{col1} = $new_val_1;
$obj->{col2} = $new_val_2;
. . .
$opj-> update ;
# or
MyDB::Author-> update ({author_id => $id, col => $new_val})
# or
MyDB::Author-> update ($id, {col => $new_val})
# or (bulk update)
MyDB::Author-> update (-set => {col => $new_val},
-where => %condition)

Transaction
MyDB-> do_transaction (sub {
my $author = MyDB::Author->fetch($author_id,
{ -for => "read only"}
);
my $distribs = $author->distribs( -for => 'update');
foreach my $distrib (@$distribs) {
my $id = $distrib->{distrib_id};
MyDB::Distrib-> update ($id, {col => $val});
}
});
can be nested
can involve several dbh
no savepoints (yet)

Other features

Named placeholders / bind()
# introduce named placeholders
$statement-> prepare (-where => {
col1 => '?:foo ',
col2 => {"<" => '?:bar '},
col3 => {">" => '?:bar '},
col3 => 1234,
});
# fill placeholders with values
$statement-> bind (foo => 99, bar => 88, other => 77);
$statement-> bind ($hashref);
$sql @bind SELECT * FROM .. WHERE col1 = ? AND col2 < ? AND col3 = ? -- -- 1234 ?:foo ?:bar

Types (inflate/deflate)
# declare a Type
My::DB -> Type ( Multivalue =>
from_DB => sub {$_[ 0 ] = [split /;/ , $_[ 0 ]] } ,
to_DB => sub {$_[ 0 ] = join ";" , @$_[ 0 ] },
);
# apply it to some columns in a table
My::DB::Author->metadm-> define_column_type (
Multivalue => qw/hobbies languages/,
);

Auto_expand
# declare auto-expansions
MyDB::Author -> define_auto_expand (qw/distributions/);
MyDB::Distribution -> define_auto_expand (qw/modules/);
# apply to an object (automatically fetches all modules of all distributions of that author)
$author-> auto_expand ();
# use the data tree
use YAML; print Dump($author);

Schema localization
{
# a kind of "local MyDB";
my $guard = MyDB-> localize_state ();
# temporary change class data
MyDB->dbh($new_dbh, %new_options);
do_some_work_with_new_dbh();
} # automatically restore previous state

Schema generator
perl -MDBIx::DataModel::Schema::Generator
-e " fromDBI ('dbi:connection:string')" --
-schema My::New::Schema > My/New/Schema.pm
perl -MDBIx::DataModel::Schema::Generator
-e " fromDBIxClass ('Some::DBIC::Schema')" --
-schema My::New::Schema > My/New/Schema.pm

Auto_insert / Auto_update / No_update
$table-> auto_insert_columns ( created_by =>
sub { $ENV{REMOTE_USER} . ", " . localtime }
);
$table-> auto_update_columns (
modified_by => sub {… }
);
$table-> no_update_columns (qw/ row_id /);
 can also be declared for the whole schema

Extending / customizing DBIx::DataModel
Schema hooks for
SQL dialects (join syntax, alias syntax, limit / offset, etc.)
last_insert_id
Ad hoc subclasses for
SQL::Abstract
Table
Join
Statements
Statement callbacks
Extending table classes
additional methods
redefining _singleInsert method

Conclusion

Strengths
centralized definitions of tables & associations
efficiency
improved API for SQL::Abstract
clear conceptual distinction between
data sources (tables and joinss),
database statements (stateful objects representing SQL queries)
data rows (lightweight blessed hashrefs)
concise and flexible syntax for joins
used in production for mission-critical app
(running Geneva courts)

Limitations
tiny community
no schema versioning
no object caching nor 'dirty columns'
no 'cascaded update' nor 'insert or create'

Lots of documentation
SYNOPSIS AND DESCRIPTION
DESIGN
QUICKSTART
REFERENCE
COOKBOOK
MISC
INTERNALS
GLOSSARY

THANK YOU FOR YOUR ATTENTION

Bonus slides

ORM: What for ?
[catalyst list] On Thu, 2006-06-08, Steve wrote:
Not intending to start any sort of rancorous discussion,
but I was wondering whether someone could illuminate
me a little?
I'm comfortable with SQL, and with DBI. I write basic
SQL that runs just fine on all databases, or more
complex SQL when I want to target a single database
(ususally postgresql).
What value does an ORM add for a user like me?

ORM useful for …
dynamic SQL
navigation between tables
generate complex SQL queries from Perl datastructures
better than phrasebook or string concatenation
automatic data conversions (inflation / deflation)
expansion of tree data structures coded in the relational model
transaction encapsulation
data validation
computed fields
caching
…
 See Also : http://lists.scsys.co.uk/pipermail/catalyst/2006-June

Many-to-many implementation author_id author_name e_mail 1 * 1 * * * Author distrib_id module_id Dependency distrib_id distrib_name d_release author_id Distribution module_id module_name distrib_id Module 1 1 link table for n-to-n association

Writing SQL SQL is too low-level, I don't ever want to see it SQL is the most important part of my application, I won't let anybody write it for me

Why hashref instead of OO accessors ?
Perl builtin rich API for hashes (keys, values, slices, string interpolation)
good for import / export in YAML/XML/JSON
easier to follow steps in Perl debugger
faster than OO accessor methods
visually clear distinction between lvalue / rvalue
my $val = $hashref->{column};
$hashref->{column} = $val;
visually clear distinction between
$row->{column} / $row->remote_table()

Callback example
WITH RECURSIVE nodetree(level, id, pid, sort) AS ( SELECT 1, id, parent, '{1}'::int[] FROM nodes WHERE parent IS NULL UNION SELECT level+1,p.id, parent, sort||p.id FROM nodetree pr JOIN nodes p ON p.parent = pr.id
) SELECT * FROM nodetree ORDER BY sort;
my $with_clause = "WITH RECURSIVE …"; DBIx::DataModel->Schema('Tst') ->Table(qw/Nodetree nodetree id/); my $result = Tst::Nodetree-> select ( -post_SQL => sub {my $sql = shift; $sql =~ s/^/$with_clause/; return $sql, @_ }, -orderBy => 'sort', );

Verbose form for definitions
DBIx::DataModel-> define_schema (
class => "My::DB"
);
My::DB->metadm-> define_table (
class => "Author",
db_name => "author",
primary_key => "author_id"
):

New features in 2.0
metaclasses
client can query about tables, associations, types, etc.
method namespace for regular objects is not polluted by meta-methods
single-schema / multi-schema mode
misc additions
bulk update & bulk delete
support for table inheritance
arbitrary clauses in joins
API changes
perlish_method_names()
SQL generation moved to SQL::Abstract::More
Params::Validate everywhere
deprecated Autoload()
compatibility layer : use DBIx::DataModel –compatibility => 1.0

Migration to v2.0
deploy DBIx::DataModel 2.0
-compatibility => 1.0
test
change client code according to new API
test
suppress compatibility layer

DBIx::DataModel history
2005 : 1st CPAN publication (v0.10, 16.09.05)
2006 : YAPC::EU::06 Birmingham presentation
2008 : heavy refactoring (v1.03, 23.09.08)
statement object
implicit schema name
2011: heavy refactoring
metaobject layer
multi-schema mode
API renaming

http://anainc.tumblr.com	123
http://b.ana.io	41
http://anaioco.tumblr.com	19
http://www.slideshare.net	16

DBIx-DataModel v2.0 in detail

by ldami on Jun 13, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 199

Statistics

DBIx-DataModel v2.0 in detail — Presentation Transcript