PofEAA and
SQLAlchemy
INADA Naoki
@methane
Public Activity
MessagePack-Python
JSON like binary protocol
wsaccel
WebSocket accelerator for Tornado, ws4py
MyKaze
PyMyS...
Online Game Developer
R&D and Architect
Tuning server side middleware and application
KLab Inc.
SQLAlchemy
Great O/R Mapper
for Python
Patterns of
Enterprise Application
Architecture
Design patterns
good to know
for Web Programmers
SQLAlchemy
Quickly
(from tutorial)
● Dialects -- Various DB-API wrapper
● Engine -- Connection management
● Schema and Types
● SQL expression
● and O/R Mappe...
Create Engine
from sqlalchemy import create_engine
engine = create_engine('sqlite:///:memory:',
echo=True)
# Using engine ...
Define Object and Schema
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, S...
Create table on DB
Base.metadata.create_all(engine)
Output:
CREATE TABLE users (
id INTEGER NOT NULL,
name VARCHAR,
email ...
Create an instance
>>> ed_user = User(name='ed',
... email='ed@example.com',
... password='edspassword')
...
>>> ed_user.p...
Save it
from sqlalchemy.orm import sessionmaker
Session = sessionmaker(bind=engine)
session = Session()
session.add(ed_use...
Too complicated?
Why not just
engine.save(user)
What's session?
Session is Unit of Work
Unit of Work is
a Pattern
in PofEAA
Part 1. The Narratives
3. Mapping to RDB
Part 2. The Patterns
10. Data Source Architectural Patterns
11. O/R Behavioral Pa...
Part 1. The Narratives
3. Mapping to RDB
Part 2. The Patterns
10. Data Source Architectural Patterns
11. O/R Behavioral Pa...
11. O/R Behavioral Patterns
● Unit of Work
● Identity map
● Lazy load
Unit of Work
Target: Maintains set of objects to save
in transaction.
How:
1. Wrap transaction.
2. Keep track of new, modi...
Why Unit of Work
Remove boilerplate saving code from domain
logic.
Avoid objects saved too often.
Unit of Work in SQLAlchemy
# Create an Unit of Work
session = Session()
# Get connection and start transaction
ed_user = s...
Object states
user = User(...)
# Transient (not saved)
session.add(user)
# pending (saved)
session.flush()
# persistent (s...
Flushing
Execute insert / update /delete query.
Session flushes automatically when querying
to avoid inconsistent result. ...
expire -- cause reload
session.expire(user)
print(user.name) # Cause reloading
commit() # expires all objects for consiste...
Contextual Session
from sqlalchemy.orm import (sessionmaker,
scoped_session)
Session = scoped_session(
sessionmaker(bind=e...
11. O/R Behavioral Patterns
● Unit of Work
● Identity map
● Lazy load
Target:
Avoid two objects for one record.
How:
Keep mapping of (table, pk) -> object.
Check the map when querying.
Bonus: ...
Identity map in SQLAlchemy
Session has identity map.
# Looks up id map after querying
session.query(User).filter(User.id==...
11. O/R Behavioral Patterns
● Unit of Work
● Identity map
● Lazy load
Lazy Load
Load relation or heavy column on demand.
Lazy loading relationship
http://docs.sqlalchemy.
org/en/rel_0_8/orm/loading.html
SQLAlchemy uses lazy loading by default ...
Lazy loading big column
http://docs.sqlalchemy.
org/en/rel_0_8/orm/mapper_config.
html#deferred-column-loading
SQLAlchemy ...
O/R Structural Pattern
12. O/R Structural Pattern
● Identity Field
● Foreign Key Mapping
● Association Table Mapping
● Dependent Mapping
● Embedd...
Object have PK in database as a field.
class Player(Base):
id = Column(INTEGER, primary=True)
Identity Field
12. O/R Structural Pattern
● Identity Field
● Foreign Key Mapping
● Association Table Mapping
● Dependent Mapping
● Embedd...
Map one to one or one to many relation to
Object reference.
Not:
session.query(Address).get(user.address_id)
Yes:
user.add...
With property
from werkzeug.utils.cached_property
class User(Base):
...
@cached_property
def address(self):
session.query(...
relationship
class User(Base):
...
addresses = relationship('Address',
backref='user')
class Address(Base):
...
user_id = ...
12. O/R Structural Pattern
● Identity Field
● Foreign Key Mapping
● Association Table Mapping
● Dependent Mapping
● Embedd...
Association Table Mapping
Mapping many-to-many association table to
object reference.
In SQLAlchemy:
Pass “secondary” argu...
12. O/R Structural Pattern
● Identity Field
● Foreign Key Mapping
● Association Table Mapping
● Dependent Mapping
● Embedd...
Dependent Mapping
O/R map ownership without identity.
PofEAA says:
I don’t recommend Dependent Mapping
if you’re using Uni...
12. O/R Structural Pattern
● Identity Field
● Foreign Key Mapping
● Association Table Mapping
● Dependent Mapping
● Embedd...
Embedded Value
Map an object to some columns in a row.
class Duration:
start_date
end_date
class Account:
duration
CREATE ...
Embedded Value by property
@property
def duration(self):
return Duration(
self.start_date, self.end_date)
@duration.setter...
Embedded Value by Composite
http://docs.sqlalchemy.org/en/rel_0_8/orm/mapper_config.html#mapper-
composite
class Account(B...
12. O/R Structural Pattern
● Identity Field
● Foreign Key Mapping
● Association Table Mapping
● Dependent Mapping
● Embedd...
Serialized LOB
Serialize objects and save to XLOB column.
Using property
_info = Column('info', BLOB)
@property
def info(self):
return json.loads(self._info)
@info.setter
def _set_...
Custom Type
SQLAlchemy provides PickleType for serialized
LOB.
You can define custom type via TypeDecorator:
http://docs.s...
12. O/R Structural Pattern
● Identity Field
● Foreign Key Mapping
● Association Table Mapping
● Dependent Mapping
● Embedd...
Single Table Inheritance
Player
SoccerPlayer BaseballPlayer
CREATE TABLE player(
id INTEGER PRIMARY KEY,
type INTEGER NOT ...
Single Table Inheritance in SQLA
class Player(Base):
__tablename__ = ‘player’
id = Column(INTEGER, primary_key=True)
posit...
Defining columns in subclass
Defining columns in subclass may cause
conflict.
SQLA provides way to avoid it. See below.
ht...
Class Table Inheritance
Player
SoccerPlayer BaseballPlayer
CREATE TABLE player(
id INTEGER PRIMARY KEY,
type INTEGER NOT N...
Class table inheritance in SQLA
SQLA call it “Joined table inheritance”
http://docs.sqlalchemy.org/en/latest/orm/extension...
class Player(Base):
__tablename__ = 'player'
id = Column(INTEGER, primary_key=True)
type = Column(INTEGER, nullable=False)...
Concrete Table Inheritance
Player
SoccerPlayer BaseballPlayer
CREATE TABLE soccer_player(
id INTEGER PRIMARY KEY,
name VAR...
Mix-in
class BasePlayer(object):
id = Column(INTEGER, primary_key=True)
name = Column(VARCHAR(32), nullable=False)
class S...
Concrete Table Inheritance in SQLA
http://docs.sqlalchemy.org/en/latest/orm/inheritance.
html#concrete-table-inheritance
10. Data Source Architectural Patterns
● Table Data Gateway
● Row Data Gateway
● Active Record
● Data Mapper
● Architectur...
Table Data Gateway
Target: Split querying from your domain logic
How: Create gateway for each table.
class PlayerGateway:
...
Row Data Gateway
Target: Split querying from your domain logic
How: Create gateway for each record.
class PlayerGateway:
@...
Active Record
Row Data Gateway with Domain Logic
class Player:
@classmethod
def find_by_id(cls, id)
...
def birthday(self)...
Data Mapper
Target: Split out column/attribute mapping code
from your domain logic.
class PersonMapper(BaseMapper):
def ma...
Architectural Patterns
and SQLAlchemy
My personal consideration
Querying in SQLAlchemy
Unit of Work handles most of update and insert
queries.
Easy query can be written very easy.
(Build...
Table Gateway?
Most tables doesn’t require Table Gateway.
When there are some complex queries,
Table Gateway is good to ha...
Row Data Gateway?
Active Record?
Separating model object and row data gateway
cause additional complexity.
For example, do...
Active Record drawbacks
Problem: Good table design is not good class
design always.
My answer: Use structural patterns lik...
Fat Model (™)
Person.register(...) # class method
person.unregister()
person.request_friend(other_person_id)
...
Simple ActiveRecord & Service
class AccountService:
def register(self, name, email, age,...)
def unregister(self, person_i...
Finder in Service
When you use service classes, you may be able to
put finders there and don’t use Table Gateway.
class Pe...
Anemic Domain Model
http://www.martinfowler.
com/bliki/AnemicDomainModel.html
P: Implementing all domain logic in service ...
When you design large
application or framework,
PofEAA helps you lot.
PofEAA and SQLAlchemy
Upcoming SlideShare
Loading in...5
×

PofEAA and SQLAlchemy

11,397

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
11,397
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
16
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

PofEAA and SQLAlchemy

  1. 1. PofEAA and SQLAlchemy INADA Naoki @methane
  2. 2. Public Activity MessagePack-Python JSON like binary protocol wsaccel WebSocket accelerator for Tornado, ws4py MyKaze PyMySQL in Tornado (WIP) Python for PHPer Quick guide of Python for PHPer (Japanese)
  3. 3. Online Game Developer R&D and Architect Tuning server side middleware and application KLab Inc.
  4. 4. SQLAlchemy
  5. 5. Great O/R Mapper for Python
  6. 6. Patterns of Enterprise Application Architecture
  7. 7. Design patterns good to know for Web Programmers
  8. 8. SQLAlchemy Quickly (from tutorial)
  9. 9. ● Dialects -- Various DB-API wrapper ● Engine -- Connection management ● Schema and Types ● SQL expression ● and O/R Mapper SQLAlchemy features
  10. 10. Create Engine from sqlalchemy import create_engine engine = create_engine('sqlite:///:memory:', echo=True) # Using engine without O/R mapper con = engine.connect() with con.begin() as trx: con.execute('SELECT 1+1') con.close()
  11. 11. Define Object and Schema from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import Column, Integer, String Base = declarative_base() class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) name = Column(String) email = Column(String, nullable=False, unique=True) password = Column(String, nullable=False)
  12. 12. Create table on DB Base.metadata.create_all(engine) Output: CREATE TABLE users ( id INTEGER NOT NULL, name VARCHAR, email VARCHAR NOT NULL, password VARCHAR NOT NULL, PRIMARY KEY (id), UNIQUE (email) )
  13. 13. Create an instance >>> ed_user = User(name='ed', ... email='ed@example.com', ... password='edspassword') ... >>> ed_user.password 'edspassword' >>> str(ed_user.id) 'None'
  14. 14. Save it from sqlalchemy.orm import sessionmaker Session = sessionmaker(bind=engine) session = Session() session.add(ed_user) session.commit()
  15. 15. Too complicated?
  16. 16. Why not just engine.save(user) What's session?
  17. 17. Session is Unit of Work
  18. 18. Unit of Work is a Pattern in PofEAA
  19. 19. Part 1. The Narratives 3. Mapping to RDB Part 2. The Patterns 10. Data Source Architectural Patterns 11. O/R Behavioral Patterns 12. O/R Structural Patterns 13. O/R Metadata Mapping Patterns O/R chapters in the P of EAA
  20. 20. Part 1. The Narratives 3. Mapping to RDB Part 2. The Patterns 10. Data Source Architectural Patterns 11. O/R Behavioral Patterns 12. O/R Structural Patterns 13. O/R Metadata Mapping Patterns O/R chapters in the P of EAA
  21. 21. 11. O/R Behavioral Patterns ● Unit of Work ● Identity map ● Lazy load
  22. 22. Unit of Work Target: Maintains set of objects to save in transaction. How: 1. Wrap transaction. 2. Keep track of new, modified and deleted objects. 3. Save objects when commit
  23. 23. Why Unit of Work Remove boilerplate saving code from domain logic. Avoid objects saved too often.
  24. 24. Unit of Work in SQLAlchemy # Create an Unit of Work session = Session() # Get connection and start transaction ed_user = session.query(User). filter(User.name=='ed').one() ed_user.name='E.D.' session.commit() # save ed_user and commit
  25. 25. Object states user = User(...) # Transient (not saved) session.add(user) # pending (saved) session.flush() # persistent (saved if modified) session.expunge(user) # Detached (not saved)
  26. 26. Flushing Execute insert / update /delete query. Session flushes automatically when querying to avoid inconsistent result. (autoflush) You can stop autoflush. You can manually flush.
  27. 27. expire -- cause reload session.expire(user) print(user.name) # Cause reloading commit() # expires all objects for consistency. You can avoid this by expire_on_commit=False.
  28. 28. Contextual Session from sqlalchemy.orm import (sessionmaker, scoped_session) Session = scoped_session( sessionmaker(bind=engine)) Session.add(ed_user) Session.commit() ... Session.remove()
  29. 29. 11. O/R Behavioral Patterns ● Unit of Work ● Identity map ● Lazy load
  30. 30. Target: Avoid two objects for one record. How: Keep mapping of (table, pk) -> object. Check the map when querying. Bonus: It acts like a easy cache Identity map
  31. 31. Identity map in SQLAlchemy Session has identity map. # Looks up id map after querying session.query(User).filter(User.id==3).one() # Looks up id map before and after querying. session.query(User).get(3) NOTE: Id map is weak reference by default.
  32. 32. 11. O/R Behavioral Patterns ● Unit of Work ● Identity map ● Lazy load
  33. 33. Lazy Load Load relation or heavy column on demand.
  34. 34. Lazy loading relationship http://docs.sqlalchemy. org/en/rel_0_8/orm/loading.html SQLAlchemy uses lazy loading by default for relationship. You can choose eager loading strategy.
  35. 35. Lazy loading big column http://docs.sqlalchemy. org/en/rel_0_8/orm/mapper_config. html#deferred-column-loading SQLAlchemy supports deferred column to lazy loading big BLOB/TEXT.
  36. 36. O/R Structural Pattern
  37. 37. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  38. 38. Object have PK in database as a field. class Player(Base): id = Column(INTEGER, primary=True) Identity Field
  39. 39. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  40. 40. Map one to one or one to many relation to Object reference. Not: session.query(Address).get(user.address_id) Yes: user.address Foreign Key Mapping
  41. 41. With property from werkzeug.utils.cached_property class User(Base): ... @cached_property def address(self): session.query(Address). get(self.address_id) This is enough to achieve ``user.address``. But no SQLA support.
  42. 42. relationship class User(Base): ... addresses = relationship('Address', backref='user') class Address(Base): ... user_id = Column(Integer, ForeignKey('user.id'))
  43. 43. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  44. 44. Association Table Mapping Mapping many-to-many association table to object reference. In SQLAlchemy: Pass “secondary” argument to relationship() http://docs.sqlalchemy.org/en/rel_0_8/orm/tutorial.html#building-a-many- to-many-relationship
  45. 45. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  46. 46. Dependent Mapping O/R map ownership without identity. PofEAA says: I don’t recommend Dependent Mapping if you’re using Unit of Work.
  47. 47. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  48. 48. Embedded Value Map an object to some columns in a row. class Duration: start_date end_date class Account: duration CREATE TABLE account ( … start_date DATE, end_date DATE, … );
  49. 49. Embedded Value by property @property def duration(self): return Duration( self.start_date, self.end_date) @duration.setter def duration(self, duration): self.start_date = duration.start self.end_date = duration.end
  50. 50. Embedded Value by Composite http://docs.sqlalchemy.org/en/rel_0_8/orm/mapper_config.html#mapper- composite class Account(Base): ... start_date = Column(DATE) end_date = Column(DATE) duration = composite(Duration, start_date, end_date) class Duration(namedtuple(‘Duration’, ‘start end’)): def __composite_value__(self): return self
  51. 51. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  52. 52. Serialized LOB Serialize objects and save to XLOB column.
  53. 53. Using property _info = Column('info', BLOB) @property def info(self): return json.loads(self._info) @info.setter def _set_info(self, info): self._info = json.dumps(self)
  54. 54. Custom Type SQLAlchemy provides PickleType for serialized LOB. You can define custom type via TypeDecorator: http://docs.sqlalchemy.org/en/rel_0_8/core/types. html#marshal-json-strings
  55. 55. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  56. 56. Single Table Inheritance Player SoccerPlayer BaseballPlayer CREATE TABLE player( id INTEGER PRIMARY KEY, type INTEGER NOT NULL, position INTEGER, ) All classes saved into one table.
  57. 57. Single Table Inheritance in SQLA class Player(Base): __tablename__ = ‘player’ id = Column(INTEGER, primary_key=True) position = Column(INTEGER) type = Column(INTEGER, nullable=False) __mapper_args__ = {‘polymorphic_on’: type} SOCCER_PLAYER = 1 BASEBALL_PLAYER = 2 class SoccerPlayer(Player): __mapper_args__ = { ‘polymorhic_identity’: Player.SOCCER_PLAYER}
  58. 58. Defining columns in subclass Defining columns in subclass may cause conflict. SQLA provides way to avoid it. See below. http://docs.sqlalchemy. org/en/latest/orm/extensions/declarative.html#resolving- column-conflicts
  59. 59. Class Table Inheritance Player SoccerPlayer BaseballPlayer CREATE TABLE player( id INTEGER PRIMARY KEY, type INTEGER NOT NULL ) CREATE TABLE soccer_player( id INTEGER PRIMARY KEY, position INTEGER NOT NULL ) CREATE TABLE baseball_player( id INTEGER PRIMARY KEY, position INTEGER NOT NULL ) Tables for each classes.
  60. 60. Class table inheritance in SQLA SQLA call it “Joined table inheritance” http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative.html#joined- table-inheritance
  61. 61. class Player(Base): __tablename__ = 'player' id = Column(INTEGER, primary_key=True) type = Column(INTEGER, nullable=False) SOCCER_PLAYER = 1 BASEBALL_PLAYER = 2 __mapper_args__ = {'polymorphic_on': type} class SoccerPlayer(Player): __tablename__ = 'soccer_player' id = Column(ForeignKey('player.id'), primary_key=True) position = Column(INTEGER, nullable=False) __mapper_args__ = { 'polymorhic_identity': Player.SOCCER_PLAYER}
  62. 62. Concrete Table Inheritance Player SoccerPlayer BaseballPlayer CREATE TABLE soccer_player( id INTEGER PRIMARY KEY, name VARCHAR(32) NOT NULL, position INTEGER NOT NULL ) CREATE TABLE baseball_player( id INTEGER PRIMARY KEY, name VARCHAR(32) NOT NULL, position INTEGER NOT NULL ) Tables for each concrete classes
  63. 63. Mix-in class BasePlayer(object): id = Column(INTEGER, primary_key=True) name = Column(VARCHAR(32), nullable=False) class SoccerPlayer(BasePlayer, Base): __tablename__ = ‘soccer_player’ position = Column(INTEGER, nullable=False) Pros) Simple. Cons) You can’t get support from SQLAlchemy
  64. 64. Concrete Table Inheritance in SQLA http://docs.sqlalchemy.org/en/latest/orm/inheritance. html#concrete-table-inheritance
  65. 65. 10. Data Source Architectural Patterns ● Table Data Gateway ● Row Data Gateway ● Active Record ● Data Mapper ● Architectural Pattern and SQLAlchemy
  66. 66. Table Data Gateway Target: Split querying from your domain logic How: Create gateway for each table. class PlayerGateway: def find_by_id(self, id) def find_by_name(self, name) def update(self, id, name=None, age=None) def insert(self, id, name, age)
  67. 67. Row Data Gateway Target: Split querying from your domain logic How: Create gateway for each record. class PlayerGateway: @classmethod def find_by_id(cls, id): … return cls(id, name, age) … def insert(self) def update(self)
  68. 68. Active Record Row Data Gateway with Domain Logic class Player: @classmethod def find_by_id(cls, id) ... def birthday(self): self.age += 1
  69. 69. Data Mapper Target: Split out column/attribute mapping code from your domain logic. class PersonMapper(BaseMapper): def map(self, row): return Person(id=row.id, name=row.name, age=row.age)
  70. 70. Architectural Patterns and SQLAlchemy My personal consideration
  71. 71. Querying in SQLAlchemy Unit of Work handles most of update and insert queries. Easy query can be written very easy. (Building complex query is hard)
  72. 72. Table Gateway? Most tables doesn’t require Table Gateway. When there are some complex queries, Table Gateway is good to have.
  73. 73. Row Data Gateway? Active Record? Separating model object and row data gateway cause additional complexity. For example, does your model need identity map? Person Person PersonRow
  74. 74. Active Record drawbacks Problem: Good table design is not good class design always. My answer: Use structural patterns like Embedded Value. P: Mixing database access and domain logic is bad idea. A: Use Table Gateway to separate them.
  75. 75. Fat Model (™) Person.register(...) # class method person.unregister() person.request_friend(other_person_id) ...
  76. 76. Simple ActiveRecord & Service class AccountService: def register(self, name, email, age,...) def unregister(self, person_id) class FriendService: def request(self, from_person_id, to_person_id)
  77. 77. Finder in Service When you use service classes, you may be able to put finders there and don’t use Table Gateway. class PersonService: ... def find_by_name(self, name): return Session.query(Person). filter_by(name=name).all()
  78. 78. Anemic Domain Model http://www.martinfowler. com/bliki/AnemicDomainModel.html P: Implementing all domain logic in service is not a OOP. A: Simple Active Record + Medium sized classes + Service
  79. 79. When you design large application or framework, PofEAA helps you lot.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×