• Like
PofEAA and SQLAlchemy
Upcoming SlideShare
Loading in...5
×
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
10,845
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
13
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. PofEAA and SQLAlchemy INADA Naoki @methane
  • 2. Public Activity MessagePack-Python JSON like binary protocol wsaccel WebSocket accelerator for Tornado, ws4py MyKaze PyMySQL in Tornado (WIP) Python for PHPer Quick guide of Python for PHPer (Japanese)
  • 3. Online Game Developer R&D and Architect Tuning server side middleware and application KLab Inc.
  • 4. SQLAlchemy
  • 5. Great O/R Mapper for Python
  • 6. Patterns of Enterprise Application Architecture
  • 7. Design patterns good to know for Web Programmers
  • 8. SQLAlchemy Quickly (from tutorial)
  • 9. ● Dialects -- Various DB-API wrapper ● Engine -- Connection management ● Schema and Types ● SQL expression ● and O/R Mapper SQLAlchemy features
  • 10. Create Engine from sqlalchemy import create_engine engine = create_engine('sqlite:///:memory:', echo=True) # Using engine without O/R mapper con = engine.connect() with con.begin() as trx: con.execute('SELECT 1+1') con.close()
  • 11. Define Object and Schema from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import Column, Integer, String Base = declarative_base() class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) name = Column(String) email = Column(String, nullable=False, unique=True) password = Column(String, nullable=False)
  • 12. Create table on DB Base.metadata.create_all(engine) Output: CREATE TABLE users ( id INTEGER NOT NULL, name VARCHAR, email VARCHAR NOT NULL, password VARCHAR NOT NULL, PRIMARY KEY (id), UNIQUE (email) )
  • 13. Create an instance >>> ed_user = User(name='ed', ... email='ed@example.com', ... password='edspassword') ... >>> ed_user.password 'edspassword' >>> str(ed_user.id) 'None'
  • 14. Save it from sqlalchemy.orm import sessionmaker Session = sessionmaker(bind=engine) session = Session() session.add(ed_user) session.commit()
  • 15. Too complicated?
  • 16. Why not just engine.save(user) What's session?
  • 17. Session is Unit of Work
  • 18. Unit of Work is a Pattern in PofEAA
  • 19. Part 1. The Narratives 3. Mapping to RDB Part 2. The Patterns 10. Data Source Architectural Patterns 11. O/R Behavioral Patterns 12. O/R Structural Patterns 13. O/R Metadata Mapping Patterns O/R chapters in the P of EAA
  • 20. Part 1. The Narratives 3. Mapping to RDB Part 2. The Patterns 10. Data Source Architectural Patterns 11. O/R Behavioral Patterns 12. O/R Structural Patterns 13. O/R Metadata Mapping Patterns O/R chapters in the P of EAA
  • 21. 11. O/R Behavioral Patterns ● Unit of Work ● Identity map ● Lazy load
  • 22. Unit of Work Target: Maintains set of objects to save in transaction. How: 1. Wrap transaction. 2. Keep track of new, modified and deleted objects. 3. Save objects when commit
  • 23. Why Unit of Work Remove boilerplate saving code from domain logic. Avoid objects saved too often.
  • 24. Unit of Work in SQLAlchemy # Create an Unit of Work session = Session() # Get connection and start transaction ed_user = session.query(User). filter(User.name=='ed').one() ed_user.name='E.D.' session.commit() # save ed_user and commit
  • 25. Object states user = User(...) # Transient (not saved) session.add(user) # pending (saved) session.flush() # persistent (saved if modified) session.expunge(user) # Detached (not saved)
  • 26. Flushing Execute insert / update /delete query. Session flushes automatically when querying to avoid inconsistent result. (autoflush) You can stop autoflush. You can manually flush.
  • 27. expire -- cause reload session.expire(user) print(user.name) # Cause reloading commit() # expires all objects for consistency. You can avoid this by expire_on_commit=False.
  • 28. Contextual Session from sqlalchemy.orm import (sessionmaker, scoped_session) Session = scoped_session( sessionmaker(bind=engine)) Session.add(ed_user) Session.commit() ... Session.remove()
  • 29. 11. O/R Behavioral Patterns ● Unit of Work ● Identity map ● Lazy load
  • 30. Target: Avoid two objects for one record. How: Keep mapping of (table, pk) -> object. Check the map when querying. Bonus: It acts like a easy cache Identity map
  • 31. Identity map in SQLAlchemy Session has identity map. # Looks up id map after querying session.query(User).filter(User.id==3).one() # Looks up id map before and after querying. session.query(User).get(3) NOTE: Id map is weak reference by default.
  • 32. 11. O/R Behavioral Patterns ● Unit of Work ● Identity map ● Lazy load
  • 33. Lazy Load Load relation or heavy column on demand.
  • 34. Lazy loading relationship http://docs.sqlalchemy. org/en/rel_0_8/orm/loading.html SQLAlchemy uses lazy loading by default for relationship. You can choose eager loading strategy.
  • 35. Lazy loading big column http://docs.sqlalchemy. org/en/rel_0_8/orm/mapper_config. html#deferred-column-loading SQLAlchemy supports deferred column to lazy loading big BLOB/TEXT.
  • 36. O/R Structural Pattern
  • 37. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  • 38. Object have PK in database as a field. class Player(Base): id = Column(INTEGER, primary=True) Identity Field
  • 39. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  • 40. Map one to one or one to many relation to Object reference. Not: session.query(Address).get(user.address_id) Yes: user.address Foreign Key Mapping
  • 41. With property from werkzeug.utils.cached_property class User(Base): ... @cached_property def address(self): session.query(Address). get(self.address_id) This is enough to achieve ``user.address``. But no SQLA support.
  • 42. relationship class User(Base): ... addresses = relationship('Address', backref='user') class Address(Base): ... user_id = Column(Integer, ForeignKey('user.id'))
  • 43. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  • 44. Association Table Mapping Mapping many-to-many association table to object reference. In SQLAlchemy: Pass “secondary” argument to relationship() http://docs.sqlalchemy.org/en/rel_0_8/orm/tutorial.html#building-a-many- to-many-relationship
  • 45. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  • 46. Dependent Mapping O/R map ownership without identity. PofEAA says: I don’t recommend Dependent Mapping if you’re using Unit of Work.
  • 47. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  • 48. Embedded Value Map an object to some columns in a row. class Duration: start_date end_date class Account: duration CREATE TABLE account ( … start_date DATE, end_date DATE, … );
  • 49. Embedded Value by property @property def duration(self): return Duration( self.start_date, self.end_date) @duration.setter def duration(self, duration): self.start_date = duration.start self.end_date = duration.end
  • 50. Embedded Value by Composite http://docs.sqlalchemy.org/en/rel_0_8/orm/mapper_config.html#mapper- composite class Account(Base): ... start_date = Column(DATE) end_date = Column(DATE) duration = composite(Duration, start_date, end_date) class Duration(namedtuple(‘Duration’, ‘start end’)): def __composite_value__(self): return self
  • 51. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  • 52. Serialized LOB Serialize objects and save to XLOB column.
  • 53. Using property _info = Column('info', BLOB) @property def info(self): return json.loads(self._info) @info.setter def _set_info(self, info): self._info = json.dumps(self)
  • 54. Custom Type SQLAlchemy provides PickleType for serialized LOB. You can define custom type via TypeDecorator: http://docs.sqlalchemy.org/en/rel_0_8/core/types. html#marshal-json-strings
  • 55. 12. O/R Structural Pattern ● Identity Field ● Foreign Key Mapping ● Association Table Mapping ● Dependent Mapping ● Embedded Value ● Serialized LOB ● Single Table Inheritance ● Class Table Inheritance ● Concrete Table Inheritance ● Inheritance Mappers
  • 56. Single Table Inheritance Player SoccerPlayer BaseballPlayer CREATE TABLE player( id INTEGER PRIMARY KEY, type INTEGER NOT NULL, position INTEGER, ) All classes saved into one table.
  • 57. Single Table Inheritance in SQLA class Player(Base): __tablename__ = ‘player’ id = Column(INTEGER, primary_key=True) position = Column(INTEGER) type = Column(INTEGER, nullable=False) __mapper_args__ = {‘polymorphic_on’: type} SOCCER_PLAYER = 1 BASEBALL_PLAYER = 2 class SoccerPlayer(Player): __mapper_args__ = { ‘polymorhic_identity’: Player.SOCCER_PLAYER}
  • 58. Defining columns in subclass Defining columns in subclass may cause conflict. SQLA provides way to avoid it. See below. http://docs.sqlalchemy. org/en/latest/orm/extensions/declarative.html#resolving- column-conflicts
  • 59. Class Table Inheritance Player SoccerPlayer BaseballPlayer CREATE TABLE player( id INTEGER PRIMARY KEY, type INTEGER NOT NULL ) CREATE TABLE soccer_player( id INTEGER PRIMARY KEY, position INTEGER NOT NULL ) CREATE TABLE baseball_player( id INTEGER PRIMARY KEY, position INTEGER NOT NULL ) Tables for each classes.
  • 60. Class table inheritance in SQLA SQLA call it “Joined table inheritance” http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative.html#joined- table-inheritance
  • 61. class Player(Base): __tablename__ = 'player' id = Column(INTEGER, primary_key=True) type = Column(INTEGER, nullable=False) SOCCER_PLAYER = 1 BASEBALL_PLAYER = 2 __mapper_args__ = {'polymorphic_on': type} class SoccerPlayer(Player): __tablename__ = 'soccer_player' id = Column(ForeignKey('player.id'), primary_key=True) position = Column(INTEGER, nullable=False) __mapper_args__ = { 'polymorhic_identity': Player.SOCCER_PLAYER}
  • 62. Concrete Table Inheritance Player SoccerPlayer BaseballPlayer CREATE TABLE soccer_player( id INTEGER PRIMARY KEY, name VARCHAR(32) NOT NULL, position INTEGER NOT NULL ) CREATE TABLE baseball_player( id INTEGER PRIMARY KEY, name VARCHAR(32) NOT NULL, position INTEGER NOT NULL ) Tables for each concrete classes
  • 63. Mix-in class BasePlayer(object): id = Column(INTEGER, primary_key=True) name = Column(VARCHAR(32), nullable=False) class SoccerPlayer(BasePlayer, Base): __tablename__ = ‘soccer_player’ position = Column(INTEGER, nullable=False) Pros) Simple. Cons) You can’t get support from SQLAlchemy
  • 64. Concrete Table Inheritance in SQLA http://docs.sqlalchemy.org/en/latest/orm/inheritance. html#concrete-table-inheritance
  • 65. 10. Data Source Architectural Patterns ● Table Data Gateway ● Row Data Gateway ● Active Record ● Data Mapper ● Architectural Pattern and SQLAlchemy
  • 66. Table Data Gateway Target: Split querying from your domain logic How: Create gateway for each table. class PlayerGateway: def find_by_id(self, id) def find_by_name(self, name) def update(self, id, name=None, age=None) def insert(self, id, name, age)
  • 67. Row Data Gateway Target: Split querying from your domain logic How: Create gateway for each record. class PlayerGateway: @classmethod def find_by_id(cls, id): … return cls(id, name, age) … def insert(self) def update(self)
  • 68. Active Record Row Data Gateway with Domain Logic class Player: @classmethod def find_by_id(cls, id) ... def birthday(self): self.age += 1
  • 69. Data Mapper Target: Split out column/attribute mapping code from your domain logic. class PersonMapper(BaseMapper): def map(self, row): return Person(id=row.id, name=row.name, age=row.age)
  • 70. Architectural Patterns and SQLAlchemy My personal consideration
  • 71. Querying in SQLAlchemy Unit of Work handles most of update and insert queries. Easy query can be written very easy. (Building complex query is hard)
  • 72. Table Gateway? Most tables doesn’t require Table Gateway. When there are some complex queries, Table Gateway is good to have.
  • 73. Row Data Gateway? Active Record? Separating model object and row data gateway cause additional complexity. For example, does your model need identity map? Person Person PersonRow
  • 74. Active Record drawbacks Problem: Good table design is not good class design always. My answer: Use structural patterns like Embedded Value. P: Mixing database access and domain logic is bad idea. A: Use Table Gateway to separate them.
  • 75. Fat Model (™) Person.register(...) # class method person.unregister() person.request_friend(other_person_id) ...
  • 76. Simple ActiveRecord & Service class AccountService: def register(self, name, email, age,...) def unregister(self, person_id) class FriendService: def request(self, from_person_id, to_person_id)
  • 77. Finder in Service When you use service classes, you may be able to put finders there and don’t use Table Gateway. class PersonService: ... def find_by_name(self, name): return Session.query(Person). filter_by(name=name).all()
  • 78. Anemic Domain Model http://www.martinfowler. com/bliki/AnemicDomainModel.html P: Implementing all domain logic in service is not a OOP. A: Simple Active Record + Medium sized classes + Service
  • 79. When you design large application or framework, PofEAA helps you lot.