1
MoSQLMoSQL
MoskyMosky
2
More than SQL, but Less than ORMMore than SQL, but Less than ORM
MoSQLMoSQL
3
OutlineOutline
● Why not SQL?Why not SQL?
● Why ORM?Why ORM?
● MoSQLMoSQL
– SQL BuildersSQL Builders
– Model of Result SetModel of Result Set
● ConclusionConclusion
4
Why not SQL?Why not SQL?
5
SQL SyntaxSQL Syntax
● SELECT * FROM article;SELECT * FROM article;
● SELECT * FROM article LIMIT 1;SELECT * FROM article LIMIT 1;
● add “ ORDER BY created ”?add “ ORDER BY created ”?
● add “ OFFSET 10 ”?add “ OFFSET 10 ”?
● add “ GROUP BY author ”?add “ GROUP BY author ”?
● Is “ UPDATE article WHERE title='SQL' SETIs “ UPDATE article WHERE title='SQL' SET
title='ORM' ” correct?title='ORM' ” correct?
6
!@#$%!@#$%
7
SQL InjectionSQL Injection
● ') or '1'='1') or '1'='1
● ' or true; --' or true; --
● ' or 1=1; --' or 1=1; --
● ' or 2=2; --' or 2=2; --
● ' or 'str'='str'; --' or 'str'='str'; --
● ……
8
It may be hacker friendly.It may be hacker friendly.
9
SQL seems ancient, but ...SQL seems ancient, but ...
10
using SQL is theusing SQL is the FASTESTFASTEST way.way.
11
Why ORM?Why ORM?
12
ORM SyntaxORM Syntax
class User(Base):class User(Base):
__tablename__ = 'users'__tablename__ = 'users'
name = Column(String)name = Column(String)
fullname = Column(String)fullname = Column(String)
password = Column(String)password = Column(String)
13
ORM Syntax (cont.)ORM Syntax (cont.)
>>> fake_user = User('fakeuser', 'Invalid',>>> fake_user = User('fakeuser', 'Invalid',
'12345')'12345')
>>> session.add(fake_user)>>> session.add(fake_user)
>>> for row in session.query(User,>>> for row in session.query(User,
User.name).all():User.name).all():
... print row.User, row.name... print row.User, row.name
14
hmmm …hmmm …
15
SQL InjectionSQL Injection
● ' or true; --' or true; --
● ' or 1=1; --' or 1=1; --
● ' or 1=1; #' or 1=1; #
● ' or 1=1; /*' or 1=1; /*
● ') or '1'='1') or '1'='1
● ……
● SaferSafer
16
It's good!It's good!
17
ORM seems modern, but ...ORM seems modern, but ...
18
the most of ORMs are SLOW.the most of ORMs are SLOW.
19
SQL < ______ < ORMSQL < ______ < ORM
20
SQL < MoSQL < ORMSQL < MoSQL < ORM
21
SQL BuildersSQL Builders
22
SQL Builders (cont.)SQL Builders (cont.)
>>> from mosql.build import *>>> from mosql.build import *
>>>>>> select('pycon')select('pycon')
SELECT * FROM "pycon"SELECT * FROM "pycon"
>>> select('pycon',>>> select('pycon', {'id': 'mosky'}{'id': 'mosky'}))
SELECT * FROM "pycon" WHERE "id" = 'mosky'SELECT * FROM "pycon" WHERE "id" = 'mosky'
23
SQL Builders (cont.)SQL Builders (cont.)
>>> insert('pycon',>>> insert('pycon', {'yr': 2013, 'id': 'masky'}{'yr': 2013, 'id': 'masky'}))
INSERT INTO "pycon" ("id", "yr") VALUES ('masky', 2013)INSERT INTO "pycon" ("id", "yr") VALUES ('masky', 2013)
>>> update('pycon',>>> update('pycon',
...... where={'id': 'masky'}where={'id': 'masky'},,
...... set ={'id': 'mosky'}set ={'id': 'mosky'}
... )... )
UPDATE "pycon" SET "id"='mosky' WHERE "id" = 'masky'UPDATE "pycon" SET "id"='mosky' WHERE "id" = 'masky'
24
SQL Builders (cont.)SQL Builders (cont.)
● insert(table,insert(table, setset, …), …)
● select(table,select(table, wherewhere, …), …)
● update(table,update(table, wherewhere,, setset, …), …)
● delete(table,delete(table, wherewhere, …), …)
● ......
25
If you like it,If you like it,
26
sudo pip install mosqlsudo pip install mosql
27
Model of Result SetModel of Result Set
28
Model: Configure ConnectionModel: Configure Connection
import psycopg2.poolimport psycopg2.pool
from mosql.result import Modelfrom mosql.result import Model
pool = psycopg2.pool.SimpleConnectionPool(1, 5,pool = psycopg2.pool.SimpleConnectionPool(1, 5,
database='mosky')database='mosky')
class PostgreSQL(Model):class PostgreSQL(Model):
getconn = pool.getconngetconn = pool.getconn
putconn = pool.putconnputconn = pool.putconn
29
Model: Set the Name of TableModel: Set the Name of Table
class Person(PostgreSQL):class Person(PostgreSQL):
table = 'person'table = 'person'
>>> Person.select(>>> Person.select({'person_id': 'mosky'}{'person_id': 'mosky'}))
{'name': ['Mosky Liu'], 'person_id': ['mosky']}{'name': ['Mosky Liu'], 'person_id': ['mosky']}
>>> Person.where(person_id=>>> Person.where(person_id=('andy', 'mosky')('andy', 'mosky')))
{'name': ['Andy Warhol', 'Mosky Liu'], 'person_id':{'name': ['Andy Warhol', 'Mosky Liu'], 'person_id':
['andy', 'mosky']}['andy', 'mosky']}
30
Model: Make QueriesModel: Make Queries
Person.Person.selectselect({'person_id': 'mosky'})({'person_id': 'mosky'})
Person.Person.insertinsert({'person_id': 'tina'})({'person_id': 'tina'})
Person.Person.updateupdate((
where={'person_id': 'mosky'},where={'person_id': 'mosky'},
set ={'name' : 'Yiyu Liu'}set ={'name' : 'Yiyu Liu'}
))
Person.Person.deletedelete({'person_id': 'tina'})({'person_id': 'tina'})
31
Model: Squash ColumnsModel: Squash Columns
class Person(PostgreSQL):class Person(PostgreSQL):
table = 'person'table = 'person'
squashed = set(['person_id', 'name'])squashed = set(['person_id', 'name'])
>>> Person.select({'person_id': 'mosky'})>>> Person.select({'person_id': 'mosky'})
{'name':{'name': 'Mosky Liu''Mosky Liu', 'person_id':, 'person_id': 'mosky''mosky'}}
>>> Person.where(person_id=('andy', 'mosky'))>>> Person.where(person_id=('andy', 'mosky'))
{'name':{'name': 'Andy Warhol''Andy Warhol', 'person_id':, 'person_id': 'andy''andy'}}
32
Model: ArrangeModel: Arrange
class Person(PostgreSQL):class Person(PostgreSQL):
......
arrange_by = ('person_id', )arrange_by = ('person_id', )
>>> for person in Person.arrange(>>> for person in Person.arrange({'person_id':{'person_id':
('andy', 'mosky')}('andy', 'mosky')}):):
... print person... print person
{'name': 'Andy Warhol', 'person_id': 'andy'}{'name': 'Andy Warhol', 'person_id': 'andy'}
{'name': 'Mosky Liu', 'person_id': 'mosky'}{'name': 'Mosky Liu', 'person_id': 'mosky'}
33
Model: Arrange (cont.)Model: Arrange (cont.)
>>> for detail in>>> for detail in DetailDetail.arrange({'person_id':.arrange({'person_id':
('mosky', 'andy')}):('mosky', 'andy')}):
... print detail... print detail
......
{'detail_id': [5],{'detail_id': [5],
'key': 'email','key': 'email',
'person_id': 'andy','person_id': 'andy',
'val': ['andy@gmail.com']}'val': ['andy@gmail.com']}
......
34
Model: FindModel: Find
class Person(PostgreSQL):class Person(PostgreSQL):
......
arrange_by = ('person_id', )arrange_by = ('person_id', )
>>> for person in Person.>>> for person in Person.findfind((person_id=('andy',person_id=('andy',
'mosky')'mosky')):):
... print person... print person
{'name': 'Andy Warhol', 'person_id': 'andy'}{'name': 'Andy Warhol', 'person_id': 'andy'}
{'name': 'Mosky Liu', 'person_id': 'mosky'}{'name': 'Mosky Liu', 'person_id': 'mosky'}
35
Model: Identify a RowModel: Identify a Row
class Person(PostgreSQL):class Person(PostgreSQL):
......
ident_by = ('person_id', )ident_by = ('person_id', )
36
Model: ModificationModel: Modification
>>> p = Person.where(person_id='mosky')>>> p = Person.where(person_id='mosky')
>>>>>> p['name'] = 'Yiyu Liu'p['name'] = 'Yiyu Liu'
>>>>>> p.name = 'Yiyu Liu'p.name = 'Yiyu Liu'
>>> p.save()>>> p.save()
>>> d =>>> d = DetailDetail.where(.where(person_id='mosky', key='email'person_id='mosky', key='email'))
>>>>>> p['val'][0] = '<modified email>'p['val'][0] = '<modified email>'
>>>>>> p.val[0] = '<modified email>'p.val[0] = '<modified email>'
>>> p.save()>>> p.save()
37
Model: Pop and AppendModel: Pop and Append
>>> d = Detail.where(>>> d = Detail.where(person_id='mosky', key='email'person_id='mosky', key='email'))
>>>>>> p.pop(-1)p.pop(-1)
>>>>>> p.append({'val': '<new mail>'})p.append({'val': '<new mail>'})
>>> p.save()>>> p.save()
38
Model: Default ClausesModel: Default Clauses
class Person(PostgreSQL):class Person(PostgreSQL):
......
clauses = dict(clauses = dict(
order_by=('person_id', )order_by=('person_id', )
))
39
PerformancePerformance
●
AboutAbout 4x4x faster than SQLAlchemy.faster than SQLAlchemy.
● Just a little bit slower than pure SQL.Just a little bit slower than pure SQL.
40
SecuritySecurity
● Security by default.Security by default.
● Use escaping technique.Use escaping technique.
● Prevent SQL injection from both valuePrevent SQL injection from both value
and identifier.and identifier.
● Passed the tests fromPassed the tests from sqlmapsqlmap at level=5at level=5
and risk=3.and risk=3.
41
ConclusionConclusion
● Easy-to-LearnEasy-to-Learn
● ConvenientConvenient
● FasterFaster
● SecureSecure
● sudo pip install mosqlsudo pip install mosql
● http://mosql.mosky.tw/http://mosql.mosky.tw/
● Welcome to fork!Welcome to fork!

MoSQL: More than SQL, but less than ORM

  • 1.
  • 2.
    2 More than SQL,but Less than ORMMore than SQL, but Less than ORM MoSQLMoSQL
  • 3.
    3 OutlineOutline ● Why notSQL?Why not SQL? ● Why ORM?Why ORM? ● MoSQLMoSQL – SQL BuildersSQL Builders – Model of Result SetModel of Result Set ● ConclusionConclusion
  • 4.
  • 5.
    5 SQL SyntaxSQL Syntax ●SELECT * FROM article;SELECT * FROM article; ● SELECT * FROM article LIMIT 1;SELECT * FROM article LIMIT 1; ● add “ ORDER BY created ”?add “ ORDER BY created ”? ● add “ OFFSET 10 ”?add “ OFFSET 10 ”? ● add “ GROUP BY author ”?add “ GROUP BY author ”? ● Is “ UPDATE article WHERE title='SQL' SETIs “ UPDATE article WHERE title='SQL' SET title='ORM' ” correct?title='ORM' ” correct?
  • 6.
  • 7.
    7 SQL InjectionSQL Injection ●') or '1'='1') or '1'='1 ● ' or true; --' or true; -- ● ' or 1=1; --' or 1=1; -- ● ' or 2=2; --' or 2=2; -- ● ' or 'str'='str'; --' or 'str'='str'; -- ● ……
  • 8.
    8 It may behacker friendly.It may be hacker friendly.
  • 9.
    9 SQL seems ancient,but ...SQL seems ancient, but ...
  • 10.
    10 using SQL istheusing SQL is the FASTESTFASTEST way.way.
  • 11.
  • 12.
    12 ORM SyntaxORM Syntax classUser(Base):class User(Base): __tablename__ = 'users'__tablename__ = 'users' name = Column(String)name = Column(String) fullname = Column(String)fullname = Column(String) password = Column(String)password = Column(String)
  • 13.
    13 ORM Syntax (cont.)ORMSyntax (cont.) >>> fake_user = User('fakeuser', 'Invalid',>>> fake_user = User('fakeuser', 'Invalid', '12345')'12345') >>> session.add(fake_user)>>> session.add(fake_user) >>> for row in session.query(User,>>> for row in session.query(User, User.name).all():User.name).all(): ... print row.User, row.name... print row.User, row.name
  • 14.
  • 15.
    15 SQL InjectionSQL Injection ●' or true; --' or true; -- ● ' or 1=1; --' or 1=1; -- ● ' or 1=1; #' or 1=1; # ● ' or 1=1; /*' or 1=1; /* ● ') or '1'='1') or '1'='1 ● …… ● SaferSafer
  • 16.
  • 17.
    17 ORM seems modern,but ...ORM seems modern, but ...
  • 18.
    18 the most ofORMs are SLOW.the most of ORMs are SLOW.
  • 19.
    19 SQL < ______< ORMSQL < ______ < ORM
  • 20.
    20 SQL < MoSQL< ORMSQL < MoSQL < ORM
  • 21.
  • 22.
    22 SQL Builders (cont.)SQLBuilders (cont.) >>> from mosql.build import *>>> from mosql.build import * >>>>>> select('pycon')select('pycon') SELECT * FROM "pycon"SELECT * FROM "pycon" >>> select('pycon',>>> select('pycon', {'id': 'mosky'}{'id': 'mosky'})) SELECT * FROM "pycon" WHERE "id" = 'mosky'SELECT * FROM "pycon" WHERE "id" = 'mosky'
  • 23.
    23 SQL Builders (cont.)SQLBuilders (cont.) >>> insert('pycon',>>> insert('pycon', {'yr': 2013, 'id': 'masky'}{'yr': 2013, 'id': 'masky'})) INSERT INTO "pycon" ("id", "yr") VALUES ('masky', 2013)INSERT INTO "pycon" ("id", "yr") VALUES ('masky', 2013) >>> update('pycon',>>> update('pycon', ...... where={'id': 'masky'}where={'id': 'masky'},, ...... set ={'id': 'mosky'}set ={'id': 'mosky'} ... )... ) UPDATE "pycon" SET "id"='mosky' WHERE "id" = 'masky'UPDATE "pycon" SET "id"='mosky' WHERE "id" = 'masky'
  • 24.
    24 SQL Builders (cont.)SQLBuilders (cont.) ● insert(table,insert(table, setset, …), …) ● select(table,select(table, wherewhere, …), …) ● update(table,update(table, wherewhere,, setset, …), …) ● delete(table,delete(table, wherewhere, …), …) ● ......
  • 25.
    25 If you likeit,If you like it,
  • 26.
    26 sudo pip installmosqlsudo pip install mosql
  • 27.
    27 Model of ResultSetModel of Result Set
  • 28.
    28 Model: Configure ConnectionModel:Configure Connection import psycopg2.poolimport psycopg2.pool from mosql.result import Modelfrom mosql.result import Model pool = psycopg2.pool.SimpleConnectionPool(1, 5,pool = psycopg2.pool.SimpleConnectionPool(1, 5, database='mosky')database='mosky') class PostgreSQL(Model):class PostgreSQL(Model): getconn = pool.getconngetconn = pool.getconn putconn = pool.putconnputconn = pool.putconn
  • 29.
    29 Model: Set theName of TableModel: Set the Name of Table class Person(PostgreSQL):class Person(PostgreSQL): table = 'person'table = 'person' >>> Person.select(>>> Person.select({'person_id': 'mosky'}{'person_id': 'mosky'})) {'name': ['Mosky Liu'], 'person_id': ['mosky']}{'name': ['Mosky Liu'], 'person_id': ['mosky']} >>> Person.where(person_id=>>> Person.where(person_id=('andy', 'mosky')('andy', 'mosky'))) {'name': ['Andy Warhol', 'Mosky Liu'], 'person_id':{'name': ['Andy Warhol', 'Mosky Liu'], 'person_id': ['andy', 'mosky']}['andy', 'mosky']}
  • 30.
    30 Model: Make QueriesModel:Make Queries Person.Person.selectselect({'person_id': 'mosky'})({'person_id': 'mosky'}) Person.Person.insertinsert({'person_id': 'tina'})({'person_id': 'tina'}) Person.Person.updateupdate(( where={'person_id': 'mosky'},where={'person_id': 'mosky'}, set ={'name' : 'Yiyu Liu'}set ={'name' : 'Yiyu Liu'} )) Person.Person.deletedelete({'person_id': 'tina'})({'person_id': 'tina'})
  • 31.
    31 Model: Squash ColumnsModel:Squash Columns class Person(PostgreSQL):class Person(PostgreSQL): table = 'person'table = 'person' squashed = set(['person_id', 'name'])squashed = set(['person_id', 'name']) >>> Person.select({'person_id': 'mosky'})>>> Person.select({'person_id': 'mosky'}) {'name':{'name': 'Mosky Liu''Mosky Liu', 'person_id':, 'person_id': 'mosky''mosky'}} >>> Person.where(person_id=('andy', 'mosky'))>>> Person.where(person_id=('andy', 'mosky')) {'name':{'name': 'Andy Warhol''Andy Warhol', 'person_id':, 'person_id': 'andy''andy'}}
  • 32.
    32 Model: ArrangeModel: Arrange classPerson(PostgreSQL):class Person(PostgreSQL): ...... arrange_by = ('person_id', )arrange_by = ('person_id', ) >>> for person in Person.arrange(>>> for person in Person.arrange({'person_id':{'person_id': ('andy', 'mosky')}('andy', 'mosky')}):): ... print person... print person {'name': 'Andy Warhol', 'person_id': 'andy'}{'name': 'Andy Warhol', 'person_id': 'andy'} {'name': 'Mosky Liu', 'person_id': 'mosky'}{'name': 'Mosky Liu', 'person_id': 'mosky'}
  • 33.
    33 Model: Arrange (cont.)Model:Arrange (cont.) >>> for detail in>>> for detail in DetailDetail.arrange({'person_id':.arrange({'person_id': ('mosky', 'andy')}):('mosky', 'andy')}): ... print detail... print detail ...... {'detail_id': [5],{'detail_id': [5], 'key': 'email','key': 'email', 'person_id': 'andy','person_id': 'andy', 'val': ['andy@gmail.com']}'val': ['andy@gmail.com']} ......
  • 34.
    34 Model: FindModel: Find classPerson(PostgreSQL):class Person(PostgreSQL): ...... arrange_by = ('person_id', )arrange_by = ('person_id', ) >>> for person in Person.>>> for person in Person.findfind((person_id=('andy',person_id=('andy', 'mosky')'mosky')):): ... print person... print person {'name': 'Andy Warhol', 'person_id': 'andy'}{'name': 'Andy Warhol', 'person_id': 'andy'} {'name': 'Mosky Liu', 'person_id': 'mosky'}{'name': 'Mosky Liu', 'person_id': 'mosky'}
  • 35.
    35 Model: Identify aRowModel: Identify a Row class Person(PostgreSQL):class Person(PostgreSQL): ...... ident_by = ('person_id', )ident_by = ('person_id', )
  • 36.
    36 Model: ModificationModel: Modification >>>p = Person.where(person_id='mosky')>>> p = Person.where(person_id='mosky') >>>>>> p['name'] = 'Yiyu Liu'p['name'] = 'Yiyu Liu' >>>>>> p.name = 'Yiyu Liu'p.name = 'Yiyu Liu' >>> p.save()>>> p.save() >>> d =>>> d = DetailDetail.where(.where(person_id='mosky', key='email'person_id='mosky', key='email')) >>>>>> p['val'][0] = '<modified email>'p['val'][0] = '<modified email>' >>>>>> p.val[0] = '<modified email>'p.val[0] = '<modified email>' >>> p.save()>>> p.save()
  • 37.
    37 Model: Pop andAppendModel: Pop and Append >>> d = Detail.where(>>> d = Detail.where(person_id='mosky', key='email'person_id='mosky', key='email')) >>>>>> p.pop(-1)p.pop(-1) >>>>>> p.append({'val': '<new mail>'})p.append({'val': '<new mail>'}) >>> p.save()>>> p.save()
  • 38.
    38 Model: Default ClausesModel:Default Clauses class Person(PostgreSQL):class Person(PostgreSQL): ...... clauses = dict(clauses = dict( order_by=('person_id', )order_by=('person_id', ) ))
  • 39.
    39 PerformancePerformance ● AboutAbout 4x4x fasterthan SQLAlchemy.faster than SQLAlchemy. ● Just a little bit slower than pure SQL.Just a little bit slower than pure SQL.
  • 40.
    40 SecuritySecurity ● Security bydefault.Security by default. ● Use escaping technique.Use escaping technique. ● Prevent SQL injection from both valuePrevent SQL injection from both value and identifier.and identifier. ● Passed the tests fromPassed the tests from sqlmapsqlmap at level=5at level=5 and risk=3.and risk=3.
  • 41.
    41 ConclusionConclusion ● Easy-to-LearnEasy-to-Learn ● ConvenientConvenient ●FasterFaster ● SecureSecure ● sudo pip install mosqlsudo pip install mosql ● http://mosql.mosky.tw/http://mosql.mosky.tw/ ● Welcome to fork!Welcome to fork!