Successfully reported this slideshow.

Fantastic DSL in Python

15

Share

Loading in …3
×
1 of 61
1 of 61

Fantastic DSL in Python

15

Share

[PyConJP2012] Problems and solutions about internal DSL design in Python. PythonでDSLを設計する上での問題点と解決策。 Video: http://www.youtube.com/watch?v=l8ptNmtB0G8

[PyConJP2012] Problems and solutions about internal DSL design in Python. PythonでDSLを設計する上での問題点と解決策。 Video: http://www.youtube.com/watch?v=l8ptNmtB0G8

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Fantastic DSL in Python

  1. 1. PyConJP 2012 Fantastic DSL in Python Makoto Kuwata <kwa@kuwata-lab.com> http://www.kuwata-lab.com/ 2012-09-16 (Sun) Video: http://www.youtube.com/watch?v=l8ptNmtB0G8
  2. 2. Agenda •About DSL •Fantastic 'with' statement •Fantastic 'for' statement •Fantastic Decorator •Fantastic Operator
  3. 3. About DSL
  4. 4. What is DSL (Domain Specific Language?) DSL Google Search
  5. 5. Internal DSL v.s. External DSL •Internal DSL (内部DSL) •Written inside an existing host language 既存の言語を使って、その言語の文法を使って書かれたDSL •No need to implement parser パーサを書く必要なし •Some lang are not good at internal DSL 言語によっては苦手な場合がある •External DSL (外部DSL) Python? •Original, indendent syntax 独自の文法 •Need to implement parser パーサを実装する必要あり •All languages have same power about external DSL 言語による得手不得手は、ほぼない
  6. 6. Fantastic 'with' Statement
  7. 7. What is 'with' statement? •Enforces finishing ## Python 2.4 process f = open('file') 終了処理を強制する機能 try: •ex: close opened file text = f.read() certainly finally: 例:ファイルを必ず閉じる f.close() ## Python 2.5 or later with open('file') as f: text = f.read()
  8. 8. How to use 'with' statement in DSL? •Appends pre- Append pre-process process and post- 前処理を追加 process around } block x=1 ブロックに前処理と後処理をくっ y=2 code block つける z=3 Append post-process 後処理を追加
  9. 9. Case Study: Change Directory (Kook) ## before ## after chdir('build') with chdir('build'): cp('../file', 'file') cp('../file', 'file') system('cmd...') system('cmd...') chdir('..')
  10. 10. Case Study: Start/Stop time (Benchmarker) ## before ## after t1 = time.time() bm = Benchmarker() for i in xrange(N): with bm('func'): func() for i in xrange(N): t2 = time.time() func() print('func: %s' % (t2-t1))
  11. 11. Case Study: Form Builder ## before <form action="/" method="POST"> <input type="hidden" name="_csrf" value="${request.csrf()}" /> <input type="text" ... /> <input type="submit" ... /> </form>
  12. 12. Case Study: Form Builder ## after % with FormBuilder(request, "/") as fb: <input type="text" ... /> <input type="submit" ... /> % endwith
  13. 13. Pitfall: Variable Scope with test("arithmetic operation"): with test("1+1 should be 1"): x = 1+1 assert x == 2 with test("1-1 should be 0"): x = 1-1 Identical variable because assert x == 0 scope is not independented スコープが独立してないので変数も共有される
  14. 14. Pitfall: Cannot Skip Block! <div> % with cache("key", 60*60*1): <ul> % for item in get_items(): <li>${item}</li> % endfor You may want to skip block </ul> when cache is not expired, % endwith but Python doesn't allow it! </div> キャッシュが生きている間はブロックを スキップしたいんだけど、実はできない
  15. 15. Fantastic 'for' Statement
  16. 16. What is 'for' statement? •Iterates code block ## repeat 100 times ブロックを繰り返す機能 for i in xrange(100): •ex: repeat block for N print("i=%s" % i) times 例:ブロックをN回実行 ## read lines from file with open('file') as f: •ex: read each line for line in f: from opened file print(line) 例:オープンしたファイルから 1行ずつ読み込む
  17. 17. How to use 'for' statement in DSL? #1 •Appends pre- Append pre-process process and post- 前処理を追加 process around } iteration block x=1 繰り返し用のブロックに y=2 iteration block 前処理と後処理をくっつける z=3 Append post-process 後処理を追加
  18. 18. How to use 'for' statement in DSL? #1 •Appends pre- def gfunc(): process and post- process around iteration block .... .... }pre-process 繰り返し用のブロックに for i in xrange(n): } exec block 前処理と後処理をくっつける yield .... .... } post-process
  19. 19. Case Study: Start/Stop time (Benchmarker) ## before ## after t1 = time.time() bm = Benchmarker( for i in xrange(N): repeat=N) func1() for _ in bm('func'): t2 = time.time() func1() print('func: %s' % (t2-t1))
  20. 20. How to use 'for' statement in DSL? #2 •Iterate code block def gfunc(): only once ブロックを1回だけ繰り返す .... .... } pre-process yield } exec block (only once!) •Emulates 'with' stmt .... by 'for' stmt .... } post-process with文をfor文でエミュレート
  21. 21. Case Study: Change Directory (Kook) ## before ## after chdir('tmp') for _ in chdir('tmp'): cp('../file', 'file') cp('../file', 'file') system('cmd...') system('cmd...') chdir('..') Use 'for' as alternative of 'with' in Python 2.4 Python2.4では 'with' が使えない ので、代わりとして 'for' を使う
  22. 22. How to use 'for' statement in DSL? #3 •Iterate block in 0 or def gfunc(): 1 time ブロックを0回または1回だけ 繰り返す .... .... }pre-process if condition: } •Emulates 'if' stmt by 'for' stmt to skip yield exec block block according to .... condition if文をfor文でエミュレートし、条 .... } post-process 件によってブロックをスキップ
  23. 23. Case Study: Fragment Cache (Tenjin) <div> % for _ in cache("key", 60*60*1): <ul> % for item in get_items(): <li>${item}</li> % endfor You can skip block when </ul> cache is not expired! Wow! % endfor with文と違い、キャッシュが生きている </div> ときにブロックをスキップできる! ステキ!
  24. 24. Case Study: Java and JSP <div> <% for (int _: cache("key",60)) { %> <ul> <% for (Item item: getItems()) { %> <li>${item}</li> <% } %> </ul> <% } %> </div>
  25. 25. Fantastic Decorator
  26. 26. What is Decorator? •Higher-order function class Hello(object): to manipulate declaring function or @classmethod class def fn(cls): 宣言中の関数やクラスを操作する print("Hello") 高階関数 •ex: @classmethod(), ## above is same as @property() def fn(cls): 例:@classmethod(), print("Hello") @property fn = classmethod(fn)
  27. 27. How to use Decorator in DSL? #1 •Mark function with def deco(func): or without args func._mark = True 関数に目印をつける return func def deco_with(arg): def deco(func): func._mark = arg return func deco
  28. 28. Case Study: Marking and Options (Kook) @recipe def test(c): system("python -m oktest test") @recipe("*.o") @ingreds("$(1).c", if_exist="$(1).h") def file_o(c): system(c%'gcc -o $(product) $(ingred)')
  29. 29. How to use Decorator in DSL? #2 •Syntax sugar to def fn(arg): pass function arg print(arg) 引数として関数を渡すときの シンタックスシュガー func(x, y, fn) @func(x, y) def fn(arg): print(arg)
  30. 30. Case Study: Event Handler ## before ## after def _(): @button.onclick do() def _(): some() do() thing() some() button.onclick(_) thing() More readable because @onclick appears first ブロックより先に@onclickが きているのでより読みやすい
  31. 31. Case Study: Database Transaction ## before ## after def _(): @transaction do() def _(): some() do() thing() some() transaction(_) thing() More readable because @transaction appears first ブロックより先に@transactionが 現れるので、より読みやすい
  32. 32. Case Study: unittest ## before ## after def _(): @assertRaise2(Err) do() def _(): some() do() thing() some() assertRaise(Err, _) thing() More readable because assertRailse() appears first assertRaises2()のほうがブロックより 先に現れるので、より読みやすい
  33. 33. How to use Decorator in DSL? #3 •Alternative syntax of with func(arg) as x: with- or for- .... statement with文やfor文の代替 •Independent scope! 独立したスコープ! @func(arg) •Nested scope! def _(x): 入れ子になったスコープ! ....
  34. 34. Case Study: open() ## before ## after with open('file', @open('file', 'w') 'w') as f: def _(f): f.write("....") f.write("....")
  35. 35. How to use Decorator in DSL? #4 •Alternative syntax of class Foo(object): class definition def meth1(self): class定義の代替 .... •More natural representation of inheritance structure @classdef 継承構造のより自然な表現 def _(): •More natural scope @methoddef より自然なスコープ def _(): ....
  36. 36. Class Inheritance class Parent(object): .... class Child(Parent): .... Inheritance structure class Baby(Child): is not represented .... visually 継承構造が視覚的には 表現されていない
  37. 37. Nested Class Definition class Parent(object): class Child(Parent): Represents class Baby(Child): inheritance structure .... visually, but not possible in Python! こう書けば継承構造が視 覚的に表現できるけど、 Pythonではこう書けない!
  38. 38. Nested Structure by with-statement with define('Parent'): with define('Child'): with define('Baby'): Represents structure .... visually, but scopes are not independent 構造は視覚的に表現でき ているが、スコープが独 立していない (共有される)
  39. 39. Nested Structure with Decorator @defclass def Parent(): @defclass def Child(): Represents structure visually, and scopes @defclass are not shared def Baby(): 継承構造を視覚的に表現 できるし、スコープも共 .... 有されない (独立している)
  40. 40. Strange Scope Rule (for beginner) in Class Definition 1: class Outer(object): 2: VAR = 1 3: 4: class Inner(object): 5: # error 6: print(VAR) Why I can't access 7: # error, too to outer variable? 8: print(Outer.VAR) どうして外側の変数に アクセスできないの?
  41. 41. Natural Scope Rule (for beginner) with Function 1: def outer(): 2: VAR = 1 3: 4: def middle(): 5: print(VAR) # OK More natual 6: scope rule 7: def inner(): (especially for beginner) 8: print(VAR) # OK より自然なスコープルール (特に初心者にとっては)
  42. 42. Case Study: RSpec describe("Class1") do describe("#method1()") do it("should return None") do Class1.new.method1().should be_nil end end end
  43. 43. Case Study: RSpec class Class1Test(TestCase): class method1Test(Class1Test): def test_should_return_None(self): assert Class1().method1() is None
  44. 44. Case Study: RSpec @describe("Class1") def _(): @describe("#method1()") def _(): @it("should return None") def _(self): assert Class1().method1() is None
  45. 45. Case Study: RSpec @describe("Class1") def _(): Variables in outer are... def this(): 外側の関数で定義された変数に... return Class1() @describe("#method1()") def _(): @it("should return None") def _(self): assert this().method1() is None Accessable from inner! 内側の関数から自然にアクセスできる!
  46. 46. Fantastic Operator
  47. 47. Operator Override 1: class Node(object): 2: def __init__(self, op, left, right): 3: self.op, self.left, self.right = 4: Overrides '==' operator op, left, right 5: 6: def __eq__(self, other): 7: return Node('==', self, other) 8: def __ne__(self, other): 9: return Node('!=', self, other) Overrides '!=' operator
  48. 48. Expression Evaluation v.s. AST Expression Evaluation x+1 2 AST (Abstract Syntax Tree) + x+1 x 1 参考: http://www.slideshare.net/kwatch/dsl-presentation
  49. 49. Case Study: Modern O/R Mapper (SQLAlchemy) Python SQL == x == 1 x=1 x 1 == x == None x is null x None
  50. 50. Case Study: Assertion (Oktest) Assertion When Failed ## Python AssertionError: assert x == y (no message) ## Nose AssertionError: ok_(x == y) (no message) ## Oktest AssertionError: ok (x) == y 1 == 2 : failed Shows actual & expected values 失敗時に、実際値と期待値を表示してくれる
  51. 51. Case Study: Assertion (Oktest) Oktest Oktest() returns AssertionObject ok() はAssertionObject のインスタンスを返す >>> ok (1+1) <oktest.AssertionObject object> >>> ok (1+1).__eq__(1) Traceback (most recent call last): ... AssertionError: 2 == 1 : failed. Overrides '==' operator '==' 演算子をオーバーライドしている 参考: http://goo.gl/C6Eun
  52. 52. Case Study: String Interpolation (Kook) CMD = 'rst2html -i utf-8' @recipe('*.html') @ingreds('$(1).txt') Low readability def file_html(c): 可読性が低い system("%s %s > %s" % (CMD, c.ingreds[0], c.product)) # or cmd,prod,ingred = CC,c.product,c.ingreds[0] system("%(cmd)s %(ingred)s > %(prod)s" % locals()) Name only, No expression 名前のみで式が使えない
  53. 53. Case Study: String Interpolation (Kook) http://www.kuwata-lab.com/kook/pykook-users-guide.html#cookbook-prod CMD = 'rst2html -i utf-8' @recipe('hello') @ingreds('hello.c') def compile(c): prod, ingreds = c.product, c.ingreds system(c%"$(CMD) $(ingreds[0]) > $(prod)") Operator Indexing override Both global and available 演算子オーバーライド local var available 添字も利用可能 グローバル変数とローカル 変数の両方が利用可能
  54. 54. Case Study: Method Chain Finishing Problem AppointmentBuilder() .From(1300) .To(1400) .For("Dental") .done() Method chain requires end of chain. メソッドチェーンでは、チェーンの終わりを明示する必要がある
  55. 55. Case Study: Method Chain Finishing Problem - AppointmentBuilder() .From(1300) .To(1400) .For("Dental") Unary operator works at end of chain! 単項演算子はチェーンの終わりに実行される! Very similar to unordered list or bullet! 見た目が箇条書きそっくり!
  56. 56. Reflected operator class Foo(object): def __add__(self, other): ... foo + 1 def __radd__(self, other): ... 1 + foo
  57. 57. Reflected operator def __add__(self, other): ... int.__add__ = __add__ 1 + Foo() Not possible ;< Pythonではできない class Foo(object): def __radd__(self, other): ... No problem :) 1 + Foo() 問題なし
  58. 58. Case Study: Should DSL http://www.should-dsl.info/ def test1(self): self.player |should| have(11).cards def test2(self): self.player.name |should| equal_to('John Doe') Instead of adding method to any object, use bit operator which has low priority ビットOR演算子の優先順位が低いことを利用して、任意のオブジェクトに メソッドを追加できないというPythonの欠点を克服した、絶妙なDSL
  59. 59. One more thing...
  60. 60. DSL設計者に贈る言葉 「キモイ」は褒め言葉! 古参のイチャモンは新規性の証し! オレが書きたいように書けないのはPythonが悪い!
  61. 61. おしまい

×