Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fantastic DSL in Python


Published on

[PyConJP2012] Problems and solutions about internal DSL design in Python. PythonでDSLを設計する上での問題点と解決策。 Video:

Published in: Technology
  • Be the first to comment

Fantastic DSL in Python

  1. 1. PyConJP 2012Fantastic DSLin PythonMakoto Kuwata <> (Sun)Video:
  2. 2. Agenda•About DSL•Fantastic with statement•Fantastic for statement•Fantastic Decorator•Fantastic Operator
  3. 3. About DSL
  4. 4. What is DSL (Domain Specific Language?) DSL Google Search
  5. 5. Internal DSL v.s. External DSL•Internal DSL (内部DSL) •Written inside an existing host language 既存の言語を使って、その言語の文法を使って書かれたDSL •No need to implement parser パーサを書く必要なし •Some lang are not good at internal DSL 言語によっては苦手な場合がある•External DSL (外部DSL) Python? •Original, indendent syntax 独自の文法 •Need to implement parser パーサを実装する必要あり •All languages have same power about external DSL 言語による得手不得手は、ほぼない
  6. 6. Fantastic with Statement
  7. 7. What is with statement?•Enforces finishing ## Python 2.4 process f = open(file) 終了処理を強制する機能 try: •ex: close opened file text = certainly finally: 例:ファイルを必ず閉じる f.close() ## Python 2.5 or later with open(file) as f: text =
  8. 8. How to use with statement in DSL?•Appends pre- Append pre-process process and post- 前処理を追加 process around } block x=1 ブロックに前処理と後処理をくっ y=2 code block つける z=3 Append post-process 後処理を追加
  9. 9. Case Study: Change Directory (Kook)## before ## afterchdir(build) with chdir(build):cp(../file, file) cp(../file, file)system(cmd...) system(cmd...)chdir(..)
  10. 10. Case Study: Start/Stop time (Benchmarker)## before ## aftert1 = time.time() bm = Benchmarker()for i in xrange(N): with bm(func): func() for i in xrange(N):t2 = time.time() func()print(func: %s % (t2-t1))
  11. 11. Case Study: Form Builder ## before <form action="/" method="POST"> <input type="hidden" name="_csrf" value="${request.csrf()}" /> <input type="text" ... /> <input type="submit" ... /> </form>
  12. 12. Case Study: Form Builder ## after % with FormBuilder(request, "/") as fb: <input type="text" ... /> <input type="submit" ... /> % endwith
  13. 13. Pitfall: Variable Scope with test("arithmetic operation"): with test("1+1 should be 1"): x = 1+1 assert x == 2 with test("1-1 should be 0"): x = 1-1 Identical variable because assert x == 0 scope is not independented スコープが独立してないので変数も共有される
  14. 14. Pitfall: Cannot Skip Block! <div> % with cache("key", 60*60*1): <ul> % for item in get_items(): <li>${item}</li> % endfor You may want to skip block </ul> when cache is not expired, % endwith but Python doesnt allow it! </div> キャッシュが生きている間はブロックを スキップしたいんだけど、実はできない
  15. 15. Fantastic for Statement
  16. 16. What is for statement?•Iterates code block ## repeat 100 times ブロックを繰り返す機能 for i in xrange(100): •ex: repeat block for N print("i=%s" % i) times 例:ブロックをN回実行 ## read lines from file with open(file) as f: •ex: read each line for line in f: from opened file print(line) 例:オープンしたファイルから 1行ずつ読み込む
  17. 17. How to use for statement in DSL? #1•Appends pre- Append pre-process process and post- 前処理を追加 process around } iteration block x=1 繰り返し用のブロックに y=2 iteration block 前処理と後処理をくっつける z=3 Append post-process 後処理を追加
  18. 18. How to use for statement in DSL? #1•Appends pre- def gfunc(): process and post- process around iteration block .... .... }pre-process 繰り返し用のブロックに for i in xrange(n): } exec block 前処理と後処理をくっつける yield .... .... } post-process
  19. 19. Case Study: Start/Stop time (Benchmarker)## before ## aftert1 = time.time() bm = Benchmarker(for i in xrange(N): repeat=N) func1() for _ in bm(func):t2 = time.time() func1()print(func: %s % (t2-t1))
  20. 20. How to use for statement in DSL? #2•Iterate code block def gfunc(): only once ブロックを1回だけ繰り返す .... .... } pre-process yield } exec block (only once!)•Emulates with stmt .... by for stmt .... } post-process with文をfor文でエミュレート
  21. 21. Case Study: Change Directory (Kook)## before ## afterchdir(tmp) for _ in chdir(tmp):cp(../file, file) cp(../file, file)system(cmd...) system(cmd...)chdir(..) Use for as alternative of with in Python 2.4 Python2.4では with が使えない ので、代わりとして for を使う
  22. 22. How to use for statement in DSL? #3•Iterate block in 0 or def gfunc(): 1 time ブロックを0回または1回だけ 繰り返す .... .... }pre-process if condition: }•Emulates if stmt by for stmt to skip yield exec block block according to .... condition if文をfor文でエミュレートし、条 .... } post-process 件によってブロックをスキップ
  23. 23. Case Study: Fragment Cache (Tenjin) <div> % for _ in cache("key", 60*60*1): <ul> % for item in get_items(): <li>${item}</li> % endfor You can skip block when </ul> cache is not expired! Wow! % endfor with文と違い、キャッシュが生きている </div> ときにブロックをスキップできる! ステキ!
  24. 24. Case Study: Java and JSP <div> <% for (int _: cache("key",60)) { %> <ul> <% for (Item item: getItems()) { %> <li>${item}</li> <% } %> </ul> <% } %> </div>
  25. 25. Fantastic Decorator
  26. 26. What is Decorator?•Higher-order function class Hello(object): to manipulate declaring function or @classmethod class def fn(cls): 宣言中の関数やクラスを操作する print("Hello") 高階関数 •ex: @classmethod(), ## above is same as @property() def fn(cls): 例:@classmethod(), print("Hello") @property fn = classmethod(fn)
  27. 27. How to use Decorator in DSL? #1•Mark function with def deco(func): or without args func._mark = True 関数に目印をつける return func def deco_with(arg): def deco(func): func._mark = arg return func deco
  28. 28. Case Study: Marking and Options (Kook)@recipedef test(c): system("python -m oktest test")@recipe("*.o")@ingreds("$(1).c", if_exist="$(1).h")def file_o(c): system(c%gcc -o $(product) $(ingred))
  29. 29. How to use Decorator in DSL? #2•Syntax sugar to def fn(arg): pass function arg print(arg) 引数として関数を渡すときの シンタックスシュガー func(x, y, fn) @func(x, y) def fn(arg): print(arg)
  30. 30. Case Study: Event Handler## before ## afterdef _(): @button.onclick do() def _(): some() do() thing() some()button.onclick(_) thing() More readable because @onclick appears first ブロックより先に@onclickが きているのでより読みやすい
  31. 31. Case Study: Database Transaction## before ## afterdef _(): @transaction do() def _(): some() do() thing() some()transaction(_) thing() More readable because @transaction appears first ブロックより先に@transactionが 現れるので、より読みやすい
  32. 32. Case Study: unittest## before ## afterdef _(): @assertRaise2(Err) do() def _(): some() do() thing() some()assertRaise(Err, _) thing() More readable because assertRailse() appears first assertRaises2()のほうがブロックより 先に現れるので、より読みやすい
  33. 33. How to use Decorator in DSL? #3•Alternative syntax of with func(arg) as x: with- or for- .... statement with文やfor文の代替 •Independent scope! 独立したスコープ! @func(arg) •Nested scope! def _(x): 入れ子になったスコープ! ....
  34. 34. Case Study: open()## before ## afterwith open(file, @open(file, w)w) as f: def _(f): f.write("....") f.write("....")
  35. 35. How to use Decorator in DSL? #4•Alternative syntax of class Foo(object): class definition def meth1(self): class定義の代替 .... •More natural representation of inheritance structure @classdef 継承構造のより自然な表現 def _(): •More natural scope @methoddef より自然なスコープ def _(): ....
  36. 36. Class Inheritanceclass Parent(object): ....class Child(Parent): .... Inheritance structureclass Baby(Child): is not represented .... visually 継承構造が視覚的には 表現されていない
  37. 37. Nested Class Definitionclass Parent(object): class Child(Parent): Represents class Baby(Child): inheritance structure .... visually, but not possible in Python! こう書けば継承構造が視 覚的に表現できるけど、 Pythonではこう書けない!
  38. 38. Nested Structure by with-statementwith define(Parent): with define(Child): with define(Baby): Represents structure .... visually, but scopes are not independent 構造は視覚的に表現でき ているが、スコープが独 立していない (共有される)
  39. 39. Nested Structure with Decorator@defclassdef Parent(): @defclass def Child(): Represents structure visually, and scopes @defclass are not shared def Baby(): 継承構造を視覚的に表現 できるし、スコープも共 .... 有されない (独立している)
  40. 40. Strange Scope Rule (for beginner) in Class Definition1: class Outer(object):2: VAR = 13:4: class Inner(object):5: # error6: print(VAR) Why I cant access7: # error, too to outer variable?8: print(Outer.VAR) どうして外側の変数に アクセスできないの?
  41. 41. Natural Scope Rule (for beginner) with Function1: def outer():2: VAR = 13:4: def middle():5: print(VAR) # OK More natual6: scope rule7: def inner(): (especially for beginner)8: print(VAR) # OK より自然なスコープルール (特に初心者にとっては)
  42. 42. Case Study: RSpecdescribe("Class1") do describe("#method1()") do it("should return None") do be_nil end endend
  43. 43. Case Study: RSpecclass Class1Test(TestCase): class method1Test(Class1Test): def test_should_return_None(self): assert Class1().method1() is None
  44. 44. Case Study: RSpec@describe("Class1")def _(): @describe("#method1()") def _(): @it("should return None") def _(self): assert Class1().method1() is None
  45. 45. Case Study: RSpec@describe("Class1")def _(): Variables in outer are... def this(): 外側の関数で定義された変数に... return Class1() @describe("#method1()") def _(): @it("should return None") def _(self): assert this().method1() is None Accessable from inner! 内側の関数から自然にアクセスできる!
  46. 46. Fantastic Operator
  47. 47. Operator Override 1: class Node(object): 2: def __init__(self, op, left, right): 3: self.op, self.left, self.right = 4: Overrides == operator op, left, right 5: 6: def __eq__(self, other): 7: return Node(==, self, other) 8: def __ne__(self, other): 9: return Node(!=, self, other) Overrides != operator
  48. 48. Expression Evaluation v.s. ASTExpression Evaluation x+1 2AST (Abstract Syntax Tree) + x+1 x 1参考:
  49. 49. Case Study: Modern O/R Mapper (SQLAlchemy) Python SQL == x == 1 x=1 x 1 ==x == None x is null x None
  50. 50. Case Study: Assertion (Oktest)Assertion When Failed ## Python AssertionError: assert x == y (no message) ## Nose AssertionError: ok_(x == y) (no message) ## Oktest AssertionError: ok (x) == y 1 == 2 : failed Shows actual & expected values 失敗時に、実際値と期待値を表示してくれる
  51. 51. Case Study: Assertion (Oktest)Oktest Oktest() returns AssertionObject ok() はAssertionObject のインスタンスを返す >>> ok (1+1) <oktest.AssertionObject object> >>> ok (1+1).__eq__(1) Traceback (most recent call last): ... AssertionError: 2 == 1 : failed. Overrides == operator == 演算子をオーバーライドしている参考:
  52. 52. Case Study: String Interpolation (Kook)CMD = rst2html -i utf-8@recipe(*.html)@ingreds($(1).txt) Low readabilitydef file_html(c): 可読性が低い system("%s %s > %s" % (CMD, c.ingreds[0], c.product)) # or cmd,prod,ingred = CC,c.product,c.ingreds[0] system("%(cmd)s %(ingred)s > %(prod)s" % locals()) Name only, No expression 名前のみで式が使えない
  53. 53. Case Study: String Interpolation (Kook) CMD = rst2html -i utf-8 @recipe(hello) @ingreds(hello.c) def compile(c): prod, ingreds = c.product, c.ingreds system(c%"$(CMD) $(ingreds[0]) > $(prod)") Operator Indexing override Both global and available演算子オーバーライド local var available 添字も利用可能 グローバル変数とローカル 変数の両方が利用可能
  54. 54. Case Study: Method Chain Finishing Problem AppointmentBuilder() .From(1300) .To(1400) .For("Dental") .done() Method chain requires end of chain. メソッドチェーンでは、チェーンの終わりを明示する必要がある
  55. 55. Case Study: Method Chain Finishing Problem - AppointmentBuilder() .From(1300) .To(1400) .For("Dental") Unary operator works at end of chain! 単項演算子はチェーンの終わりに実行される! Very similar to unordered list or bullet! 見た目が箇条書きそっくり!
  56. 56. Reflected operator class Foo(object): def __add__(self, other): ... foo + 1 def __radd__(self, other): ... 1 + foo
  57. 57. Reflected operator def __add__(self, other): ... int.__add__ = __add__ 1 + Foo() Not possible ;< Pythonではできない class Foo(object): def __radd__(self, other): ... No problem :) 1 + Foo() 問題なし
  58. 58. Case Study: Should DSL test1(self): self.player |should| have(11).cardsdef test2(self): |should| equal_to(John Doe) Instead of adding method to any object, use bit operator which has low priority ビットOR演算子の優先順位が低いことを利用して、任意のオブジェクトに メソッドを追加できないというPythonの欠点を克服した、絶妙なDSL
  59. 59. One more thing...
  60. 60. DSL設計者に贈る言葉「キモイ」は褒め言葉!古参のイチャモンは新規性の証し!オレが書きたいように書けないのはPythonが悪い!
  61. 61. おしまい