MYPY: A PYTHON VARIANT
WITH SEAMLESS DYNAMIC
  AND STATIC TYPING
                 Jukka Lehtosalo
 University of Cambridge Computer Laboratory
SPEAKER BIO
2000-2006 Software engineer (QPR Software and Kielikone)

2007-2009 Development manager (Kielikone)

2009→ PhD research (University of Cambridge)

Projects spanning web apps, Windows apps, mobile
apps,distributed apps, document databases, information retrieval,
pattern recognition, natural language processing, data mining,
garbage collectors, compilers, localization tools, automated testing
tools etc.
mypy is an experimental
    Python variant
    with seamless
dynamic and static typing
Mostly, but not 100%
 compatible with
       Python

           mypy is an experimental
               Python variant
               with seamless
           dynamic and static typing
Research project,
              in development

mypy is an experimental
    Python variant
    with seamless
dynamic and static typing
mypy is an experimental
              Python variant
              with seamless
          dynamic and static typing

  Freely mix static and dynamic (duck) types
Combine the benefits of static and dynamic typing
LANGUAGE GAP

    Python         Java, C++, C, C#




 Scripting, web
development, UI,    Heavy lifting
  prototyping
VISION

     mypy
                                 mypy
 (dynamic typing)
                             (static typing)
    Python
                                    Java, C++, C, C#


 Scripting, web
development, UI,             Heavy lifting
  prototyping
WHAT DOES IT LOOK LIKE?
mypy, dynamic typing    mypy, static typing

def fib(n):             void fib(int n):
    a, b = 0, 1             a, b = 0, 1
    while a < n:            while a < n:
        print(a)                print(a)
        a, b = b, a+b           a, b = b, a+b
WHAT DOES IT LOOK LIKE?
 mypy, dynamic typing    mypy, static typing

 def fib(n):             void fib(int n):
     a, b = 0, 1             a, b = 0, 1
     while a < n:            while a < n:
         print(a)                print(a)
         a, b = b, a+b           a, b = b, a+b

 Dynamically
typed function
WHAT DOES IT LOOK LIKE?
mypy, dynamic typing           mypy, static typing

def fib(n):                    void fib(int n):
    a, b = 0, 1                    a, b = 0, 1
    while a < n:                   while a < n:
        print(a)            Static     print(a)
        a, b = b, a+b                  a, b = b, a+b
                        type signature
WHAT DOES IT LOOK LIKE?
mypy, dynamic typing       mypy, static typing

def fib(n):                void fib(int n):
    a, b = 0, 1                a, b = 0, 1
    while a < n:               while a < n:
        print(a)                   print(a)
        a, b = b, a+b              a, b = b, a+b

                        Type inference
DYNAMIC VS STATIC TYPING
mypy, dynamic typing

def err():
    return 1 + ‘x’     # no error (not called)
DYNAMIC VS STATIC TYPING
mypy, dynamic typing

def err():
    return 1 + ‘x’     # runtime error (when called)
err()
DYNAMIC VS STATIC TYPING
mypy, dynamic typing

def err():
    return 1 + ‘x’     # runtime error (when called)
err()


mypy, static typing

int err():
    return 1 + ‘x’     # compile (static) error
TYPES

int, str, bool, MyClass        any
     simple type                     dynamic (duck) type

list<int>, dict<str, int>      func<str, int>
     generic type                  function type

tuple<int, str>
    tuple type
EXAMPLE SCRIPT
import sys, re

if not sys.argv[1:]:
    raise RuntimeError('Usage: wordfreq FILE')

dict<str, int> d = {}

s = open(sys.argv[1]).read()
for word in re.sub('W', ' ', s).split():
    d[word] = d.get(word, 0) + 1

# Use list comprehension
l = [(freq, word) for word, freq in d.items()]

for freq, word in sorted(l):
    print('%-6d %s' % (freq, word))
EXAMPLE SCRIPT
import sys, re

if not sys.argv[1:]:
    raise RuntimeError('Usage: wordfreq FILE')

dict<str, int> d = {}

s = open(sys.argv[1]).read()
for word in re.sub('W', ' ', s).split():
    d[word] = d.get(word, 0) + 1

# Use list comprehension
l = [(freq, word) for word, freq in d.items()]

for freq, word in sorted(l):
    print('%-6d %s' % (freq, word))
EXAMPLE CLASS
             dynamic typing
class BankAccount:
    def __init__(self, initial_balance=0):
        self.balance = initial_balance
    def deposit(self, amount):
        self.balance += amount
    def withdraw(self, amount):
        self.balance -= amount
    def overdrawn(self):
        return self.balance < 0

my_account = BankAccount(15)
my_account.withdraw(5)
print(my_account.balance)      (source: Python Wiki)
EXAMPLE CLASS
               static typing
class BankAccount:
    void __init__(self, int initial_balance=0):
        self.balance = initial_balance
    void deposit(self, int amount):
        self.balance += amount
    void withdraw(self, int amount):
        self.balance -= amount
    bool overdrawn(self):
        return self.balance < 0

my_account = BankAccount(15)
my_account.withdraw(5)
print(my_account.balance)
EXAMPLE CLASS
               static typing
class BankAccount:
    void __init__(self, int initial_balance=0):
        self.balance = initial_balance
    void deposit(self, int amount):
        self.balance += amount
    void withdraw(self, int amount):
        self.balance -= amount
    bool overdrawn(self):
        return self.balance < 0

my_account = BankAccount(15)
my_account.withdraw(5)
print(my_account.balance)
MYPY LIBRARY SUPPORT

                Mypy program
 Mypy VM      Native mypy libraries

Translation
                object mapping
   layer

CPython VM      Python libraries
NO GIL

CPython uses Global Interpreter Lock
only one thread can usually run at a time
NO GIL

    CPython uses Global Interpreter Lock
    only one thread can usually run at a time

     Not ok in the multicore era!

Mypy will have a      new ✝, custom VM         without GIL.

    ✝   Based on the Alore VM (http://www.alorelang.org/)
WHY STATIC TYPING?
PERFORMANCE
                                          mypy
speed




                                      static typing,
                                     ahead-of-time
                                        compiler

                                          PyPy
                                      JIT compiler

                                       CPython
                                      interpreter
                                t
        (conceptual, not in scale)
TYPE CHECKING
Static type checker pinpoints errors in your
program, before running it

some_function(foo[n], self.bar(x))
                               ^
StaticError: Argument has invalid type list<str>,
  list<int> expected


➠ Less debugging
➠ Easier maintenance and refactoring
TOOL SUPPORT

Static type information makes many tools more powerful:

 • IDEs   with reliable and precise code completion

 • Refactoring    tools

 • Static   analysis and linting tools


 ➠ Better productivity and quality
SEAMLESS DYNAMIC AND
    STATIC TYPING
        Use the best tool for the job

 Use dynamic typing when it makes sense
  Use static typing when it makes sense

 As real programs are not uniform: use both
Static typing may be a win
 • if efficiency is important
 • if your project is large or complex
 • if you need to maintain code for several years
Even when the above are true, dynamic typing is useful
• for tests
• for the user interface
• for glue code
• for extensions, scripts and plugins
Dynamically typed
      program



Statically typed libraries
hot spot

             Sprinkle static types
           for better performance
Statically typed interfaces


   Dynamically typed
   implementations
Evolve programs from dynamic to static typing




v 0.5         v 1.0                 v 2.0
IMPLEMENTATION PLAN
                All dates tentative!
Phase 1(2012)
Mypy type checker + mypy-to-Python translator
Phase 2 (2013)
Compiler to native code (C/LLVM back end), new VM
Phase N (?)
More Python features
IDE
JVM back end?
Android port? (etc.)
MYPY VS OTHER PROJECTS
CYTHON
•   Cython is a Python variant that   cpdef int sum3d(int[:, :, :] a):
    compiles to Python C modules          cdef int total = 0
                                          I = a.shape[0]
                                          J = a.shape[1]
•   Cython can speed up tight             K = a.shape[2]
    loops by a lot (100 x or more)        for i in range(I):
    by using type declarations                for j in range(J):
                                                  for k in range(K):
                                                       total += a[i, j, k]
•   Cython programs can also call         return total
    C functions directly
                                           (adapted from Cython Users Guide)
•   Similar to Pyrex
MYPY VS CYTHON
   Mypy (planned)                Cython

Powerful type system       Simple type system

                       Speedups mainly to low-level
General code speedup
                                 code

  New, efficient VM            CPython VM

      No GIL                GIL still a problem
SHED SKIN


• Compiler   for a statically typed subset of Python

• Large   speedups to some programs (100x or more)

• Code    looks like Python (explicit type declarations not needed)

 • Whole-program      type inference
MYPY VS SHED SKIN
      Mypy (planned)                    Shed Skin

Large set of Python features   Restricted Python feature set

 Dynamic and static typing           Static typing only

  Modular, (relatively) fast    Very slow compilation for
        compiles                     large programs

  Python lib access layer         Not many Python libs
MYPY VS RPYTHON


• RPython   is a Python subset used by PyPy to implement the JIT
 compiler

• RPython   uses whole-program type inference (like Shed Skin)

• Differences   vs mypy: similar to mypy vs Shed Skin
CONCLUSION
       Tell me what you think!

All help and feedback is very welcome!

              Web site:
http://www.mypy-lang.org
           Thank you!
ANDROID: OVERVIEW

• Over   500 M Android devices activated

• Android   native APIs mostly based on Java (Dalvik)

• Performance   important for apps

 • Cheap     devices with low-end CPUs

 • UI   responsiveness key to good user experience

 • Battery   conservation
MYPY ON ANDROID?

   Android Wishlist
   1. Python syntax + libs               Can’t have all of these
   2. Good performance                         right now
  3. Easy native API access



• Mypy    is a potential solution (by compiling to Dalvik)

• But   there’s a lot of work to do...

Mypy pycon-fi-2012

  • 1.
    MYPY: A PYTHONVARIANT WITH SEAMLESS DYNAMIC AND STATIC TYPING Jukka Lehtosalo University of Cambridge Computer Laboratory
  • 2.
    SPEAKER BIO 2000-2006 Softwareengineer (QPR Software and Kielikone) 2007-2009 Development manager (Kielikone) 2009→ PhD research (University of Cambridge) Projects spanning web apps, Windows apps, mobile apps,distributed apps, document databases, information retrieval, pattern recognition, natural language processing, data mining, garbage collectors, compilers, localization tools, automated testing tools etc.
  • 3.
    mypy is anexperimental Python variant with seamless dynamic and static typing
  • 4.
    Mostly, but not100% compatible with Python mypy is an experimental Python variant with seamless dynamic and static typing
  • 5.
    Research project, in development mypy is an experimental Python variant with seamless dynamic and static typing
  • 6.
    mypy is anexperimental Python variant with seamless dynamic and static typing Freely mix static and dynamic (duck) types Combine the benefits of static and dynamic typing
  • 7.
    LANGUAGE GAP Python Java, C++, C, C# Scripting, web development, UI, Heavy lifting prototyping
  • 8.
    VISION mypy mypy (dynamic typing) (static typing) Python Java, C++, C, C# Scripting, web development, UI, Heavy lifting prototyping
  • 9.
    WHAT DOES ITLOOK LIKE? mypy, dynamic typing mypy, static typing def fib(n): void fib(int n): a, b = 0, 1 a, b = 0, 1 while a < n: while a < n: print(a) print(a) a, b = b, a+b a, b = b, a+b
  • 10.
    WHAT DOES ITLOOK LIKE? mypy, dynamic typing mypy, static typing def fib(n): void fib(int n): a, b = 0, 1 a, b = 0, 1 while a < n: while a < n: print(a) print(a) a, b = b, a+b a, b = b, a+b Dynamically typed function
  • 11.
    WHAT DOES ITLOOK LIKE? mypy, dynamic typing mypy, static typing def fib(n): void fib(int n): a, b = 0, 1 a, b = 0, 1 while a < n: while a < n: print(a) Static print(a) a, b = b, a+b a, b = b, a+b type signature
  • 12.
    WHAT DOES ITLOOK LIKE? mypy, dynamic typing mypy, static typing def fib(n): void fib(int n): a, b = 0, 1 a, b = 0, 1 while a < n: while a < n: print(a) print(a) a, b = b, a+b a, b = b, a+b Type inference
  • 13.
    DYNAMIC VS STATICTYPING mypy, dynamic typing def err(): return 1 + ‘x’ # no error (not called)
  • 14.
    DYNAMIC VS STATICTYPING mypy, dynamic typing def err(): return 1 + ‘x’ # runtime error (when called) err()
  • 15.
    DYNAMIC VS STATICTYPING mypy, dynamic typing def err(): return 1 + ‘x’ # runtime error (when called) err() mypy, static typing int err(): return 1 + ‘x’ # compile (static) error
  • 16.
    TYPES int, str, bool,MyClass any simple type dynamic (duck) type list<int>, dict<str, int> func<str, int> generic type function type tuple<int, str> tuple type
  • 17.
    EXAMPLE SCRIPT import sys,re if not sys.argv[1:]: raise RuntimeError('Usage: wordfreq FILE') dict<str, int> d = {} s = open(sys.argv[1]).read() for word in re.sub('W', ' ', s).split(): d[word] = d.get(word, 0) + 1 # Use list comprehension l = [(freq, word) for word, freq in d.items()] for freq, word in sorted(l): print('%-6d %s' % (freq, word))
  • 18.
    EXAMPLE SCRIPT import sys,re if not sys.argv[1:]: raise RuntimeError('Usage: wordfreq FILE') dict<str, int> d = {} s = open(sys.argv[1]).read() for word in re.sub('W', ' ', s).split(): d[word] = d.get(word, 0) + 1 # Use list comprehension l = [(freq, word) for word, freq in d.items()] for freq, word in sorted(l): print('%-6d %s' % (freq, word))
  • 19.
    EXAMPLE CLASS dynamic typing class BankAccount: def __init__(self, initial_balance=0): self.balance = initial_balance def deposit(self, amount): self.balance += amount def withdraw(self, amount): self.balance -= amount def overdrawn(self): return self.balance < 0 my_account = BankAccount(15) my_account.withdraw(5) print(my_account.balance) (source: Python Wiki)
  • 20.
    EXAMPLE CLASS static typing class BankAccount: void __init__(self, int initial_balance=0): self.balance = initial_balance void deposit(self, int amount): self.balance += amount void withdraw(self, int amount): self.balance -= amount bool overdrawn(self): return self.balance < 0 my_account = BankAccount(15) my_account.withdraw(5) print(my_account.balance)
  • 21.
    EXAMPLE CLASS static typing class BankAccount: void __init__(self, int initial_balance=0): self.balance = initial_balance void deposit(self, int amount): self.balance += amount void withdraw(self, int amount): self.balance -= amount bool overdrawn(self): return self.balance < 0 my_account = BankAccount(15) my_account.withdraw(5) print(my_account.balance)
  • 22.
    MYPY LIBRARY SUPPORT Mypy program Mypy VM Native mypy libraries Translation object mapping layer CPython VM Python libraries
  • 23.
    NO GIL CPython usesGlobal Interpreter Lock only one thread can usually run at a time
  • 24.
    NO GIL CPython uses Global Interpreter Lock only one thread can usually run at a time Not ok in the multicore era! Mypy will have a new ✝, custom VM without GIL. ✝ Based on the Alore VM (http://www.alorelang.org/)
  • 25.
  • 26.
    PERFORMANCE mypy speed static typing, ahead-of-time compiler PyPy JIT compiler CPython interpreter t (conceptual, not in scale)
  • 27.
    TYPE CHECKING Static typechecker pinpoints errors in your program, before running it some_function(foo[n], self.bar(x)) ^ StaticError: Argument has invalid type list<str>, list<int> expected ➠ Less debugging ➠ Easier maintenance and refactoring
  • 28.
    TOOL SUPPORT Static typeinformation makes many tools more powerful: • IDEs with reliable and precise code completion • Refactoring tools • Static analysis and linting tools ➠ Better productivity and quality
  • 29.
    SEAMLESS DYNAMIC AND STATIC TYPING Use the best tool for the job Use dynamic typing when it makes sense Use static typing when it makes sense As real programs are not uniform: use both
  • 30.
    Static typing maybe a win • if efficiency is important • if your project is large or complex • if you need to maintain code for several years Even when the above are true, dynamic typing is useful • for tests • for the user interface • for glue code • for extensions, scripts and plugins
  • 31.
    Dynamically typed program Statically typed libraries
  • 32.
    hot spot Sprinkle static types for better performance
  • 33.
    Statically typed interfaces Dynamically typed implementations
  • 34.
    Evolve programs fromdynamic to static typing v 0.5 v 1.0 v 2.0
  • 35.
    IMPLEMENTATION PLAN All dates tentative! Phase 1(2012) Mypy type checker + mypy-to-Python translator Phase 2 (2013) Compiler to native code (C/LLVM back end), new VM Phase N (?) More Python features IDE JVM back end? Android port? (etc.)
  • 36.
    MYPY VS OTHERPROJECTS
  • 37.
    CYTHON • Cython is a Python variant that cpdef int sum3d(int[:, :, :] a): compiles to Python C modules cdef int total = 0 I = a.shape[0] J = a.shape[1] • Cython can speed up tight K = a.shape[2] loops by a lot (100 x or more) for i in range(I): by using type declarations for j in range(J): for k in range(K): total += a[i, j, k] • Cython programs can also call return total C functions directly (adapted from Cython Users Guide) • Similar to Pyrex
  • 38.
    MYPY VS CYTHON Mypy (planned) Cython Powerful type system Simple type system Speedups mainly to low-level General code speedup code New, efficient VM CPython VM No GIL GIL still a problem
  • 39.
    SHED SKIN • Compiler for a statically typed subset of Python • Large speedups to some programs (100x or more) • Code looks like Python (explicit type declarations not needed) • Whole-program type inference
  • 40.
    MYPY VS SHEDSKIN Mypy (planned) Shed Skin Large set of Python features Restricted Python feature set Dynamic and static typing Static typing only Modular, (relatively) fast Very slow compilation for compiles large programs Python lib access layer Not many Python libs
  • 41.
    MYPY VS RPYTHON •RPython is a Python subset used by PyPy to implement the JIT compiler • RPython uses whole-program type inference (like Shed Skin) • Differences vs mypy: similar to mypy vs Shed Skin
  • 42.
    CONCLUSION Tell me what you think! All help and feedback is very welcome! Web site: http://www.mypy-lang.org Thank you!
  • 43.
    ANDROID: OVERVIEW • Over 500 M Android devices activated • Android native APIs mostly based on Java (Dalvik) • Performance important for apps • Cheap devices with low-end CPUs • UI responsiveness key to good user experience • Battery conservation
  • 44.
    MYPY ON ANDROID? Android Wishlist 1. Python syntax + libs Can’t have all of these 2. Good performance right now 3. Easy native API access • Mypy is a potential solution (by compiling to Dalvik) • But there’s a lot of work to do...

Editor's Notes