Slides from Advaned Python lectures I gave recently in Haifa Linux club Advanced python, Part 1: - Decorators - Descriptors - Metaclasses - Multiple inheritance

1. Topics in Advanced Python
No agenda – jumping in...
© 2014 Zaar Hai tech.zarmory.com

2. Decorators
Decorator is a function that accepts a function and returns
another function
from functools import wraps
def deco(f):
@wraps(f)
def wrapper(*args, **kwargs):
print "started at %s" % time.time()
rv = f(*args, **kwargs)
print "finished at %s" % time.time()
return wrapper
@deco
def sleeper():
"Sleeper function"
time.sleep(2)
>>> sleeper()
started at 1384245972.01
finished at 1384245974.01
>>> print sleeper.__name__
sleeper
wraps takes care about wrapped aux data. Without it:
>>> print sleeper.__name__
wrapper
Important when reading online help of someone else's code
2 © 2014 Zaar Hai tech.zarmory.com

3. Decorator usage examples
To return cached values
To implement retries
To guard methods/functions with a lock
To validate input/output of the function
To limit function execution by a certain timeout
Decorators greatly help with code reuse and
Make code explicitly readable
3 © 2014 Zaar Hai tech.zarmory.com

4. Decorator is just a function
You can call decorators “inline” in your code
@deco
def sleeper(): pass
sleeper()
deco(sleeper)()
@deco1
@deco2
def sleeper(): pass
sleeper()
deco1(deco2((sleeper))()
Similarly, decorators themselves can accept parameters
@deco(5)
def sleeper(): pass
sleeper()
deco(5)(sleeper)()
Note: The invocation order is a bit different
Note:@deco(5) is executed when the code is imported
4 © 2014 Zaar Hai tech.zarmory.com

5. Decorator with parameter - example
$ cat mydeco.py
def deco(p):
print "got %s" % p
def real_deco(f):
@wraps(f)
def wrapper(*args, **kwargs):
print "decorating"
return f(*args, **kwargs)
return wrapper
return real_deco
@deco(5)
def hello(): pass
>>> import mydeco # outer function runs during import
got 5
>>> mydeco.hello() # decorator runs when function executed
decorating
>>>
5 © 2014 Zaar Hai tech.zarmory.com

6. Decorator can be a class
Some times its useful to implement decorators as a class
class deco(object):
def __init__(self, p):
self.p = p
print "got %s" % p
def __call__(self, f):
@wraps(f)
def wrapper(*args, **kwargs):
print "decorating"
return f(*args, **kwargs)
return wrapper
__call__ is a special method that is invoked when you try
calling an object as if it was a function
6 © 2014 Zaar Hai tech.zarmory.com

7. How does @property decorator work?
@property – one of the most common decorators in Python
class A(object):
@property
def a(self):
return "a"
>>> A().a
'a'
But how does it work?
7 © 2014 Zaar Hai tech.zarmory.com

8. Meet descriptors
Descriptor is a protocol for accessing object attributes
class A(object):
def __init__(self):
self.a = 1
>>> f = A(); f.a
When you access a, what actually happens is
v = self.__dict__["a"]
if hasattr(v, '__get__'):
return v.__get__(self)
else:
return v
I.e. when attribute defines __get__ (or __set__) methods,
they are called to produce (or set) actual attribute value
8 © 2014 Zaar Hai tech.zarmory.com

9. Back to the @property
@property implements both decorator and descriptor
semantics like this:
class Property(object):
def __init__(self, getter):
self.getter = getter
def __get__(self, obj, csl):
return self.getter(obj)
class A(object):
@Property
def a(self):
return "a"
>>> o = A(); o.a
'a'
>>> type(o.a)
<type 'str'>
9 © 2014 Zaar Hai tech.zarmory.com

10. Descriptors in action
One-time caching of method's return values
class OneTime(object):
def __init__(self, func):
self.func = func
def __get__(self, obj, cls):
to_augment = obj or cls
rv = self.func(to_augment)
pname = self.func.__name__
setattr(to_augment, pname, rv)
return rv
class BigMath(object):
@OneTime
def big_data(self):
print "working hard...."
return 42
Reusable – follows DRY principle
More on this here and here
class BigMath(object):
def get_big_data(self):
if hasattr(self, "_rv"):
return self._rv
self._rv = self._get_data()
return self._rv
def _get_data(self):
print "working hard...."
return 42
Nice, but leads to a lot
of copy/paste code
10 © 2014 Zaar Hai tech.zarmory.com

11. © 2014 Zaar Hai tech.zarmory.com
Multiple inheritance
“LEGO” goes on

12. M-I in action - MixIns
“MixIn” is a programming concept about creating aux class
that enriches functionality of the main class
class SteeringWheel(object): ...
class HornMixIn(object):
"""Mix-in class to enable horn functionality"""
def horn(self, *args, **kwargs):
print "move over!"
class CruiseControlMixIn(object):
def cc_set(self): …
def cc_cancel(self): ...
def cc_restore(self):...
class Mazda2SteeringWheel(CruiseControlMixIn, HornMixIn, SteeringWheel):
"""Sometimes there is nothing to configure here at all"""
12 © 2014 Zaar Hai tech.zarmory.com

13. M-I in action – MixIns (continued)
Here is another example
class Synchronized(object):
def __init__(self, *args, **kwargs):
self._sync_lock = threading.Lock()
super(Synchronized, self).__init__(*args, **kwargs)
@staticmethod
def synchronized(func):
@wraps(func)
def wrapper(self, *args, **kwargs):
with self._sync_lock:
return func(self, *args, **kwargs)
return wrapper
class Writer(Syncrhonized, Db):
@Synchronized.synchronized
def write(self, o):
self.db.write(o)
As we see, for a lot of real-live examples M-I can be used
straight-forward and its behavior in Python “makes sense”
13 © 2014 Zaar Hai tech.zarmory.com

14. “New style” classes
Not exactly “new” - since Python 2.2
Always inherit from object
Really, Always inherit from object
Use super to call ancestor methods
class Foo(object):
def bar(self): pass
class Bar(Foo):
def bar(self):
return super(Bar, self).bar()
class Foo:
def bar(self): pass
class Bar(Foo):
def bar(self):
return Foo.bar(self)
Following the above technique will make your and others' life
much easier
14 © 2014 Zaar Hai tech.zarmory.com

15. Multiple inheritance and MRO
class A(object):
def __init__(self):
super(A, self).__init__()
print "A"
class B(object):
def __init__(self):
super(B, self).__init__()
print "B"
class C(A,B):
def __init__(self):
super(C, self).__init__()
print "C"
What will be output of running C()?
“A B C”?
“B A C”?
How do we arrive from super(A..) to the method of B?
15 © 2014 Zaar Hai tech.zarmory.com

16. Multiple inheritance and MRO
MRO – method resolution order
Python utilizes C3 algorithm
>>> C.__mro__
(<class 'my2.C'>, <class 'my2.A'>, <class 'my2.B'>, <type 'object'>)
And the answer to the previous question is “B A C”
For most real-life example I've seen, C3 logic “makes sense”
Without inheriting from object and using super, one had to
do MRO by himself in the code.
Bottom line – super() does not return ancestor, but the next
member of MRO chain
16 © 2014 Zaar Hai tech.zarmory.com

17. MRO pitfalls
class A(object): pass
class B(object): pass
class C(A,B): pass
class D(B,A): pass
class E(C,D): pass
Where super(E, self).__init__() will end up?
17 © 2014 Zaar Hai tech.zarmory.com

18. MRO pitfalls
class A(object): pass
class B(object): pass
class C(A,B): pass
class D(B,A): pass
class E(C,D): pass
Where super(E, self).__init__() will end up?
>>> class E(C,D): pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
Cannot create a consistent method resolution
order (MRO) for bases B, A
There are cases when C3 would (and should) fail to prevent a
naive programmer from creating a mess
18 © 2014 Zaar Hai tech.zarmory.com

19. type function revised
All of you are familiar with type:
>>> class A(object): pass
>>> a = A()
>>> type(a)
<class '__main__.A'>
BTW, don't use type(a) == A, but isinstance(a, A)
What type(A) will print?
19 © 2014 Zaar Hai tech.zarmory.com

20. type function revised
>>> type(A)
<type 'type'>
In Python, object model hierarchy has two levels:
Objects are instances of their Class
Classes are instances of their Type – AKA metaclass
All classes by default are instances of type metaclass, but we
can define our own metaclasses of course:
class MetaA(type): pass
class A(object):
__metaclass__ = MetaA
>>> type(A)
<class '__main__.MetaA'>
Metaclass can augment class creation. More on this later
20 © 2014 Zaar Hai tech.zarmory.com

21. Yet another face of type function
type can be used to create new classes on the fly
The syntax: type(name, bases, dict)
class Actor(object):
def get(self, req):
return self.val
for c in [Volumes, Hosts]:
class Conf(object): pass
class Volumes(Conf):
url = '/volumes'
val = 5
class Hosts(Conf):
url = '/hosts'
val = 4
HandlerClass = type('Handler', (Actor, c), {})
HttpServer.addUrl(c.url, HandlerClass)
HttpServer.start()
Allows really decoupled design. Yet flexible and efficient.
21 © 2014 Zaar Hai tech.zarmory.com

22. Metaclass example – proper Enum
class UserRoles(Enum):
root = 10
user = 20
user__desc = "Regular users"
Nobody = 30
>>> UserRoles.root
10
>>> UserRoles.root.name
'root' # Wow, I have a name!
>>> UserRoles.user.desc
'Regular users' # And even description
>>> UserRoles.root == 10
True # I look and quack just like the native type I was assigned to
>>> role = UserRoles(10) # Cast me! (with the DB value for example)
>>> role; role.name; role == 10
10
'root'
True
>>> role == UserRoles.root
True
Magic? May be; but a simple one
22 © 2014 Zaar Hai tech.zarmory.com

23. Magic revealed
class EnumMeta(type):
def __new__(mcs, name, base, d):
cls = super(EnumMeta, mcs).__new__(mcs, name, base, d)
for k, v in cls.__dict__.items():
pname = getattr(cls, k+"__name", k)
pdesc = getattr(cls, k+"__desc", "")
n = type(v).__name__
prop = type(n, (type(v),), {"name" : pname, "desc" : pdesc})
p = prop(v)
setattr(cls, k, p)
return cls
class Enum(object):
__metaclass__ = EnumMeta
Full story here
23 © 2014 Zaar Hai tech.zarmory.com

24. © 2014 Zaar Hai tech.zarmory.com
Thank you …