Python Objects Mykola Kharechko, Developer © Quintagroup, 2008

Python Objects

Mykola Kharechko, Developer

© Quintagroup, 2008

Table of contents Objects in memory Objects at python level Objects in python virtual machine

Objects in memory

Objects at python level

Objects in python virtual machine

Objects in memory Object structure Python memory management Garbage collection

Object structure

Python memory management

Garbage collection

Object structure

Objects with same class

Python Memory Management

Allocation Strategy For small requests, the allocator sub-allocates <Big> blocks of memory. Requests greater than 256 bytes are routed to the system's allocator.

Pymalloc structure Arena 256kb Pool 4kb Fixed size block

Fixed size block * Request in bytes Size of allocated block Size class idx * ------------------------------------------------------------------------------ * 1-8 8 0 * 9-16 16 1 * 17-24 24 2 * 25-32 32 3 * 33-40 40 4 * 41-48 48 5 * 49-56 56 6 * 57-64 64 7 * 65-72 72 8 * ... ... ... * 241-248 248 30 * 249-256 256 31

Garbage Collection Python performs garbage collection via reference counting and a cyclic garbage collector that is able to detect and break reference cycles.

Example >>> import gc >>> gc.disable() # 2.9mb >>> a = range(1000000) # 18mb >>> del a # 14mb >>> gc.enable() # 14mb >>> gc.collect() # 14mb 0 >>> gc.disable() ‏ >>> a = range(1000000) # 18mb >>> a.append(a) # 18mb >>> del a # 18mb >>> gc.enable() # 18mb >>> gc.collect() # 14 mb 1 >>>

Weak references weakref module weakref.ref class weakref.proxy class weakref.WeakKeyDictionary

weakref module

weakref.ref class

weakref.proxy class

weakref.WeakKeyDictionary

weakref example >>> from weakref import ref, proxy >>> class A(object):pass ... >>> a = A() ‏ >>> ref_a = ref(a) ‏ >>> a.a = 16 >>> ref_a().a 16 >>> proxy_a = proxy(a) ‏ >>> proxy_a.a 16 >>> proxy_a <weakproxy at 0x94e25f4 to A at 0x94e0fcc> >>> del a >>> ref_a() ‏ >>> ref_a().a Traceback (most recent call last): File &quot;<stdin>&quot;, line 1, in <module> AttributeError: 'NoneType' object has no attribute 'a' >>> proxy_a <weakproxy at 0x94e23c4 to NoneType at 0x81479b8>

List object typedef struct { PyObject_VAR_HEAD // ob_size /* Vector of pointers to list elements. list[0] is ob_item[0], etc. */ PyObject **ob_item; // 0 <= ob_size <= allocated // len(list) == ob_size Py_ssize_t allocated; } PyListObject;

List creation PyObject * PyList_New(Py_ssize_t size) ‏ { PyListObject *op; size_t nbytes; .... nbytes = size * sizeof(PyObject *); ..... op->ob_item = (PyObject **) PyMem_MALLOC(nbytes); memset(op->ob_item, 0, nbytes); ... op->ob_size = size; op->allocated = size; ... }

Integer objects typedef struct { PyObject_HEAD long ob_ival; } PyIntObject;

Integer optimization. Problem >>> a = -2 >>> b = -2 >>> id(a) == id(b) ‏ True >>> a = 300 >>> b = 300 >>> id(a) == id(b) ‏ False

Integer optimization. Solution #define NSMALLPOSINTS 257 #define NSMALLNEGINTS 5 /* References to small integers are saved in this array so that they can be shared. The integers that are saved are those in the range -NSMALLNEGINTS (inclusive) to NSMALLPOSINTS (not inclusive). */ static PyIntObject * small_ints[NSMALLNEGINTS + NSMALLPOSINTS];

Objects at python level Objects creation Object attributes Specific attributes

Objects creation

Object attributes

Specific attributes

Objects creation Constructor: obj = type->tp_new(type, args, kwds); type = Py_TYPE(obj); type->tp_init(obj, args, kwds); Classes created by metaclasses

Object attributes Attributes stored in __dict__ or in object when __slots__ used Attribute can be accessed via descriptor protocol The most famous descriptor - property

Specific attributes __getattribute__ -access to all attrs __setattr__ -opposite to __getattribute__ Descriptor protocol: __get__, __set__, __delete__ Pickler protocol: __getstate__, __setstate__, __reduce__

__getattribute__ -access to all attrs

__setattr__ -opposite to __getattribute__

Descriptor protocol: __get__, __set__, __delete__

Pickler protocol: __getstate__, __setstate__, __reduce__

Objects in PVM typedef struct { PyObject_HEAD int co_argcount; /* #arguments, except *args */ int co_nlocals; /* #local variables */ int co_stacksize; /* #entries needed for evaluation stack */ int co_flags;/* CO_..., see below */ PyObject *co_code; /* instruction opcodes */ PyObject *co_consts; /* list (constants used) */ PyObject *co_names;/* list of strings (names used) */ PyObject *co_varnames;/* tuple of strings (local variable names) */ PyObject *co_freevars;/* tuple of strings (free variable names) */ PyObject *co_cellvars; /* tuple of strings (cell variable names) */ /* The rest doesn't count for hash/cmp */ PyObject *co_filename;/* string (where it was loaded from) */ PyObject *co_name;/* string (name, for reference) */ int co_firstlineno;/* first source line number */ PyObject *co_lnotab;/* string (encoding addr<->lineno mapping) */ void *co_zombieframe; /* for optimization only (see frameobject.c) */ } PyCodeObject;

CodeObject example File test.py a = 17 b = a + 3 print b

CodeObject Example >>> import test 34 >>> pyc = open('test.pyc') ‏ >>> pyc.read(8) ‏ 'xb3xf2
>Cx8eH' >>> from marshal import load >>> co = load(pyc) ‏ >>> dir(co) ‏ [... 'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames'] >>> co.co_consts (17, None) ‏ >>> co.co_names ('a', 'b') ‏

CodeObject example >>> from dis import dis >>> dis(co) ‏ 1 0 LOAD_CONST 0 (17) ‏ 3 STORE_NAME 0 (a) ‏ 2 6 LOAD_NAME 0 (a) ‏ 9 LOAD_CONST 0 (17) ‏ 12 BINARY_ADD 13 STORE_NAME 1 (b) ‏ 3 16 LOAD_NAME 1 (b) ‏ 19 PRINT_ITEM 20 PRINT_NEWLINE 21 LOAD_CONST 1 (None) ‏ 24 RETURN_VALUE