Python advance

Python Advance
Deepak Chandella
Email: deepchandila07@gmail.com

Agenda
 Comprehensions
 Lambda with (map, filter and reduce)
 Context managers
 Iterator, Generators, Decorators
 Python GIL and multiprocessing and multithreading
 Python WSGI
 Python Unittests

Comprehensions
Comprehensions in Python provide us with a short and concise way to construct new
sequences (such as lists, set, dictionary etc.) using sequences which have been already
defined. Python supports the following 4 types of comprehensions.
 List Comprehensions
 Dictionary Comprehensions
 Set Comprehensions
 Generator Comprehensions

List Comprehensions
 For example, if we want to create an output list which contains only the even
numbers which are present in the input list.
input_list = [1, 2, 3, 4, 4, 5, 6, 7, 7]
list_comp = [var for var in input_list if var % 2 == 0]
print("Output List:", list_comp)
Note: that list comprehension may or may not contain an if condition. List
comprehensions can contain multiple for (nested list comprehensions).

Dictionary Comprehensions
 For example, if we want to create an output dictionary which contains only the odd
numbers that are present in the input list as keys and their cubes as values.
input_list = [1,2,3,4,5,6,7]
dict_comp = {var:var ** 3 for var in input_list if var % 2 != 0}
print("Output Dictionary:", dict_comp)

Set Comprehensions
 Set comprehensions are pretty similar to list comprehensions.The only difference
between them is that set comprehensions use curly brackets { }.
 For example, if we want to create an output set which contains only the even
numbers that are present in the input list.
input_list = [1, 2, 3, 4, 4, 5, 6, 6, 6, 7, 7]
set_comp = {var for var in input_list if var % 2 == 0}
print("Output Set:", set_comp)

Generator Comprehensions
 Generator Comprehensions are very similar to list comprehensions. One difference
between them is that generator comprehensions use circular brackets whereas list
comprehensions use square brackets.
 The major difference between them is that generators don’t allocate memory for the
whole list. Instead, they generate each value one by one which is why they are memory
efficient.
input_list = [1, 2, 3, 4, 4, 5, 6, 7, 7]
output_gen = (var for var in input_list if var % 2 == 0)
print("generator object :", output_gen)
for var in output_gen:
print(var, end = ' ')

Lambda
In Python, Lambda functions are anonymous functions which means function
without a name. Lambda definition does not include a “return” statement, it always
contains an expression which is returned.
cube = lambda x: x*x*x
print(cube(3))
Note: Lambda functions can be used along with built-in functions like filter(), map()
and reduce().

Use of lambda with filter
 The filter() function in Python takes in a function and a list as arguments.This
offers an elegant way to filter out all the elements of a sequence “sequence”, for
which the function returnsTrue.
li = [5, 7, 22, 97, 54, 62, 77, 23, 73, 61]
final_list = list(filter(lambda x: (x%2 != 0) , li))
print(final_list)

Use of lambda with map
 The map() function in Python takes in a function and a list as argument.The
function is called with a lambda function and a list and a new list is returned which
contains all the lambda modified items returned by that function for each item.
li = [5, 7, 22, 97, 54, 62, 77, 23, 73, 61]
final_list = list(map(lambda x: x*2 , li))
print(final_list)

Use of lambda with reduce
 The reduce() function in Python takes in a function and a list as argument.
The function is called with a lambda function and a list and a new reduced result is
returned.This performs a repetitive operation over the pairs of the list.This is a part
of functools module.
from functools import reduce
li = [5, 8, 10, 20, 50, 100]
sum = reduce((lambda x, y: x + y), li)
print (sum)

Context Manager
 Managing Resources
In any programming language, the usage of resources like file operations or database
connections is very common. But these resources are limited in supply. Therefore, the
main problem lies in making sure to release these resources after usage. If they are not
released then it will lead to resource leakage and may cause the system to either slow
down or crash. It would be very helpful if user have a mechanism for the automatic
setup and teardown of resources.
In Python, it can be achieved by the usage of context managers which facilitate the
proper handling of resources.

Error scenario for resources
Let’s take the example of file management. When a file is opened, a file descriptor is consumed which
is a limited resource. Only a certain number of files can be opened by a process at a time.
file_descriptors = []
for x in range(100000):
file_descriptors.append(open('test.txt', 'w'))
Output:
Traceback (most recent call last):
File "context.py", line 3, in OSError: [Errno 24] Too many open files:
'test.txt‘
An error message saying that too many files are open. The above example is a case of file descriptor
leakage. It happens because there are too many open files and they are not closed.

Managing Resources using Context Manager
Python provides an easy way to manage resources i.e Context Managers.
The ”with”keyword is used. When it gets evaluated it should result in an object that
performs context management.
with open("test.txt") as f:
data = f.read()

Creating a Context Manager
When creating context managers using classes, user need to ensure that the class has the
methods: __enter__() and __exit__(). The __enter__() returns the resource
that needs to be managed and the __exit__() does not return anything but performs
the cleanup operations.
class FileManager():
def __init__(self, filename, mode):
self.filename = filename
self.mode = mode
self.file = None
def __enter__(self):
self.file = open(self.filename, self.mode)
return self.file
def __exit__(self, exc_type, exc_value, exc_traceback):
self.file.close()

Loading a file with context manager
loading a file with context manager created in previous slide
with FileManager('test.txt', 'w') as f:
f.write('Test')
print(f.closed)

Iterator
 An iterator is an object that contains a countable number of values.
 An iterator is an object that can be iterated upon, meaning that you can traverse
through all the values.
 Technically, in Python, an iterator is an object which implements the iterator
protocol, which consist of the methods __iter__() and __next__()

Iterator vs Iterable
 Lists, tuples, dictionaries, and sets are all iterable objects. They are
iterable containers which you can get an iterator from.
 All these objects have a iter() method which is used to get an iterator:
For example:
mytuple = ("apple", "banana", "cherry")
myit = iter(mytuple)
print(next(myit))
print(next(myit))
print(next(myit))

Creating a Iterator
 To create an object/class as an iterator you have to implement the
methods __iter__() and __next__() to your object.
 The __iter__() method acts similar, you can do operations (initializing etc.),
but must always return the iterator object itself.
 The __next__() method also allows you to do operations, and must return
the next item in the sequence.
 To prevent the iteration to go on forever, we can use
the StopIteration statement.

Creating a Iterator
Below is a simple Python program that creates iterator type that iterates from 10 to given
limit. For example, if limit is 15, then it prints 10 11 12 13 14 15. And if limit is 5, then it
prints nothing.
class Test:
def __init__(self, limit):
self.limit = limit
def __iter__(self):
self.x = 10
return self
def next(self):
x = self.x
if x > self.limit:
raise StopIteration
self.x = x + 1;
return x

Creating a Iterator
Using iterators created in previous slide…
 Prints numbers from 10 to 15
for i in Test(15):
print(i)
 Prints nothing
for i in Test(5):
print(i)

Generator
 There is a lot of overhead in building an iterator. We have to implement a class
with __iter__() and __next__() method, keep track of internal states,
raise StopIteration when there was no values to be returned etc.
 This is both lengthy and counter intuitive. Generator comes into rescue in such
situations.
 Python generators are a simple way of creating iterators. All the overhead we
mentioned above are automatically handled by generators in Python.
 Simply speaking, a generator is a function that returns an object (iterator) which we
can iterate over (one value at a time).

Creating a Generator
 It is fairly simple to create a generator in Python. It is as easy as defining a normal
function with yield statement instead of a return statement.
 If a function contains at least one yield statement (it may contain
other yield or return statements), it becomes a generator function.
Both yield and return will return some value from a function.
 The difference is that, while a return statement terminates a function entirely, yield
statement pauses the function saving all its states and later continues from there on
successive calls.

Generator example
 For example:
def my_gen():
n = 1
print('This is printed first')
yield n
n += 1
print('This is printed second')
yield n
n += 1
print('This is printed at last')
yield n

Generator example
 Try previous example on interpreter by calling one by one.
 Or try it with for loop e.g.
for item in my_gen():
print(item)

Generator example2
 Generator example that reverse a string
def rev_str(my_str):
length = len(my_str)
for i in range(length - 1,-1,-1):
yield my_str[i]
for char in rev_str("hello"):
print(char)

When to use yield instead of return
 Generator Return sends a specified value back to its caller whereas Yield can
produce a sequence of values. We should use yield when we want to iterate over a
sequence, but don’t want to store the entire sequence in memory.

Decorator
 Decorators are very powerful and useful tool in Python since it allows
programmers to modify the behavior of function or class.
 Decorators allow us to wrap another function in order to extend the behavior of
wrapped function, without permanently modifying it.
 In Decorators, functions are taken as the argument into another function and then
called inside the wrapper function.
 Python allows you to use decorators in a simpler way with the @ symbol,
sometimes called the “pie” syntax.

Decorator example
 For example
def my_decorator(func):
def wrapper():
print("Something is happening before the function is
called.")
func()
print("Something is happening after the function is
called.")
return wrapper
@my_decorator
def say_hello():
print(“Hellloooo!”)

Decorator example2
 For example, timer decorator to count time of execution.
import functools
import time
def timer(func):
"""Print the runtime of the decorated function""“
@functools.wraps(func)
def wrapper_timer(*args, **kwargs):
start_time = time.perf_counter() # 1
value = func(*args, **kwargs)
end_time = time.perf_counter() # 2
run_time = end_time - start_time # 3
print(f"Finished {func.__name__!r} in {run_time:.4f} secs")
return value
return wrapper_timer
@timer
def waste_some_time(num_times):
for _ in range(num_times):
sum([i**2 for i in range(10000)])

Decorator example2
 Execute the timer decorator with diff-2 values.
waste_some_time(1)
waste_some_time(999)
Check the output…

Decorator Appendix
 The @functools.wrapsdecorator uses the
function functools.update_wrapper()to update
special attributes like __name__and __doc__ that are used in the
introspection.
 Try below on interpriter.
>>> waste_some_time
>>> waste_some_time.__name__
>>> help(waste_some_time)

Python GIL
 What is GIL
Python Global Interpreter Lock (GIL) is a type of process lock which is used by python
whenever it deals with processes. Generally, Python only uses one thread to execute
the set of written statements. This means that in python only one thread will be
executed at a time.

Python GIL
 What problem did the GIL solve for Python
Python uses reference counting for memory management. It means that objects created in Python have
a reference count variable that keeps track of the number of references that point to the object. When
this count reaches zero, the memory occupied by the object is released.
For example:
import sys
a = []
b = a
sys.getrefcount(a)
NOTE: output of above program is 3, because the list object was referenced by a, b and the argument
passed to sys.getrefcount()
This reference counter variable needed to be protected, because sometimes two threads increase or
decrease its value simultaneously by doing that it may lead to memory leaked so in order to protect
thread we add locks to all data structures that are shared across threads but sometimes by adding locks
there exists a multiple locks which lead to another problem that is deadlock. In order to avoid memory
leaked and deadlocks problem, we used single lock on the interpreter that is Global Interpreter
Lock(GIL).

Python GIL
 Does GIL really impact multithreaded program

How to deal with GIL
 Multi-processing vs multi-threading
The most popular way is to use a multi-processing approach where you use multiple processes instead
of threads. Each Python process gets its own Python interpreter and memory space so the GIL won’t be a
problem. Python has a multiprocessing module which lets us create processes easily
 Alternative Python interpreters
Python has multiple interpreter implementations. CPython, Jython, IronPython and PyPy, written in C,
Java, C# and Python respectively, are the most popular ones. GIL exists only in the original Python
implementation that is CPython.
 Just wait it out
While many Python users take advantage of the single-threaded performance benefits of GIL. The
multi-threading programmers don’t have to fret as some of the brightest minds in the Python
community are working to remove the GIL from CPython. One such attempt is known as the Gilectomy.

Python WSGI
WSGI is not a server, a python module, a framework, an API or any kind of software. It
is just an interface specification by which server and application communicate. Both
server and application interface sides are specified in the PEP 3333.
Beneath Django, Flask, Bottle and every other Python web framework, lies the Web
Server Gateway Interface or WSGI for short. WSGI is to Python what Servlets are to
Java — a common specification for web servers that allows different web servers and
application frameworks to interact based on a common API. However, as with most
things, the Python version is considerably simpler.

Python Unittests
Unit Testing is the first level of software testing where the smallest testable parts of a
software are tested. This is used to validate that each unit of the software performs as
designed. For example.
import unittest
class SimpleTest(unittest.TestCase):
def test(self):
self.assertTrue(True)
if __name__ == '__main__':
unittest.main()

Python Unittests
There are three types of possible test outcomes :
 OK – This means that all the tests are passed.
 FAIL – This means that the test did not pass and an AssertionError exception is
raised.
 ERROR – This means that the test raises an exception other than AssertionError.

Python Unittests
Basic assertions used in the code
assertEqual() – This statement is used to check if the result
obtained is equal to the expected result.
assertTrue() / assertFalse() – This statement is
used to verify if a given statement is true or false.
assertRaises() – This statement is used to raise a specific
exception.

Python advance

More Related Content

What's hot

Similar to Python advance

Recently uploaded

Python advance