release_python_day2_slides_201606.pdf

Python Programming
4-days class
PaulYang
2016/06

Agenda of Python Programming
Day 1 - Basic Python
Day 2 - Advanced Python
Day 3 - Web Scraping with Python
Day 4 - Web application with python

Advanced Python
Python - Day2
PaulYang
2016/06

Objectives
• How to code and run your Python program more efficient ?
• How to process data, such as search “specific pattern”
from string ?
• How to make you code more maintainable and readable
for mid/large project ?

Courses agenda – Day2
1. Error Handling
2. Generator function, List processing/comprehension and generator
expression
3. Functional Tools: map, lambda, filter, reduce…more
4. Module and Package
5. Useful Packages Intro: os, collection, sys, json…more
6. Text processing – Regular Expression and Unicode
7. OO programming

Open-Process-Close
7
Standard open-process-close
• Not only to text but also apply to most database or whatever file
#1.INPUT
the_file = open('dialogue.txt')
#2.PROCESS
#do_something to data of file
#3.CLOSE
the_file.close()

Open-Process-Close
8
Question: how to separate actor and each line
Actor’s spoken line
Actor
dialogue_chinese.txt

Open-Process-Close
9
Man: 憑你的智慧，我很難跟你解釋！
: separator
line.split(“:”)
Man 憑你的智慧，我很難跟你解釋！
(role, line_spoken)= line.split(“:”)

Exercise 7
10
data = open("dialogue_chinese.txt", encoding="utf-8")
for line in data:
(role,line_spoken) = line.split(":")
print(role,end="")
print("說: ", end="")
print(line_spoken, end="")
• File->NewFile -> Type-in in IDLE3, press F5

Exercise 7
11
• You should run into ValueError and python crashes
Exception

Exercise 7 - problems
12
• Split(“:”) doesn’t know how to deal with the third part of
string
• Complain “too many value”
Second delimiter “:”

Exercise 7 – a fix
13
• maxsplit

Exercise 7 – a fix
14
• assign “1” to the arg of split() - maxsplit
for line in data:
(role,line_spoken) = line.split(":",maxsplit=1)
print(role,end="")

15
Again run into ValueError and python crashes

16
• Split(“:”) expects “:” , doesn’t know how to deal with wrong
format
• Complain “not enough values”
no delimiter : is found

Decisions to make
17
If not line.find(“:”) == -1
#dosomething
Print()
• Put extra control logic to check and correct split error
• Endless check if data format changes again
• too many flows cause code unreadable

Decisions to make
18
• Let error happens and deal with exception
Python tries to
execute your code
Normal code flow
Crash
Python tries but fails
Recover back to execute
The exception is
handled
Code continues working
code flow with exception

Try/Except
try:
#do your operations here;
except ExceptionI:
#If there is ExceptionI, then execute this block.
except ExceptionII:
#If there is ExceptionII, then execute this block.
else:
#If there is no exception then execute this block.
finally:
#This would always be executed.

Exercise 8 - A fix by Try/Except
for line in data:
try:
print(role,end="")
except:
pass
data.close()
Do nothing

Exercise 8 – another problem
def print_file():
filename = input("輸入要開啟的檔名:")
data = open(filename, encoding="utf-8")
for line in data:
try:
print(role,end="")
except:
pass
data.close()
print_file()
輸入要開啟的檔名: diagoue.txt
What if file doesn’t exist – you put incorrect filename

Exercise 8 – one way to fix
import os
def print_file():
if os.path.exists(filename):
for line in data:
try:
print(role,end="")
except:
pass
data.close()
else:
print("檔名不存在")
>>> print_file()
>>> 輸入要開啟的檔名: diagoue.txt
檔名不存在
• Extra check-logic version

def print_file():
try:
for line in data:
try:
print(role,end="")
except:
pass
data.close()
except:
print("檔名不存在")
>>> print_file()
>>> 輸入要開啟的檔名: diagoue.txt
檔名不存在
• Try-except version
Less is better!!
Complexity creates more problem and
prevent you from making right thing

try:
print(role,end="")
except:
pass
• Be specific for the exception you gonna catch
This bypass all runtime error. you
may not notice other bugs and don’t
know how to handle afterward
try:
for line in data:
try:
print(role,end="")
except ValueError:
pass
data.close()
except IOError as err:
print(err)

Exceptions
Content credited: http://www.tutorialspoint.com/python/python_exceptions.htm

Full handling clause
• Try / Finally / else clauses
try:
fh = open("testfile", "w")
fh.write("This is my test file for exception handling!!")
finally:
print("Error: can't find file or read data")
try:
fh = open("testfile.txt", "w")
fh.write("This is my test file for exception handling!!")
except IOError as e:
print("Error: can't find file or write data")
print(e)
else:
print ("Written content in the file successfully")

Raising an Exception
You can raise exception to halt python or create your custom-made exception
def password_check(passwd):
if len(passwd) < 1:
raise Exception("PasswordFormatError: length is {}".format(len(passwd)))
if passwd.find('*') != -1:
raise Exception("PasswordFormatError: contains *")
password = ""
try:
password_check(password)
except Exception as e:
print(e)
PasswordFormatError: length is 0

The following exceptions are used mostly as base classes for
other exceptions.
Content credited: https://docs.python.org/3/library/exceptions.html#exception-hierarchy

• User defined exception - MyAppLookUpError
class MyAppLookupError(LookupError):
'''raise this when there's a lookup error for my app'''
if len(passwd) < 1:
raise MyAppLookupError("MyAppPasswordFormatError: length is
{}".format(len(passwd)))
if passwd.find('*') != -1:
raise MyAppLookupError("MyAppPasswordFormatError: contains *")
password = ""
try:
password_check(password)
except Exception as e:
print(e)
Inherit from baseclass : BaseException

Assertion
• An assertion is a sanity-check - An expression is tested, and if the result
comes up false, an exception is raised.
• assert Expression[, Arguments]
assert type(passwd) is str, "passwd is not a string: %r" % passwd
return passwd
print(password_check("2323r23"))
print(password_check("abc3d3"))
print(password_check(0.2344))
AssertionError: passwd is not a string: 0.2344
Arg is executed when
not Expression ( Expression != True)

Generator function, List comprehension
and generator expression
(2-2)

• function that produces a sequence of results instead of a single value
• using the yield statement instead of return statement
• Behavior is quite different than normal func
• Return an generator object.
Generator function
No output was produced
Rerun an object rather
than a value

• The function only executes on next() or forloop
• A generator is a one-time operation,
Generator function
>>> a = countdown(3)
>>> next(a)
Counting down from 3
3
>>> next(a)
2
>>> next(a)
2
>>> next(a)
1
>>> next(a)
One-time, exception
once it’s consumed

We want to process a big file. How can we do in ?
• Ex. get the sum of series of number
Ex.
[0,1,2,3,4]
[0,1,2,3,4,5,6,7,8,9]
[0,1,2,3,4,………………,100000]
A huge file, such as big dataset -- 1GB cvs/txt file
<!!!> Raspberrypi 2 Memory: 1 GB
Huge Data Question

def firstn_list(n):
num, nums = 0, []
while num <= n:
nums.append(num)
num += 1
return nums
>>> sequnce_nums = firstn_list(10000)#10**5
>>> sum(sequnce_nums)
49995000
By normal function
• Let’s create a list and sum each up
• [0,1,2,3,4,………………,10000]
Here nums contains 10000
num variable in the memory
0
0 1 2
1
..
2 …
9999
10000
nums[]
Run

def firstn_generator(n):
num = 0
while num < n:
yield num
num += 1
>>>sequnce_nums = firstn_generator(10000)#10**5
>>>sum(sequnce_nums)
49995000
By generators function
1 2 … 10000
Run
1st
Run
2nd
Run
..th
Run
10000
num num num num

Key difference
• firstn_list(10**8 ) produces huge memory consumption once because
create 10**8 variables in the list
• firstn_generator(10**8 ) only consumes the size of 1 local
variable every time when invoked accordingly

python performance analysis
IPython shortcuts
%load_ext memory_profiler
%load_ext line_profiler
%timeit
%memit
%prun
%mprun
How much time does my app take?
How much memory does my app use?

Setup
$ pip install line-profiler
$ pip install psutil
$ pip install memory_profiler
$ ipython profile create
#Edit three files below
~/.ipython/extensions/line_profiler_ext.py
~/.ipython/extensions/memory_profiler_ext.py
~/.ipython/profile_default/ipython_config.py
Ipython notebook
http://pynash.org/2013/03/06/timing-and-profiling/

http://localhost:8888/notebooks/Python_performance_analysis.ipynb

Exercise 9 – measure time/mem
%load_ext memory_profiler
%load_ext line_profiler
def my_func_t(n):
a = [1] * (10 ** 6) * n
print (a)
b = [2] * (2 * 10 ** 7) * n
del b
return a
%timeit -r 1 my_func_t(3)
%memit -r 1 my_func_t(3)

def firstn_list(n):
num, nums = 0, []
while num < n:
nums.append(num)
num += 1
return nums
%timeit –n 3 sum(firstn_list(10 ** 6))
%memit sum(firstn_list(10 ** 6))

def firstn_generator(n):
num = 0
while num < n:
yield num
num += 1
%timeit –n 3 sum(firstn_generator(10 ** 6))
%memit sum(firstn_generator(10 ** 6))

Faster and smaller usage
By generator func

Recap
• A generator function is mainly a way to optimize memory use
and distribute one-time work to the one in different time scale
• save more space and run faster than normal function

List Processing/Comprehensions

list-processing features
S = ['IBM',50,91.10]
s.append(x) # Append x to end of s
s.extend(t) # Add items in t to end of s
s.count(x) # Count occurences of x in s
s.index(x) # Return index of x in s
s.insert(i,x) # Insert x at index i
s.remove(x) # Remove first occurence of x
s.reverse() # Reverses items in list
s.sort() # Sort items in s in-place
• Python has a lot of list-processing features

list-processing features
x = [1, 2, -3, 4, -5]
a = []
for i in x:
if i > 0:
a.append(2*i)
• If we want to create a new list in which each element is
doubled in the old list when old value is not negative
• One way use forloop iterate
5 lines of code

List Comprehensions
• General syntax
[expression for x in s if condition]
• What it means
result = []
for x in s:
if condition:
result.append(expression)
• Can be used anywhere a sequence is expected
>>> a = [1,2,3,4]
>>> sum([x*x for x in a])
30

List Comprehensions
>>> x = [1, 2, -3, 4, -5]
>>> a = [i*2 for i in x if i > 0]
>>> b
[2,8,4,20]
• Same case by list comprehension
2 lines of code

List Comprehensions
• ASCII and str interchange between map
>>> ord('s')
115
>>> res = []
>>> for x in 'spam':
res.append(ord(x))
>>> res
[115, 112, 97, 109]
Content: http://www.ascii-code.com/
>>> res = list(map(ord, 'spam'))
>>> res
[115, 112, 97, 109]
>>> res = [ord(x) for x in 'spam']
>>> res
[115, 112, 97, 109]
Stand list
processing
Use map() to
combine ord()
Use map() to
combine ord()

List Comprehensions
• nested loop: we want to add each element from a ,b
a, b = [0, 1, 2] , [100,200,300]
>>> res = []
>>> for x in a:
for y in b:
res.append(x+y)
>>> res
[100, 200, 300, 101, 201, 301, 102, 202, 302]
>>>[x+y for x in a for y in b]
>>> res
[100, 200, 300, 101, 201, 301, 102, 202, 302]
Use 2 forloops
Finish on 1 line

Exercise 10 – transfer for to list comp (1)
>>>result = []
>>>for x in ‘spam’:
for y in ‘SPAM’:
result.append(x+y)
>>>result
• Intervene ‘spam’ with ‘SPAM’ to [sS,sP,sA sS,Ps,pP……]?
Normal forloop way

Exercise 10 – transfer for to list comp (2)
>>>result = []
>>>for x in range(5):
if x % 2 == 0
for y in range(5):
if y % 2 == 1:
result.append((x,y))
• Intervene ‘even number’ with ‘odd number’ in one to five to
[(0,1),(0,3),(2,1)…..]
even
odd

Exercise 10– Answer(1)
>>> [x + y for x in 'spam' for y in 'SPAM']
['sS', 'sP', 'sA', 'sM', 'pS', 'pP', 'pA', 'pM',
'aS', 'aP', 'aA', 'aM', 'mS', 'mP', 'mA', 'mM']
Intervene ‘spam’ with ‘SPAM’ to

Exercise 10– Answer(2)
>>> [(x, y) for x in range(5) if x % 2 == 0 for y in range(5) if y % 2 == 1]
[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
Intervene even and odd number

Recap
• Don’t abuse the complicated list comp
Python’s import this motto:
Simple is better than complex. code conciseness is a much less
important goal than code readability
• Performance (depend on the pattern)
list comprehensions > map calls > for loop (PVM)

Generator Expressions
•General syntax
(expression for x in s if condition)
•What it means
for x in s:
if condition:
yield expression

•A generated version of a list comprehension but
produced one at a time
>>> a = [1,2,3,4]
>>> b = (2*x for x in a)
>>> b
<generator object at 0x58760>
>>> for i in b: print b,
...
2 4 6 8

•Different from a list comp
>>> a = [1,2,3,4]
>>> b = [2*x for x in a]
>>> b
[2, 4, 6, 8]
>>> c = (2*x for x in a)
>>>

•The parens () can be dropped if used as a single
function argument
sum((x*x for x in s))
Generator expression
sum(x*x for x in s)

>>> G = (x ** 2 for x in range(4))
>>> iter(G) is G # iter(G) optional: __iter__ returns self
True
>>> next(G) # Generator objects: automatic methods
0
>>> next(G)
1
>>> next(G)
4
>>> next(G)
9
>>> next(G)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>> G
<generator object <genexpr> at 0x00000000029A8318>

Exercise 11 – make Generator
wwwlog = open("access-log")
total = 0
for line in wwwlog:
bytestr = line.rsplit(None,1)[1]
if bytestr != '-':
total += int(bytestr)
print "Total", total
81.107.39.38 - ... "GET /ply/ply.html HTTP/1.1" 200 97238
81.107.39.38 - ... "GET /ply/ HTTP/1.1" 304 -
• Find out how many bytes of data were transferred

Exercise 11 - Generators as a Pipeline
• Each step is defined by iteration/generation
wwwlog = open("access-log")
bytecolumn = (line.rsplit(None,1)[1] for line in wwwlog)
bytes = (int(x) for x in bytecolumn if x != '-')
print "Total", sum(bytes)

Recap
• generator expressions may also run slightly slower than list
comprehensions in practice
• so they are probably best used only for very large result sets
or applications that cannot wait for full results generation.

Functional programming
What: a way to build code in which you use functions as the main building block.
Why: might lead to code that’s easier to test, debug, parallelize, and understand.
How: thinking about what programs should do instead of how, using functions as the
major unit to solve problems on computer.
Think of “y=f(x)”, not uses function
Content credited: http://nanavati.org/love_functional_programming.htm

Composition everywhere
Credited Scott Wlaschin: https://www.youtube.com/watch?v=E8I19uA-wGY

Credited Scott Wlaschin: https://www.youtube.com/watch?v=E8I19uA-wGY
Function all the way
down

•Imperative programming (С/C++, Java)
•both procedural and object-oriented
•Declarative programming
•Functional programming (Haskell, Scheme, OCaml)
•Logic programming (Prolog, Clojure core.logic)

Python is a multi paradigm programming language.
Credit FP in python : https://www.youtube.com/watch?v=goypZR_lQ7I

Calculate partially invalid string with operations:
"28+32+++32++39"
HOWTO WHATTO

expr, result = "28+32+++32++39", 0
for t in expr.split("+"):
if t != "":
res += int(t)
print result
Imperative style = actions that change state from initial state to result

from operator import add
expr = "28+32+++32++39"
print reduce(add, map(int, filter(bool, expr.split("+"))))
Functional style = apply transformation (and compositions)

Key concepts
Functions as Objects (first-class objects)
>>> def i_am_an_object(myarg):
... return myarg
...
>>> i_am_an_object(1)
1
>>> an_object_by_any_other_name = i_am_an_object
>>> an_object_by_any_other_name(2)
2
>>> i_am_an_object
<function i_am_an_object at 0x100432aa0>
>>> an_object_by_any_other_name
<function i_am_an_object at 0x100432aa0>

Key concepts
High order function
• meaning that functions can accept other functions as arguments and return
functions to the caller.
• Use function as value or data >>> def second_element(t):
... return t[1]
...
>>> zepp = [('Guitar', 'Jimmy'), ('Vocals',
'Robert'), ('Bass', 'John Paul'), ('Drums',
'John')]
>>> sorted(zepp)
[('Bass', 'John Paul'), ('Drums', 'John'),
('Guitar', 'Jimmy'), ('Vocals', 'Robert')]
>>> sorted(zepp, key=second_element)
[('Guitar', 'Jimmy'), ('Drums', 'John'),
('Bass', 'John Paul'), ('Vocals', 'Robert')]

Key concepts
Anonymous Functions (Lambdas )
• functions that are not bound to a name, lambdas instead of def
• lambda [arg1 [,arg2,.....argn]]:expression
• Use in simple func without name required
>>> def f (x): return x**2
>>> print f(8)
64
>>>
>>> g = lambda x: x**2
>>> print g(8)
64
>>> def call_func(f, *args):
... retaurn f(*args)
>>> call_func(lambda x, y: x + y, 4, 5)
9
Use lambda func as
argument

Functional tools - map
map()
• that performs some operation on each element of an iterable like []
• map(func, SEQ)
>>> def incre_num(a, b, c):
... return a*10000 + b*100 + c
...
>>> map(incre_num, (1, 2, 3), (4, 5, 6), (7, 8, 9))
[10407, 20508, 30609]
User defined func

revise the following procedure codes as functional
def increment_each(mylist)
new_list = list()
for element in mylist:
new_list.append(incre(element))
return new_list
def incre(element):
return element + 1
increment_each([0,1,2,3])
[1,2,3,4]

Use only expression without statement
map(incre,[0,1,2,3])
#or
map(lambda element: element + 1,[0,1,2,3])

A simple implementation of map()
def map(func, seq)
new_seq = list()
for element in seq:
new_seq.append(func(element))
return new_seq

Functional tools - filter
filter()
• filter out all the elements of a list, for which the function returns True
• filter(func, SEQ)
filter(lambda x: x % 2, [0,1,2,3])
[1,3]
filter(lambda x: x % 2 == 0, [0,1,2,3])
[2,4]
f returns a Boolean value, i.e. either True or False
Leave it in list If x%2 is True - get odd num
Even num
Code credited http://www.python-course.eu/lambda.php

Functional tools - reduce
reduce()
• continually applies the function func() to the sequence seq. It returns a
single value
• reduce(func, SEQ)
>>>reduce(lambda x,y: x+y, [47,11,42,13])
113
Func(Func(Func(47,11),42),13)
Code credited http://www.python-course.eu/lambda.php

Map-Reduce
Photo credited: http://paulyang0125.blogspot.tw/2015/09/part-2-explain-mapreduce-model-using.html
the same concept of Google MapReduce distributing
jobs into different machines in parallel way

More Python FP
• Functools
• Itertools
• Generators
• Decorators
More on PYTHON FP:
https://newcircle.com/bookshelf/python_fundamentals_tutorial/functional_programming
http://thecodeship.com/patterns/guide-to-python-function-decorators/
Function decorators are simply wrappers to existing functions.

Exercise 12 – revise the code using reduce()
def time_each(mylist):
result = 1
for element in mylist:
result *= element
return result
result = time_each([1,2,3,4])
print(result)

Exercise 12 – revise the code using reduce()
def time_each(mylist):
if len(mylist) == 1:
return mylist[0]
else:
return mylist[0] * time_each(mylist[1:])
result = time_each([1,2,3,4])
print(result)
Recursive version
2 = time_each([0,1,2]) = 1 * (1 * 2)
time_each([1,2]) = 1 * 2
time_each([2]) = 2
http://interactivepython.org/runestone/static/pythonds/Recursion/pythondsCalculatingtheSumofaListofNumbers.html
http://www.python-course.eu/recursive_functions.php

Exercise 12 – Answer
>>> reduce( (lambda x, y: x * y), [1, 2, 3, 4] )
24

Modules
• Simply, a module is a file consisting of Python code. A module
can define functions, classes and variables.
• A file named module.py creates a module module.
• The import modu statement will look for modu.py in the same
directory. If it isn’t found, the Python interpreter will search
for modu.py in the sys.path recursively; or raise an ImportError
exception if it isn’t found.
93

Modules
94
Content credited: http://interactivepython.org/runestone/static/pip/PythonModules/modules.html
import math
print(math.pi)
print(math.e)
print(math.sqrt(2.0))
print(math.sin(math.radians(90))) # sin of 90 degrees
3.14159265359
2.71828182846
1.41421356237
1.0

• A module provides a namespace. The module’s variables,
functions, and classes will be available to the caller through
the module’s namespace
• import, import as, from import are statements.
95
Create an alias

Package
96
Package is just a collection of modules, Any directory
with an __init__.py file is considered a Python package.
from sound.formats.wav import *
record() # no good code style
from sound.formats.wav import record
record()
import sound.formats.wav
wav.record()
A file wav.py in the directory sound/formats/ is imported with the
statement import sound.formats This statement will look for an __init__.py file
in pack, execute all of its top-level statements

OS - Operating System Interface
98
>>> import os
>>> os.getcwd() # Return the current working directory
'C:Python35'
>>> os.chdir('/server/accesslogs') # Change current
working directory
>>> os.system('mkdir today') # Run the command mkdir in
the system shell 0
More on
https://docs.python.org/3/library/os.html#module-os

Sys - System-specific parameters and
functions
99
$python demo.py one two three
>>> import sys
>>> print(sys.argv)
['demo.py', 'one', 'two', 'three']
More on https://docs.python.org/3/library/os.html#module-os
argv[0]
Script name
argv[1:]
Arg1 - n
>>> import sys
>>> sys.version
‘3.6.5 (r265:79063, Apr 16 2016, 13:57:41) n[GCC 4.4.3]'
>>> sys.version_info
(3, 6, 5, 'final', 0)
Command Line Arguments
Information on the Python Interpreter

Collection module
100
More on http://book.pythontips.com/en/latest/collections.html
This module implements some nice data structures which enhance the original
dict and tutle structure – faster and solve some native problems
defaultdict
counter
deque
namedtuple
enum.Enum

Other BIFs
101
More on https://docs.python.org/3/library/functions.html#help

Other BIFs
102
More on https://docs.python.org/3/library/functions.html#help
>>> hex(255)
'0xff'
>>> hex(-42)
'-0x2a'
letters = "ABCa"
for letter in letters:
# Get number that represents
letter in ASCII.
number = ord(letter)
print(letter, "=", number)
A = 65
B = 66
C = 67
a = 97
numbers = [97, 98, 99]
for number in numbers:
# Convert ASCII-based number
to character.
letter = chr(number)
print(number, "=", letter)
97 = a
98 = b
99 = c

Text processing
(regular Expression and Unicode)
(2-6)

Regular expression (Regex)
104
• A regular expression is a special sequence of characters that
helps you match or find other strings or sets of strings, using a
specialized syntax held in a pattern.
• Regular expressions are widely used in UNIX world.
• Especially used for detecting word pattern
re.match(pattern, string, flags=0) #flag uses | (OR)
re.search(pattern, string, flags=0)
re.sub(pattern, repl, string, max=0)
match checks for a match only at the beginning of the string,
not quite useful compared with search
Search checks for a match anywhere in the
string

105
>>> import re
>>> wordlist = ['hbasgoed', 'abhshooe', 'hbatisgoe', 'tbgoe', 'tbortgoe','abaisoed',
'abandoned', 'abased', 'abashed', 'abatised', 'abed', 'aborted','abaisogoe',
'abandongoe']
>>> [w for w in wordlist if re.search('ed$',w)]
['abaissed', 'abandoned', 'abased', 'abashed', 'abatised', 'abed', 'aborted']
• Detecting word pattern
Let's find words ending with ed using the regular expression
Let's find puzzle crossword for 8 letters with ‘a’ as its third letter and ‘y’ as sixth
>>> [w for w in wordlist if re.search('^..a..o..$',w)]
['hbasgoed', 'abaisoed']
. wildcard symbol match any single character
^ match beginning of line.
$ Matches end of line.

106
>>> wordlist = ['abjectly', 'adjuster', 'gold','golder','dejected', 'golf', 'dejectly',
'injector', 'majestic','hold', 'hole']
>>> [w for w in wordlist if re.search('^[ghi][mno][jlk][def]$',w)]
['gold', 'golf', 'hold', 'hole']
>>> [w for w in wordlist if re.search('^[ghi][mno][jlk][def]',w)]
['gold', 'golder', 'golf', 'injector', 'hold', 'hole']
Ranges and Closures
Let’s find 4 letters word in which the first ranges from ‘g’,’h’ or ‘I’, the second ‘m’, ‘n’ or ‘o’
As 4653 in T9 system
the regular expression: ^[ghi][mno][jlk][def]$
Content credited http://www.nltk.org/images/T9.png

107
>>> chat_words = ['miiinnee', 'mmmmmmmmiiiiiiiiinnnnnnnnneeeeeeee',
'mine','me','mingg','mi','ne' ]
>>> [w for w in chat_words if re.search('^m+i+n+e+$', w)]
['miiinnee', 'mmmmmmmmiiiiiiiiinnnnnnnnneeeeeeee', 'mine']
>>> [w for w in chat_words if re.search('^m*i*n*e*$', w)]
['miiinnee', 'mmmmmmmmiiiiiiiiinnnnnnnnneeeeeeee', 'mine', 'me', 'mi', 'ne']
Ranges and Closures
• + symbol means "one or more instances of the preceding item", which could be an
individual character like m, a set like [fed] or a range like [d-f]
• * symbol means "zero or more instances of the preceding item".
• The regular expression ^m*i*n*e*$ will match ^m+i+n+e+$
• e.g. me, min, and mmmmm.

108
Ranges and Closures
: escape a control character
{n,m}: Matches n or more occurrences of preceding expression
(): Groups regular expressions
python|ruby: match “python” or “ruby”
digitlist = ['123','331-points', '0.0085', 'Adopting','C$','0.05','abv','2016'
'0.1', 'JAP#','92.2','911-ppppppp''43643', '1978', 'US$','1.1253553','bread-and-
butter','PO@','10-year','1.14', 'Advancing','1.1650', 'savings-and-loan','1.17',
'1.0','10-day','331-bigpoints', 'Absorbed']
>>> [w for w in digitlist if re.search('(ed|ing)$', w)]
['Adopting', 'Advancing', 'Absorbed']
>>> [w for w in digitlist if re.search('^[0-9]+.[0-9]+$', w)]
['0.0085', '0.05', '20160.1', '92.2', '1.1253553', '1.14', '1.1650', '1.17', '1.0']
>>> [w for w in digitlist if re.search('^[A-Z]+$$', w)]
['C$', 'US$']
>>> [w for w in digitlist if re.search('^[0-9]{4}$', w)]
['1978']
>>> [w for w in digitlist if re.search('^[0-9]+-[a-z]{3,5}$', w)]
['10-year', '10-day']

109
>>> word1 = 'smartdog smatdog ldoedog ddwwlaj'
>>> re.findall('dog',word1)
['dog', 'dog', 'dog']
>>> word = 'supercalifragilisticexpialidocious'
>>> re.findall('[aeiou]', word)
['u', 'e', 'a', 'i', 'a', 'i', 'i', 'i', 'e', 'i', 'a', 'i', 'o', 'i', 'o',
'u']
Findall()
finds all (non-overlapping) matches of the given regular expression

110
http://www.tutorialspoint.com/python/python_reg_expressions.htm

111
http://www.tutorialspoint.com/python/python_reg_expressions.htm
option Flags
line = "Cats are smarter than dogsn dfssss www";
>>> bool(re.search( 'dogs$', line, re.I))
False
>>> bool(re.search( 'dogs$', line, re.I|re.M))
True

112
Deal with Chinese characters in python
>>> chinese_string = "恭喜發財"
>>> pattern = "發財"
>>> re.search(pattern,chinese_string)
<_sre.SRE_Match object; span=(2, 4), match='發財'>
>>> new_pattern = "新喜"
>>> new_c_string =
re.sub(pattern,new_pattern,chinese_string)
>>> print(new_c_string)
恭喜新喜
Found it
Substitute it with new
pattern

Before Unicode
113
In the beginning, computing was mainly centered in North America
and done in English. Characters were stored one-per-byte by using
either
a -> 01100001(x61)
In other parts of the world, different ways of storing their characters were
invented
• Japan: various JIS encodings
• Taiwan: BIG5
• China: GB 18030
ASCII (7 bits)
EBCDIC (8 bits) #mainly used in IBM mainframe

Unicode
114
Where do you use UNICODE?
- ANYWHERE

Unicode
115
Let’s open UTF-8 file
A string of byte
1 byte = 8bits
This string is encoded in UTF-8 format
http://farmdev.com/talks/unicode/

Unicode
116
ASCII – one chaterter is 1 byte
We can not encode it by ascii

Unicode
118
Python3
Python 3 always stores text strings as sequences of Unicode code points
• Str == Unicode string is now in default
• No Unicode type
http://wolfprojects.altervista.org/talks/unicode-and-python-3/

Unicode
119
IO FILE -> console -> IOFILE

Unicode
120
Reading and writing IO
with open('unicode.txt', encoding='utf-8') as f:
for line in f: print(repr(line))
with open("/tmp/sample.utf8", mode="w", encoding="utf8") as f:
for s in slist:
print(s, file=f)
# OR
open('filename', 'wb').write(s.encode('utf-8'))
Reading from UTF-8 encoded
file
Writing

Unicode
121
(Chinese)codepoint value <-> ByteString in UTF-8/16/BIG5 <->Unicode

Class
• Everything in Python is object
• Class is the template of a object, which is basically a collection of attributes
and methods as similar as variables and functions
• As functional program (FP) object oriented programming (OOP) is a
programming paradigm, making use of “object” (an instance) as main
building block to organize your code.
• The four major principles of object orientation are:
• Encapsulation
• Data Abstraction
• Polymorphism
• Inheritance

Class
• the recipe (class ) to bake a cake (an object
of this class)
• If you bake a cake you need ingredients and
instructions to bake the cake. Accordingly a
class needs variables and methods.
• There are class variables, which have the
same value in all methods (class variables
are also known as static member variables –
every object share only the one) and there
are instance variables, which have normally
different values for different objects.

Class
• Uses the class statement creates a new class definition
class Class_Name(object):
#sometodo

Class
• Employee Class (a blueprint to employee data)
class Employee:
"""Common base class for all employees"""
empCount = 0
def __init__(self, name, salary):
self.name = name
self.salary = salary
Employee.empCount += 1
def displayCount(self):
print("Total Employee %d" % Employee.empCount)
def displayEmployee(self):
print("Name : ", self.name, ", Salary: ", self.salary)
Class Variable
Classname.var
Memebr variable
Different in each object

Class
• Class Instantiation (create a object) & attribute access
>>> paul = Employee('Paul',10000)
>>> paul.displayCount()
Total Employee 1
>>> paul.displayEmployee()
Name : Paul , Salary: 10000
>>> paul.salary = 20000
>>> print(paul.salary)
20000

Class
• class-related BIFs
• dir(): show all supported methods and attributes
• help(): display class info
• Isinstance(x,y): check if x is the object of y
>>> dir(paul)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__',
'__format__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__',
'displayCount', 'displayEmployee', 'empCount', 'name', 'salary']
>>> paul.__doc__
'Common base class for all employees'
>>> >>> help(paul)
Help on Employee in module __main__ object:
class Employee(builtins.object)
| Common base class for all employees
| Methods defined here:
| __init__(self, name, salary)
| Initialize self. See help(type(self)) for accurate signature.
| displayCount(self)
>>> >>> isinstance(paul,Employee)
True

__init__()
• Instantiation: use constructor to create instance(實體) which run
__init__() method. We can override what we need by redefine it
self.name = name
>>> type(paul)
<class '__main__.Employee'>
>>> type(Employee)
<class 'type'>
Self is a reserved word
To represent itself
Here is “paul” afterward
the type of paul is Employee class
The type of Employee is Type class

__del__()
• Python interpreter invokes this func when the object is no longer
used – release memory
class Employee:
empCount = 0
self.name = name
def __str__(self):
return str(self.name)
def __del__(self):
print("delcalled:"+ self.__str__())
>>> paul = "lol"
delcalled:Paul

Encapsulation
• encapsulation is the mechanism for restricting the access to some of an object's
components, this means that the internal representation of an object can't be seen from
outside of the objects definition.
• Access to this data is typically only achieved through special methods: Getters and Setters.
http://www.python-course.eu/object_oriented_programming.php

Encapsulation
• By default, every attribute is public as we saw before.
• you can make them either protected or private
• start with an underscore character "_" for protected member
• Start with double underscore “__” for instance variable
http://www.python-course.eu/object_oriented_programming.php
Python doesn’t really support for information hiding.
Instead Python use name mangling

Encapsulation
class Encapsulation(object):
def __init__(self, a, b, c):
self.public = a
self._protected = b
self.__private = c
>>> from encapsulation import Encapsulation
>>> x = Encapsulation(11,13,17)
>>> x.public
11
>>> x._protected
13
>>> x._protected = 23
>>> x._protected
23
>>> x.__private
File "<stdin>", line 1, in <module>
AttributeError: 'Encapsulation' object has no attribute '__private'
Can be accessed inside/outside
Supposed to be retracted from
outside but python still allow
you to access
This changes virtually nothing
Can’t be accessed outsie

Class Method/Attribute
• Use decorator “@classmethod” with parameters “cls” to tell this is class methods
class Employee:
empCount = 0
self.name = name
@classmethod
def displayCount(cls):
print("Total Employee %d" % cls.empCount)

Exercise 13 – create your first class
• Enhance Employee class to support more operations:
• To support the below attributes:
• Instance attributes: name, grades, salary, position
• class attribute: managerCount , teacherCount, otherCount,
empCount
• To support the operations below:
• methods
• display_failure_count(): return sum of the failure class (the failure score is the subject
‘s score smaller than 60)
• add_grade(grade): to put class’s score like 100, 80 , 50
• Class Methods
• display_position_count(position)
• display_employee_count()

Exercise 13 – create your first class
paul = Employee('Paul',15000,"teacher")
jack = Employee('Jack',20000,"manager")
joy = Employee('Joy',15000,"writter")
mike = Employee('mike',20000,"teacher")
#add grade
paul.add_grade(100)
paul.add_grade(80)
paul.add_grade(90)
print("%s average grade:%d" % (paul.get_name(), paul.display_avg_grade()))
#add failure grade
paul.add_grade(40)
paul.add_grade(30)
paul.add_grade(58)
print("%s failed clount:%d" % (paul.get_name(), paul.display_failure_count()))
#check statistic by accessing class attribute by class methods
Employee.display_employee_count()
Employee.display_position_count("manager")
Employee.display_position_count("teacher")
Employee.display_position_count("other")
To test your employee class

Answer
Check employee.py in exercise_13

Inheritance
Inheritance is used to indicate that one class will get most or all of its features from a
parent class. This happens implicitly whenever you write class Truck(Vehicle),
which says "Make a class Truck that inherits from Vehicle.“
The Vehicle class is the superclass
of Car and Truck. Car and Truck are subclasses of Vehicle.
Content credited: http://tutorials.jenkov.com/java/inheritance.html
classSubDemo(Demo):
#dosomething

Inheritance
• Implicit Inheritance:
• child will inherit all of its behavior from Parent.
class Parent(object):
def implicit(self):
print "PARENT implicit()"
class Child(Parent):
pass
>>> dad = Parent()
>>> son = Child()
>>> dad.implicit()
>>> son.implicit()
PARENT implicit()
PARENT implicit()
all subclasses (i.e., Child) will
automatically get those features
from base class (i.e., Parent)

Inheritance
• Override Explicitly:
• sometimes you want the child to behave differently. In this case you want to
override the function in the child, effectively replacing the functionality
def override(self):
print "PARENT override()"
def override(self):
print "CHILD override()"
>>> dad = Parent()
>>> son = Child()
>>> dad.override()
>>> son.override()
PARENT override()
CHILD override()
define a function with the same
name in child

Inheritance
• Alter
• overriding where you want to alter the behavior before or after the Parent class's
version runs.
def altered(self):
print "PARENT altered()"
def altered(self):
print "CHILD, BEFORE PARENT altered()"
super(Child, self).altered()
print "CHILD, AFTER PARENT altered()"
>>> dad = Parent()
>>> son = Child()
>>> dad.altered()
>>> son.altered()
PARENT altered()
CHILD, BEFORE PARENT altered()
PARENT altered()
CHILD, AFTER PARENT altered()
use a Python built-in function
named super to get the Parent
version to call.

Inheritance
Multiple Inheritance
A class can inherit from more than one class. All the super-classes are put in parenthesis as a
comma separated list behind the class name.

Inheritance
class CL1(object):
def __init__(self):
print("class 1 init")
def print_CL1(self):
print("class 1")
class CL2(object):
def __init__(self):
print "class 2 init"
def print_CL2(self):
print("class 2")
class CL3(CL2, CL1):
pass
>>> instance = CL3()
class2 init
>>> instance.print_CL1()
class 1
>>> instance.print_CL2()
class 2
Content credited: http://scottlobdell.me/2015/06/multiple-inheritance-python/

Inheritance
class CL1(object):
def __init__(self):
super(CL1, self).__init__()
print("class 1 init “)
class CL2(object):
def __init__(self):
print("class 2 init“)
class CL3(CL1, CL2):
def __init__(self):
print("class 3 init“)
>>> instance = CL3()
class 2 init
class 1 init
class 3 init
adding “super” in every
constructor to every constructor
gets called:

Polymorphism
Polymorphism is the ability to present the same interface for differing underlying
forms. Polymorphic functions or methods can be applied to arguments of different
types, and they can behave differently depending on the type of the arguments to
which they are applied
def f(x, y):
print("values: ", x, y)
f(42,43)
f(42, 43.7)
f(42.3,43)
f(42.0, 43.9)
f([3,5,6],(3,5))
#include <iostream>
using namespace std;
void f(int x, int y ) {
cout << "values: " << x << ", " << x << endl;
}
void f(int x, double y ) {
}
void f(double x, int y ) {
}
void f(double x, double y ) {
}
int main()
{
f(42, 43); f(42, 43.7); f(42.3,43);f(42.0,
43.9);
}
In c++. have to overload f for the
various type combinations

Polymorphism
a = "alfa"
b = (1, 2, 3, 4)
c = ['o', 'm', 'e', 'g', 'a']
>>>print(a[2])
>>>Print(b[1])
>>>Print(c[3])
f
2
g
Python uses polymorphism
extensively in built-in types. Here
we use the same indexing operator
for three different data types.

Polymorphism
class Bear(object):
def sound(self):
print "Groarrr"
class Dog(object):
def sound(self):
print "Woof woof!"
def makeSound(animalType):
animalType.sound()
bearObj = Bear()
dogObj = Dog()
makeSound(bearObj)
makeSound(dogObj)
Content credited: https://pythonspot.com/en/poylmorphism/
Polymorphism in Python with a function:

Polymorphism
class Animal:
def __init__(self, name=''):
self.name = name
def talk(self):# Abstract method, defined by convention only
raise NotImplementedError("Subclass must implement abstract method")
class Cat(Animal):
def talk(self):
print("Meow!")
class Dog(Animal):
def talk(self):
print("Woof!")
c = Cat("Missy")
c.talk()
d = Dog("Rocky")
d.talk()
Content credited: http://zetcode.com/lang/python/oop/
Polymorphism with abstract class

Reference
Python Wiki/Docs/Community:
• https://docs.python.org/3/
Python learning online resources:
• http://www.nltk.org
• http://www.python-course.eu
• https://pythonspot.com/en/poylmorphism
• https://newcircle.com/bookshelf/python_fundamentals_tutorial/functional_programming
• http://thecodeship.com/patterns/guide-to-python-function-decorators/
• https://www.youtube.com/watch?v=goypZR_lQ7I
• https://www.youtube.com/watch?v=E8I19uA-wGY
• http://tutorials.jenkov.com/java
• http://scottlobdell.me/2015/06/multiple-inheritance-python/
• http://zetcode.com/lang/python/oop

Reference
Performance measure
• http://pynash.org/2013/03/06/timing-and-profiling/
• https://www.huyng.com/posts/python-performance-analysis
• http://packages.python.org/line_profiler/
• https://pypi.python.org/pypi/memory_profiler
rsplit
• http://python-reference.readthedocs.org/en/latest/docs/str/rsplit.html

Generators Function
• The function only executes on next()
x = firstn_generator(10)
>>> x
>>>next(x)
0
>>>next(x)
1
File "<stdin>", line 1, in ?
StopIteration

release_python_day2_slides_201606.pdf

Recommended

Recommended

More Related Content

Similar to release_python_day2_slides_201606.pdf

Similar to release_python_day2_slides_201606.pdf (20)

More from Paul Yang

More from Paul Yang (20)

Recently uploaded

Recently uploaded (20)

release_python_day2_slides_201606.pdf