Python & Perl: Lecture 20

Python & Perl Lecture 20 Vladimir Kulyukin Department of Computer Science Utah State Universitywww.youtube.com/vkedco www.vkedco.blogspot.com

Outline ● File and Directory Processing ● Piping Generatorswww.youtube.com/vkedco www.vkedco.blogspot.com

Quiz Answer class TwoDPoint: def __init__(self, x=0, y=0): self._x, self._y = x, y @property def x(self): return self._x @property def y(self): return self._y @x.setter def x(self, xv): self._x = xv @y.setter def y(self, yv): self._y = yv @x.deleter def x(self): del self._x @y.deleter def y(self): del self._ywww.youtube.com/vkedco www.vkedco.blogspot.com

File & Directory Processingwww.youtube.com/vkedco www.vkedco.blogspot.com

Exploring Local Directory System ● A common scenario: find all files whose name matches a pattern that may be scattered all over a local directory system and process them ● Example 01: find all .gz archives and back them up on an external drive ● Example 02: find all .txt files and index them to be searchable ● Example 03: find all .jpg/.bmp/.png files and create an catalogue mapping a file name to its exact location on the local drivewww.youtube.com/vkedco www.vkedco.blogspot.com

OS.WALK() ● os.walk() is the workhorse of processing local file systems ● This function uses generators to walk through a local file system import os ## given some directory, startdir, os.walk ## returns a generator object that yields triples ## of the form (path, dirlist, filelist) for path, dirlist, filelist in os.walk(startdir): ## path is current directory ## dirlist is list of subdirectories of path ## filelist is list of files in pathwww.youtube.com/vkedco www.vkedco.blogspot.com

Example 01 Write a generator factory that produces generators to traverse a given directory and yield files whose names match a given pattern.www.youtube.com/vkedco www.vkedco.blogspot.com

Example 01 import os, fnmatch def file_name_factory(fnpattern, searchdir): ## 1. create a generator object for generating ## (path, dirlist, filelist) tuples for curdir, dirlist, filelist in os.walk(searchdir): ## 2. return the subset of filelist that match fnpattern for filename in fnmatch.filter(filelist, fnpattern): ## 3. yield one by one yield os.path.join(curdir, filename)www.youtube.com/vkedco www.vkedco.blogspot.com

Example 01 ### pygen is a generator object of the .py files in ### D:/Dropbox/MyShare/Python/Perl/ >>> pygen = file_name_factory(*.py, D:/Dropbox/MyShare/PythonPerl/) >>> pygen.next() D:/Dropbox/MyShare/PythonPerl/build_huffman_tree.py >>> pygen.next() D:/Dropbox/MyShare/PythonPerl/dropbox_class_announcement.py >>> pygen.next() D:/Dropbox/MyShare/PythonPerl/huffman_coding.pywww.youtube.com/vkedco www.vkedco.blogspot.com

Example 02 Write a generator factory that produces generators to traverse a given directory and yield files whose names match a given pattern provided that their size is above a given threshold.www.youtube.com/vkedco www.vkedco.blogspot.com

Example 02 def file_name_condition_factory(fnpattern, startdir, condition): dirwalker = os.walk(startdir) for curdir, dirlist, filelist in dirwalker: fnfilter = (os.path.join(curdir, fn) for fn in filelist if fnmatch.fnmatch(fn, fnpattern) and condition(os.path.join(curdir, fn))) for fn in fnfilter: yield fn def file_condition(fpath, size=100): return os.path.getsize(fpath) > sizewww.youtube.com/vkedco www.vkedco.blogspot.com

Example 02 ### odtfiles is a generator that finds all Open Office Writer files greater ### than 100 bytes >>> odtfiles = file_name_condition_factory(*.odt, D:/Dropbox/MyShare/PythonPerl/, file_condition) >>> odtfiles.next() D:/Dropbox/MyShare/PythonPerl/lectures/quizes/cs3430_quiz_03.odt >>> odtfiles.next() D:/Dropbox/MyShare/PythonPerl/lectures/quizes/cs3430_quiz_04_s12.odt >>> odtfiles.next() D:/Dropbox/MyShare/PythonPerl/lectures/quizes/cs3430_quiz_05_s12.odtwww.youtube.com/vkedco www.vkedco.blogspot.com

Example 03 Write a class DirStats with a method getStats that takes a directory and returns the directorys total size (in bytes), total number of subdirectories, and total number of files in the directory.www.youtube.com/vkedco www.vkedco.blogspot.com

Example 03 import os from os.path import join, getsize class DirStats: def __init__(self): self.totalSize, self.numDirs, self.numFiles = 0, 0, 0 def getStats(self, dirname): if os.path.isfile(dirname): return for root, dirs, files in os.walk(dirname): self.numDirs += len(dirs); self.numFiles += len(files) for f in files: self.totalSize += getsize(join(root, f)) return (self.totalSize, self.numDirs, self.numFiles)www.youtube.com/vkedco www.vkedco.blogspot.com

Example 04 Write a function that extracts the lines with url references from all .txt files in a given directory.www.youtube.com/vkedco www.vkedco.blogspot.com

Example 04: Solution ● Get a generator that yields all .txt in a given search directory (we can use file_name_factory) ● Get a generator that yields opened input streams from a generator of file names ● Get a generator that yields lines from a generator of opened input streams ● Get a generator that yields lines that match a url pattern from a generator of lines ● Take the last generator and write its yields into a filewww.youtube.com/vkedco www.vkedco.blogspot.com

Example 04 First, we compile two URL patterns (neither comprehensive nor efficient): import re urlstr01 = r(https|http)://www.([a-z]|[0-9]|[A-Z])+.(org|com|html|htm|net|edu) urlstr02 = rwww.([a-z]|[0-9]|[A-Z])+.(org|com|html|htm|net|edu) urlpat01 = re.compile(urlstr01) urlpat02 = re.compile(urlstr02)www.youtube.com/vkedco www.vkedco.blogspot.com

Example 04 Second, we get a generator that yields open input streams from a generator of filenames def text_instream_factory(gen_file_name): for fn in filenames: yield open(fn, r)www.youtube.com/vkedco www.vkedco.blogspot.com

Example 04 Third, we get a generator that yields lines from a generator of open input streams: def text_line_factory(gen_text_instream): for instrm in gen_text_instream: for line in instrm: yield line instrm.close()www.youtube.com/vkedco www.vkedco.blogspot.com

Example 04 Fourth, we get a generator that yields lines that match our urlpat from a generator of lines: def url_line_factory(lines): for ln in lines: if re.search(urlpat01, ln) or re.search(urlpat02, ln): yield lnwww.youtube.com/vkedco www.vkedco.blogspot.com

Example 04 Fifth, we put it all together: def extract_urls(startdir, resultfile): gen_file_name = file_name_factory(*.txt, startdir) gen_text_instream = text_instream_factory(gen_file_name) gen_text_line = text_line_factory(gen_text_instream) gen_url_line = url_line_factory(gen_text_line) with open(resultfile, w) as outfile: for url_line in gen_url_line: outfile.write(url_line+n) outfile.flush()www.youtube.com/vkedco www.vkedco.blogspot.com

Example 04 How Does it All Work? We have a generator pipeline: each generator feeds its output to the next one. next() next() next() gen_file_name gen_text_instream gen_text_line gen_url_line next() for url_line in gen_url_line: outfile.write(url_line+n)www.youtube.com/vkedco www.vkedco.blogspot.com

Source Code http://vkedco.blogspot.com/2012/03/python-perl-lecture-20.htmlwww.youtube.com/vkedco www.vkedco.blogspot.com

Feedback Bugs, comments to vladimir dot kulyukin at usu dot eduwww.youtube.com/vkedco www.vkedco.blogspot.com

Reading & References ● www.python.org ● www.dabeaz.com/generators ● Ch. 09, 11 in “Beginning Python: From Novice to Professional” by M. Hetlandwww.youtube.com/vkedco www.vkedco.blogspot.com

http://vkedco.blogspot.com	205
http://www.vkedco.blogspot.com	71
http://vkedco.blogspot.in	14
http://www.vkedco.blogspot.in	9
http://vkedco.blogspot.de	8
http://localhost	6
http://vkedco.blogspot.fr	3
http://vkedco.blogspot.jp	3
http://vkedco.blogspot.hu	2
http://vkedco.blogspot.ch	2
http://www.vkedco.blogspot.com.es	2
http://vkedco.blogspot.com.es	2
http://vkedco.blogspot.com.br	2
http://vkedco.blogspot.kr	1
http://vkedco.blogspot.pt	1
http://vkedco.blogspot.com.au	1
http://vkedco.blogspot.com.ar	1

Python & Perl: Lecture 20

by Vladimir Kulyukin

Accessibility

Categories

Upload Details

Usage Rights

17 Embeds 333

Statistics