Transcript of "Python cookbook, 2nd edition (2005)"
Python Cookbook, 2nd Edition
By David Ascher, Alex Martelli, Anna Ravenscroft
Pub Date: March 2005
Like its predecessor, the new edition offers a collection of solutions to
problems that Python programmers face everyday. Updated for Python
2.4, it now includes over 200 recipes that range from simple tasks, such
as working with dictionaries and list comprehensions, to complex tasks,
such as monitoring a network and building a templating system.
Python Cookbook, 2nd Edition
By David Ascher, Alex Martelli, Anna Ravenscroft
Pub Date: March 2005
The Design of the Book
The Implementation of the Book
Using the Code from This Book
Conventions Used in This Book
How to Contact Us
Recipe 1.1. ProcessingaStringOneCharacterataTime
Recipe 1.2. ConvertingBetweenCharactersandNumericCodes
Recipe 1.3. TestingWhetheranObjectIsString-like
Recipe 1.4. AligningStrings
Recipe 1.5. TrimmingSpacefromtheEndsofaString
Recipe 1.6. CombiningStrings
Recipe 1.7. ReversingaStringbyWordsorCharacters
Recipe 1.8. CheckingWhetheraStringContainsaSetofCharacters
Recipe 1.9. SimplifyingUsageofStrings'translateMethod
Recipe 1.10. FilteringaStringforaSetofCharacters
Recipe 1.11. CheckingWhetheraStringIsTextorBinary
Recipe 1.12. ControllingCase
Recipe 1.13. AccessingSubstrings
Recipe 1.14. ChangingtheIndentationofaMultilineString
Recipe 1.15. ExpandingandCompressingTabs
Recipe 1.16. InterpolatingVariablesinaString
Recipe 1.17. InterpolatingVariablesinaStringinPython2.4
Recipe 1.18. ReplacingMultiplePatternsinaSinglePass
Recipe 1.19. CheckingaStringforAnyofMultipleEndings
Recipe 1.20. HandlingInternationalTextwithUnicode
Recipe 1.21. ConvertingBetweenUnicodeandPlainStrings
This book is not a typical O'Reilly book, written as a cohesive manuscript by one or two authors.
Instead, it is a new kind of booka bold attempt at applying some principles of open source
development to book authoring. Over 300 members of the Python community contributed
materials to this book. In this Preface, we, the editors, want to give you, the reader, some
background regarding how this book came about and the processes and people involved, and
some thoughts about the implications of this new form.
The Design of the Book
In early 2000, Frank Willison, then Editor-in-Chief of O'Reilly & Associates, contacted me (David
Ascher) to find out if I wanted to write a book. Frank had been the editor for Learning Python,
which I cowrote with Mark Lutz. Since I had just taken a job at what was then considered a Perl
shop (ActiveState), I didn't have the bandwidth necessary to write another book, and plans for
the project were gently shelved. Periodically, however, Frank would send me an email or chat
with me at a conference regarding some of the book topics we had discussed. One of Frank's
ideas was to create a Python Cookbook, based on the concept first used by Tom Christiansen and
Nathan Torkington with the Perl Cookbook. Frank wanted to replicate the success of the Perl
Cookbook, but he wanted a broader set of people to provide input. He thought that, much as in a
real cookbook, a larger set of authors would provide for a greater range of tastes. The quality, in
his vision, would be ensured by the oversight of a technical editor, combined with O'Reilly's
editorial review process.
Frank and Dick Hardt, ActiveState's CEO, realized that Frank's goal could be combined with
ActiveState's goal of creating a community site for open source programmers, called the
ActiveState Programmer's Network (ASPN). ActiveState had a popular web site, with the
infrastructure required to host a wide variety of content, but it wasn't in the business of creating
original content. ActiveState always felt that the open source communities were the best sources
of accurate and up-to-date content, even if sometimes that content was hard to find.
The O'Reilly and ActiveState teams quickly realized that the two goals were aligned and that a
joint venture would be the best way to achieve the following key objectives:
Creating an online repository of Python recipes by Python programmers for Python
Publishing a book containing the best of those recipes, accompanied by overviews and
background material written by key Python figures
Learning what it would take to create a book with a different authoring model
At the same time, two other activities were happening. First, those of us at ActiveState, including
Paul Prescod, were actively looking for "stars" to join ActiveState's development team. One of
the candidates being recruited was the famous (but unknown to us, at the time) Alex Martelli.
Alex was famous because of his numerous and exhaustive postings on the Python mailing list,
where he exhibited an unending patience for explaining Python's subtleties and joys to the
increasing audience of Python programmers. He was unknown because he lived in Italy and,
since he was a relative newcomer to the Python community, none of the old Python hands had
ever met himtheir paths had not happened to cross back in the 1980s when Alex lived in the
United States, working for IBM Research and enthusiastically using and promoting other high-
level languages (at the time, mostly IBM's Rexx).
ActiveState wooed Alex, trying to convince him to move to Vancouver. We came quite close, but
his employer put some golden handcuffs on him, and somehow Vancouver's weather couldn't
compete with Italy's. Alex stayed in Italy, much to my disappointment. As it happened, Alex was
also at that time negotiating with O'Reilly about writing a book. Alex wanted to write a cookbook,
but O'Reilly explained that the cookbook was already signed. Later, Alex and O'Reilly signed a
contract for Python in Nutshell.
The second ongoing activity was the creation of the Python Software Foundation. For a variety of
reasons, best left to discussion over beers at a conference, everyone in the Python community
wanted to create a non-profit organization that would be the holder of Python's intellectual
property, to ensure that Python would be on a legally strong footing. However, such an
organization needed both financial support and buy-in from the Python community to be
Given all these parameters, the various parties agreed to the following plan:
ActiveState would build an online cookbook, a mechanism by which anyone could submit a
recipe (i.e., a snippet of Python code addressing a particular problem, accompanied by a
discussion of the recipe, much like a description of why one should use cream of tartar
when whipping egg whites). To foster a community of authors and encourage peer review,
the web site would also let readers of the recipes suggest changes, ask questions, and so
As part of my ActiveState job, I would edit and ensure the quality of the recipes. Alex
Martelli joined the project as a co-editor when the material was being prepared for
publication, and, with Anna Martelli Ravenscroft, took over as primary editor for the second
O'Reilly would publish the best recipes as the Python Cookbook.
In lieu of author royalties for the recipes, a portion of the proceeds from the book sales
would be donated to the Python Software Foundation.
The Implementation of the Book
The online cookbook (at http://aspn.activestate.com/ASPN/Cookbook/Python/) was the entry
point for the recipes. Users got free accounts, filled in a form, and presto, their recipes became
part of the cookbook. Thousands of people read the recipes, and some added comments, and so,
in the publishing equivalent of peer review, the recipes matured and grew. While it was
predictable that the chance of getting your name in print would get people attracted to the online
cookbook, the ongoing success of the cookbook, with dozens of recipes added monthly and more
and more references to it on the newsgroups, is a testament to the value it brings to the
readersvalue which is provided by the recipe authors.
Starting from the materials available on the site, the implementation of the book was mostly a
question of selecting, merging, ordering, and editing the materials. A few more details about this
part of the work are in the "Organization" section of this Preface.
Using the Code from This Book
This book is here to help you get your job done. In general, you may use the code in this book in
your programs and documentation. You do not need to contact us for permission unless you're
reproducing a significant portion of the code. For example, writing a program that uses several
chunks of code from this book does not require permission. Selling or distributing a CD-ROM of
code taken from O'Reilly books does require permission. Answering a question by citing this book
and quoting example code does not require permission. Incorporating a significant amount of
code from this book into your product's documentation does require permission. We appreciate,
but do not require, attribution. An attribution usually includes the title, author, publisher, and
ISBN. For example: "Python Cookbook, 2d ed., by Alex Martelli, Anna Martelli Ravenscroft, and
David Ascher (O'Reilly Media, 2005) 0-596-00797-3." If you feel your use of code from this book
falls outside fair use or the permission given above, feel free to contact us at
We expect that you know at least some Python. This book does not attempt to teach Python as a
whole; rather, it presents some specific techniques and concepts (and occasionally tricks) for
dealing with particular tasks. If you are looking for an introduction to Python, consider some of
the books described in the Further Reading section of this Preface. However, you don't need to
know a lot of Python to find this book helpful. Chapters include recipes demonstrating the best
techniques for accomplishing some elementary and general tasks, as well as more complex or
specialized ones. We have also added sidebars, here and there, to clarify certain concepts which
are used in the book and which you may have heard of, but which might still be unclear to you.
However, this is definitely not a book just for beginners. The main target audience is the whole
Python community, mostly made up of pretty good programmers, neither newbies nor wizards.
And if you do already know a lot about Python, you may be in for a pleasant surprise! We've
included recipes that explore some the newest and least well-known areas of Python. You might
very well learn a few thingswe did! Regardless of where you fall along the spectrum of Python
expertise, and more generally of programming skill, we believe you will get something valuable
from this book.
If you already own the first edition, you may be wondering whether you need this second edition,
too. We think the answer is "yes." The first edition had 245 recipes; we kept 146 of those (with
lots of editing in almost all cases), and added 192 new ones, for a total of 338 recipes in this
second edition. So, over half of the recipes in this edition are completely new, and all the recipes
are updated to apply to today's Pythonreleases 2.3 and 2.4. Indeed, this update is the main
factor which lets us have almost 100 more recipes in a book of about the same size. The first
edition covered all versions from 1.5.2 (and sometimes earlier) to 2.2; this one focuses firmly on
2.3 and 2.4. Thanks to the greater power of today's Python, and, even more, thanks to the fact
that this edition avoids the "historical" treatises about how you had to do things in Python
versions released 5 or more years ago, we were able to provide substantially more currently
relevant recipes and information in roughly the same amount of space.
This book has 20 chapters. Each chapter is devoted to a particular kind of recipe, such as
algorithms, text processing, databases, and so on. The 1st edition had 17 chapters. There have
been improvements to Python, both language and library, and to the corpus of recipes the
Python community has posted to the cookbook site, that convinced us to add three entirely new
chapters: on the iterators and generators introduced in Python 2.3; on Python's support for time
and money operations, both old and new; and on new, advanced tools introduced in Python 2.2
and following releases (custom descriptors, decorators, metaclasses). Each chapter contains an
introduction, written by an expert in the field, followed by recipes selected from the online
cookbook (in some casesabout 5% of this book's recipesa few new recipes were specially written
for this volume) and edited to fit the book's formatting and style requirements. Alex (with some
help from Anna) did the vast majority of the selectiondetermining which recipes from the first
edition to keep and update, and selecting new recipes to add, or merge with others, from the
nearly 1,000 available on the site (so, if a recipe you posted to the cookbook site didn't get into
this printed edition, it's his fault!). He also decided which subjects just had to be covered and
thus might need specially written recipesalthough he couldn't manage to get quite all of the
specially written recipes he wanted, so anything that's missing, and wasn't on the cookbook site,
might not be entirely his fault.
Once the selection was complete, the work turned to editing the recipes, and to merging multiple
recipes, as well as incorporating important contents from many significant comments posted
about the recipes. This proved to be quite a challenge, just as it had been for the first edition,
but even more so. The recipes varied widely in their organization, level of completeness, and
sophistication. With over 300 authors involved, over 300 different "voices" were included in the
text. We have striven to maintain a variety of styles to reflect the true nature of this book, the
book written by the entire Python community. However, we edited each recipe, sometimes quite
considerably, to make it as accessible and useful as possible, ensuring enough uniformity in
structure and presentation to maximize the usability of the book as a whole. Most recipes, both
from the first edition and from the online site, had to be updated, sometimes heavily, to take
advantage of new tools and better approaches developed since those recipes were originally
posted. We also carefully reconsidered (and slightly altered) the ordering of chapters, and the
placement and ordering of recipes within chapters; our goal in this reordering was to maximize
the book's usefulness for both newcomers to Python and seasoned veterans, and, also, for both
readers tackling the book sequentially, cover to cover, and ones just dipping in, in "random
access" fashion, to look for help on some specific area.
While the book should thus definitely be accessible "by hops and jumps," we nevertheless believe
a first sequential skim will amply repay the modest time you, the reader, invest in it. On such a
skim, skip every recipe that you have trouble following or that is of no current interest to you.
Despite the skipping, you'll still get a sense of how the whole book hangs together and of where
certain subjects are covered, which will stand you in good stead both for later in-depth
sequential reading, if that's your choice, and for "random access" reading. To further help you
get a sense of what's where in the book, here's a capsule summary of each chapter's contents,
and equally capsule bios of the Python experts who were so kind as to take on the task of writing
the chapters' "Introduction" sections.
Chapter 1, introduction by Fred L. Drake, Jr.
This chapter contains recipes for manipulating text in a variety of ways, including
combining, filtering, and formatting strings, substituting variables throughout a text
document, and dealing with Unicode.
Fred Drake is a member of the PythonLabs group, working on Python development. A
father of three, Fred is best known in the Python community for single-handedly
maintaining the official documentation. Fred is a co-author of Python & XML (O'Reilly).
Chapter 2, introduction by Mark Lutz
This chapter presents techniques for working with data in files and for manipulating files
and directories within the filesystem, including specific file formats and archive formats
such as tar and zip.
Mark Lutz is well known to most Python users as the most prolific author of Python books,
including Programming Python, Python Pocket Reference, and Learning Python (all from
O'Reilly), which he co-authored with David Ascher. Mark is also a leading Python trainer,
spreading the Python gospel throughout the world.
Chapter 3, introduction by Gustavo Niemeyer and Facundo Batista
This chapter (new in this edition) presents tools and techniques for working with dates,
times, decimal numbers, and some other money-related issues.
Gustavo Niemeyer is the author of the third-party dateutil module, as well as a variety of
other Python extensions and projects. Gustavo lives in Brazil. Facundo Batista is the author
of the Decimal PEP 327, and of the standard library module decimal, which brought
floating-point decimal support to Python 2.4. He lives in Argentina. The editors were
delighted to bring them together for this introduction.
Chapter 4, introduction by David Ascher
This chapter includes recipes for many common techniques that can be used anywhere, or
that don't really fit into any of the other, more specific recipe categories.
David Ascher is a co-editor of this volume. David's background spans physics, vision
research, scientific visualization, computer graphics, a variety of programming languages,
co-authoring Learning Python (O'Reilly), teaching Python, and these days, a slew of
technical and nontechnical tasks such as managing the ActiveState team. David also gets
roped into organizing Python conferences on a regular basis.
Chapter 5, introduction by Tim Peters
This chapter covers techniques for searching and sorting in Python. Many of the recipes
explore creative uses of the stable and fast list.sort in conjunction with the decorate-
sort-undecorate (DSU) idiom (newly built in with Python 2.4), while others demonstrate
the power of heapq, bisect, and other Python searching and sorting tools.
Tim Peters, also known as the tim-bot, is one of the mythological figures of the Python
world. He is the oracle, channeling Guido van Rossum when Guido is busy, channeling the
IEEE-754 floating-point committee when anyone asks anything remotely relevant, and
appearing conservative while pushing for a constant evolution in the language. Tim is a
member of the PythonLabs team.
Chapter 6, introduction by Alex Martelli
This chapter offers a wide range of recipes that demonstrate the power of object-oriented
programming with Python, including fundamental techniques such as delegating and
controlling attribute access via special methods, intermediate ones such as the
implementation of various design patterns, and some simple but useful applications of
advanced concepts, such as custom metaclasses, which are covered in greater depth in
Alex Martelli, also known as the martelli-bot, is a co-editor of this volume. After almost a
decade with IBM Research, then a bit more than that with think3, inc., Alex now works as a
freelance consultant, most recently for AB Strakt, a Swedish Python-centered firm. He also
edits and writes Python articles and books, including Python in a Nutshell (O'Reilly) and,
occasionally, research works on the game of contract bridge.
Chapter 7, introduction by Aaron Watters
This chapter presents Python techniques for persistence, including serialization approaches
and interaction with various databases.
Aaron Watters was one of the earliest advocates of Python and is an expert in databases.
He's known for having been the lead author on the first book on Python (Internet
Programming with Python, M&T Books, now out of print), and he has authored many
widely used Python extensions, such as kjBuckets and kwParsing. Aaron currently works
as a freelance consultant.
Chapter 8, introduction by Mark Hammond
This chapter includes a collection of recipes that assist with the debugging and testing
process, from customizing error logging and traceback information, to unit testing with
custom modules, unittest and doctest.
Mark Hammond is best known for his work supporting Python on the Windows platform.
With Greg Stein, he built an incredible library of modules interfacing Python to a wide
variety of APIs, libraries, and component models such as COM. He is also an expert
designer and builder of developer tools, most notably Pythonwin and Komodo. Finally,
Mark is an expert at debugging even the most messy systemsduring Komodo
development, for example, Mark was often called upon to debug problems that spanned
is also co-author, with Andy Robinson, of Python Programming on Win32 (O'Reilly).
Chapter 9, introduction by Greg Wilson
This chapter covers a variety of techniques for concurrent programming, including threads,
queues, and multiple processes.
Greg Wilson writes children's books, as well as books on parallel programming and data
crunching. When he's not doing that, he's a contributing editor with Doctor Dobb's Journal,
an adjunct professor in Computer Science at the University of Toronto, and a freelance
software developer. Greg was the original driving force behind the Software Carpentry
project, and he recently received a grant from the Python Software Foundation to develop
Pythonic course material for computational scientists and engineers.
Chapter 10, introduction by Donn Cave
This chapter includes recipes for a number of common system administration tasks, from
generating passwords and interacting with the Windows registry, to handling mailbox and
web server issues.
Donn Cave is a software engineer at the University of Washington's central computer site.
Over the years, Donn has proven to be a fount of information on comp.lang.python on all
matters related to system calls, Unix, system administration, files, signals, and the like.
Chapter 11, introduction by Fredrik Lundh
This chapter contains recipes for common GUI tasks, mostly with Tkinter, but also a
smattering of wxPython, Qt, image processing, and GUI recipes specific to Jython (for
JVMJava Virtual Machine), Mac OS X, and IronPython (for dotNET).
Fredrik Lundh, also known as the eff-bot, is the CTO of Secret Labs AB, a Swedish Python-
focused company providing a variety of products and technologies. Fredrik is the world's
leading expert on Tkinter (the most popular GUI toolkit for Python), as well as the main
author of the Python Imaging Library (PIL). He is also the author of Python Standard
Library (O'Reilly), which is a good complement to this volume and focuses on the modules
in the standard Python library. Finally, he is a prolific contributor to comp.lang.python,
helping novices and experts alik