Your SlideShare is downloading. ×
The Art of Unix Programming     by Eric Steven Raymond
The Art of Unix Programmingby Eric Steven RaymondCopyright © 2003 Eric S. RaymondThis book and its on-line version are dis...
DedicationTo Ken Thompson and Dennis Ritchie, because you inspired me.                                             i
Table of Contents     Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...
The Art                   of Unix                   Program-                   ming            Rule of Robustness: Robustn...
The Art                             of Unix                             Program-                             ming         ...
The Art of Unix Programming           Case Study: Unix Password File Format . . . . . . . . . . . . . . . . . . . . . . . ...
The Art                   of Unix                   Program-                   ming7. Multiprogramming . . . . . . . . . ....
The Art                   of Unix                   Program-                   ming            Case Study: ascii . . . . ....
The Art of Unix                   Programming                   The ed pattern . . . . . . . . . . . . . . . . . . . . . ....
The Art                    of Unix                    Program-                    ming      Unix’s Cornucopia of Languages...
The Art of Unix Programming                  Emacs and Runtime Debugging . . . . . . . . . . . . . . . . . . . . . . . . ....
The Art of Unix Programming      The Zoo of Unix Documentation Formats . . . . . . . . . . . . . . . . . . . . . . . . . ....
The Art of Unix Programming            Problems in the Design of Unix . . . . . . . . . . . . . . . . . . . . . . . . . . ...
List of Figures      2.1. The PDP-7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....
List of Tables      8.1. Regular-expression examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...
List of Examples     5.1. Password file example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...
PrefaceUnix is not so much an operating system as an oral history.--<author>NealStephenson</author>There is a vast differe...
PrefaceBecause this book is aimed at transmitting culture, it includes much more in the way of history andfolklore and asi...
PrefaceYou should read this book if you are a Unix user with novice-level up to middle-level skills inthe operating system...
PrefaceAbsorbing these examples will help solidify the principles you learn into semi-instinctive workingknowledge. Ideall...
PrefaceThe Pragmatic Programmer [Hunt-Thomas] is a witty and wise disquisition on good design practicepitched at a slightl...
Prefaceprograms with essentially the same function, and we are discussing generic properties of all ofthem. Emacs, for exa...
Prefacemutt                          The mutt mail user agent is the current best-of-breed among text-                    ...
PrefaceRaskin showed me where the Rule of Least Surprise comes from. The UIUC System ArchitectureGroup contributed useful ...
Part I. Context
Chapter 1. PhilosophyPhilosophy Matters      Those who do not understand Unix are condemned to reinvent it, poorly.      -...
Chapter 1. Philosophyeven video display terminals, and contemporaneous with the first semiconductor memories. Of allproduct...
Chapter 1. PhilosophyOne of the many consequences of the exponential power-versus-time curve in computing, and thecorrespo...
Chapter 1. Philosophyeven by a decade of vendor mismanagement. Today the Unix community itself has taken controlof the tec...
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
The art of unix programming
Upcoming SlideShare
Loading in...5
×

The art of unix programming

35,015

Published on

The art of unix programming

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
35,015
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
98
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "The art of unix programming"

  1. 1. The Art of Unix Programming by Eric Steven Raymond
  2. 2. The Art of Unix Programmingby Eric Steven RaymondCopyright © 2003 Eric S. RaymondThis book and its on-line version are distributed under the terms of the Creative Commons Attribution-NoDerivs 1.0 license,with the additional proviso that the right to publish it on paper for sale or other for-profit use is reserved to Pearson Education,Inc. A reference copy of this license may be found at http://creativecommons.org/licenses/by-nd/1.0/legalcode.AIX, AS/400, DB/2, OS/2, System/360, MVS, VM/CMS, and IBM PC are trademarks of IBM. Alpha, DEC, VAX, HP-UX,PDP, TOPS-10, TOPS-20, VMS, and VT-100 are trademarks of Compaq. Amiga and AmigaOS are trademarks of Amiga,Inc. Apple, Macintosh, MacOS, Newton, OpenDoc, and OpenStep are trademarks of Apple Computers, Inc. ClearCaseis a trademark of Rational Software, Inc. Ethernet is a trademark of 3COM, Inc. Excel, MS-DOS, Microsoft Windowsand PowerPoint are trademarks of Microsoft, Inc. Java. J2EE, JavaScript, NeWS, and Solaris are trademarks of SunMicrosystems. SPARC is a trademark of SPARC international. Informix is a trademark of Informix software. Itaniumis a trademark of Intel. Linux is a trademark of Linus Torvalds. Netscape is a trademark of AOL. PDF and PostScript aretrademarks of Adobe, Inc. UNIX is a trademark of The Open Group.The photograph of Ken and Dennis in Chapter 2 appears courtesy of Bell Labs/Lucent Technologies.The epigraph on the Portability chapter is from the Bell System Technical Journal, v57 #6 part 2 (July-Aug. 1978) pp.2021-2048 and is reproduced with the permission of Bell Labs/Lucent Technologies.
  3. 3. DedicationTo Ken Thompson and Dennis Ritchie, because you inspired me. i
  4. 4. Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi Who Should Read This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii How to Use This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Related References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Conventions Used in This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Our Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Author’s Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii I. Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1. Philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Culture? What Culture? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 The Durability of Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 The Case against Learning Unix Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 What Unix Gets Wrong . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 What Unix Gets Right . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Open-Source Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Cross-Platform Portability and Open Standards . . . . . . . . . . . . . . . . . . . . . 29 The Internet and the World Wide Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 The Open-Source Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Flexibility All the Way Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Unix Is Fun to Hack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 The Lessons of Unix Can Be Applied Elsewhere . . . . . . . . . . . . . . . . . . . . 32 Basics of the Unix Philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Rule of Modularity: Write simple parts connected by clean interfaces. 36 Rule of Clarity: Clarity is better than cleverness. . . . . . . . . . . . . . . . . . . . 36 Rule of Composition: Design programs to be connected with other pro- grams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Rule of Separation: Separate policy from mechanism; separate interfaces from engines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Rule of Simplicity: Design for simplicity; add complexity only where you must. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Rule of Parsimony: Write a big program only when it is clear by demon- stration that nothing else will do. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Rule of Transparency: Design for visibility to make inspection and debug- ging easier. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 ii
  5. 5. The Art of Unix Program- ming Rule of Robustness: Robustness is the child of transparency and simplicity. 41 Rule of Representation: Fold knowledge into data, so program logic can be stupid and robust. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Rule of Least Surprise: In interface design, always do the least surprising thing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Rule of Silence: When a program has nothing surprising to say, it should say nothing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Rule of Repair: Repair what you can — but when you must fail, fail noisily and as soon as possible. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Rule of Economy: Programmer time is expensive; conserve it in preference to machine time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Rule of Generation: Avoid hand-hacking; write programs to write programs when you can. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Rule of Optimization: Prototype before polishing. Get it working before you optimize it. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Rule of Diversity: Distrust all claims for one true way. . . . . . . . . . . . . . . 47 Rule of Extensibility: Design for the future, because it will be here sooner than you think. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 The Unix Philosophy in One Lesson . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Applying the Unix Philosophy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Attitude Matters Too . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502. History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Origins and History of Unix, 1969-1995 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Genesis: 1969–1971 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Exodus: 1971–1980 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 TCP/IP and the Unix Wars: 1980-1990 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Blows against the Empire: 1991-1995 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Origins and History of the Hackers, 1961-1995 . . . . . . . . . . . . . . . . . . . . . . . . . . 66 At Play in the Groves of Academe: 1961-1980 . . . . . . . . . . . . . . . . . . . . . 67 Internet Fusion and the Free Software Movement: 1981-1991 . . . . . . . . 68 Linux and the Pragmatist Reaction: 1991-1998 . . . . . . . . . . . . . . . . . . . . . 71 The Open-Source Movement: 1998 and Onward . . . . . . . . . . . . . . . . . . . . . . . . . 73 The Lessons of Unix History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753. Contrasts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 The Elements of Operating-System Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 What Is the Operating System’s Unifying Idea? . . . . . . . . . . . . . . . . . . . . 76 iii
  6. 6. The Art of Unix Program- ming Multitasking Capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Cooperating Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Internal Boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 File Attributes and Record Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Binary File Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Preferred User Interface Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Entry Barriers to Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Operating-System Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 VMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 MacOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 OS/2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Windows NT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 BeOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 MVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 VM/CMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 What Goes Around, Comes Around . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103II. Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4. Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Encapsulation and Optimal Module Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Compactness and Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Compactness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 The SPOT Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Compactness and the Strong Single Center . . . . . . . . . . . . . . . . . . . . . . . . 116 The Value of Detachment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Software Is a Many-Layered Thing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Top-Down versus Bottom-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Glue Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Case Study: C Considered as Thin Glue . . . . . . . . . . . . . . . . . . . . . . . . . . 122 Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Case Study: GIMP Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Unix and Object-Oriented Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Coding for Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5. Textuality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 The Importance of Being Textual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 iv
  7. 7. The Art of Unix Programming Case Study: Unix Password File Format . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Case Study: .newsrc Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Case Study: The PNG Graphics File Format . . . . . . . . . . . . . . . . . . . . . . 136 Data File Metaformats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 DSV Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 RFC 822 Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Cookie-Jar Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Record-Jar Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 XML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Windows INI Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Unix Textual File Format Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 The Pros and Cons of File Compression . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Application Protocol Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 Case Study: SMTP, a Simple Socket Protocol . . . . . . . . . . . . . . . . . . . . . 149 Case Study: POP3, the Post Office Protocol . . . . . . . . . . . . . . . . . . . . . . . 150 Case Study: IMAP, the Internet Message Access Protocol . . . . . . . . . . 151 Application Protocol Metaformats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 The Classical Internet Application Metaprotocol . . . . . . . . . . . . . . . . . . . 153 HTTP as a Universal Application Protocol . . . . . . . . . . . . . . . . . . . . . . . . 154 BEEP: Blocks Extensible Exchange Protocol . . . . . . . . . . . . . . . . . . . . . . 156 XML-RPC, SOAP, and Jabber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1576. Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Studying Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 Case Study: audacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Case Study: fetchmail’s -v option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 Case Study: GCC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Case Study: kmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Case Study: SNG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Case Study: The Terminfo Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Case Study: Freeciv Data Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 Designing for Transparency and Discoverability . . . . . . . . . . . . . . . . . . . . . . . . 179 The Zen of Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 Coding for Transparency and Discoverability . . . . . . . . . . . . . . . . . . . . . . 181 Transparency and Avoiding Overprotectiveness . . . . . . . . . . . . . . . . . . . . 182 Transparency and Editable Representations . . . . . . . . . . . . . . . . . . . . . . . 183 Transparency, Fault Diagnosis, and Fault Recovery . . . . . . . . . . . . . . . . 185 Designing for Maintainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 v
  8. 8. The Art of Unix Program- ming7. Multiprogramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 Separating Complexity Control from Performance Tuning . . . . . . . . . . . . . . . 189 Taxonomy of Unix IPC Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 Handing off Tasks to Specialist Programs . . . . . . . . . . . . . . . . . . . . . . . . . 191 Pipes, Redirection, and Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 Wrappers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Security Wrappers and Bernstein Chaining . . . . . . . . . . . . . . . . . . . . . . . . 198 Slave Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Peer-to-Peer Inter-Process Communication . . . . . . . . . . . . . . . . . . . . . . . . 200 Problems and Methods to Avoid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 Obsolescent Unix IPC Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 Remote Procedure Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 Threads — Threat or Menace? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 Process Partitioning at the Design Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2138. Minilanguages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Understanding the Taxonomy of Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Applying Minilanguages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Case Study: sng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Case Study: Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Case Study: Glade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Case Study: m4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Case Study: XSLT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 Case Study: The Documenter’s Workbench Tools . . . . . . . . . . . . . . . . . . 229 Case Study: fetchmail Run-Control Syntax . . . . . . . . . . . . . . . . . . . . . . . . 234 Case Study: awk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Case Study: PostScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Case Study: bc and dc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 Case Study: Emacs Lisp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 Case Study: JavaScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 Designing Minilanguages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Choosing the Right Complexity Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Extending and Embedding Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 Writing a Custom Grammar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Macros — Beware! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Language or Application Protocol? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2479. Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 Data-Driven Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 vi
  9. 9. The Art of Unix Program- ming Case Study: ascii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 Case Study: Statistical Spam Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 Case Study: Metaclass Hacking in fetchmailconf . . . . . . . . . . . . . . . . . . 253 Ad-hoc Code Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Case Study: Generating Code for the ascii Displays . . . . . . . . . . . . . . . . 259 Case Study: Generating HTML Code for a Tabular List . . . . . . . . . . . . 26210. Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 What Should Be Configurable? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Where Configurations Live . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Run-Control Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 Case Study: The .netrc File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 Portability to Other Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 System Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 User Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 When to Use Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 Portability to Other Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Command-Line Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 The -a to -z of Command-Line Options . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 Portability to Other Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 How to Choose among the Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 Case Study: fetchmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 Case Study: The XFree86 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 On Breaking These Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28711. Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 Applying the Rule of Least Surprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 History of Interface Design on Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Evaluating Interface Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Tradeoffs between CLI and Visual Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 Case Study: Two Ways to Write a Calculator Program . . . . . . . . . . . . . . 298 Transparency, Expressiveness, and Configurability . . . . . . . . . . . . . . . . . . . . . . 300 Unix Interface Design Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 The Filter Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 The Cantrip Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 The Source Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 The Sink Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 The Compiler Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 vii
  10. 10. The Art of Unix Programming The ed pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 The Roguelike Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 The ‘Separated Engine and Interface’ Pattern . . . . . . . . . . . . . . . . . . . . . . 310 The CLI Server Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 Language-Based Interface Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Applying Unix Interface-Design Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 The Polyvalent-Program Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 The Web Browser as a Universal Front End . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 Silence Is Golden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 12. Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 Don’t Just Do Something, Stand There! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 Measure before Optimizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 Nonlocality Considered Harmful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 Throughput vs. Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 Batching Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 Overlapping Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 Caching Operation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 13. Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Speaking of Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 The Three Sources of Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Tradeoffs between Interface and Implementation Complexity . . . . . . . 337 Essential, Optional, and Accidental Complexity . . . . . . . . . . . . . . . . . . . 339 Mapping Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 When Simplicity Is Not Enough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 A Tale of Five Editors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 ed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 vi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 Sam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Emacs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Wily . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 The Right Size for an Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 Identifying the Complexity Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 Compromise Doesn’t Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Is Emacs an Argument against the Unix Tradition? . . . . . . . . . . . . . . . . 355 The Right Size of Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357III. Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 14. Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360 viii
  11. 11. The Art of Unix Program- ming Unix’s Cornucopia of Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360 Why Not C? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 Interpreted Languages and Mixed Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 Language Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 C ............................................................. 364 C++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368 Perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 Tcl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 Java . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 Emacs Lisp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 Trends for the Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Choosing an X Toolkit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38515. Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388 A Developer-Friendly Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388 Choosing an Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389 Useful Things to Know about vi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389 Useful Things to Know about Emacs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 The Antireligious Choice: Using Both . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 Special-Purpose Code Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 yacc and lex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 Case Study: Glade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 make: Automating Your Recipes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 Basic Theory of make . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 make in Non-C/C++ Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 Utility Productions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 Generating Makefiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 Version-Control Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 Why Version Control? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404 Version Control by Hand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 Automated Version Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 Unix Tools for Version Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 Runtime Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 Combining Tools with Emacs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 Emacs and make . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 ix
  12. 12. The Art of Unix Programming Emacs and Runtime Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 Emacs and Version Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 Emacs and Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 Like an IDE, Only Better . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 16. Reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 The Tale of J. Random Newbie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 Transparency as the Key to Reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 From Reuse to Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 The Best Things in Life Are Open . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 Where to Look? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 Issues in Using Open-Source Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 Licensing Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 What Qualifies as Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426 Standard Open-Source Licenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 When You Need a Lawyer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430IV. Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432 17. Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 Evolution of C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 Early History of C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 C Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436 Unix Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437 Standards and the Unix Wars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 The Ghost at the Victory Banquet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Unix Standards in the Open-Source World . . . . . . . . . . . . . . . . . . . . . . . . 442 IETF and the RFC Standards Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 Specifications as DNA, Code as RNA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446 Programming for Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449 Portability and Choice of Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449 Avoiding System Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 Tools for Portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454 Internationalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454 Portability, Open Standards, and Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . 455 18. Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457 Documentation Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457 The Unix Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459 The Large-Document Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 460 Cultural Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 x
  13. 13. The Art of Unix Programming The Zoo of Unix Documentation Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462 troff and the Documenter’s Workbench Tools . . . . . . . . . . . . . . . . . . . . . . 462 TeX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464 Texinfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465 POD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465 HTML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 DocBook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 The Present Chaos and a Possible Way Out . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 DocBook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 Document Type Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 Other DTDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 The DocBook Toolchain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470 Migration Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473 Editing Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 Related Standards and Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 SGML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 XML-DocBook References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476 Best Practices for Writing Unix Documentation . . . . . . . . . . . . . . . . . . . . . . . . . 47619. Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Unix and Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Best Practices for Working with Open-Source Developers . . . . . . . . . . . . . . . . 482 Good Patching Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482 Good Project- and Archive-Naming Practice . . . . . . . . . . . . . . . . . . . . . . 486 Good Development Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489 Good Distribution-Making Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493 Good Communication Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 The Logic of Licenses: How to Pick One . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499 Why You Should Use a Standard License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500 Varieties of Open-Source Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501 MIT or X Consortium License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501 BSD Classic License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501 Artistic License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502 General Public License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502 Mozilla Public License . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50220. Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504 Essence and Accident in Unix Tradition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504 Plan 9: The Way the Future Was . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507 xi
  14. 14. The Art of Unix Programming Problems in the Design of Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509 A Unix File Is Just a Big Bag of Bytes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509 Unix Support for GUIs Is Weak . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510 File Deletion Is Forever . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511 Unix Assumes a Static File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512 The Design of Job Control Was Badly Botched . . . . . . . . . . . . . . . . . . . . 512 The Unix API Doesn’t Use Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . 513 ioctl2 and fcntl2 Are an Embarrassment . . . . . . . . . . . . . . . . . . . . . . . . . . 514 The Unix Security Model May Be Too Primitive . . . . . . . . . . . . . . . . . . . 514 Unix Has Too Many Different Kinds of Names . . . . . . . . . . . . . . . . . . . . 515 File Systems Might Be Considered Harmful . . . . . . . . . . . . . . . . . . . . . . . 515 Towards a Global Internet Address Space . . . . . . . . . . . . . . . . . . . . . . . . . 515 Problems in the Environment of Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515 Problems in the Culture of Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518 Reasons to Believe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521A. Glossary of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522B. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526C. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 536D. Rootless Root . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538 Editor’s Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538 Master Foo and the Ten Thousand Lines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539 Master Foo and the Script Kiddie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540 Master Foo Discourses on the Two Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541 Master Foo and the Methodologist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542 Master Foo Discourses on the Graphical User Interface . . . . . . . . . . . . . . . . . . . . . . . 543 Master Foo and the Unix Zealot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544 Master Foo Discourses on the Unix-Nature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 544 Master Foo and the End User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545 xii
  15. 15. List of Figures 2.1. The PDP-7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.1. Schematic history of timesharing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.1. Qualitative plot of defect count and density vs. module size. . . . . . . . . . . . . . . . . . . . . 109 4.2. Caller/callee relationships in GIMP with a plugin loaded. . . . . . . . . . . . . . . . . . . . . . . 125 6.1. Screen shot of audacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 6.2. Screen shot of kmail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 6.3. Main window of a Freeciv game. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 8.1. Taxonomy of languages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 11.1. The xcalc GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 11.2. Screen shot of the original Rogue game. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 11.3. The Xcdroast GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 11.4. Caller/callee relationships in a polyvalent program. . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 13.1. Sources and kinds of complexity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 18.1. Processing structural documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468 18.2. Present-day XML-DocBook toolchain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470 18.3. Future XML-DocBook toolchain with FOP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472 xiii
  16. 16. List of Tables 8.1. Regular-expression examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 8.2. Introduction to regular-expression operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 14.1. Language choices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 14.2. Summary of X Toolkits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 xiv
  17. 17. List of Examples 5.1. Password file example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 5.2. A .newsrc example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.3. A fortune file example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 5.4. Basic data for three planets in a record-jar format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 5.5. An XML example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 5.6. A .INI file example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.7. An SMTP session example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 5.8. A POP3 example session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 5.9. An IMAP session example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 6.1. An example fetchmail -v transcript. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 6.2. An SNG Example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 7.1. The pic2graph pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 8.1. Glade Hello, World. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 8.2. A sample m4 macro. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 8.3. A sample XSLT program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 8.4. Taxonomy of languages — the pic source. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 8.5. Synthetic example of a fetchmailrc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 8.6. RSA implementation using dc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 9.1. Example of fetchmailrc syntax. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 9.2. Python structure dump of a fetchmail configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 9.3. copy_instance metaclass code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 9.4. Calling context for copy_instance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 9.5. ascii usage screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 9.6. Desired output format for the star table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 9.7. Master form of the star table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 10.1. A .netrc example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 10.2. X configuration example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 18.1. groff1 markup example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462 18.2. man markup example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 19.1. tar archive maker production. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493 xv
  18. 18. PrefaceUnix is not so much an operating system as an oral history.--<author>NealStephenson</author>There is a vast difference between knowledge and expertise. Knowledge lets you deduce the rightthing to do; expertise makes the right thing a reflex, hardly requiring conscious thought at all.This book has a lot of knowledge in it, but it is mainly about expertise. It is going to try to teachyou the things about Unix development that Unix experts know, but aren’t aware that they know.It is therefore less about technicalia and more about shared culture than most Unix books — bothexplicit and implicit culture, both conscious and unconscious traditions. It is not a ‘how-to’ book,it is a ‘why-to’ book.The why-to has great practical importance, because far too much software is poorly designed. Muchof it suffers from bloat, is exceedingly hard to maintain, and is too difficult to port to new platformsor extend in ways the original programmers didn’t anticipate. These problems are symptoms of baddesign. We hope that readers of this book will learn something of what Unix has to teach aboutgood design.This book is divided into four parts: Context, Design, Tools, and Community. The first part(Context) is philosophy and history, to help provide foundation and motivation for what follows.The second part (Design) unfolds the principles of the Unix philosophy into more specific adviceabout design and implementation. The third part (Tools) focuses on the software Unix providesfor helping you solve problems. The fourth part (Community) is about the human-to-humantransactions and agreements that make the Unix culture so effective at what it does.Because this is a book about shared culture, I never planned to write it alone. You will notice thatthe text includes guest appearances by prominent Unix developers, the shapers of the Unix tradition.The book went through an extended public review process during which I invited these luminariesto comment on and argue with the text. Rather than submerging the results of that review processin the final version, these guests were encouraged to speak with their own voices, amplifying anddeveloping and even disagreeing with the main line of the text.In this book, when I use the editorial ‘we’ it is not to pretend omniscience but to reflect the fact thatit attempts to articulate the expertise of an entire community. xvi
  19. 19. PrefaceBecause this book is aimed at transmitting culture, it includes much more in the way of history andfolklore and asides than is normal for a technical book. Enjoy; these things, too, are part of youreducation as a Unix programmer. No single one of the historical details is vital, but the gestalt ofthem all is important. We think it makes a more interesting story this way. More importantly,understanding where Unix came from and how it got the way it is will help you develop an intuitivefeel for the Unix style.For the same reason, we refuse to write as if history is over. You will find an unusually large numberof references to the time of writing in this book. We do not wish to pretend that current practicereflects some sort of timeless and perfectly logical outcome of preordained destiny. References totime of writing are meant as an alert to the reader two or three or five years hence that the associatedstatements of fact may have become dated and should be double-checked.Other things this book is not is neither a C tutorial, nor a guide to the Unix commands and API. Itis not a reference for sed or yacc or Perl or Python. It’s not a network programming primer, nor anexhaustive guide to the mysteries of X. It’s not a tour of Unix’s internals and architecture, either.Other books cover these specifics better, and this book points you at them as appropriate.Beyond all these technical specifics, the Unix culture has an unwritten engineering tradition that hasdeveloped over literally millions of man-years1 of skilled effort. This book is written in the beliefthat understanding that tradition, and adding its design patterns to your toolkit, will help you becomea better programmer and designer.Cultures consist of people, and the traditional way to learn Unix culture is from other people andthrough the folklore, by osmosis. This book is not a substitute for person-to-person acculturation,but it can help accelerate the process by allowing you to tap the experience of others.Who Should Read This BookYou should read this book if you are an experienced Unix programmer who is often in the positionof either educating novice programmers or debating partisans of other operating systems, and youfind it hard to articulate the benefits of the Unix approach.You should read this book if you are a C, C++, or Java programmer with experience on otheroperating systems and you are about to start a Unix-based project.1 The three and a half decades between 1969 and 2003 is a long time. Going by the historical trend curve in number of Unixsites during that period, probably somewhere upwards of fifty million man-years have been plowed into Unix developmentworldwide. xvii
  20. 20. PrefaceYou should read this book if you are a Unix user with novice-level up to middle-level skills inthe operating system, but little development experience, and want to learn how to design softwareeffectively under Unix.You should read this book if you are a non-Unix programmer who has figured out that the Unixtradition might have something to teach you. We believe you’re right, and that the Unix philosophycan be exported to other operating systems. So we will pay more attention to non-Unix environments(especially Microsoft operating systems) than is usual in a Unix book; and when tools and casestudies are portable, we say so.You should read this book if you are an application architect considering platforms or implemen-tation strategies for a major general-market or vertical application. It will help you understandthe strengths of Unix as a development platform, and of the Unix tradition of open source as adevelopment method.You should not read this book if what you are looking for is the details of C coding or how to usethe Unix kernel API. There are many good books on these topics; Advanced Programming in theUnix Environment [Stevens92] is classic among explorations of the Unix API, and The Practice ofProgramming [Kernighan-Pike99] is recommended reading for all C programmers (indeed for allprogrammers in any language).How to Use This BookThis book is both practical and philosophical. Some parts are aphoristic and general, others willexamine specific case studies in Unix development. We will precede or follow general principlesand aphorisms with examples that illustrate them: examples drawn not from toy demonstrationprograms but rather from real working code that is in use every day.We have deliberately avoided filling the book with lots of code or specification-file examples, eventhough in many places this might have made it easier to write (and in some places perhaps easierto read!). Most books about programming give too many low-level details and examples, but fail atgiving the reader a high-level feel for what is really going on. In this book, we prefer to err in theopposite direction.Therefore, while you will often be invited to read code and specification files, relatively few areactually included in the book. Instead, we point you at examples on the Web. xviii
  21. 21. PrefaceAbsorbing these examples will help solidify the principles you learn into semi-instinctive workingknowledge. Ideally, you should read this book near the console of a running Unix system, with a Webbrowser handy. Any Unix will do, but the software case studies are more likely to be preinstalledand immediately available for inspection on a Linux system. The pointers in the book are invitationsto browse and experiment. Introduction of these pointers is paced so that wandering off to explorefor a while won’t break up exposition that has to be continuous.Note: While we have made every effort to cite URLs that should remain stable and usable, there isno way we can guarantee this. If you find that a cited link has gone stale, use common sense and doa phrase search with your favorite Web search engine. Where possible we suggest ways to do thisnear the URLs we cite.Most abbreviations used in this book are expanded at first use. For convenience, we have alsoprovided a glossary in an appendix.References are usually by author name. Numbered footnotes are for URLs that would intrude onthe text or that we suspect might be perishable; also for asides, war stories, and jokes.2To make this book more accessible to less technical readers, we invited some non-programmers toread it and identify terms that seemed both obscure and necessary to the flow of exposition. Wealso use footnotes for definitions of elementary terms that an experienced programmer is unlikely toneed.Related ReferencesSome famous papers and a few books by Unix’s early developers have mined this territory before.Kernighan and Pike’s The Unix Programming Environment [Kernighan-Pike84] stands out amongthese and is rightly considered a classic. But today it shows its age a bit; it doesn’t cover the Internet,and the World Wide Web or the new wave of interpreted languages like Perl, Tcl, and Python.About halfway into the composition of this book, we learned of Mike Gancarz’s The Unix Philoso-phy [Gancarz]. This book is excellent within its range, but did not attempt to cover the full spectrumof topics we felt needed to be addressed. Nevertheless we are grateful to the author for the reminderthat the very simplest Unix design patterns have been the most persistent and successful ones.2 This particular footnote is dedicated to Terry Pratchett, whose use of footnotes is quite...inspiring. xix
  22. 22. PrefaceThe Pragmatic Programmer [Hunt-Thomas] is a witty and wise disquisition on good design practicepitched at a slightly different level of the software-design craft (more about coding, less about higher-level partitioning of problems) than this book. The authors’ philosophy is an outgrowth of Unixexperience, and it is an excellent complement to this book.The Practice of Programming [Kernighan-Pike99] covers some of the same ground as The Prag-matic Programmer from a position deep within the Unix tradition.Finally (and with admitted intent to provoke) we recommend Zen Flesh, Zen Bones [Reps-Senzaki],an important collection of Zen Buddhist primary sources. References to Zen are scatteredthroughout this book. They are included because Zen provides a vocabulary for addressing someideas that turn out to be very important for software design but are otherwise very difficult to hold inthe mind. Readers with religious attachments are invited to consider Zen not as a religion but as atherapeutic form of mental discipline — which, in its purest non-theistic forms, is exactly what Zenis.Conventions Used in This BookThe term “UNIX” is technically and legally a trademark of The Open Group, and should formallybe used only for operating systems which are certified to have passed The Open Group’s elaboratestandards-conformance tests. In this book we use “Unix” in the looser sense widely current amongprogrammers, to refer to any operating system (whether formally Unix-branded or not) that is eithergenetically descended from Bell Labs’s ancestral Unix code or written in close imitation of itsdescendants. In particular, Linux (from which we draw most of our examples) is a Unix underthis definition.This book employs the Unix manual page convention of tagging Unix facilities with a followingmanual section in parentheses, usually on first introduction when we want to emphasize that thisis a Unix command. Thus, for example, read “munger(1)” as “the ‘munger’ program, which willbe documented in section 1 (user tools) of the Unix manual pages, if it’s present on your system”.Section 2 is C system calls, section 3 is C library calls, section 5 is file formats and protocols, section8 is system administration tools. Other sections vary among Unixes but are not cited in this book.For more, type man 1 man at your Unix shell prompt (older System V Unixes may require man -s1 man).Sometimes we mention a Unix application (such as Emacs, without a manual-section suffix andcapitalized. This is a clue that the name actually represents a well-established family of Unix xx
  23. 23. Prefaceprograms with essentially the same function, and we are discussing generic properties of all ofthem. Emacs, for example, includes xemacs.At various points later in this book we refer to ‘old school’ and ‘new school’ methods. Aswith rap music, new-school starts about 1990. In this context, it’s associated with the rise ofscripting languages, GUIs, open-source Unixes, and the Web. Old-school refers to the pre-1990(and especially pre-1985) world of expensive (shared) computers, proprietary Unixes, scripting inshell, and C everywhere. This difference is worth pointing out because cheaper and less memory-constrained machines have wrought some significant changes on the Unix programming style.Our Case StudiesA lot of books on programming rely on toy examples constructed specifically to prove a point. Thisone won’t. Our case studies will be real, pre-existing pieces of software that are in production useevery day. Here are some of the major ones:cdrtools/xcdroast These two separate projects are usually used together. The cdrtools package is a set of CLI tools for writing CD-ROMs; Web search for “cdrtools”. The xcdroast application is a GUI front end for cdrtools; see the xcdroast project site [http://www.xcdroast.org/].fetchmail The fetchmail program retrieves mail from remote-mail servers using the POP3 or IMAP post-office protocols. See the fetchmail home page [http://www.catb.org/~esr/fetchmail] (or search for “fetchmail” on the Web).GIMP The GIMP (GNU Image Manipulation Program) is a full-featured paint, draw, and image-manipulation program that can edit a huge variety of graphical formats in sophisticated ways. Sources are avail- able from the GIMP home page [http://www.gimp.org/] (or search for "GIMP" on the Web). xxi
  24. 24. Prefacemutt The mutt mail user agent is the current best-of-breed among text- based Unix electronic mail agents, with notably good support for MIME (Multipurpose Internet Mail Extensions) and the use of privacy aids such as PGP (Pretty Good Privacy) and GPG (GNU Privacy Guard). Source code and executable binaries are available at the Mutt project site [http://www.mutt.org].xmlto The xmlto command renders DocBook and other XML docu- ments in various output formats, including HTML and text and PostScript. For sources and documentation, see the xmlto project site [http://cyberelk.net/tim/xmlto/].To minimize the amount of code the user needs to read to understand the examples, we have triedto choose case studies that can be used more than once, ideally to illustrate several different designprinciples and practices. For this same reason, many of the examples are from my projects. Noclaim that these are the best possible ones is implied, merely that I find them sufficiently familiar tobe useful for multiple expository purposes.Author’s AcknowledgementsThe guest contributors (Ken Arnold, Steven M. Bellovin, Stuart Feldman, Jim Gettys, Steve Johnson,Brian Kernighan, David Korn, Mike Lesk, Doug McIlroy, Marshall Kirk McKusick, Keith Packard,Henry Spencer, and Ken Thompson) added a great deal of value to this book. Doug McIlroy, inparticular, went far beyond the call of duty in the thoroughness of his critique and the depth of hiscontributions, displaying the same care and dedication to excellence which he brought to managingthe original Unix research group thirty years ago.Special thanks go to Rob Landley and to my wife Catherine Raymond, both of whom deliveredintensive line-by-line critiques of manuscript drafts. Rob’s insightful and attentive commentaryactually inspired more than one entire chapter in the final manuscript, and he had a lot to do withits present organization and range; if he had written all the text he pushed me to improve, I wouldhave to call him a co-author. Cathy was my test audience representing non-technical readers; to theextent this book is accessible to people who aren’t already programmers, that’s largely her doing.This book benefited from discussions with many other people over the five years it took me to writeit. Mark M. Miller helped me achieve enlightenment about threads. John Cowan supplied someinsights about interface design patterns and drafted the case studies of wily and VM/CMS, and Jef xxii
  25. 25. PrefaceRaskin showed me where the Rule of Least Surprise comes from. The UIUC System ArchitectureGroup contributed useful feedback on early chapters. The sections on What Unix Gets Wrong andFlexibility in Depth were directly inspired by their review. Russell J. Nelson contributed the materialon Bernstein chaining in Chapter 7. Jay Maynard contributed most of the material in the MVS casestudy in Chapter 3. Les Hatton provided many helpful comments on the Languages chapter andmotivated the portion of Chapter 4 on Optimal Module Size. David A. Wheeler contributed manyperceptive criticisms and some case-study material, especially in the Design part. Russ Cox helpeddevelop the survey of Plan 9. Dennis Ritchie corrected me on some historical points about C.Hundreds of Unix programmers, far too many to list here, contributed advice and comments duringthe book’s public review period between January and June of 2003. As always, I found the processof open peer review over the Web both intensely challenging and intensely rewarding. Also asalways, responsibility for any errors in the resulting work remains my own.The expository style and some of the concerns of this book have been influenced by the designpatterns movement; indeed, I flirted with the idea of titling the book Unix Design Patterns. I didn’t,because I disagree with some of the implicit central dogmas of the movement and don’t feel the needto use all its formal apparatus or accept its cultural baggage. Nevertheless, my approach has certainlybeen influenced by Christopher Alexander’s work3 (especially The Timeless Way of Building and APattern Language), and I owe the Gang of Four and other members of their school a large debt ofgratitude for showing me how it is possible to use Alexander’s insights to talk about software designat a high level without merely uttering vague and useless generalities. Interested readers should seeDesign Patterns: Elements of Reusable Object-Oriented Software [GangOfFour] for an introductionto design patterns.The title of this book is, of course, a reference to Donald Knuth’s The Art of Computer Programming.While not specifically associated with the Unix tradition, Knuth has been an influence on us all.Editors with vision and imagination aren’t as common as they should be. Mark Taub is one; he sawmerit in a stalled project and skillfully nudged me into finishing it. Copy editors with a good earfor prose style and enough ability to improve writing that isn’t like theirs are even less common,but Mary Lou Nohr makes that grade. Jerry Votta seized on my concept for the cover and made itlook better than I had imagined. The whole crew at Addison-Wesley gets high marks for makingthe editorial and production process as painless as possible, and for cheerfully accommodating mycontrol-freak tendencies not just over the text but deep into the details of the book’s visual design,art, and marketing.3 An appreciation of Alexander’s work, with links to on-line versions of significant portions, may be found at Some Notes onChristopher Alexander [http://www.math.utsa.edu/sphere/salingar/Chris.text.html]. xxiii
  26. 26. Part I. Context
  27. 27. Chapter 1. PhilosophyPhilosophy Matters Those who do not understand Unix are condemned to reinvent it, poorly. -- <author>HenrySpencer</author> Usenet signature, November 1987 Culture? What Culture? This is a book about Unix programming, but in it we’re going to toss around the words ‘culture’, ‘art’, and ‘philosophy’ a lot. If you are not a programmer, or you are a programmer who has had little contact with the Unix world, this may seem strange. But Unix has a culture; it has a distinctive art of programming; and it carries with it a powerful design philosophy. Understanding these traditions will help you build better software, even if you’re developing for a non-Unix platform. Every branch of engineering and design has technical cultures. In most kinds of engineering, the unwritten traditions of the field are parts of a working practitioner’s education as important as (and, as experience grows, often more important than) the official handbooks and textbooks. Senior engineers develop huge bodies of implicit knowledge, which they pass to their juniors by (as Zen Buddhists put it) “a special transmission, outside the scriptures”. Software engineering is generally an exception to this rule; technology has changed so rapidly, software environments have come and gone so quickly, that technical cultures have been weak and ephemeral. There are, however, exceptions to this exception. A very few software technologies have proved durable enough to evolve strong technical cultures, distinctive arts, and an associated design philosophy transmitted across generations of engineers. The Unix culture is one of these. The Internet culture is another — or, in the twenty-first century, arguably the same one. The two have grown increasingly difficult to separate since the early 1980s, and in this book we won’t try particularly hard. The Durability of Unix Unix was born in 1969 and has been in continuous production use ever since. That’s several geologic eras by computer-industry standards — older than the PC or workstations or microprocessors or 25
  28. 28. Chapter 1. Philosophyeven video display terminals, and contemporaneous with the first semiconductor memories. Of allproduction timesharing systems today, only IBM’s VM/CMS can claim to have existed longer, andUnix machines have provided hundreds of thousands of times more service hours; indeed, Unix hasprobably supported more computing than all other timesharing systems put together.Unix has found use on a wider variety of machines than any other operating system can claim. Fromsupercomputers to handhelds and embedded networking hardware, through workstations and serversand PCs and minicomputers, Unix has probably seen more architectures and more odd hardware thanany three other operating systems combined.Unix has supported a mind-bogglingly wide spectrum of uses. No other operating system has shonesimultaneously as a research vehicle, a friendly host for technical custom applications, a platformfor commercial-off-the-shelf business software, and a vital component technology of the Internet.Confident predictions that Unix would wither away, or be crowded out by other operating systems,have been made yearly since its infancy. And yet Unix, in its present-day avatars as Linux and BSDand Solaris and MacOS X and half a dozen other variants, seems stronger than ever today. Robert Metcalf [the inventor of Ethernet] says that if something comes along to replace Ethernet, it will be called “Ethernet”, so therefore Ethernet will never die.4 Unix has already undergone several such transformations. — <author>KenThompson</author>At least one of Unix’s central technologies — the C language — has been widely naturalizedelsewhere. Indeed it is now hard to imagine doing software engineering without C as a ubiquitouscommon language of systems programming. Unix also introduced both the now-ubiquitous tree-shaped file namespace with directory nodes and the pipeline for connecting programs.Unix’s durability and adaptability have been nothing short of astonishing. Other technologies havecome and gone like mayflies. Machines have increased a thousandfold in power, languages havemutated, industry practice has gone through multiple revolutions — and Unix hangs in there, stillproducing, still paying the bills, and still commanding loyalty from many of the best and brightestsoftware technologists on the planet. 4 In fact, Ethernet has already been replaced by a different technology with the same name — twice. Once when coax was replaced with twisted pair, and a second time when gigabit Ethernet came in. 26
  29. 29. Chapter 1. PhilosophyOne of the many consequences of the exponential power-versus-time curve in computing, and thecorresponding pace of software development, is that 50% of what one knows becomes obsolete overevery 18 months. Unix does not abolish this phenomenon, but does do a good job of containing it.There’s a bedrock of unchanging basics — languages, system calls, and tool invocations — that onecan actually keep using for years, even decades. Elsewhere it is impossible to predict what will bestable; even entire operating systems cycle out of use. Under Unix, there is a fairly sharp distinctionbetween transient knowledge and lasting knowledge, and one can know ahead of time (with about90% certainty) which category something is likely to fall in when one learns it. Thus the loyaltyUnix commands.Much of Unix’s stability and success has to be attributed to its inherent strengths, to design decisionsKen Thompson, Dennis Ritchie, Brian Kernighan, Doug McIlroy, Rob Pike and other early Unixdevelopers made back at the beginning; decisions that have been proven sound over and over. Butjust as much is due to the design philosophy, art of programming, and technical culture that grew uparound Unix in the early days. This tradition has continuously and successfully propagated itself insymbiosis with Unix ever since.The Case against Learning Unix CultureUnix’s durability and its technical culture are certainly of interest to people who already like Unix,and perhaps to historians of technology. But Unix’s original application as a general-purposetimesharing system for mid-sized and larger computers is rapidly receding into the mists of history,killed off by personal workstations. And there is certainly room for doubt that it will ever achievesuccess in the mainstream business-desktop market now dominated by Microsoft.Outsiders have frequently dismissed Unix as an academic toy or a hacker’s sandbox. One well-known polemic, the Unix Hater’s Handbook [Garfinkel], follows an antagonistic line nearly as oldas Unix itself in writing its devotees off as a cult religion of freaks and losers. Certainly the colossaland repeated blunders of AT&T, Sun, Novell, and other commercial vendors and standards consortiain mispositioning and mismarketing Unix have become legendary.Even from within the Unix world, Unix has seemed to be teetering on the brink of universality forso long as to raise the suspicion that it will never actually get there. A skeptical outside observer’sconclusion might be that Unix is too useful to die but too awkward to break out of the back room; aperpetual niche operating system.What confounds the skeptics’ case is, more than anything else, the rise of Linux and other open-source Unixes (such as the modern BSD variants). Unix’s culture proved too vital to be smothered 27
  30. 30. Chapter 1. Philosophyeven by a decade of vendor mismanagement. Today the Unix community itself has taken controlof the technology and marketing, and is rapidly and visibly solving Unix’s problems (in ways we’llexamine in more detail in Chapter 20).What Unix Gets WrongFor a design that dates from 1969, it is remarkably difficult to identify design choices in Unix thatare unequivocally wrong. There are several popular candidates, but each is still a subject of spiriteddebate not merely among Unix fans but across the wider community of people who think about anddesign operating systems.Unix files have no structure above byte level. File deletion is irrevocable. The Unix security modelis arguably too primitive. Job control is botched. There are too many different kinds of names forthings. Having a file system at all may have been the wrong choice. We will discuss these technicalissues in Chapter 20.But perhaps the most enduring objections to Unix are consequences of a feature of its philosophyfirst made explicit by the designers of the X windowing system. X strives to provide “mechanism,not policy”, supporting an extremely general set of graphics operations and deferring decisions abouttoolkits and interface look-and-feel (the policy) up to application level. Unix’s other system-levelservices display similar tendencies; final choices about behavior are pushed as far toward the useras possible. Unix users can choose among multiple shells. Unix programs normally provide manybehavior options and sport elaborate preference facilities.This tendency reflects Unix’s heritage as an operating system designed primarily for technical users,and a consequent belief that users know better than operating-system designers what their own needsare. This tenet was firmly established at Bell Labs by Dick Hamming5 who insisted in the 1950s when computers were rare and expensive, that open-shop computing, where customers wrote their own programs, was imperative, because “it is better to solve the right problem the wrong way than the wrong problem the right way”. — <author>DougMcIlroy</author> 5 Yes, the Hamming of ‘Hamming distance’ and ‘Hamming code’. 28

×