3. Preface
Twelve years have passed since the first edition of Modern Compiler Design. For
many computer science subjects this would be more than a life time, but since com-
piler design is probably the most mature computer science subject, it is different.
An adult person develops more slowly and differently than a toddler or a teenager,
and so does compiler design. The present book reflects that.
Improvements to the book fall into two groups: presentation and content. The
‘look and feel’ of the book has been modernized, but more importantly we have
rearranged significant parts of the book to present them in a more structured manner:
large chapters have been split and the optimizing code generation techniques have
been collected in a separate chapter. Based on reader feedback and experiences in
teaching from this book, both by ourselves and others, material has been expanded,
clarified, modified, or deleted in a large number of places. We hope that as a result
of this the reader feels that the book does a better job of making compiler design
and construction accessible.
The book adds new material to cover the developments in compiler design and
construction over the last twelve years. Overall the standard compiling techniques
and paradigms have stood the test of time, but still new and often surprising opti-
mization techniques have been invented; existing ones have been improved; and old
ones have gained prominence. Examples of the first are: procedural abstraction, in
which routines are recognized in the code and replaced by routine calls to reduce
size; binary rewriting, in which optimizations are applied to the binary code; and
just-in-time compilation, in which parts of the compilation are delayed to improve
the perceived speed of the program. An example of the second is a technique which
extends optimal code generation through exhaustive search, previously available for
tiny blocks only, to moderate-size basic blocks. And an example of the third is tail
recursion removal, indispensable for the compilation of functional languages. These
developments are mainly described in Chapter 9.
Although syntax analysis is the one but oldest branch of compiler construction
(lexical analysis being the oldest), even in that area innovation has taken place.
Generalized (non-deterministic) LR parsing, developed between 1984 and 1994, is
now used in compilers. It is covered in Section 3.5.8.
New hardware requirements have necessitated new compiler developments. The
main examples are the need for size reduction of the object code, both to fit the code
into small embedded systems and to reduce transmission times; and for lower power
v
4. vi Preface
consumption, to extend battery life and to reduce electricity bills. Dynamic memory
allocation in embedded systems requires a balance between speed and thrift, and the
question is how compiler design can help. These subjects are covered in Sections
9.2, 9.3, and 10.2.8, respectively.
With age comes legacy. There is much legacy code around, code which is so
old that it can no longer be modified and recompiled with reasonable effort. If the
source code is still available but there is no compiler any more, recompilation must
start with a grammar of the source code. For fifty years programmers and compiler
designers have used grammars to produce and analyze programs; now large legacy
programs are used to produce grammars for them. The recovery of the grammar
from legacy source code is discussed in Section 3.6. If just the binary executable
program is left, it must be disassembled or even decompiled. For fifty years com-
piler designers have been called upon to design compilers and assemblers to convert
source programs to binary code; now they are called upon to design disassemblers
and decompilers, to roll back the assembly and compilation process. The required
techniques are treated in Sections 8.4 and 8.5.
The bibliography
The literature list has been updated, but its usefulness is more limited than before,
for two reasons. The first is that by the time it appears in print, the Internet can pro-
vide more up-to-date and more to-the-point information, in larger quantities, than a
printed text can hope to achieve. It is our contention that anybody who has under-
stood a larger part of the ideas explained in this book is able to evaluate Internet
information on compiler design.
The second is that many of the papers we refer to are available only to those
fortunate enough to have login facilities at an institute with sufficient budget to
obtain subscriptions to the larger publishers; they are no longer available to just
anyone who walks into a university library. Both phenomena point to paradigm
shifts with which readers, authors, publishers and librarians will have to cope.
The structure of the book
This book is conceptually divided into two parts. The first, comprising Chapters 1
through 10, is concerned with techniques for program processing in general; it in-
cludes a chapter on memory management, both in the compiler and in the generated
code. The second part, Chapters 11 through 14, covers the specific techniques re-
quired by the various programming paradigms. The interactions between the parts
of the book are outlined in the adjacent table. The leftmost column shows the four
phases of compiler construction: analysis, context handling, synthesis, and run-time
systems. Chapters in this column cover both the manual and the automatic creation
5. Preface vii
of the pertinent software but tend to emphasize automatic generation. The other
columns show the four paradigms covered in this book; for each paradigm an ex-
ample of a subject treated by each of the phases is shown. These chapters tend to
contain manual techniques only, all automatic techniques having been delegated to
Chapters 2 through 9.
in imperative
and object-
oriented
programs
(Chapter 11)
in functional
programs
(Chapter 12)
in logic
programs
(Chapter 13)
in parallel/
distributed
programs
(Chapter 14)
How to do:
analysis
(Chapters 2 & 3)
−− −− −− −−
context
handling
(Chapters 4 & 5)
identifier
identification
polymorphic
type checking
static rule
matching
Linda static
analysis
synthesis
(Chapters 6–9)
code for
while-
statement
code for list
comprehension
structure
unification
marshaling
run-time
systems
(no chapter)
stack reduction
machine
Warren
Abstract
Machine
replication
The scientific mind would like the table to be nice and square, with all boxes
filled —in short “orthogonal”— but we see that the top right entries are missing
and that there is no chapter for “run-time systems” in the leftmost column. The top
right entries would cover such things as the special subjects in the program text
analysis of logic languages, but present text analysis techniques are powerful and
flexible enough and languages similar enough to handle all language paradigms:
there is nothing to be said there, for lack of problems. The chapter missing from
the leftmost column would discuss manual and automatic techniques for creating
run-time systems. Unfortunately there is little or no theory on this subject: run-time
systems are still crafted by hand by programmers on an intuitive basis; there is
nothing to be said there, for lack of solutions.
Chapter 1 introduces the reader to compiler design by examining a simple tradi-
tional modular compiler/interpreter in detail. Several high-level aspects of compiler
construction are discussed, followed by a short history of compiler construction and
introductions to formal grammars and closure algorithms.
Chapters 2 and 3 treat the program text analysis phase of a compiler: the conver-
sion of the program text to an abstract syntax tree. Techniques for lexical analysis,
lexical identification of tokens, and syntax analysis are discussed.
Chapters 4 and 5 cover the second phase of a compiler: context handling. Sev-
eral methods of context handling are discussed: automated ones using attribute
grammars, manual ones using L-attributed and S-attributed grammars, and semi-
automated ones using symbolic interpretation and data-flow analysis.
6. viii Preface
Chapters 6 through 9 cover the synthesis phase of a compiler, covering both in-
terpretation and code generation. The chapters on code generation are mainly con-
cerned with machine code generation; the intermediate code required for paradigm-
specific constructs is treated in Chapters 11 through 14.
Chapter 10 concerns memory management techniques, both for use in the com-
piler and in the generated program.
Chapters 11 through 14 address the special problems in compiling for the various
paradigms – imperative, object-oriented, functional, logic, and parallel/distributed.
Compilers for imperative and object-oriented programs are similar enough to be
treated together in one chapter, Chapter 11.
Appendix B contains hints and answers to a selection of the exercises in the
book. Such exercises are marked by a followed the page number on which the
answer appears. A larger set of answers can be found on Springer’s Internet page;
the corresponding exercises are marked by www.
Several subjects in this book are treated in a non-traditional way, and some words
of justification may be in order.
Lexical analysis is based on the same dotted items that are traditionally reserved
for bottom-up syntax analysis, rather than on Thompson’s NFA construction. We
see the dotted item as the essential tool in bottom-up pattern matching, unifying
lexical analysis, LR syntax analysis, bottom-up code generation and peep-hole op-
timization. The traditional lexical algorithms are just low-level implementations of
item manipulation. We consider the different treatment of lexical and syntax analy-
sis to be a historical artifact. Also, the difference between the lexical and the syntax
levels tends to disappear in modern software.
Considerable attention is being paid to attribute grammars, in spite of the fact
that their impact on compiler design has been limited. Yet they are the only known
way of automating context handling, and we hope that the present treatment will
help to lower the threshold of their application.
Functions as first-class data are covered in much greater depth in this book than is
usual in compiler design books. After a good start in Algol 60, functions lost much
status as manipulatable data in languages like C, Pascal, and Ada, although Ada
95 rehabilitated them somewhat. The implementation of some modern concepts,
for example functional and logic languages, iterators, and continuations, however,
requires functions to be manipulated as normal data. The fundamental aspects of
the implementation are covered in the chapter on imperative and object-oriented
languages; specifics are given in the chapters on the various other paradigms.
Additional material, including more answers to exercises, and all diagrams and
all code from the book, are available through Springer’s Internet page.
Use as a course book
The book contains far too much material for a compiler design course of 13 lectures
of two hours each, as given at our university, so a selection has to be made. An
7. Preface ix
introductory, more traditional course can be obtained by including, for example,
Chapter 1;
Chapter 2 up to 2.7; 2.10; 2.11; Chapter 3 up to 3.4.5; 3.5 up to 3.5.7;
Chapter 4 up to 4.1.3; 4.2.1 up to 4.3; Chapter 5 up to 5.2.2; 5.3;
Chapter 6; Chapter 7 up to 9.1.1; 9.1.4 up to 9.1.4.4; 7.3;
Chapter 10 up to 10.1.2; 10.2 up to 10.2.4;
Chapter 11 up to 11.2.3.2; 11.2.4 up to 11.2.10; 11.4 up to 11.4.2.3.
A more advanced course would include all of Chapters 1 to 11, possibly exclud-
ing Chapter 4. This could be augmented by one of Chapters 12 to 14.
An advanced course would skip much of the introductory material and concen-
trate on the parts omitted in the introductory course, Chapter 4 and all of Chapters
10 to 14.
Acknowledgments
We owe many thanks to the following people, who supplied us with help, remarks,
wishes, and food for thought for this Second Edition: Ingmar Alting, José Fortes,
Bert Huijben, Jonathan Joubert, Sara Kalvala, Frank Lippes, Paul S. Moulson, Pras-
ant K. Patra, Carlo Perassi, Marco Rossi, Mooly Sagiv, Gert Jan Schoneveld, Ajay
Singh, Evert Wattel, and Freek Zindel. Their input ranged from simple corrections to
detailed suggestions to massive criticism. Special thanks go to Stefanie Scherzinger,
whose thorough and thoughtful criticism of our outline code format induced us to
improve it considerably; any remaining imperfections should be attributed to stub-
bornness on the part of the authors. The presentation of the program code snippets
in the book profited greatly from Carsten Heinz’s listings package; we thank
him for making the package available to the public.
We are grateful to Ann Kostant, Melissa Fearon, and Courtney Clark of Springer
US, who, through fast and competent work, have cleared many obstacles that stood
in the way of publishing this book. We thank them for their effort and pleasant
cooperation.
We mourn the death of Irina Athanasiu, who did not live long enough to lend her
expertise in embedded systems to this book.
We thank the Faculteit der Exacte Wetenschappen of the Vrije Universiteit for
their support and the use of their equipment.
Amsterdam, Dick Grune
March 2012 Kees van Reeuwijk
Henri E. Bal
Ceriel J.H. Jacobs
Delft, Koen G. Langendoen
8. x Preface
Abridged Preface to the First Edition (2000)
In the 1980s and 1990s, while the world was witnessing the rise of the PC and
the Internet on the front pages of the daily newspapers, compiler design methods
developed with less fanfare, developments seen mainly in the technical journals, and
–more importantly– in the compilers that are used to process today’s software. These
developments were driven partly by the advent of new programming paradigms,
partly by a better understanding of code generation techniques, and partly by the
introduction of faster machines with large amounts of memory.
The field of programming languages has grown to include, besides the tradi-
tional imperative paradigm, the object-oriented, functional, logical, and parallel/dis-
tributed paradigms, which inspire novel compilation techniques and which often
require more extensive run-time systems than do imperative languages. BURS tech-
niques (Bottom-Up Rewriting Systems) have evolved into very powerful code gen-
eration techniques which cope superbly with the complex machine instruction sets
of present-day machines. And the speed and memory size of modern machines allow
compilation techniques and programming language features that were unthinkable
before. Modern compiler design methods meet these challenges head-on.
The audience
Our audience are students with enough experience to have at least used a compiler
occasionally and to have given some thought to the concept of compilation. When
these students leave the university, they will have to be familiar with language pro-
cessors for each of the modern paradigms, using modern techniques. Although cur-
riculum requirements in many universities may have been lagging behind in this
respect, graduates entering the job market cannot afford to ignore these develop-
ments.
Experience has shown us that a considerable number of techniques traditionally
taught in compiler construction are special cases of more fundamental techniques.
Often these special techniques work for imperative languages only; the fundamental
techniques have a much wider application. An example is the stack as an optimized
representation for activation records in strictly last-in-first-out languages. Therefore,
this book
• focuses on principles and techniques of wide application, carefully distinguish-
ing between the essential (= material that has a high chance of being useful to
the student) and the incidental (= material that will benefit the student only in
exceptional cases);
• provides a first level of implementation details and optimizations;
• augments the explanations by pointers for further study.
The student, after having finished the book, can expect to:
9. Preface xi
• have obtained a thorough understanding of the concepts of modern compiler de-
sign and construction, and some familiarity with their practical application;
• be able to start participating in the construction of a language processor for each
of the modern paradigms with a minimal training period;
• be able to read the literature.
The first two provide a firm basis; the third provides potential for growth.
Acknowledgments
We owe many thanks to the following people, who were willing to spend time and
effort on reading drafts of our book and to supply us with many useful and some-
times very detailed comments: Mirjam Bakker, Raoul Bhoedjang, Wilfred Dittmer,
Thomer M. Gil, Ben N. Hasnai, Bert Huijben, Jaco A. Imthorn, John Romein, Tim
Rühl, and the anonymous reviewers. We thank Ronald Veldema for the Pentium
code segments.
We are grateful to Simon Plumtree, Gaynor Redvers-Mutton, Dawn Booth, and
Jane Kerr of John Wiley Sons Ltd, for their help and encouragement in writing
this book. Lambert Meertens kindly provided information on an older ABC com-
piler, and Ralph Griswold on an Icon compiler.
We thank the Faculteit Wiskunde en Informatica (now part of the Faculteit der
Exacte Wetenschappen) of the Vrije Universiteit for their support and the use of
their equipment.
Dick Grune dick@cs.vu.nl, http://www.cs.vu.nl/~dick
Henri E. Bal bal@cs.vu.nl, http://www.cs.vu.nl/~bal
Ceriel J.H. Jacobs ceriel@cs.vu.nl, http://www.cs.vu.nl/~ceriel
Koen G. Langendoen koen@pds.twi.tudelft.nl, http://pds.twi.tudelft.nl/~koen
Amsterdam, May 2000
22. 2 1 Introduction
the baron pulled himself from a swamp by his hair plait, rather than by his boot-
straps [14].
Program
Compiler
txt
txt
X
Compiler
exe
exe
Y
=
=
Executable compiler
code
Compiler source text
Input in
code for target
Executable program
language
in implementation
in source language
Program source text
some input format
machine
Output in
some output format
Fig. 1.1: Compiling and running a compiler
Compilation does not differ fundamentally from file conversion but it does differ
in degree. The main aspect of conversion is that the input has a property called
semantics—its “meaning”—which must be preserved by the process. The structure
of the input and its semantics can be simple, as, for example in a file conversion
program which converts EBCDIC to ASCII; they can be moderate, as in an WAV
to MP3 converter, which has to preserve the acoustic impression, its semantics;
or they can be considerable, as in a compiler, which has to faithfully express the
semantics of the input program in an often extremely different output format. In the
final analysis, a compiler is just a giant file conversion program.
The compiler can work its magic because of two factors:
• the input is in a language and consequently has a structure, which is described in
the language reference manual;
• the semantics of the input is described in terms of and is attached to that same
structure.
These factors enable the compiler to “understand” the program and to collect its
semantics in a semantic representation. The same two factors exist with respect
to the target language. This allows the compiler to rephrase the collected semantics
in terms of the target language. How all this is done in detail is the subject of this
book.
23. 1 Introduction 3
Compiler
txt
Source
text
Front−
end
(analysis)
Semantic
represent−
ation (synthesis)
end
Back−
exe
Executable
code
Fig. 1.2: Conceptual structure of a compiler
The part of a compiler that performs the analysis of the source language text
is called the front-end, and the part that does the target language synthesis is the
back-end; see Figure 1.2. If the compiler has a very clean design, the front-end is
totally unaware of the target language and the back-end is totally unaware of the
source language: the only thing they have in common is knowledge of the semantic
representation. There are technical reasons why such a strict separation is inefficient,
and in practice even the best-structured compilers compromise.
The above description immediately suggests another mode of operation for a
compiler: if all required input data are available, the compiler could perform the
actions specified by the semantic representation rather than re-express them in a
different form. The code-generating back-end is then replaced by an interpreting
back-end, and the whole program is called an interpreter. There are several reasons
for doing this, some fundamental and some more opportunistic.
One fundamental reason is that an interpreter is normally written in a high-level
language and will therefore run on most machine types, whereas generated object
code will only run on machines of the target type: in other words, portability is
increased. Another is that writing an interpreter is much less work than writing a
back-end.
A third reason for using an interpreter rather than a compiler is that perform-
ing the actions straight from the semantic representation allows better error check-
ing and reporting. This is not fundamentally so, but is a consequence of the fact
that compilers (front-end/back-end combinations) are expected to generate efficient
code. As a result, most back-ends throw away any information that is not essential
to the program execution in order to gain speed; this includes much information that
could have been useful in giving good diagnostics, for example source code and its
line numbers.
A fourth reason is the increased security that can be achieved by interpreters;
this effect has played an important role in Java’s rise to fame. Again, this increased
security is not fundamental since there is no reason why compiled code could not do
the same checks an interpreter can. Yet it is considerably easier to convince oneself
that an interpreter does not play dirty tricks than that there are no booby traps hidden
in binary executable code.
A fifth reason is the ease with which an interpreter can handle new program
code generated by the running program itself. An interpreter can treat the new code
exactly as all other code. Compiled code must, however, invoke a compiler (if avail-
able), and load and link the newly compiled code to the running program (if pos-
24. 4 1 Introduction
sible). In fact, if a programming language allows new code to be constructed in
a running program, the use of an interpreter is almost unavoidable. Conversely, if
the language is typically implemented by an interpreter, the language might as well
allow new code to be constructed in a running program.
Why is a compiler called a compiler?
The original meaning of “to compile” is “to select representative material and add it to a
collection”; makers of compilation CDs use the term in its proper meaning. In its early days
programming language translation was viewed in the same way: when the input contained
for example “a + b”, a prefabricated code fragment “load a in register; add b to register”
was selected and added to the output. A compiler compiled a list of code fragments to be
added to the translated program. Today’s compilers, especially those for the non-imperative
programming paradigms, often perform much more radical transformations on the input
program.
It should be pointed out that there is no fundamental difference between using a
compiler and using an interpreter. In both cases the program text is processed into
an intermediate form, which is then interpreted by some interpreting mechanism. In
compilation,
• the program processing is considerable;
• the resulting intermediate form, machine-specific binary executable code, is low-
level;
• the interpreting mechanism is the hardware CPU; and
• program execution is relatively fast.
In interpretation,
• the program processing is minimal to moderate;
• the resulting intermediate form, some system-specific data structure, is high- to
medium-level;
• the interpreting mechanism is a (software) program; and
• program execution is relatively slow.
These relationships are summarized graphically in Figure 1.3. Section 7.5.1
shows how a fairly smooth shift from interpreter to compiler can be made.
After considering the question of why one should study compiler construction
(Section 1.1) we will look at simple but complete demonstration compiler (Section
1.2); survey the structure of a more realistic compiler (Section 1.3); and consider
possible compiler architectures (Section 1.4). This is followed by short sections on
the properties of a good compiler (1.5), portability and retargetability (1.6), and
the history of compiler construction (1.7). Next are two more theoretical subjects:
an introduction to context-free grammars (Section 1.8), and a general closure algo-
rithm (Section 1.9). A brief explanation of the various code forms used in the book
(Section 1.10) concludes this introductory chapter.
25. 1.1 Why study compiler construction? 5
code
Source
code
Executable
Machine
Interpreter
code
Source Intermediate
code
processing
preprocessing
processing
preprocessing
Compilation
Interpretation
Fig. 1.3: Comparison of a compiler and an interpreter
Occasionally, the structure of the text will be summarized in a “roadmap”, as
shown for this chapter.
Roadmap
1 Introduction 1
1.1 Why study compiler construction? 5
1.2 A simple traditional modular compiler/interpreter 9
1.3 The structure of a more realistic compiler 22
1.4 Compiler architectures 26
1.5 Properties of a good compiler 31
1.6 Portability and retargetability 32
1.7 A short history of compiler construction 33
1.8 Grammars 34
1.9 Closure algorithms 41
1.10 The code forms used in this book 46
1.1 Why study compiler construction?
There are a number of objective reasons why studying compiler construction is a
good idea:
• compiler construction is a very successful branch of computer science, and one
of the earliest to earn that predicate;
26. 6 1 Introduction
• given its similarity to file conversion, it has wider application than just compilers;
• it contains many generally useful algorithms in a realistic setting.
We will have a closer look at each of these below. The main subjective reason to
study compiler construction is of course plain curiosity: it is fascinating to see how
compilers manage to do what they do.
1.1.1 Compiler construction is very successful
Compiler construction is a very successful branch of computer science. Some of
the reasons for this are the proper structuring of the problem, the judicious use of
formalisms, and the use of tools wherever possible.
1.1.1.1 Proper structuring of the problem
Compilers analyze their input, construct a semantic representation, and synthesize
their output from it. This analysis–synthesis paradigm is very powerful and widely
applicable. A program for tallying word lengths in a text could for example consist
of a front-end which analyzes the text and constructs internally a table of (length,
frequency) pairs, and a back-end which then prints this table. Extending this pro-
gram, one could replace the text-analyzing front-end by a module that collects file
sizes in a file system; alternatively, or additionally, one could replace the back-end
by a module that produces a bar graph rather than a printed table; we use the word
“module” here to emphasize the exchangeability of the parts. In total, four pro-
grams have already resulted, all centered around the semantic representation and
each reusing lots of code from the others.
Likewise, without the strict separation of analysis and synthesis phases, program-
ming languages and compiler construction would not be where they are today. With-
out it, each new language would require a completely new set of compilers for all
interesting machines—or die for lack of support. With it, a new front-end for that
language suffices, to be combined with the existing back-ends for the current ma-
chines: for L languages and M machines, L front-ends and M back-ends are needed,
requiring L+M modules, rather than L×M programs. See Figure 1.4.
It should be noted immediately, however, that this strict separation is not com-
pletely free of charge. If, for example, a front-end knows it is analyzing for a ma-
chine with special machine instructions for multi-way jumps, it can probably an-
alyze case/switch statements so that they can benefit from these machine instruc-
tions. Similarly, if a back-end knows it is generating code for a language which has
no nested routine declarations, it can generate simpler code for routine calls. Many
professional compilers are integrated compilers for one programming language and
one machine architecture, using a semantic representation which derives from the
source language and which may already contain elements of the target machine.
27. 1.1 Why study compiler construction? 7
L M
Language 1
Language 2
Back−ends for
Front−ends for
Machine
Language
Machine 2
Machine 1
Semantic
represent−
ation
Fig. 1.4: Creating compilers for L languages and M machines
Still, the structuring has played and still plays a large role in the rapid introduction
of new languages and new machines.
1.1.1.2 Judicious use of formalisms
For some parts of compiler construction excellent standardized formalisms have
been developed, which greatly reduce the effort to produce these parts. The best
examples are regular expressions and context-free grammars, used in lexical and
syntactic analysis. Enough theory about these has been developed from the 1960s
onwards to fill an entire course, but the practical aspects can be taught and under-
stood without going too deeply into the theory. We will consider these formalisms
and their applications in Chapters 2 and 3.
Attribute grammars are a formalism that can be used for handling the context, the
long-distance relations in a program that link, for example, the use of a variable to its
declaration. Since attribute grammars are capable of describing the full semantics of
a language, their use can be extended to interpretation or code generation, although
other techniques are perhaps more usual. There is much theory about them, but
they are less well standardized than regular expressions and context-free grammars.
Attribute grammars are covered in Section 4.1.
Manual object code generation for a given machine involves a lot of nitty-gritty
programming, but the process can be automated, for example by using pattern
matching and dynamic programming. Quite a number of formalisms have been de-
signed for the description of target code, both at the assembly and the binary level,
but none of these has gained wide acceptance to date and each compiler writing
system has its own version. Automated code generation is treated in Section 9.1.4.
28. 8 1 Introduction
1.1.1.3 Use of program-generating tools
Once one has the proper formalism in which to describe what a program should
do, one can generate a program from it, using a program generator. Examples are
lexical analyzers generated from regular descriptions of the input, parsers generated
from grammars (syntax descriptions), and code generators generated from machine
descriptions. All these are generally more reliable and easier to debug than their
handwritten counterparts; they are often more efficient too.
Generating programs rather than writing them by hand has several advantages:
• The input to a program generator is of a much higher level of abstraction than the
handwritten program would be. The programmer needs to specify less, and the
tools take responsibility for much error-prone housekeeping. This increases the
chances that the program will be correct. For example, it would be cumbersome
to write parse tables by hand.
• The use of program-generating tools allows increased flexibility and modifiabil-
ity. For example, if during the design phase of a language a small change in the
syntax is considered, a handwritten parser would be a major stumbling block
to any such change. With a generated parser, one would just change the syntax
description and generate a new parser.
• Pre-canned or tailored code can be added to the generated program, enhancing
its power at hardly any cost. For example, input error handling is usually a dif-
ficult affair in handwritten parsers; a generated parser can include tailored error
correction code with no effort on the part of the programmer.
• A formal description can sometimes be used to generate more than one type of
program. For example, once we have written a grammar for a language with the
purpose of generating a parser from it, we may use it to generate a syntax-directed
editor, a special-purpose program text editor that guides and supports the user in
editing programs in that language.
In summary, generated programs may be slightly more or slightly less efficient than
handwritten ones, but generating them is so much more efficient than writing them
by hand that whenever the possibility exists, generating a program is almost always
to be preferred.
The technique of creating compilers by program-generating tools was pioneered
by Brooker et al. in 1963 [51], and its importance has continually risen since. Pro-
grams that generate parts of a compiler are sometimes called compiler compilers,
although this is clearly a misnomer. Yet, the term lingers on.
1.1.2 Compiler construction has a wide applicability
Compiler construction techniques can be and are applied outside compiler construc-
tion in its strictest sense. Alternatively, more programming can be considered com-
piler construction than one would traditionally assume. Examples are reading struc-
29. 1.2 A simple traditional modular compiler/interpreter 9
tured data, rapid introduction of new file formats, and general file conversion prob-
lems. Also, many programs use configuration or specification files which require
processing that is very similar to compilation, if not just compilation under another
name.
If input data has a clear structure it is generally possible to write a grammar
for it. Using a parser generator, a parser can then be generated automatically. Such
techniques can, for example, be applied to rapidly create “read” routines for HTML
files, PostScript files, etc. This also facilitates the rapid introduction of new formats.
Examples of file conversion systems that have profited considerably from compiler
construction techniques are TeX text formatters, which convert TeX text to dvi for-
mat, and PostScript interpreters, which convert PostScript text to image rendering
instructions for a specific printer.
1.1.3 Compilers contain generally useful algorithms
A third reason to study compiler construction lies in the generally useful data struc-
tures and algorithms compilers contain. Examples are hashing, precomputed tables,
the stack mechanism, garbage collection, dynamic programming, and graph algo-
rithms. Although each of these can be studied in isolation, it is educationally more
valuable and satisfying to do so in a meaningful context.
1.2 A simple traditional modular compiler/interpreter
In this section we will show and discuss a simple demo compiler and interpreter, to
introduce the concepts involved and to set the framework for the rest of the book.
Turning to Figure 1.2, we see that the heart of a compiler is the semantic represen-
tation of the program being compiled. This semantic representation takes the form
of a data structure, called the “intermediate code” of the compiler. There are many
possibilities for the form of the intermediate code; two usual choices are linked lists
of pseudo-instructions and annotated abstract syntax trees. We will concentrate here
on the latter, since the semantics is primarily attached to the syntax tree.
1.2.1 The abstract syntax tree
The syntax tree of a program text is a data structure which shows precisely how
the various segments of the program text are to be viewed in terms of the grammar.
The syntax tree can be obtained through a process called “parsing”; in other words,
30. 10 1 Introduction
parsing1 is the process of structuring a text according to a given grammar. For this
reason, syntax trees are also called parse trees; we will use the terms interchange-
ably, with a slight preference for “parse tree” when the emphasis is on the actual
parsing. Conversely, parsing is also called syntax analysis, but this has the problem
that there is no corresponding verb “to syntax-analyze”. The parser can be written
by hand if the grammar is very small and simple; for larger and/or more complicated
grammars it can be generated by a parser generator. Parser generators are discussed
in Chapter 3.
The exact form of the parse tree as required by the grammar is often not the
most convenient one for further processing, so usually a modified form of it is used,
called an abstract syntax tree, or AST. Detailed information about the semantics
can be attached to the nodes in this tree through annotations, which are stored in
additional data fields in the nodes; hence the term annotated abstract syntax tree.
Since unannotated ASTs are of limited use, ASTs are always more or less annotated
in practice, and the abbreviation “AST” is used also for annotated ASTs.
Examples of annotations are type information (“this assignment node concerns
a Boolean array assignment”) and optimization information (“this expression does
not contain a function call”). The first kind is related to the semantics as described in
the manual, and is used, among other things, for context error checking. The second
kind is not related to anything in the manual but may be important for the code
generation phase. The annotations in a node are also called the attributes of that
node and since a node represents a grammar symbol, one also says that the grammar
symbol has the corresponding attributes. It is the task of the context handling module
to determine and place the annotations or attributes.
Figure 1.5 shows the expression b*b − 4*a*c as a parse tree; the grammar used
for expression is similar to those found in the Pascal, Modula-2, or C manuals:
expression → expression ’+’ term | expression ’−’ term | term
term → term ’*’ factor | term ’/’ factor | factor
factor → identifier | constant | ’(’ expression ’)’
Figure 1.6 shows the same expression as an AST and Figure 1.7 shows it as an
annotated AST in which possible type and location information has been added. The
precise nature of the information is not important at this point. What is important is
that we see a shift in emphasis from syntactic structure to semantic contents.
Usually the grammar of a programming language is not specified in terms of
input characters but of input “tokens”. Examples of input tokens are identifiers (for
example length or a5), strings (Hello!, !@#), numbers (0, 123e−5), keywords
(begin, real), compound operators (++, :=), separators (;, [), etc. Input tokens may
be and sometimes must be separated by white space, which is otherwise ignored. So
before feeding the input program text to the parser, it must be divided into tokens.
Doing so is the task of the lexical analyzer; the activity itself is sometimes called
“to tokenize”, but the literary value of that word is doubtful.
1 In linguistic and educational contexts, the verb “to parse” is also used for the determination of
word classes: determining that in “to go by” the word “by” is an adverb and in “by the way” it is a
preposition. In computer science the word is used exclusively to refer to syntax analysis.
31. 1.2 A simple traditional modular compiler/interpreter 11
expression
identifier constant
factor identifier factor identifier
term factor term factor identifier
term term factor
expression term
’*’
’b’
’b’
’4’
’a’
’c’
’*’
’*’
’−’
Fig. 1.5: The expression b*b − 4*a*c as a parse tree
’b’ ’b’
’*’
’−’
’*’
’*’ ’c’
’a’
’4’
Fig. 1.6: The expression b*b − 4*a*c as an AST
32. 12 1 Introduction
type: real
loc: var b
type: real
loc: tmp1
type: real
loc: tmp1
type: real
loc: var b
type: real
loc: const
type: real
loc: var a
loc: tmp2
type: real type: real
loc: var c
type: real
loc: tmp2
’b’
’*’
’−’
’b’
’4’ ’a’
’c’
’*’
’*’
Fig. 1.7: The expression b*b − 4*a*c as an annotated AST
1.2.2 Structure of the demo compiler
We see that the front-end in Figure 1.2 must at least contain a lexical analyzer, a
syntax analyzer (parser), and a context handler, in that order. This leads us to the
structure of the demo compiler/interpreter shown in Figure 1.8.
Syntax
analysis
Lexical
analysis
Code
Context
handling
generation
Interpretation
code
(AST)
Intermediate
Fig. 1.8: Structure of the demo compiler/interpreter
The back-end allows two intuitively different implementations: a code generator
and an interpreter. Both use the AST, the first for generating machine code, the
second for performing the implied actions immediately.
33. 1.2 A simple traditional modular compiler/interpreter 13
1.2.3 The language for the demo compiler
To keep the example small and to avoid the host of detailed problems that marks
much of compiler writing, we will base our demonstration compiler on fully paren-
thesized expressions with operands of one digit. An arithmetic expression is “fully
parenthesized” if each operator plus its operands is enclosed in a set of parenthe-
ses and no other parentheses occur. This makes parsing almost trivial, since each
open parenthesis signals the start of a lower level in the parse tree and each close
parenthesis signals the return to the previous, higher level: a fully parenthesized
expression can be seen as a linear notation of a parse tree.
expression → digit | ’(’ expression operator expression ’)’
operator → ’+’ | ’*’
digit → ’0’ | ’1’ | ’2’ | ’3’ | ’4’ | ’5’ | ’6’ | ’7’ | ’8’ | ’9’
Fig. 1.9: Grammar for simple fully parenthesized expressions
To simplify things even further, we will have only two operators, + and *. On the
other hand, we will allow white space, including tabs and newlines, in the input. The
grammar in Figure 1.9 produces such forms as 3, (5+8), and (2*((3*4)+9)).
Even this almost trivial language allows us to demonstrate the basic principles of
both compiler and interpreter construction, with the exception of context handling:
the language just has no context to handle.
#include parser.h /* for type AST_node */
#include backend.h /* for Process() */
#include error .h /* for Error() */
int main(void) {
AST_node *icode;
if (!Parse_program(icode)) Error(No top−level expression);
Process(icode);
return 0;
}
Fig. 1.10: Driver for the demo compiler
Figure 1.10 shows the driver of the compiler/interpreter, in C. It starts by includ-
ing the definition of the syntax analyzer, to obtain the definitions of type AST_node
and of the routine Parse_program(), which reads the program and constructs the
AST. Next it includes the definition of the back-end, to obtain the definition of the
routine Process(), for which either a code generator or an interpreter can be linked
in. It then calls the front-end and, if it succeeds, the back-end.
34. 14 1 Introduction
(It should be pointed out that the condensed layout used for the program texts
in the following sections is not really favored by any of the authors but is solely
intended to keep each program text on a single page. Also, the #include commands
for various system routines have been omitted.)
1.2.4 Lexical analysis for the demo compiler
The tokens in our language are (, ), +, *, and digit. Intuitively, these are five different
tokens, but actually digit consists of ten tokens, for a total of 14. Our intuition is
based on the fact that the parser does not care exactly which digit it sees; so as
far as the parser is concerned, all digits are one and the same token: they form a
token class. On the other hand, the back-end is interested in exactly which digit is
present in the input, so we have to preserve the digit after all. We therefore split the
information about a token into two parts, the class of the token and its representation.
This is reflected in the definition of the type Token_type in Figure 1.11, which has
two fields, one for the class of the token and one for its representation.
/* Define class constants */
/* Values 0−255 are reserved for ASCII characters */
#define EoF 256
#define DIGIT 257
typedef struct {int class; char repr;} Token_type;
extern Token_type Token;
extern void get_next_token(void);
Fig. 1.11: Header file lex.h for the demo lexical analyzer
For token classes that contain only one token which is also an ASCII character
(for example +), the class is the ASCII value of the character itself. The class of
digits is DIGIT, which is defined in lex.h as 257, and the repr field is set to the
representation of the digit. The class of the pseudo-token end-of-file is EoF, which
is defined as 256; it is useful to treat the end of the file as a genuine token. These
numbers over 255 are chosen to avoid collisions with any ASCII values of single
characters.
The representation of a token has at least two important uses. First, it is processed
in one or more phases after the parser to produce semantic information; examples
are a numeric value produced from an integer token, and an identification in some
form from an identifier token. Second, it is used in error messages, to display the
exact form of the token. In this role the representation is useful for all tokens, not just
for those that carry semantic information, since it enables any part of the compiler
to produce directly the correct printable version of any token.
35. 1.2 A simple traditional modular compiler/interpreter 15
The representation of a token is usually a string, implemented as a pointer, but in
our demo compiler all tokens are single characters, so a field of type char suffices.
The implementation of the demo lexical analyzer, as shown in Figure 1.12,
defines a global variable Token and a procedure get_next_token(). A call to
get_next_token() skips possible layout characters (white space) and stores the next
single character as a (class, repr) pair in Token. A global variable is appropriate here,
since the corresponding input file is also global. In summary, a stream of tokens can
be obtained by calling get_next_token() repeatedly.
#include lex.h /* for self check */
/* PRIVATE */
static int Is_layout_char(int ch) {
switch (ch) {
case ’ ’ : case ’ t ’ : case ’n’: return 1;
default: return 0;
}
}
/* PUBLIC */
Token_type Token;
void get_next_token(void) {
int ch;
/* get a non−layout character: */
do {
ch = getchar();
if (ch 0) {
Token.class = EoF; Token.repr = ’#’;
return;
}
} while (Is_layout_char(ch));
/* classify it : */
if ( ’0’ = ch ch = ’9’) {Token.class = DIGIT;}
else {Token.class = ch;}
Token.repr = ch;
}
Fig. 1.12: Lexical analyzer for the demo compiler
1.2.5 Syntax analysis for the demo compiler
It is the task of syntax analysis to structure the input into an AST. The grammar in
Figure 1.9 is so simple that this can be done by two simple Boolean read routines,
Parse_operator() for the non-terminal operator and Parse_expression() for the non-
36. 16 1 Introduction
terminal expression. Both routines are shown in Figure 1.13 and the driver of the
parser, which contains the initial call to Parse_expression(), is in Figure 1.14.
static int Parse_operator(Operator *oper) {
if (Token.class == ’+’) {
*oper = ’+’; get_next_token(); return 1;
}
if (Token.class == ’*’ ) {
*oper = ’*’ ; get_next_token(); return 1;
}
return 0;
}
static int Parse_expression(Expression **expr_p) {
Expression *expr = *expr_p = new_expression();
/* try to parse a digit : */
if (Token.class == DIGIT) {
expr−type = ’D’; expr−value = Token.repr − ’0’;
get_next_token();
return 1;
}
/* try to parse a parenthesized expression: */
if (Token.class == ’( ’ ) {
expr−type = ’P’;
get_next_token();
if (!Parse_expression(expr−left)) {
Error(Missing expression);
}
if (!Parse_operator(expr−oper)) {
Error(Missing operator);
}
if (!Parse_expression(expr−right)) {
Error(Missing expression);
}
if (Token.class != ’ ) ’ ) {
Error(Missing ) );
}
get_next_token();
return 1;
}
/* failed on both attempts */
free_expression(expr); return 0;
}
Fig. 1.13: Parsing routines for the demo compiler
Each of the routines tries to read the syntactic construct it is named after, using
the following strategy. The routine for the non-terminal N tries to read the alter-
37. 1.2 A simple traditional modular compiler/interpreter 17
#include stdlib .h
#include lex.h
#include error .h /* for Error() */
#include parser.h /* for self check */
/* PRIVATE */
static Expression *new_expression(void) {
return (Expression *)malloc(sizeof (Expression));
}
static void free_expression(Expression *expr) {free((void *)expr);}
static int Parse_operator(Operator *oper_p);
static int Parse_expression(Expression **expr_p);
/* PUBLIC */
int Parse_program(AST_node **icode_p) {
Expression *expr;
get_next_token(); /* start the lexical analyzer */
if (Parse_expression(expr)) {
if (Token.class != EoF) {
Error(Garbage after end of program);
}
*icode_p = expr;
return 1;
}
return 0;
}
Fig. 1.14: Parser environment for the demo compiler
natives of N in order. For each alternative A it tries to read its first member A1. If
A1 is found present, the routine assumes that A is the correct alternative and it then
requires the presence of the other members of A. This assumption is not always
warranted, which is why this parsing method is quite weak. But for the grammar of
Figure 1.9 the assumption holds.
If the routine succeeds in reading the syntactic construct in this way, it yields
a pointer to the corresponding AST as an output parameter, and returns a 1 for
success; the output parameter is implemented as a pointer to the location where the
output value must be stored, a usual technique in C. If the routine fails to find the
first member of any alternative of N, it does not consume any input, does not set
its output parameter, and returns a 0 for failure. And if it gets stuck in the middle it
stops with a syntax error message.
The C template used for a rule
P → A1 A2 . . . An | B1 B2 . . . | . . .
is presented in Figure 1.15. More detailed code is required if any of Ai, Bi, . . . , is a
terminal symbol; see the examples in Figure 1.13. An error in the input is detected
when we require a certain syntactic construct and find it is not there. We then give
38. 18 1 Introduction
an error message by calling Error() with an appropriate message; this routine does
not return and terminates the program, after displaying the message to the user.
int P(...) {
/* try to parse the alternative A1 A2 ... An */
if (A1(...)) {
if (!A2(...)) Error(Missing A2);
...
if (!An(...)) Error(Missing An);
return 1;
}
/* try to parse the alternative B1 B2 ... */
if (B1(...)) {
if (!B2(...)) Error(Missing B2);
...
return 1;
}
...
/* failed to find any alternative of P */
return 0;
}
Fig. 1.15: A C template for the grammar rule P → A1A2...An|B1B2...|...
This approach to parsing is called “recursive descent parsing”, because a set of
routines descend recursively to construct the parse tree. It is a rather weak parsing
method and makes for inferior error diagnostics, but is, if applicable at all, very sim-
ple to implement. Much stronger parsing methods are discussed in Chapter 3, but
recursive descent is sufficient for our present needs. The recursive descent parsing
presented here is not to be confused with the much stronger predictive recursive
descent parsing, which is discussed amply in Section 3.4.1. The latter is an imple-
mentation of LL(1) parsing, and includes having look-ahead sets to base decisions
on.
Although in theory we should have different node types for the ASTs of different
syntactic constructs, it is more convenient to group them in broad classes and have
only one node type for each of these classes. This is one of the differences between
the parse tree, which follows the grammar faithfully, and the AST, which serves the
convenience of the compiler writer. More in particular, in our example all nodes in
an expression are of type Expression, and, since we have only expressions, that is
the only possibility for the type of AST_node. To differentiate the nodes of type
Expression, each such node contains a type attribute, set with a characteristic value:
’D’ for a digit and ’P’ for a parenthesized expression. The type attribute tells us how
to interpret the fields in the rest of the node. Such interpretation is needed in the
code generator and the interpreter. The header file with the definition of node type
Expression is shown in Figure 1.16.
The syntax analysis module shown in Figure 1.14 defines a single Boolean rou-
tine Parse_program() which tries to read the program as an expression by calling
39. 1.2 A simple traditional modular compiler/interpreter 19
typedef int Operator;
typedef struct _expression {
char type; /* ’D’ or ’P’ */
int value; /* for ’D’ */
struct _expression *left , * right ; /* for ’P’ */
Operator oper; /* for ’P’ */
} Expression;
typedef Expression AST_node; /* the top node is an Expression */
extern int Parse_program(AST_node **);
Fig. 1.16: Parser header file for the demo compiler
Parse_expression() and, if it succeeds, converts the pointer to the expression to a
pointer to AST_node, which it subsequently yields as its output parameter. It also
checks if the input is indeed finished after the expression.
Figure 1.17 shows the AST that results from parsing the expression
(2*((3*4)+9)). Depending on the value of the type attribute, a node contains
either a value attribute or three attributes left, oper, and right. In the diagram, the
non-applicable attributes have been crossed out in each node.
’D’
’D’ ’D’
’D’
type
oper
value
’P’
’P’
’P’
*
*
+
2
3 4
9
left right
Fig. 1.17: An AST for the expression (2*((3*4)+9))
40. 20 1 Introduction
1.2.6 Context handling for the demo compiler
As mentioned before, there is no context to handle in our simple language. We could
have introduced the need for some context handling in the form of a context check
by allowing the logical values t and f as additional operands (for true and false) and
defining + as logical or and * as logical and. The context check would then be that
the operands must be either both numeric or both logical. Alternatively, we could
have collected optimization information, for example by doing all arithmetic that
can be done at compile time. Both would have required code that is very similar to
that shown in the code generation and interpretation sections below. (Also, the op-
timization proposed above would have made the code generation and interpretation
trivial!)
1.2.7 Code generation for the demo compiler
The code generator receives the AST (actually a pointer to it) and generates code
from it for a simple stack machine. This machine has four instructions, which work
on integers:
PUSH n pushes the integer n onto the stack
ADD replaces the topmost two elements by their sum
MULT replaces the topmost two elements by their product
PRINT pops the top element and prints its value
The module, which is shown in Figure 1.18, defines one routine Process() with one
parameter, a pointer to the AST. Its purpose is to emit—to add to the object file—
code with the same semantics as the AST. It first generates code for the expression
by calling Code_gen_expression() and then emits a PRINT instruction. When run,
the code for the expression will leave its value on the top of the stack where PRINT
will find it; at the end of the program run the stack will again be empty (provided
the machine started with an empty stack).
The routine Code_gen_expression() checks the type attribute of its parameter to
see if it is a digit node or a parenthesized expression node. In both cases it has to
generate code to put the eventual value on the top of the stack. If the input node is a
digit node, the routine obtains the value directly from the node and generates code
to push it onto the stack: it emits a PUSH instruction. Otherwise the input node is a
parenthesized expression node; the routine first has to generate code for the left and
right operands recursively, and then emit an ADD or MULT instruction.
When run with the expression (2*((3*4)+9)) as input, the compiler that
results from combining the above modules produces the following code:
41. 1.2 A simple traditional modular compiler/interpreter 21
#include parser.h /* for types AST_node and Expression */
#include backend.h /* for self check */
/* PRIVATE */
static void Code_gen_expression(Expression *expr) {
switch (expr−type) {
case ’D’:
printf (PUSH %dn, expr−value);
break;
case ’P’:
Code_gen_expression(expr−left);
Code_gen_expression(expr−right);
switch (expr−oper) {
case ’+’: printf (ADDn); break;
case ’*’ : printf (MULTn); break;
}
break;
}
}
/* PUBLIC */
void Process(AST_node *icode) {
Code_gen_expression(icode); printf(PRINTn);
}
Fig. 1.18: Code generation back-end for the demo compiler
PUSH 2
PUSH 3
PUSH 4
MULT
PUSH 9
ADD
MULT
PRINT
1.2.8 Interpretation for the demo compiler
The interpreter (see Figure 1.19) is very similar to the code generator. Both perform
a depth-first scan of the AST, but where the code generator emits code to have the
actions performed by a machine at a later time, the interpreter performs the actions
right away. The extra set of braces ({. . . }) after case ’P’: is needed because we need
two local variables and the C language does not allow declarations in the case parts
of a switch statement.
Note that the code generator code (Figure 1.18) and the interpreter code (Figure
1.19) share the same module definition file (called a “header file” in C), backend.h,
shown in Figure 1.20. This is possible because they both implement the same inter-
face: a single routine Process(AST_node *). Further on we will see an example of
a different type of interpreter (Section 6.3) and two other code generators (Section
42. 22 1 Introduction
#include parser.h /* for types AST_node and Expression */
#include backend.h /* for self check */
/* PRIVATE */
static int Interpret_expression(Expression *expr) {
switch (expr−type) {
case ’D’:
return expr−value;
break;
case ’P’: {
int e_left = Interpret_expression(expr−left);
int e_right = Interpret_expression(expr−right);
switch (expr−oper) {
case ’+’: return e_left + e_right;
case ’*’ : return e_left * e_right;
}}
break;
}
}
/* PUBLIC */
void Process(AST_node *icode) {
printf (%dn, Interpret_expression(icode));
}
Fig. 1.19: Interpreter back-end for the demo compiler
7.5.1), each using this same interface. Another module that implements the back-
end interface meaningfully might be a module that displays the AST graphically.
Each of these can be combined with the lexical and syntax modules, to produce a
program processor.
extern void Process(AST_node *);
Fig. 1.20: Common back-end header for code generator and interpreter
1.3 The structure of a more realistic compiler
Figure 1.8 showed that in order to describe the demo compiler we had to decompose
the front-end into three modules and that the back-end could stay as a single module.
It will be clear that this is not sufficient for a real-world compiler. A more realistic
picture is shown in Figure 1.21, in which front-end and back-end each consists of
five modules. In addition to these, the compiler will contain modules for symbol
table handling and error reporting; these modules will be called upon by almost all
other modules.
