The document summarizes the key ideas from the book "The Programmer's Apprentice" which aimed to provide software engineers with intelligent computer assistance. It discusses the concepts of clichés, the Plan Calculus representation, and overlays which allow relationships between clichés to be represented. The system demonstrated capabilities like shared knowledge, automated documentation, and programming language independence through the use of these concepts.

1. 1
The Programmer’s
Apprentice
Charles Rich
Richard C. Waters
Presented by
Yann-Gaël Guéhéneuc

2. 2
2/40
Charles Rich, Richard C. Waters ;
The Programmer’s Apprentice ;
ACM Press – Frontier Series,
Addison-Wesley,
ISBN 0-201-52425-2,
1990.

3. 3
3/40
Why this book?
It covers software engineering and
artificial intelligence
Theoretical and practical
Notion of “patterns” in 1975!
– 12 years before 1987…
– 20 years before the GoF…
Much inspiration for our projects
In particular, they tackled problems similar to ours, except that their solutions are
for Lisp and Ada, while our projects relate to object-oriented programming
languages.

4. 4
4/40
Summary
Context and philosophy
Goal
Key notions
Overview of the results
Summary
What the future should have been…
Limits? And now?

5. 5
5/40
Context and philosophy
1975 – 1990
Artificial Intelligence
Software engineering
LISP and Ada
IBM’s Harlan Mills idea of
chief programmer teams
Study how software engineers analyze, synthesize, modify, specify, verify, and
document.
Artificial intelligence techniques to support software engineers.
Software engineering as a domain for investigating fundamental issues in
knowledge representation and reasoning.
Expert software engineers are surrounded by support staffs of human assistants
(junior programmers, testers, documentation writers, program librarians).

6. 6
6/40
Goal (1/2)
“[…] to provide software engineers with
intelligent assistance.” (p. 2)
“The intended interaction between a
software engineer […] is modeled after
interaction with a human assistant.”
(p. 2)
Here and in the following, all bold emphasises mine.
“Our goal is to provide every software engineer with a support team in the form
of an intelligent computer program […].” (p. 2)
“The engineer will make the important design decision and communicate them to
the Apprentice. The Apprentice will check for consistency and reasonableness
[…].” (p. 3)
Differences with a compiler. (p. 3)
No effort on the human interface, the communication with the Apprentice takes
place in the development environment through a specific command area. (p. 3-4)
Incremental automation through access to both the Apprentice and the underlying
development environment. (p. 4)

7. 7
7/40
Goal (2/2)
“Between the engineer and the
Apprentice […], the key to effective
cooperation is shared knowledge.” (p. 4)
Shared knowledge
– About the system being worked on
– About concepts of software engineering
⇒ Clichés
“This knowledge is of two kinds.” (p. 4)
“[…] this information […] must be stored on-line and in a form that can be
reasoned about.” (p. 4)
“It would be impossibly tedious if the Apprentice required engineers to describe
everything from first principles.” (p. 4)

8. 8
8/40
Key notion: Clichés (1/7)
In all engineering domains exist
standard, well-understood, primitive
building blocks
That’s a statement…

9. 9
9/40
Clichés (2/7)
Engineering methods
– Uniform general methods
• Finite element method (ME)
• Floyd-Hoare verification method (CS)
– Inspection methods
• Analysis and synthesis by inspection
Primitive building blocks
+ Inspection methods ⇒ Clichés
Uniform general methods: From first principles (from the ground up, each time).
Inspection methods: Pattern recognition.
Primitive building blocks + Inspection methods ⇒ Notion of clichés.

10. 10
10/40
Clichés (3/7)
Space of software clichés
Graphics
programming Statistical
programming
System
programming
Hardware
Geometry
Statistics
Basictechniques
I/O
methods
Numerical
methods
“The shaded intersection of the three areas represents basic software-engineering
techniques […].” (p. 19)
“The overlap between each pair of elliptical areas […] represents clichés of
intermediate generality.” (p. 20)
“The remaining part of each area […] represents the most specialized clichés.
Note that as software knowledge becomes more specialized, it begins to overlap
with domain knowledge (the dashed circles).” (p. 20)

11. 11
11/40
Clichés (4/7)
Examples
– Information system
– Linear search
“[…] a cliché contains both fixed parts
and parts that vary from one
occurrence of the cliché to the next […],
[it] may also include constraints […].”
(p. 12)
Clichés form the shared technical vocabulary of a discipline.
In literary criticism, cliché is negative; In software engineering, “the repeated use
of the same forms of expression is desirable.” (p. 12)
“Concepts similar to the idea of clichés appear in software engineering in the
work of …” (p. 12)

12. 12
12/40
Clichés (5/7)
Algorithmic clichés ≈ Program templates
“Non-algorithmic clichés are
represented […] using standard frame-
based knowledge representation
techniques […].” (p. 21)
Program templates with special features.
Frame-based knowledge representation techniques?

13. 13
13/40
Clichés (6/7)
Algorithmic clichés are more than
– Program templates
– Abstract syntax trees
Clichés result from an evolution of languages:
“The evolution of programming language is driven by at least two major forces.
One is the desire to move an entire class of decision making from the
programmer to the compiler. […] Language evolution is also driven by the
recognition of common patterns of usage (clichés) in the current language, which
are then incorporated into the syntax of the next language. […] From this point of
view, clichés can be viewed as recommendations for language design. […]
Moreover, this evolutionary process never stops—new clichés always arise at the
next higher level.” (p. 32)

14. 14
14/40
Clichés (7/7)
Key properties of Algorithmic clichés
– Canonical form
– Language independence
– Convenient manipulation
– Expressiveness
A representation of clichés
⇒ The Plan Calculus
“The scenarios […] may create the impression that an algorithmic cliché can be
represented most directly as a program template, i.e., a program text fragment
with holes to be filled in. […] program templates lack four key properties
essential to a knowledge representation for algorithmic clichés: canonical form,
language independence, convenience of manipulation, and expressiveness.”
(p. 23)
Canonical form: Abstract the semantics of program templates.
Language independence: Translation is a language independent form.
(Amortization of clichés libraries over the broadest user community)
(Automation of translation from subsets of Fortran, Cobol, Lisp, and Ada to the
Plan Calculus)
Convenient manipulation:Combination of clichés.
Expressiveness: Constraints on how the holes can be filled in.

15. 15
15/40
Key notion: Plan Calculus (1/6)
“The Plan Calculus is intended to be a
blueprint language for software.” (p. 29)
Formal representation for
– Programs
– Algorithmic clichés
No just another very high-level
programming language
“[…] it depends on exactly what is meant by the term programming language.”
(p. 30)
Programming language:
“To describe computations precisely enough to be executed by a machine”
+
“To serve as a communication medium between program writers and human
readers”
vs. Plan Calculus:
“To describe algorithmic clichés in a canonical, easy-to-combine, and language-
independent form.”
+
“To serve as a medium for the automated manipulation of programs.”
So, both compatible and not compatible.
Detail is often irrelevant to the algorithmic content of the code.
Information must be two-dimensional.
“How best to use graphics in software engineering is still an open research
question.” (p. 31)
The Plan Calculus is like a wide-spectrum language (very high-level languages).

16. 16
16/40
Plan Calculus (2/6)
Representation based on a graph
– Data flow
– Control flow
Genealogy
Subroutines Program generators
MacrosFlowcharts Program schemas
Flowchart schemas Logic
Program transformation Data abstractionFormal grammars
Plan calculus
“[…] it represents data and control flow explicitly.” (p. 24)
“[…] the kind of changes one typically wants to apply to an algorithm
correspond to changes that can be easily applied to a graph […].” (p. 27)
“In the semantics of a conventional programming language, every assignment
statement is a side effect. However, most assignment statements are merely part
of a pattern of assignments and variable references that implement the movement
of data from where it is produced to where it is used.” (p. 27)
“The plan for the enumeration cliché […] captures these constraints by restricting
the types of the corresponding boxes.” (p. 27-28)
Also, various levels of abstraction…

17. 17
17/40
Plan Calculus (3/6)
Parallel with electrical engineering plans
– “[…] a part is […] an operation, a test, or a
datum.” (p. 29)
– “Connections […] are […] data flow and
control flow.” (p. 29)
– “Constraints […] include the preconditions
and postconditions […] and the invariants
of data representations.” (p. 30)
Parts and connections may be more or less abstract.
“These constraints are expressed in the Programmer’s Apprentice using a
standard logical language.” (p. 30)

18. 18
18/40
Plan Calculus (4/6)
Parallel with artificial intelligence plans
– “[…] partial ordering of operations by data
flow and control flow.” (p. 30)
– “[…] symbolic evaluation of plans […]
very similar to symbolic evaluation […] in
planning in artificial intelligence.” (p. 30)
“[…] rather than forcing an arbitrary total ordering.” (p. 30)

19. 19
19/40
Plan Calculus (5/6)
Limits
– Canonical form: “Syntactically distinct
plans should also be semantically
distinct.” (p. 32)
– Non-local flow: Database updates/queries,
exceptional control flow
– Other kind of knowledge: Performance
properties of algorithms, domain
knowledge
Non-local flow: “Harel’s statecharts […] provide a nice solution to this problem
within a graphical formalism.” (p. 34)
Performance properties of algorithms: “Recent work in this area by Kant […]
starts with a program representation similar to the Plan Calculus.” (p. 34)
Domain knowledge: “[…] the problem of representing domain knowledge in
software is in principle equivalent to the general problem of knowledge
representation in artificial intelligence.” (p. 34-35)

20. 20
20/40
Plan Calculus (6/6)
The Plan Calculus defines parts and
connections
⇒ Plan diagrams
Plan diagrams represent clichés within the Plan Calculus .

21. 21
21/40
Key notion: Plan Diagrams (1/2)
The
Negate-if-negative
Cliché
if:negative
T F
then:negate
end:join
T F
“[…] control flow is depicted using arcs with double crosshatch marks between
exit points and entry points.” (p. 43)

22. 22
22/40
Plan Diagrams (2/2)
Parallel execution model
Hierarchical plans
Data plans
Side effects
Recursively defined plans
Relationships between clichés
⇒ Overlays
Parallel execution model: “The meaning of a plan diagram is defined formally as
the set of computation sequences the plan allows […]. However, a useful
intuitive model for plan diagrams is to imagine their direct execution as parallel
data-flow programs.” (p. 46)
Hierarchical: “Plan may contain other plans as parts.” (p. 48)
Data plans: “A data plan […] defines a […] collection of input/output
specifications for standard data structure accessors:
• A constructor […].
• A selector for each part […].
• A alterant for each part […].”
“The naming conventions for these accessors, their inputs, and their outputs, are
illustrated in Figure 4.10.” (p. 51)
“[…] a hierarchical plan may have a mixture of data and computational parts.”
(p. 51)
Side effects: “The formal statement of this condition involves use of a situational
calculus for modeling mutable objects […].” (p. 56)
Recursively defined plans: “Hierarchical plans can be recursively defined, i.e.,
the type of a subplan can be the same as the type of the plan.” (p. 57)
Relationships between clichés: “[…] a sequence can also be viewed as a list in
which the head is the first term o the sequence and the tail is the subsequence
starting as the second term. Similarly, Lisp list are an implementation of lists in
which the head is the car and tail is the cdr of a given cons cell. The facility in
the Plan Calculus for constructing views is called overlays […].” (p. 29)

23. 23
23/40
Key notion: Overlays (1/4)
“Programming knowledge includes
understanding […] relationships
between […] clichés […]” (p. 63)
“[…] such as how an instance of one cliché can be viewed as an instance of
another.” (p. 63)
“An overlay is a pair of plans with a set of correspondences between their parts.”
(p. 63)
“Basically, an overlay asserts than an instance of a plan on the left, viewed
through the correspondences, satisfies the constraints of the plan on the right.”
(p. 63)

24. 24
24/40
Overlays (2/4)
Implementing a specification
if:negative
T F
then:negate
end:join
T F
absolute-value
“The knowledge encoded in an overlay can be used in both the analysis and
synthesis of programs.” (p. 65)
“The repeated application of this recognition process can be thought of as parsing
[…].” (p. 65)
“[…] synthesis by inspection corresponds to using grammar to generate
sentences.” (p. 66)

25. 25
25/40
Overlays (3/4)
Optimization
– “[…] matching the right side and
replacing it by the left side […]” (p.71)
Implementing an abstract data structure
– “This means that the old indexed sequence
of bump-and-update viewed as a list
corresponds to the old input of push […]”
(p. 69)
Overlays also are useful for optimizations and the implementation of abstract
data structures.

26. 26
26/40
Overlays (4/4)
Temporal abstraction
next:apply-function
next:
input:
input:
continue:
generation
generation
generate
output:
*
“• Generate – given a
function (op) and an
initial value (init),
generate produces the
sequence [init, op(init),
op(op(init)), …].” (p. 76)
“A temporal correspondence links a data part in an iterative plan on the left side
of an overlay with a part of type sequence on the right side. The first term of this
temporal sequence is equal to the indicated part if the iterative plan; the second
term is equal to the same part of the recursively defined subplan; and so on
recursively.” (p. 77)
“The benefit of temporal overlays is that they make it possible to view a program
composed of iteration clichés as a composition of sequence functions, which is
much easier to understand and to reason about.” (p. 77)

27. 27
27/40
Overview of the results (1/6)
“The principal benefit of KBEmacs is
illustrated by the ration between the
size of the program […] and the size of
the command […].” (p. 79)
“This ration is seldom less than 2 to 1 and, as shown by the implementation of
the 57-line Ada program UNIT_REPAIR_REPORT using 6 commands […] can
reach 10 to 1 when constructing large programs.” (p. 79)

28. 28
28/40
Overview of the results (2/6)
“KBEmacs demonstrates […] key
capabilities […].” (p. 79)
– Shared knowledge
– Propagation of decisions
– Support for evolution
– Explanation
– Automation of details
– Escape to the surrounding environment
– Programming-language independence
Shared knowledge: Clichés.
Propagation of decisions: Using constraints, KBEmacs determines some of the
consequences of the design decisions.
Support for evolution: Traceability! Higher-level program representation and
manipulation.
Explanation: Automated documentation + Clichés traceability!
Automation of details: KBEmacs ensures clichés are combined correctly.
Escape to the surrounding environment: Text- and syntax-based editing
intermixed with knowledge-based editing.
Programming-language independence: Because of Plan Calculus but no program
translation!

29. 29
29/40
Overview of the results (3/6)
“[…] KBEmacs automatically applies
temporal abstraction to all iterative
programs.” (p. 77)
“KBEmacs tries to forestall problems
[…] by not taking the exact order of
program statements too literally.” (p. 142)
“The code […] is malformed because the expression used to fill the role refers to
the variable UNIT_KEY before it has been given a value. […] However, while
the title role is first […], there is no reason why this has to be so. The two
computations are not ordered by control flow or data flow.” (p. 142)

30. 30
30/40
Overview of the results (4/6)
Architecture of KBEmacs
Interface
Coder
Analyzer
Text editor Plan editor
Program text Program plan
Cliché library
Analyzer: “First, whenever the programmer textually modifies the program being
worked on, the analyzer creates a new plan for the program so that knowledge-
based editing can continue. Second, the analyzer is used to process cliché
definitions to create plans that are stored in the cliché library.” (p. 152)
Coder: “The coder is used to create new program text whenever the plan for the
program being edited is changed.” (p. 156)

31. 31
31/40
Overview of the results (5/6)
“[…] the central weakness of
KBEmacs—it has no general-purpose
reasoning facilities.” (p. 84)
– Error detection
– Reasonableness
– Limited cliché library
– Incomplete analysis
– (Needs reimplementation)
Error detection, no general-purpose reasoning:
“All of KBEmacs’s reasoning is achieved via special-purpose procedures.”
(p. 84)
“[…] KBEmacs has no general way to evaluate the reasonableness of a
programmer’s commands.” (p. 84)
“KBEmacs implements Plan Calculus constraints as procedures, rather than using
a general-purpose logical language.” (p. 84)
Limited cliché library: no support for “taxonomic relationships between clichés
such as specialization and extension.” (p. 84)
Incomplete analysis: Traceability! “This means that after the program text has
been modified, KBEmacs’s ability to explain the program and to modify the
program using knowledge-based commands is subsequently reduced. (p. 85)

32. 32
32/40
Overview of the results (6/6)
Recognizer
– Automated cliché recognition
• Syntactic variation
• Unrecognizable code
• Implementation variation
• Noncontiguousness
• Overlapping implementations
– Flow-graph parser
“There are both practical and theoretical motivations for automating cliché
recognition.” (p. 171)
“[…] a flow-graph parser developed by Brotsky, which is a generalization of
Earley’s algorithm […] for string parsing.” (p. 173)

33. 33
33/40
Summary
Goal: Programmer’s Apprentice
– Clichés
– Plan Calculus
– Plan diagrams
– Overlays
Implementation: KBEmacs

34. 34
34/40
What the future should have
been…
Full operational KBEmacs
Recognizer using comments and
mnemonic identifiers
“[…] a general-purpose reasoning
system (Cake)” (p. 84)
Teleological structure
Design and Requirements
Apprentices
Plan Calculus
Frames
Algebraic reasoning
Propositional logic
Comments and mnemonic identifiers: “The PUDSY system […] uses variables
names as hints to the clichés that might be present.” (p. 188)
Teleological structure:
“The teleological structure of a plan is the set of dependencies between the
preconditions and postconditions of the specifications for its parts.” (p. 195)

35. 35
35/40
Limits? (1/4)
Clichés
– Implementation-level, parameterized
(micro-, mini-) frameworks
– Many of them
• several hundreds just for the basic techniques
• only a few dozen implemented…
“We estimate that perhaps a thousand clichés are required to reach a reasonable
degree of completeness in basic techniques.” (p. 21)
“The second limitation of KBEmacs’s current cliché library is that it contains
only a few dozen clichés.” (p. 84)

36. 36
36/40
Limits? (2/4)
Clichés
– Cliché implementation (plan diagrams) is
language-dependent! Idioms…
“[…] a textual syntax is provided for
representing roles. […] special forms (in
Lisp and Ada) are provided for defining the
plan for a cliché.” (p. 148)
– Only program templates?
Cliché implementation is language-dependent!?
“This section presents several Ada clichés and contrasts them with their Lisp
counterparts.” (p. 118)

37. 37
37/40
Limits? (3/4)
Clichés and Plan Calculus
– Cliché’s complexity increases as cliché’s
size grows
– Clichés are not really ready for object-
orientation (future research…)
– Side effects express outside of the
language
Side effects: “The formal statement of this condition involves use of a situational
calculus for modeling mutable objects […].” (p. 56)

38. 38
38/40
Limits? (4/4)
Overlays
– How to count the number of elements in a
sequence?
KBEmacs
– Text-based operations/visualisation
– Fuzziness
– Demonstration system
Fuzziness: “KBEmacs tries to forestall problems […] by not taking the exact
order of program statements too literally.” (p. 142)
Demonstration system: “It is useful to distinguish three levels of systems:
demonstrations, prototypes, and fully operational systems.” (p. 168)

39. 39
39/40
And now? (1/2)
Among many others…
– The CatalogExplorer, Explainer, and
Modifier framework (Fisher, 1992)
– The System Architect’s Assistant
(Kramer, 1993)
– Task oriented software understanding
(Erdem, 1998)
– The PatternsBox/Ptidej projects (1999–?)
PatternsBox: http://www.emn.fr/albin/
Ptidej: http://www.yann-gael.gueheneuc.net/

40. 40
40/40
And now? (2/2)
Goal: Programmer’s Apprentice
– Clichés
– Plan Calculus
– Plan diagrams
– Overlays
Implementation: KBEmacs
Design patterns
+
Meta-modelling techniques
+
Constraints
PatternsBox
Ptidej
Design patterns and design defects