Lexical Analysis and Parsing
+
using C+
ii
Lexical Analysis and Parsing
+
using C+

Bruno R. Preiss
B.A.Sc., M.A.Sc., Ph.D., P.Eng.

MMIV
Copyright c 2004 by Bruno R. Preiss.
All rights reserved. No part of this publication may be reproduced, stored in a retri...
To Someone
vi
Contents
Preface

I

xiii

Lexis—The Program

1 Lexis Input Specifications
1.1 Lexis Token Specification . .
1.1.1 File . . ...
Contents

viii

7 CompressedDFA Class
7.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2...
Contents

ix

13 SymbolTable Class
13.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13.2 ...
Contents

x

21 ParseTable Class
171
21.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171...
Contents

xi

28 Stack Class Template
219
28.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...
Contents

xii
34.2.1 Declarations . . .
34.2.2 Definitions . . .
34.3 CodeFragment Class . .
34.3.1 Declarations . . .
34.3...
Preface
The preface goes here.
Bruno R. Preiss
Waterloo, Canada
March 7, 2004
xiv

Preface
Part I

Lexis—The Program
Chapter 1

Lexis Input Specifications
 

1.1

Lexis Token Specification

Lexis auxiliary definitions 1 ≡
1 blank = ’ ’.
2 tab...
Chapter 1. Lexis Input Specifications

4

Lexis token definitions 2 ≡
1 circumflex = ’^’.
2 colon = ’:’.
3 dollar = ’$’.
4 e...
1.2. Lexis Grammar Specification
¶5

Lexis start symbol definition
1 file.

5

5

≡

This code is used in ¶54.

Lexis gramma...
Chapter 1. Lexis Input Specifications

6
¶11

Lexis grammar productions 11 ≡
1 tokenDefinitionsPart =
2
‘‘
3
tokenDefinitio...
1.2. Lexis Grammar Specification
Lexis grammar productions 14 ≡
1 expression =
2
alternation
3
‘‘
4
expression case 1
5
’’
...
Chapter 1. Lexis Input Specifications

8
¶17

Lexis grammar productions 17 ≡
1 item =
2
questionMark
3
‘‘
4
item case 1 act...
1.2. Lexis Grammar Specification
¶20

9

ParserData method declarations 20 ≡
1 protected: ParserData();
This code is used i...
Chapter 1. Lexis Input Specifications

10
ItemAttributes method declarations
1 public: ItemAttributes();

25

≡

¶25

This ...
1.2. Lexis Grammar Specification

11

languageDefinitionPart case 1 actions
1 $$ = "UserDefined";

30

≡

¶30

31

≡

¶31

...
Chapter 1. Lexis Input Specifications

12
¶37

tokenDefinition case 1 actions 37 ≡
1 $expression->setAction(Lexis::discard)...
1.2. Lexis Grammar Specification
¶44

13

alternation case 2 actions 44 ≡
1 NFA* result = new NFA(*$alternation | *$alterna...
Chapter 1. Lexis Input Specifications

14
¶50

item case 4 actions 50 ≡
1 string src($characterClass.getLexeme());
2 bool c...
1.2. Lexis Grammar Specification
item case 6 actions 52 ≡
1 NFA* result = new NFA(!*$expression);
2 delete $expression;
3 $...
16

Chapter 1. Lexis Input Specifications
Chapter 2

State Class
 

2.1

Declarations

State class declaration 55 ≡
1 class State
2 {
3
State class field declaration...
Chapter 2. State Class

18
 

2.2
¶58

Definitions

State class inline method definitions
1 inline State::State() :
2
number...
2.2. Definitions
State method definitions 65 ≡
1 State& State::operator=(State const& s)
2 {
3
anchor = s.anchor;
4
action =...
Chapter 2. State Class

20
 

2.3
¶71

Files

File: Lexis/State.h 71 ≡
1 #if !defined(_Lexis_State_h)
2 #define _Lexis_Sta...
Chapter 3

NFAState Class
 

3.1

Declarations

NFAState class declaration 73 ≡
1 class NFAState : public State
2 {
3
Labe...
Chapter 3. NFAState Class

22
¶76

NFAState field
1 private:
2 private:
3 private:
4 private:
5 private:

declarations 76 ≡...
3.2. Definitions
¶81

23

NFAState class inline method definitions 81 ≡
1 inline ostream& operator<<(ostream& stream, NFASta...
Chapter 3. NFAState Class

24
¶85

NFAState method definitions 85 ≡
1 void NFAState::connect(char lab, StateNumber nxt)
2 {...
3.2. Definitions
NFAState method definitions 89 ≡
1 ostream& NFAState::put(ostream& s) const
2 {
3
s << "NFAState {";
4
Stat...
Chapter 3. NFAState Class

26
 

3.3
¶90

Files

File: Lexis/NFAState.h 90 ≡
1 #if !defined(_Lexis_NFAState_h)
2 #define _...
Chapter 4

NFA Class
 

4.1

Declarations

NFA class declaration 92 ≡
1 class NFA
2 {
3
NFA class field declarations 93
4
N...
Chapter 4. NFA Class

28

NFA class method declarations 94 ≡
1 public: explicit NFA(StateNumber = 1);
2 public: explicit N...
4.2. Definitions
NFA class inline method definitions 99 ≡
1 inline ostream& operator<<(ostream& stream, NFA const& nfa)
2
{ ...
Chapter 4. NFA Class

30

NFA class method definitions 103 ≡
1 NFA::NFA(NFA const& nfa) :
2
numberOfStates(nfa.numberOfStat...
4.2. Definitions
¶108

31

NFA class method definitions 108 ≡
1 void NFA::setAction(Action a)
2 {
3
assert(numberOfStates > ...
Chapter 4. NFA Class

32
¶111

NFA class method definitions 111 ≡
1 Set NFA::getEpsilonClosure(Set const& inputSet) const
2...
4.2. Definitions
NFA class method definitions 112 ≡
1 Set NFA::getMoveSet(Set const& inputSet, char c) const
2 {
3
Set resul...
Chapter 4. NFA Class

34
¶115

NFA class method definitions 115 ≡
1 string NFA::getDominantToken(Set const& inputSet) const...
4.3. Files

35

NFA class method definitions 118 ≡
1 NFA operator!(NFA const& nfa)
2 {
3
StateNumber const size = nfa.getNu...
Chapter 4. NFA Class

36
¶121

File: Lexis/NFA.cc 121 ≡
1 #include <iostream>
2 #include "Lexis/NFA.h"
3 #include "Lexis/N...
Chapter 5

DFAState Class
 

5.1

Declarations

DFAState class declaration 122 ≡
1 class DFAState : public State
2 {
3
DFA...
Chapter 5. DFAState Class

38
 

5.2

Definitions

DFAState class inline method definitions 125 ≡
1 inline Set DFAState::get...
5.3. Files

39

DFAState class method definitions 131 ≡
1 ostream& DFAState::put(ostream& s) const
2 {
3
s << "DFAState {";...
Chapter 5. DFAState Class

40
¶133

File: Lexis/DFAState.cc 133 ≡
1 #include <iostream>
2 #include "Lexis/DFAState.h"
3 #i...
Chapter 6

DFA State
 

6.1

Declarations

DFA class declaration 134 ≡
1 class DFA
2 {
3
DFA constants 135
4
DFA class fiel...
Chapter 6. DFA State

42

DFA class method declarations 138 ≡
1 public: explicit DFA(NFA const&);
2 public: ~DFA();
3 publ...
6.2. Definitions

43

DFA class method definitions 143 ≡
1 DFA::DFA(NFA const& nfa) :
2
numberOfStates(0),
3
state(maximumNu...
Chapter 6. DFA State

44
¶144

DFA class method definitions 144 ≡
1 DFA::~DFA()
2 {
3
for (StateNumber i = 0; i < numberOfS...
6.2. Definitions
DFA class method definitions 147 ≡
1 StateNumber DFA::initializeGroups(
2
vector<Set>& group,
3
vector<Stat...
Chapter 6. DFA State

46
¶148

DFA class method definitions 148 ≡
1 DFA& DFA::rebuild(StateNumber numberOfGroups,
2
vector<...
6.2. Definitions
DFA class method definitions 149 ≡
1 void DFA::minimize()
2 {
3
vector<Set> group(numberOfStates);
4
vector...
Chapter 6. DFA State

48

DFA class method definitions 150 ≡
1 bool DFA::isColumnEquivalent(char c1, char c2) const
2 {
3
f...
6.3. Files

49

File: Lexis/DFA.cc 153 ≡
1 #include <iostream>
2 #include "Lexis/DFA.h"
3 #include "Toolbox/EscapeSequence...
50

Chapter 6. DFA State
Chapter 7

CompressedDFA Class
 

7.1

Declarations

CompressedDFA class declaration 154 ≡
1 class CompressedDFA
2 {
3
Com...
Chapter 7. CompressedDFA Class

52
¶157

CompressedDFA class method declarations 157 ≡
1 public: explicit CompressedDFA(DF...
7.2. Definitions
construct columnTable and columnMap 159 ≡
1 vector<TableEntry> columnTable(maximumCharacter);
2 for (uint8...
Chapter 7. CompressedDFA Class

54
¶161

construct anchorTable, actionTable and tokenTable 161 ≡
1 for (StateNumber s = 0;...
7.2. Definitions
CompressedDFA class method definitions 165 ≡
1 ostream& CompressedDFA::putMap(ostream& stream,
2
char const...
Chapter 7. CompressedDFA Class

56
¶167

CompressedDFA class method definitions 167 ≡
1 ostream& CompressedDFA::putAnchorTa...
7.2. Definitions
CompressedDFA class method definitions 169 ≡
1 ostream& CompressedDFA::putTokenTable(ostream& stream) const...
Chapter 7. CompressedDFA Class

58
¶170

CompressedDFA class method definitions 170 ≡
1 ostream& CompressedDFA::putTransiti...
7.2. Definitions
CompressedDFA class method definitions 171 ≡
1 ostream& CompressedDFA::putTokens(ostream& stream) const
2 {...
Chapter 7. CompressedDFA Class

60

CompressedDFA class method definitions 172 ≡
1 ostream& CompressedDFA::putTables(ostrea...
7.3. Files

61

File: Lexis/CompressedDFA.cc 174 ≡
1 #include <iostream>
2 #include "Lexis/CompressedDFA.h"

¶174

3
4
5
6...
62

Chapter 7. CompressedDFA Class
Chapter 8

Lexical Analyzer Classes
 

8.1
8.1.1

Common Types and Constants
Type Declarations

Lexis type declarations 17...
Chapter 8. Lexical Analyzer Classes

64

8.1.3

Anchor Enumeration

Anchor enumeration declaration
1 enum Anchor
2 {
3
non...
8.2. Input Class Template

65

 

8.2
8.2.1

Input Class Template
Declarations

Input class template declaration 183 ≡
1 t...
Chapter 8. Lexical Analyzer Classes

66
¶188

Input class template method declarations 188 ≡
1 public: Input(istream&);
2 ...
8.2. Input Class Template
Input class template method definitions 191 ≡
1 template <typename T>
2 void Input<T>::checkAsser...
Chapter 8. Lexical Analyzer Classes

68
¶193

Input class template method definitions 193 ≡
1 template <typename T>
2 Input...
8.2. Input Class Template
Input class template method definitions 196 ≡
1 template <typename T>
2 Input<T>& Input<T>::opera...
Chapter 8. Lexical Analyzer Classes

70
¶199

Input class template method definitions 199 ≡
1 template <typename T>
2 void ...
8.3. Token Class Template

71

 

8.3
8.3.1

Token Class Template
Declarations

Token class template declaration 203 ≡
1 t...
Chapter 8. Lexical Analyzer Classes

72

8.3.2

Definitions

Token class template inline method definitions 207 ≡
1 template...
8.3. Token Class Template
Token class template method definitions 213 ≡
1 template <typename LT>
2 Token<LT>::Token(Token<L...
Chapter 8. Lexical Analyzer Classes

74
 

8.4
8.4.1
¶217

TokenStream Class Template
Declarations

TokenStream class temp...
8.4. TokenStream Class Template

8.4.2
¶223

75

Definitions

TokenStream class template inline method definitions 223 ≡
1 t...
Chapter 8. Lexical Analyzer Classes

76
¶228

TokenStream class template method definitions 228 ≡
1 template <typename T>
2...
8.4. TokenStream Class Template
TokenStream class template method definitions 230 ≡
1 template <typename T>
2 TokenStream<T...
Chapter 8. Lexical Analyzer Classes

78
¶232

normal transition processing 232 ≡
1 if(isAcceptOrDiscardState(currentState)...
8.5. Files

79

 

8.5

Files

File: Lexis/Lexis.h 235 ≡
1 #if !defined(_Lexis_Lexis_h)
2 #define _Lexis_Lexis_h

¶235

3
...
Chapter 8. Lexical Analyzer Classes

80
¶236

File: Lexis/Lexis.cc 236 ≡
1 #if !defined(_Lexis_Lexis_cc)
2 #define _Lexis_...
Chapter 9. Lexis Main Program

82

Chapter 9

Lexis Main Program
 

9.1
¶237

main Method Definition

Lexis main method defi...
9.1. main Method Definition
read the Lexis input file 238 ≡
1 cerr << "Reading input file:";
2 PathName argument(argv[1]);
3...
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Book
Upcoming SlideShare
Loading in …5
×

Book

292 views
171 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
292
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Book

  1. 1. Lexical Analysis and Parsing + using C+
  2. 2. ii
  3. 3. Lexical Analysis and Parsing + using C+ Bruno R. Preiss B.A.Sc., M.A.Sc., Ph.D., P.Eng. MMIV
  4. 4. Copyright c 2004 by Bruno R. Preiss. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the author. A This book was prepared with L TEX and reproduced from camera-ready copy supplied by the author. The book is typeset using the Computer Modern fonts designed by Donald E. Knuth with various additional glyphs designed by the author and implemented using . is a trademark of Addison Wesley Publishing Company. TEX is a trademark of the American Mathematical Society. UNIX is a registered trademark of AT&T Bell Laboratories.
  5. 5. To Someone
  6. 6. vi
  7. 7. Contents Preface I xiii Lexis—The Program 1 Lexis Input Specifications 1.1 Lexis Token Specification . . 1.1.1 File . . . . . . . . . . 1.2 Lexis Grammar Specification 1.2.1 ParserData Class . . 1.2.2 ItemAttributes Class 1.2.3 Actions . . . . . . . . 1.2.4 File . . . . . . . . . . 1 . . . . . . . 3 3 4 4 8 9 10 15 2 State Class 2.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 17 18 20 3 NFAState Class 3.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 21 22 26 4 NFA 4.1 4.2 4.3 Class Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 27 28 35 5 DFAState Class 5.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 37 38 39 6 DFA 6.1 6.2 6.3 41 41 42 48 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . State Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  8. 8. Contents viii 7 CompressedDFA Class 7.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 51 52 60 8 Lexical Analyzer Classes 8.1 Common Types and Constants 8.1.1 Type Declarations . . . 8.1.2 Constants . . . . . . . . 8.1.3 Anchor Enumeration . . 8.1.4 Action Enumeration . . 8.2 Input Class Template . . . . . 8.2.1 Declarations . . . . . . . 8.2.2 Definitions . . . . . . . 8.3 Token Class Template . . . . . 8.3.1 Declarations . . . . . . . 8.3.2 Definitions . . . . . . . 8.4 TokenStream Class Template . 8.4.1 Declarations . . . . . . . 8.4.2 Definitions . . . . . . . 8.5 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 63 63 63 64 64 65 65 66 71 71 72 74 74 75 79 9 Lexis Main Program 9.1 main Method Definition . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 82 84 II Gramatika—The Program 85 10 Gramatika Input Specifications 10.1 Gramatika Token Specifications . . 10.1.1 File . . . . . . . . . . . . . 10.2 Gramatika Grammar Specification 10.2.1 ParserData Class . . . . . 10.2.2 ItemAttributes Class . . . 10.2.3 Actions . . . . . . . . . . . 10.2.4 File . . . . . . . . . . . . . 87 87 88 88 92 93 94 97 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Symbol Class 99 11.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 11.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 11.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 12 Production Class 105 12.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 12.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 12.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
  9. 9. Contents ix 13 SymbolTable Class 13.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 113 114 123 14 Item Class 125 14.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 14.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 14.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 15 ItemSet Class and its Iterator 15.1 ItemSet Class . . . . . . . . . 15.1.1 Declarations . . . . . . 15.1.2 Definitions . . . . . . 15.2 ItemSet Iterator . . . . . . . 15.2.1 Declarations . . . . . . 15.2.2 Definitions . . . . . . 15.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 133 133 134 140 140 141 142 Iterator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 143 143 144 145 145 146 148 17 ActionRecord Class 17.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 149 150 151 16 Partition Class and its 16.1 Partition Class . . 16.1.1 Declarations . 16.1.2 Definitions . 16.2 Partition Iterator . 16.2.1 Declarations . 16.2.2 Definitions . 16.3 Files . . . . . . . . . 18 GotoRecord Class 153 18.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 18.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 18.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 19 State class 157 19.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 19.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 19.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 20 Automaton Class 165 20.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 20.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 20.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
  10. 10. Contents x 21 ParseTable Class 171 21.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 21.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 21.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 22 Parser Classes 22.1 Common Types and Constants . 22.1.1 Type Declarations . . . . 22.1.2 Constants . . . . . . . . . 22.1.3 TableEntry Struct . . . . 22.1.4 MapEntry Struct . . . . . 22.1.5 SymbolType Enumeration 22.1.6 Action Enumeration . . . 22.2 Value Class Template . . . . . . 22.2.1 Declarations . . . . . . . . 22.2.2 Definitions . . . . . . . . 22.3 StackItem Class Template . . . 22.3.1 Declarations . . . . . . . . 22.3.2 Definitions . . . . . . . . 22.4 Parser Class Template . . . . . . 22.4.1 Declarations . . . . . . . . 22.4.2 Definitions . . . . . . . . 22.5 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 181 181 181 182 182 183 183 184 184 185 186 186 187 188 188 189 193 23 Gramatika Main Program 195 23.1 main Method Definition . . . . . . . . . . . . . . . . . . . . . . . . . 195 23.2 File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 III The Toolbox 24 Character Utilities 24.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 201 201 201 202 25 String Utilities 205 25.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 25.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 25.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 26 EscapeSequence Utilities 209 26.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 26.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 26.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 27 PathName Class 215 27.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 27.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 27.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
  11. 11. Contents xi 28 Stack Class Template 219 28.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 28.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 28.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 29 Set Class and its Iterator 29.1 Set Class . . . . . . . . 29.1.1 Declarations . . . 29.1.2 Definitions . . . 29.2 Set Iterator Class . . . 29.2.1 Declarations . . . 29.2.2 Definitions . . . 29.3 Files . . . . . . . . . . . . . . . . . . 223 223 223 224 232 232 233 235 30 CharacterSet Class 30.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 237 237 238 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 List Utilities 241 31.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 31.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 31.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 32 HashTable Class and its 32.1 HashTable Class . . 32.1.1 Declarations . 32.1.2 Definitions . 32.2 HashTable Iterator . 32.2.1 Declarations . 32.2.2 Definitions . 32.3 Files . . . . . . . . . IV Iterator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Loma—The Program 33 Loma Input Specifications 33.1 Loma Token Specifications . . 33.1.1 File . . . . . . . . . . 33.2 Loma Grammar Specification . 33.2.1 ParserData Class . . 33.2.2 ItemAttributes Class 33.2.3 Actions . . . . . . . . 33.2.4 File . . . . . . . . . . 245 245 245 246 249 249 250 252 255 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 257 257 258 264 264 266 272 34 Fragment Class and its Descendants 34.1 Fragment Class . . . . . . . . . . . 34.1.1 Declarations . . . . . . . . . 34.1.2 Definitions . . . . . . . . . 34.2 TextFragment Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 273 273 274 275 . . . . . . . . . . . . . .
  12. 12. Contents xii 34.2.1 Declarations . . . 34.2.2 Definitions . . . 34.3 CodeFragment Class . . 34.3.1 Declarations . . . 34.3.2 Definitions . . . 34.4 IncludeFragment Class 34.4.1 Declarations . . . 34.4.2 Definitions . . . 34.5 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 275 276 276 276 278 278 278 280 35 Module Class and its Descendants 35.1 Module Class . . . . . . . . . . . 35.1.1 Declarations . . . . . . . . 35.1.2 Definitions . . . . . . . . 35.2 TextModule Class . . . . . . . . 35.2.1 Declarations . . . . . . . . 35.2.2 Definitions . . . . . . . . 35.3 FileModule Class . . . . . . . . 35.3.1 Declarations . . . . . . . . 35.3.2 Definitions . . . . . . . . 35.4 CodeModule Class . . . . . . . . 35.4.1 Declarations . . . . . . . . 35.4.2 Definitions . . . . . . . . 35.5 ExampleModule Class . . . . . . 35.5.1 Declarations . . . . . . . . 35.5.2 Definitions . . . . . . . . 35.6 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 283 283 284 287 287 287 288 288 288 289 289 290 292 292 292 293 36 Group Class 295 36.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 36.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 36.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 37 File Class 301 37.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 37.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 37.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 38 Loma Output Routines 38.1 Declarations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.2 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.3 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 307 307 310 39 Loma Main Program 311 39.1 File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 List of Modules 315 List of Symbols 349 List of Files 373
  13. 13. Preface The preface goes here. Bruno R. Preiss Waterloo, Canada March 7, 2004
  14. 14. xiv Preface
  15. 15. Part I Lexis—The Program
  16. 16. Chapter 1 Lexis Input Specifications   1.1 Lexis Token Specification Lexis auxiliary definitions 1 ≡ 1 blank = ’ ’. 2 tab = ’t’. 3 newline = ’n’. 4 whitespace = blank | tab | newline. 5 letter = /A-Za-z/. 6 octaldigit = /0-7/. 7 hexdigit = /0-9A-Fa-f/. 8 singleton = /ntvbrfa?’"//. 9 octalcode = octaldigit [ octaldigit [ octaldigit ] ]. 10 hexcode = ’x’ hexdigit { hexdigit }. 11 escapesequence = ’’ ( singleton | octalcode | hexcode ). This code is used in ¶3. ¶1
  17. 17. Chapter 1. Lexis Input Specifications 4 Lexis token definitions 2 ≡ 1 circumflex = ’^’. 2 colon = ’:’. 3 dollar = ’$’. 4 equal = ’=’. 5 leftBrace = ’{’. 6 leftBracket = ’[’. 7 leftParenthesis = ’(’. 8 period = ’.’. 9 questionMark = ’?’. 10 rightBrace = ’}’. 11 rightBracket = ’]’. 12 rightParenthesis = ’)’. 13 verticalBar = ’|’. 14 stringT = ’’’ { /!’/ | escapesequence } ’’’. 15 characterClass = ’/’ { /!// | escapesequence } ’/’. 16 definitions = ’Definitions’. 17 tokens = ’Tokens’. 18 language = ’Language’. 19 identifier = letter { letter }. 20 = whitespace { whitespace }. 21 = ’#’ { ? } $. ¶2 This code is used in ¶3. 1.1.1 ¶3 File File: Lexis/Lexis.lex 3 ≡ 1 Language: 2 LexisTokens. 3 Definitions: 4 Lexis auxiliary definitions 5 Tokens: 6 Lexis token definitions 2 1   1.2 ¶4 Lexis Grammar Specification Lexis non-terminal definitions 4 ≡ 1 alternation (nfa), alternative (nfa), auxiliaryDefinition, 2 auxiliaryDefinitionList, auxiliaryDefinitionsPart, 3 expression (nfa), file (nfa), item (nfa), 4 languageDefinitionPart (language), tokenDefinition (nfa), 5 tokenDefinitionList (nfa), tokenDefinitionsPart (nfa). This code is used in ¶54.
  18. 18. 1.2. Lexis Grammar Specification ¶5 Lexis start symbol definition 1 file. 5 5 ≡ This code is used in ¶54. Lexis grammar productions 6 ≡ 1 file = 2 languageDefinitionPart auxiliaryDefinitionsPart tokenDefinitionsPart 3 ‘‘ 4 file actions 29 5 ’’. ¶6 This code is used in ¶54. Lexis grammar productions 7 ≡ 1 languageDefinitionPart = 2 ‘‘ 3 languageDefinitionPart case 1 actions 4 ’’ 5 | language colon identifier period 6 ‘‘ 7 languageDefinitionPart case 2 actions 8 ’’ 9 . ¶7 30 31 This code is used in ¶54. Lexis grammar productions 8 ≡ 1 auxiliaryDefinitionsPart = 2 | definitions colon auxiliaryDefinitionList. ¶8 This code is used in ¶54. Lexis grammar productions 9 ≡ 1 auxiliaryDefinitionList = 2 | auxiliaryDefinitionList auxiliaryDefinition. ¶9 This code is used in ¶54. Lexis grammar productions 10 ≡ 1 auxiliaryDefinition = 2 identifier equal expression period 3 ‘‘ 4 auxiliaryDefinition actions 5 ’’. This code is used in ¶54. ¶10 32
  19. 19. Chapter 1. Lexis Input Specifications 6 ¶11 Lexis grammar productions 11 ≡ 1 tokenDefinitionsPart = 2 ‘‘ 3 tokenDefinitionsPart case 1 actions 4 ’’ 5 | tokens colon tokenDefinitionList 6 ‘‘ 7 tokenDefinitionsPart case 2 actions 8 ’’. 33 34 This code is used in ¶54. ¶12 Lexis grammar productions 12 ≡ 1 tokenDefinitionList = 2 ‘‘ 3 tokenDefinitionList case 1 actions 4 ’’ 5 | tokenDefinitionList tokenDefinition 6 ‘‘ 7 tokenDefinitionList case 2 actions 8 ’’. This code is used in ¶54. ¶13 Lexis grammar productions 13 ≡ 1 tokenDefinition = 2 equal expression period 3 ‘‘ 4 tokenDefinition case 1 actions 37 5 ’’ 6 | identifier equal expression period 7 ‘‘ 8 tokenDefinition case 2 actions 38 9 ’’. This code is used in ¶54. 35 36
  20. 20. 1.2. Lexis Grammar Specification Lexis grammar productions 14 ≡ 1 expression = 2 alternation 3 ‘‘ 4 expression case 1 5 ’’ 6 | circumflex alternation 7 ‘‘ 8 expression case 2 9 ’’ 10 | alternation dollar 11 ‘‘ 12 expression case 3 13 ’’ 14 | circumflex alternation 15 ‘‘ 16 expression case 4 17 ’’. 7 ¶14 actions 39 actions 40 actions 41 dollar actions 42 This code is used in ¶54. Lexis grammar productions 15 ≡ 1 alternation = 2 alternative 3 ‘‘ 4 alternation case 1 actions 43 5 ’’ 6 | alternation verticalBar alternative 7 ‘‘ 8 alternation case 2 actions 44 9 ’’. ¶15 This code is used in ¶54. Lexis grammar productions 16 ≡ 1 alternative = 2 ‘‘ 3 alternative case 1 actions 4 ’’ 5 | alternative item 6 ‘‘ 7 alternative case 2 actions 8 ’’. This code is used in ¶54. ¶16 45 46
  21. 21. Chapter 1. Lexis Input Specifications 8 ¶17 Lexis grammar productions 17 ≡ 1 item = 2 questionMark 3 ‘‘ 4 item case 1 actions 47 5 ’’ 6 | identifier 7 ‘‘ 8 item case 2 actions 48 9 ’’ 10 | stringT 11 ‘‘ 12 item case 3 actions 49 13 ’’ 14 | characterClass 15 ‘‘ 16 item case 4 actions 50 17 ’’ 18 | leftParenthesis expression rightParenthesis 19 ‘‘ 20 item case 5 actions 51 21 ’’ 22 | leftBracket expression rightBracket 23 ‘‘ 24 item case 6 actions 52 25 ’’ 26 | leftBrace expression rightBrace 27 ‘‘ 28 item case 7 actions 53 29 ’’. This code is used in ¶54. 1.2.1 ¶18 ParserData Class ParserData class declaration 18 ≡ 1 class ParserData 2 { 3 ParserData field declarations 19 4 ParserData method declarations 5 }; 20 This code is used in ¶22. ¶19 ParserData field declarations 19 ≡ 1 protected: HashTable<string,NFA*> definitionsTable; This code is used in ¶18.
  22. 22. 1.2. Lexis Grammar Specification ¶20 9 ParserData method declarations 20 ≡ 1 protected: ParserData(); This code is used in ¶18. ParserData class inline method definitions 21 ≡ 1 inline ParserData::ParserData() : 2 definitionsTable() 3 {} ¶21 This code is used in ¶22. File: Lexis/ParserData.h 22 ≡ 1 #if !defined(_Lexis_ParserData_h) 2 #define _Lexis_ParserData_h ¶22 3 4 5 6 7 #include #include #include #include <string> "Lexis/NFA.h" "Toolbox/HashTable.h" "Toolbox/String.h" 8 9 10 using namespace Toolbox; using namespace Toolbox::String; 11 12 13 14 15 16 namespace Lexis { ParserData class declaration 18 ParserData class inline method definitions } 21 17 18 #endif 1.2.2 ItemAttributes Class ItemAttributes class declaration 23 ≡ 1 class ItemAttributes 2 { 3 ItemAttributes field declarations 24 4 ItemAttributes method declarations 5 }; ¶23 25 This code is used in ¶27. ItemAttributes field declarations 1 public: NFA* nfa; 2 public: string language; This code is used in ¶23. 24 ≡ ¶24
  23. 23. Chapter 1. Lexis Input Specifications 10 ItemAttributes method declarations 1 public: ItemAttributes(); 25 ≡ ¶25 This code is used in ¶23. ¶26 ItemAttributes class inline method definitions 26 ≡ 1 inline ItemAttributes::ItemAttributes() : 2 nfa(0), 3 language() 4 {} This code is used in ¶27. ¶27 File: Lexis/ItemAttributes.h 27 ≡ 1 #if !defined(_Lexis_ItemAttributes_h) 2 #define _Lexis_ItemAttributes_h 3 4 5 #include <string> #include "Lexis/NFA.h" 6 7 8 using namespace Toolbox; using namespace Toolbox::String; 9 10 11 12 13 14 namespace Lexis { ItemAttributes class declaration 23 ItemAttributes class inline method definitions } 15 16 #endif 1.2.3 ¶28 Actions Lexis actions preamble 28 ≡ 1 #include <stdint.h> 2 #include "Lexis/ParserData.h" 3 #include "Lexis/ItemAttributes.h" 4 #include "Toolbox/CharacterSet.h" 5 #include "Toolbox/EscapeSequence.h" 6 7 using Lexis::NFA; This code is used in ¶54. ¶29 file actions 29 ≡ 1 NFA* nfa = $tokenDefinitionsPart; 2 nfa->setLanguage($languageDefinitionPart); 3 $$ = nfa; This code is used in ¶6. 26
  24. 24. 1.2. Lexis Grammar Specification 11 languageDefinitionPart case 1 actions 1 $$ = "UserDefined"; 30 ≡ ¶30 31 ≡ ¶31 This code is used in ¶7. languageDefinitionPart case 2 actions 1 $$ = $identifier.getLexeme(); This code is used in ¶7. auxiliaryDefinition actions 32 ≡ 1 HashTable<string,NFA*>::const_iterator ptr = 2 definitionsTable.find($identifier.getLexeme()), null; 3 if (ptr != null) 4 cerr << "Error: " << $identifier.getLexeme() << 5 " multiply defined.n"; 6 definitionsTable.add( 7 $identifier.getLexeme(), $expression); ¶32 This code is used in ¶10. tokenDefinitionsPart case 1 actions 1 $$ = 0; 33 ≡ ¶33 34 ≡ ¶34 This code is used in ¶11. tokenDefinitionsPart case 2 actions 1 $$ = $tokenDefinitionList; This code is used in ¶11. tokenDefinitionList case 1 actions 1 $$ = 0; 35 ≡ ¶35 This code is used in ¶12. tokenDefinitionList case 2 actions 36 ≡ 1 NFA* result = 0; 2 if ($tokenDefinitionList == 0) 3 { 4 result = $tokenDefinition; 5 } 6 else 7 { 8 assert($tokenDefinition != 0); 9 result = new NFA( 10 *$tokenDefinitionList | *$tokenDefinition); 11 delete $tokenDefinitionList; 12 delete $tokenDefinition; 13 } 14 $$ = result; This code is used in ¶12. ¶36
  25. 25. Chapter 1. Lexis Input Specifications 12 ¶37 tokenDefinition case 1 actions 37 ≡ 1 $expression->setAction(Lexis::discard); 2 $$ = $expression; This code is used in ¶13. ¶38 tokenDefinition case 2 actions 38 ≡ 1 $expression->setAction(Lexis::accept); 2 $expression->setToken($identifier.getLexeme()); 3 $$ = $expression; This code is used in ¶13. ¶39 expression case 1 actions 1 $$ = $alternation; 39 ≡ This code is used in ¶14. ¶40 expression case 2 actions 40 ≡ 1 NFA nl("n"); 2 NFA* result = new NFA(nl * *$alternation); 3 result->setAnchor(Lexis::head); 4 delete $alternation; 5 $$ = result; This code is used in ¶14. ¶41 expression case 3 actions 41 ≡ 1 NFA nl("n"); 2 NFA* result = new NFA(*$alternation * nl); 3 result->setAnchor(Lexis::tail); 4 delete $alternation; 5 $$ = result; This code is used in ¶14. ¶42 expression case 4 actions 42 ≡ 1 NFA nl("n"); 2 NFA* result = new NFA(nl * *$alternation * nl); 3 result->setAnchor(Lexis::both); 4 delete $alternation; 5 $$ = result; This code is used in ¶14. ¶43 alternation case 1 actions 1 $$ = $alternative; This code is used in ¶15. 43 ≡
  26. 26. 1.2. Lexis Grammar Specification ¶44 13 alternation case 2 actions 44 ≡ 1 NFA* result = new NFA(*$alternation | *$alternative); 2 delete $alternation; 3 delete $alternative; 4 $$ = result; This code is used in ¶15. alternative case 1 actions 1 $$ = new NFA(); 45 ≡ ¶45 This code is used in ¶16. alternative case 2 actions 46 ≡ 1 NFA* result = new NFA(*$alternative * *$item); 2 delete $alternative; 3 delete $item; 4 $$ = result; ¶46 This code is used in ¶16. item case 1 actions 47 ≡ 1 CharacterSet set; 2 set += ’n’; 3 set = ~set; 4 $$ = new NFA(set); ¶47 This code is used in ¶17. item case 2 actions 48 ≡ 1 HashTable<string,NFA*>::const_iterator ptr = 2 definitionsTable.find($identifier.getLexeme()), null; 3 NFA* result = 0; 4 if (ptr == null) 5 { 6 cerr << "Error: " << $identifier.getLexeme() << 7 " not defined.n"; 8 result = new NFA(); 9 } 10 else 11 result = new NFA(**ptr); 12 $$ = result; ¶48 This code is used in ¶17. item case 3 actions 49 ≡ 1 string src($stringT.getLexeme()); 2 3 4 $$ = new NFA(EscapeSequence::decode( src.substr(1, src.size()-2))); This code is used in ¶17. ¶49
  27. 27. Chapter 1. Lexis Input Specifications 14 ¶50 item case 4 actions 50 ≡ 1 string src($characterClass.getLexeme()); 2 bool complement = false; 3 4 5 6 7 8 9 10 11 assert(src.size() >= 2); if (src[1U] == ’!’) { complement = true; src = EscapeSequence::decode(src.substr(2, src.size()-3)); } else src = EscapeSequence::decode(src.substr(1, src.size()-2)); 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 CharacterSet dst; int16_t lastc = -1; string::iterator p(src.begin()), lim(src.end()); while (p != lim) { char c = *p++; if (c == ’-’ && lastc >= 0) { if (p != lim) { c = *p++; for (uint8_t i = lastc + 1U; i <= static_cast<uint8_t>(c); i++) dst += i; lastc = -1; } else { dst += ’-’; lastc = ’-’; } } else { dst += c; lastc = static_cast<uint8_t>(c); } } if (complement) dst = ~dst; $$ = new NFA(dst); This code is used in ¶17. ¶51 item case 5 actions 51 ≡ 1 $$ = $expression; This code is used in ¶17.
  28. 28. 1.2. Lexis Grammar Specification item case 6 actions 52 ≡ 1 NFA* result = new NFA(!*$expression); 2 delete $expression; 3 $$ = result; 15 ¶52 This code is used in ¶17. item case 7 actions 53 ≡ 1 NFA* result = new NFA(~*$expression); 2 delete $expression; 3 $$ = result; ¶53 This code is used in ¶17. 1.2.4 File File: Lexis/Lexis.gtk 54 ≡ 1 Language: 2 LexisGrammar. 3 Preamble: 4 ‘‘ 5 Lexis actions preamble 28 6 ’’. 7 Terminals: 8 "lexistab.h". 9 Nonterminals: 10 Lexis non-terminal definitions 4 11 Start: 12 Lexis start symbol definition 5 13 Productions: 14 Lexis grammar productions 6–17 ¶54
  29. 29. 16 Chapter 1. Lexis Input Specifications
  30. 30. Chapter 2 State Class   2.1 Declarations State class declaration 55 ≡ 1 class State 2 { 3 State class field declarations 56 4 State class method declarations 5 }; ¶55 57 This code is used in ¶71. State class field declarations 56 ≡ 1 protected: StateNumber number; 2 protected: Anchor anchor; 3 protected: Action action; 4 protected: string token; ¶56 This code is used in ¶55. State class method declarations 57 ≡ 1 public: State(); 2 public: State(StateNumber, Anchor, Action, string const&); 3 public: State& operator=(State const&); 4 public: void setNumber(StateNumber); 5 public: void setAnchor(Anchor); 6 public: void setAction(Action); 7 public: void setToken(string const&); 8 public: StateNumber getNumber() const; 9 public: Anchor getAnchor() const; 10 public: Action getAction() const; 11 public: string const& getToken() const; 12 public: ostream& put(ostream&) const; This code is used in ¶55. ¶57
  31. 31. Chapter 2. State Class 18   2.2 ¶58 Definitions State class inline method definitions 1 inline State::State() : 2 number(0), 3 anchor(none), 4 action(advance), 5 token() 6 {} 58 ≡ This code is used in ¶71. ¶59 State class inline method definitions 59 ≡ 1 inline State::State(StateNumber n, Anchor ancr, 2 Action actn, string const& s) : 3 number(n), 4 anchor(ancr), 5 action(actn), 6 token(s) 7 {} This code is used in ¶71. ¶60 State class inline method definitions 60 ≡ 1 inline StateNumber State::getNumber() const 2 { return number; } This code is used in ¶71. ¶61 State class inline method definitions 61 ≡ 1 inline Anchor State::getAnchor() const 2 { return anchor; } This code is used in ¶71. ¶62 State class inline method definitions 62 ≡ 1 inline Action State::getAction() const 2 { return action; } This code is used in ¶71. ¶63 State class inline method definitions 63 ≡ 1 inline string const& State::getToken() const 2 { return token; } This code is used in ¶71. ¶64 State class inline method definitions 64 ≡ 1 inline ostream& operator<<(ostream& stream, State const& state) 2 { return state.put(stream); } This code is used in ¶71.
  32. 32. 2.2. Definitions State method definitions 65 ≡ 1 State& State::operator=(State const& s) 2 { 3 anchor = s.anchor; 4 action = s.action; 5 token = s.token; 6 return *this; 7 } 19 ¶65 This code is used in ¶72. State method definitions 66 ≡ 1 void State::setNumber(StateNumber n) 2 { number = n; } ¶66 This code is used in ¶72. State method definitions 67 ≡ 1 void State::setAnchor(Anchor a) 2 { anchor = a; } ¶67 This code is used in ¶72. State method definitions 68 ≡ 1 void State::setAction(Action a) 2 { action = a; } ¶68 This code is used in ¶72. State method definitions 69 ≡ 1 void State::setToken(string const& s) 2 { token = s; } ¶69 This code is used in ¶72. State method definitions 70 ≡ 1 ostream& State::put(ostream& s) const 2 { 3 s << "number = " << number; 4 s << ", anchor = " << anchor; 5 s << ", action = " << action; 6 return s; 7 } This code is used in ¶72. ¶70
  33. 33. Chapter 2. State Class 20   2.3 ¶71 Files File: Lexis/State.h 71 ≡ 1 #if !defined(_Lexis_State_h) 2 #define _Lexis_State_h 3 4 5 6 7 #include #include #include #include <iostream> <string> <stdint.h> "Lexis/Lexis.h" 8 9 10 11 12 13 namespace Lexis { State class declaration 55 State class inline method definitions } 14 15 ¶72 #endif File: Lexis/State.cc 72 ≡ 1 #include <iostream> 2 #include "Lexis/State.h" 3 4 5 6 7 namespace Lexis { State method definitions } 65–70 58–64
  34. 34. Chapter 3 NFAState Class   3.1 Declarations NFAState class declaration 73 ≡ 1 class NFAState : public State 2 { 3 LabelType declaration 74 4 NFAState constants 75 5 NFAState field declarations 76 6 NFAState method declarations 7 }; ¶73 77 This code is used in ¶90. LabelType declaration 74 ≡ 1 public: enum LabelType 2 { 3 epsilon, 4 character, 5 characterSet 6 }; ¶74 This code is used in ¶73. NFAState constants 75 ≡ 1 private: enum Constants 2 { 3 maximumFanout = 2 4 }; This code is used in ¶73. ¶75
  35. 35. Chapter 3. NFAState Class 22 ¶76 NFAState field 1 private: 2 private: 3 private: 4 private: 5 private: declarations 76 ≡ uint8_t fanout; int16_t next[maximumFanout]; LabelType labelType; char labelChar; CharacterSet labelSet; This code is used in ¶73. NFAState method declarations 77 ≡ 1 public: NFAState(); 2 public: NFAState& operator=(NFAState const&); 3 public: void connect(StateNumber); 4 public: void connect(StateNumber, StateNumber); 5 public: void connect(char, StateNumber); 6 public: void connect(CharacterSet const&, StateNumber); 7 public: uint8_t getFanout() const; 8 public: LabelType getLabelType() const; 9 public: bool isTransitionLabelled(char) const; 10 public: StateNumber getNextState(uint8_t) const; 11 public: ostream& put(ostream&) const; ¶77 This code is used in ¶73.   3.2 ¶78 Definitions NFAState class inline method definitions 1 inline NFAState::NFAState() : 2 State(), 3 fanout(0), 4 labelType(epsilon), 5 labelChar(0), 6 labelSet() 7 {} 78 ≡ This code is used in ¶90. ¶79 NFAState class inline method definitions 79 ≡ 1 inline uint8_t NFAState::getFanout() const 2 { return fanout; } This code is used in ¶90. ¶80 NFAState class inline method definitions 80 ≡ 1 inline NFAState::LabelType NFAState::getLabelType() const 2 { return labelType; } This code is used in ¶90.
  36. 36. 3.2. Definitions ¶81 23 NFAState class inline method definitions 81 ≡ 1 inline ostream& operator<<(ostream& stream, NFAState const& state) 2 { return state.put(stream); } This code is used in ¶90. NFAState method definitions 82 ≡ 1 NFAState& NFAState::operator=(NFAState const& s) 2 { 3 State::operator=(s); 4 fanout = s.fanout; 5 for (uint8_t i = 0; i < fanout; ++i) 6 next[i] = s.next[i]; 7 labelType = s.labelType; 8 labelChar = s.labelChar; 9 labelSet = s.labelSet; 10 return *this; 11 } ¶82 This code is used in ¶91. NFAState method definitions 83 ≡ 1 void NFAState::connect(StateNumber nxt) 2 { 3 fanout = 1; 4 next[0] = nxt - number; 5 labelType = epsilon; 6 labelChar = 0; 7 labelSet = CharacterSet(0); 8 } ¶83 This code is used in ¶91. NFAState method definitions 84 ≡ 1 void NFAState::connect(StateNumber nxt0, StateNumber nxt1) 2 { 3 fanout = 2; 4 next[0] = nxt0 - number; 5 next[1] = nxt1 - number; 6 labelType = epsilon; 7 labelChar = 0; 8 labelSet = CharacterSet(0); 9 } This code is used in ¶91. ¶84
  37. 37. Chapter 3. NFAState Class 24 ¶85 NFAState method definitions 85 ≡ 1 void NFAState::connect(char lab, StateNumber nxt) 2 { 3 assert(lab != 0); 4 fanout = 1; 5 next[0] = nxt - number; 6 labelType = character; 7 labelChar = lab; 8 labelSet = CharacterSet(0); 9 } This code is used in ¶91. ¶86 NFAState method definitions 86 ≡ 1 void NFAState::connect(CharacterSet const& set, StateNumber nxt) 2 { 3 fanout = 1; 4 next[0] = nxt - number; 5 labelType = characterSet; 6 labelChar = 0; 7 labelSet = set; 8 } This code is used in ¶91. ¶87 NFAState method definitions 87 ≡ 1 bool NFAState::isTransitionLabelled(char c) const 2 { 3 switch (labelType) 4 { 5 case epsilon: return false; 6 case character: return labelChar == c; 7 case characterSet: return labelSet.contains(c); 8 } 9 assert(false); 10 return false; 11 } This code is used in ¶91. ¶88 NFAState method definitions 88 ≡ 1 StateNumber NFAState::getNextState(uint8_t i) const 2 { 3 assert(i < fanout); 4 return number + next[i]; 5 } This code is used in ¶91.
  38. 38. 3.2. Definitions NFAState method definitions 89 ≡ 1 ostream& NFAState::put(ostream& s) const 2 { 3 s << "NFAState {"; 4 State::put(s); 5 s << "n"; 6 if (fanout > 0) 7 { 8 s << " goto "; 9 uint8_t i = 0; 10 for (;;) 11 { 12 s << (number + next[i]); 13 i += 1; 14 if (i == fanout) 15 break; 16 s << ", "; 17 } 18 s << " on "; 19 switch (labelType) 20 { 21 case epsilon: 22 s << "epsilon"; break; 23 case character: 24 s << "’" << EscapeSequence::encode( 25 labelChar, ’’’) << "’"; 26 break; 27 case characterSet: 28 s << labelSet; break; 29 } 30 s << "n"; 31 } 32 return s << "}n"; 33 } This code is used in ¶91. 25 ¶89
  39. 39. Chapter 3. NFAState Class 26   3.3 ¶90 Files File: Lexis/NFAState.h 90 ≡ 1 #if !defined(_Lexis_NFAState_h) 2 #define _Lexis_NFAState_h 3 4 5 6 7 8 #include #include #include #include #include <iostream> <vector> <string> "Lexis/State.h" "Toolbox/CharacterSet.h" 9 10 using namespace Toolbox; 11 12 13 14 15 16 namespace Lexis { NFAState class declaration 73 NFAState class inline method definitions } 17 18 ¶91 #endif File: Lexis/NFAState.cc 91 ≡ 1 #include <iostream> 2 #include "Lexis/NFAState.h" 3 #include "Toolbox/EscapeSequence.h" 4 #include "Toolbox/Stack.h" 5 6 using namespace Toolbox; 7 8 9 10 11 namespace Lexis { NFAState method definitions } 82–89 78–81
  40. 40. Chapter 4 NFA Class   4.1 Declarations NFA class declaration 92 ≡ 1 class NFA 2 { 3 NFA class field declarations 93 4 NFA class method declarations 5 NFA class friends 95 6 }; ¶92 94 This code is used in ¶120. NFA class field declarations 93 ≡ 1 public: string language; 2 public: StateNumber const numberOfStates; 3 public: vector<NFAState> state; This code is used in ¶92. ¶93
  41. 41. Chapter 4. NFA Class 28 NFA class method declarations 94 ≡ 1 public: explicit NFA(StateNumber = 1); 2 public: explicit NFA(string const&); 3 public: explicit NFA(CharacterSet const&); 4 public: NFA(NFA const&); 5 public: ~NFA(); 6 public: NFAState const& operator[](StateNumber) const; 7 public: NFAState& operator[](StateNumber); 8 public: StateNumber getNumberOfStates() const; 9 public: void setAnchor(Anchor); 10 public: void setAction(Action); 11 public: void setToken(string const&); 12 public: void setLanguage(string const&); 13 public: ostream& put(ostream&) const; 14 public: Set getEpsilonClosure(Set const&) const; 15 public: Set getMoveSet(Set const&, char) const; 16 public: Action getDominantAction(Set const&) const; 17 public: Anchor getDominantAnchor(Set const&) const; 18 public: string getDominantToken(Set const&) const; 19 public: string const& getLanguage() const; ¶94 This code is used in ¶92. ¶95 NFA class friends 95 ≡ 1 friend NFA operator*(NFA 2 friend NFA operator|(NFA 3 friend NFA operator!(NFA 4 friend NFA operator~(NFA const&, NFA const&); const&, NFA const&); const&); const&); This code is used in ¶92.   4.2 ¶96 Definitions NFA class inline method definitions 96 ≡ 1 inline void NFA::setLanguage(string const& s) 2 { language = s; } This code is used in ¶120. ¶97 NFA class inline method definitions 97 ≡ 1 inline string const& NFA::getLanguage() const 2 { return language; } This code is used in ¶120. ¶98 NFA class inline method definitions 98 ≡ 1 inline StateNumber NFA::getNumberOfStates() const 2 { return numberOfStates; } This code is used in ¶120.
  42. 42. 4.2. Definitions NFA class inline method definitions 99 ≡ 1 inline ostream& operator<<(ostream& stream, NFA const& nfa) 2 { return nfa.put(stream); } 29 ¶99 This code is used in ¶120. NFA class method definitions 100 ≡ 1 NFA::NFA(StateNumber states) : 2 numberOfStates(states), 3 state(numberOfStates) 4 { 5 for (StateNumber n = 0; n < numberOfStates; ++n) 6 state[n].setNumber(n); 7 } ¶100 This code is used in ¶121. NFA class method definitions 101 ≡ 1 NFA::NFA(string const& s) : 2 numberOfStates(s.size() + 1), 3 state(numberOfStates) 4 { 5 for (StateNumber n = 0; n < numberOfStates; ++n) 6 state[n].setNumber(n); 7 for (StateNumber n = 0; n < numberOfStates - 1U; ++n) 8 state[n].connect(s[n], n + 1); 9 } ¶101 This code is used in ¶121. NFA class method definitions 102 ≡ 1 NFA::NFA(CharacterSet const& s) : 2 numberOfStates(2), 3 state(2) 4 { 5 for (StateNumber n = 0; n < numberOfStates; ++n) 6 state[n].setNumber(n); 7 state[0].connect(s, 1); 8 } This code is used in ¶121. ¶102
  43. 43. Chapter 4. NFA Class 30 NFA class method definitions 103 ≡ 1 NFA::NFA(NFA const& nfa) : 2 numberOfStates(nfa.numberOfStates), 3 state(numberOfStates) 4 { 5 for (StateNumber n = 0; n < numberOfStates; ++n) 6 { 7 state[n].setNumber(n); 8 state[n] = nfa.state[n]; 9 } 10 } This code is used in ¶121. ¶104 NFA class method definitions 1 NFA::~NFA() 2 {} 104 ≡ This code is used in ¶121. ¶105 NFA class method definitions 105 ≡ 1 NFAState const& NFA::operator[](StateNumber n) const 2 { 3 assert(n < numberOfStates); 4 return state[n]; 5 } This code is used in ¶121. ¶106 NFA class method definitions 106 ≡ 1 NFAState& NFA::operator[](StateNumber n) 2 { 3 assert(n < numberOfStates); 4 return state[n]; 5 } This code is used in ¶121. ¶107 NFA class method definitions 107 ≡ 1 void NFA::setAnchor(Anchor a) 2 { 3 assert(numberOfStates > 0); 4 state[numberOfStates - 1].setAnchor(a); 5 } This code is used in ¶121. ¶103
  44. 44. 4.2. Definitions ¶108 31 NFA class method definitions 108 ≡ 1 void NFA::setAction(Action a) 2 { 3 assert(numberOfStates > 0); 4 state[numberOfStates - 1].setAction(a); 5 } This code is used in ¶121. NFA class method definitions 109 ≡ 1 void NFA::setToken(string const& s) 2 { 3 assert(numberOfStates > 0); 4 state[numberOfStates - 1].setToken(s); 5 } ¶109 This code is used in ¶121. NFA class method definitions 110 ≡ 1 ostream& NFA::put(ostream& s) const 2 { 3 for (StateNumber n = 0; n < numberOfStates; ++n) 4 s << state[n]; 5 return s; 6 } This code is used in ¶121. ¶110
  45. 45. Chapter 4. NFA Class 32 ¶111 NFA class method definitions 111 ≡ 1 Set NFA::getEpsilonClosure(Set const& inputSet) const 2 { 3 Stack<StateNumber> stack; 4 Set result(inputSet); 5 6 for (Set::const_iterator p(inputSet.begin()), null; p != null; ++p) { stack.push(*p); } 7 8 9 10 11 while (!stack.isEmpty()) { StateNumber const s = stack.pop(); if (state[s].getLabelType() == NFAState::epsilon) { for (uint8_t i = 0; i < state[s].getFanout(); ++i) { StateNumber const next = state[s].getNextState(i); if (!result.contains(next)) { result += next; stack.push(next); } } } } return result; 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 } This code is used in ¶121.
  46. 46. 4.2. Definitions NFA class method definitions 112 ≡ 1 Set NFA::getMoveSet(Set const& inputSet, char c) const 2 { 3 Set result; 33 ¶112 4 for (Set::const_iterator p(inputSet.begin()), null; p != null; ++p) { StateNumber const s = *p; if (state[s].isTransitionLabelled(c)) { assert(state[s].getFanout() == 1); result += state[s].getNextState(0); } } return result; 5 6 7 8 9 10 11 12 13 14 15 } This code is used in ¶121. NFA class method definitions 113 ≡ 1 Action NFA::getDominantAction(Set const& inputSet) const 2 { 3 for (Set::const_iterator p(inputSet.begin()), null; p != null; ++p) 4 { 5 Action const a = state[*p].getAction(); 6 if (a != advance) 7 return a; 8 } 9 return advance; 10 } ¶113 This code is used in ¶121. NFA class method definitions 114 ≡ 1 Anchor NFA::getDominantAnchor(Set const& inputSet) const 2 { 3 for (Set::const_iterator p(inputSet.begin()), null; p != null; ++p) 4 { 5 Action const a = state[*p].getAction(); 6 if (a != advance) 7 return state[*p].getAnchor(); 8 } 9 return none; 10 } This code is used in ¶121. ¶114
  47. 47. Chapter 4. NFA Class 34 ¶115 NFA class method definitions 115 ≡ 1 string NFA::getDominantToken(Set const& inputSet) const 2 { 3 for (Set::const_iterator p(inputSet.begin()), null; p != null; ++p) 4 { 5 Action const a = state[*p].getAction(); 6 if (a != advance) 7 return state[*p].getToken(); 8 } 9 return ""; 10 } This code is used in ¶121. ¶116 NFA class method definitions 116 ≡ 1 NFA operator*(NFA const& nfa0, NFA const& nfa1) 2 { 3 StateNumber const size0 = nfa0.getNumberOfStates(); 4 StateNumber const size1 = nfa1.getNumberOfStates(); 5 StateNumber const resultSize = size0 + size1 - 1; 6 NFA result(resultSize); 7 for (StateNumber n = 0; n < size0 - 1U; ++n) 8 result[n] = nfa0[n]; 9 for (StateNumber n = 0; n < size1; ++n) 10 result[size0 - 1 + n] = nfa1[n]; 11 return result; 12 } This code is used in ¶121. ¶117 NFA class method definitions 117 ≡ 1 NFA operator|(NFA const& nfa0, NFA const& nfa1) 2 { 3 StateNumber const size0 = nfa0.getNumberOfStates(); 4 StateNumber const size1 = nfa1.getNumberOfStates(); 5 StateNumber const resultSize = size0 + size1 + 2; 6 NFA result(resultSize); 7 result[0].connect(static_cast<StateNumber>(1), 1 + size0); 8 for (StateNumber n = 0; n < size0; ++n) 9 result[1 + n] = nfa0[n]; 10 for (StateNumber n = 0; n < size1; ++n) 11 result[1 + size0 + n] = nfa1[n]; 12 result[size0].connect(resultSize - 1); 13 result[size0 + size1].connect(resultSize - 1); 14 return result; 15 } This code is used in ¶121.
  48. 48. 4.3. Files 35 NFA class method definitions 118 ≡ 1 NFA operator!(NFA const& nfa) 2 { 3 StateNumber const size = nfa.getNumberOfStates(); 4 StateNumber const resultSize = size + 1; 5 NFA result(resultSize); 6 result[0].connect(static_cast<StateNumber>(1), resultSize - 1); 7 for (StateNumber n = 0; n < size; ++n) 8 result[1 + n] = nfa[n]; 9 return result; 10 } ¶118 This code is used in ¶121. NFA class method definitions 119 ≡ 1 NFA operator~(NFA const& nfa) 2 { 3 StateNumber const size = nfa.getNumberOfStates(); 4 StateNumber const resultSize = size + 2; 5 NFA result(resultSize); 6 result[0].connect(static_cast<StateNumber>(1), resultSize - 1); 7 for (StateNumber n = 0; n < size - 1U; ++n) 8 result[1 + n] = nfa[n]; 9 result[size].connect(static_cast<StateNumber>(1), resultSize - 1); 10 return result; 11 } ¶119 This code is used in ¶121.   4.3 Files File: Lexis/NFA.h 120 ≡ 1 #if !defined(_Lexis_NFA_h) 2 #define _Lexis_NFA_h ¶120 3 4 5 6 7 8 #include #include #include #include #include <iostream> <vector> <string> "Lexis/NFAState.h" "Toolbox/CharacterSet.h" 9 10 using namespace Toolbox; 11 12 13 14 15 16 namespace Lexis { NFA class declaration 92 NFA class inline method definitions } 17 18 #endif 96–99
  49. 49. Chapter 4. NFA Class 36 ¶121 File: Lexis/NFA.cc 121 ≡ 1 #include <iostream> 2 #include "Lexis/NFA.h" 3 #include "Lexis/NFAState.h" 4 #include "Toolbox/EscapeSequence.h" 5 #include "Toolbox/Stack.h" 6 7 8 9 10 namespace Lexis { NFA class method definitions } 100–119
  50. 50. Chapter 5 DFAState Class   5.1 Declarations DFAState class declaration 122 ≡ 1 class DFAState : public State 2 { 3 DFAState class field declarations 123 4 DFAState class method declarations 124 5 }; ¶122 This code is used in ¶132. DFAState class field declarations 123 ≡ 1 private: Set const nfaStateSet; 2 private: vector<StateNumber> transition; ¶123 This code is used in ¶122. DFAState class method declarations 124 ≡ 1 public: DFAState(StateNumber, NFA const&, Set const&); 2 public: ~DFAState(); 3 public: DFAState& addTransition(char, StateNumber); 4 public: StateNumber getNextState(char) const; 5 public: Set getNFAStateSet() const; 6 public: ostream& put(ostream&) const; This code is used in ¶122. ¶124
  51. 51. Chapter 5. DFAState Class 38   5.2 Definitions DFAState class inline method definitions 125 ≡ 1 inline Set DFAState::getNFAStateSet() const 2 { return nfaStateSet; } This code is used in ¶132. ¶126 DFAState class inline method definitions 126 ≡ 1 inline ostream& operator<<(ostream& stream, DFAState const& state) 2 { return state.put(stream); } This code is used in ¶132. ¶127 DFAState class method definitions 127 ≡ 1 DFAState::DFAState(StateNumber n, NFA const& nfa, Set const& s) : 2 State(n, nfa.getDominantAnchor(s), 3 nfa.getDominantAction(s), nfa.getDominantToken(s)), 4 nfaStateSet(s), 5 transition(maximumCharacter) 6 { 7 for (uint8_t i = 0; i < maximumCharacter; ++i) 8 transition[i] = errorState; 9 } This code is used in ¶133. ¶128 DFAState class method definitions 1 DFAState::~DFAState() 2 {} 128 ≡ This code is used in ¶133. ¶129 DFAState class method definitions 129 ≡ 1 DFAState& DFAState::addTransition(char c, StateNumber to) 2 { 3 assert(static_cast<uint8_t>(c) < maximumCharacter); 4 transition[c] = to; 5 return *this; 6 } This code is used in ¶133. ¶130 DFAState class method definitions 130 ≡ 1 StateNumber DFAState::getNextState(char c) const 2 { 3 assert(static_cast<uint8_t>(c) < maximumCharacter); 4 return transition[c]; 5 } This code is used in ¶133. ¶125
  52. 52. 5.3. Files 39 DFAState class method definitions 131 ≡ 1 ostream& DFAState::put(ostream& s) const 2 { 3 s << "DFAState {"; 4 State::put(s); 5 s << "n"; 6 for (StateNumber n = 0; n < DFA::maximumNumberOfStates; ++n) 7 { 8 CharacterSet set; 9 for (uint8_t c = 0; c < maximumCharacter; ++c) 10 if (transition[c] == n) 11 set += c; 12 if (set != 0) 13 { 14 cout << " goto " << n; 15 cout << " on " << set << "n"; 16 } 17 } 18 return s << "}n"; 19 } ¶131 This code is used in ¶133.   5.3 Files File: Lexis/DFAState.h 132 ≡ 1 #if !defined(_Lexis_DFAState_h) 2 #define _Lexis_DFAState_h ¶132 3 4 5 6 7 8 #include #include #include #include #include <iostream> <string> "Lexis/State.h" "Lexis/NFA.h" "Toolbox/Set.h" 9 10 using namespace Toolbox; 11 12 13 14 15 16 namespace Lexis { DFAState class declaration 122 DFAState class inline method definitions } 17 18 #endif 125,126
  53. 53. Chapter 5. DFAState Class 40 ¶133 File: Lexis/DFAState.cc 133 ≡ 1 #include <iostream> 2 #include "Lexis/DFAState.h" 3 #include "Lexis/DFA.h" 4 #include "Toolbox/EscapeSequence.h" 5 #include "Toolbox/CharacterSet.h" 6 7 using namespace Toolbox; 8 9 10 11 12 namespace Lexis { DFAState class method definitions } 127–131
  54. 54. Chapter 6 DFA State   6.1 Declarations DFA class declaration 134 ≡ 1 class DFA 2 { 3 DFA constants 135 4 DFA class field declarations 136 5 DFA class method declarations 137,138 6 }; ¶134 This code is used in ¶152. DFA constants 135 ≡ 1 public: enum Constants 2 { 3 maximumNumberOfStates = errorState - 1 4 }; ¶135 This code is used in ¶134. DFA class field declarations 136 ≡ 1 private: StateNumber numberOfStates; 2 private: vector<DFAState*> state; 3 private: string language; ¶136 This code is used in ¶134. DFA class method declarations 137 ≡ 1 private: StateNumber getStateNumber(Set const&) const; 2 private: StateNumber initializeGroups( 3 vector<Set>&, vector<StateNumber>&) const; 4 private: DFA& rebuild( 5 StateNumber, vector<Set> const&, vector<StateNumber> const&); This code is used in ¶134. ¶137
  55. 55. Chapter 6. DFA State 42 DFA class method declarations 138 ≡ 1 public: explicit DFA(NFA const&); 2 public: ~DFA(); 3 public: void minimize(); 4 public: StateNumber getNumberOfStates() const; 5 public: DFAState const& operator[](StateNumber) const; 6 public: bool isColumnEquivalent(char, char) const; 7 public: bool isRowEquivalent(StateNumber, StateNumber) const; 8 public: string const& getLanguage() const; 9 public: ostream& put(ostream&) const; ¶138 This code is used in ¶134.   6.2 ¶139 Definitions DFA class inline method definitions 139 ≡ 1 inline StateNumber DFA::getNumberOfStates() const 2 { return numberOfStates; } This code is used in ¶152. ¶140 DFA class inline method definitions 140 ≡ 1 inline string const& DFA::getLanguage() const 2 { return language; } This code is used in ¶152. ¶141 DFA class inline method definitions 141 ≡ 1 inline ostream& operator<<(ostream& stream, DFA const& dfa) 2 { return dfa.put(stream); } This code is used in ¶152. ¶142 DFA class method definitions 142 ≡ 1 StateNumber DFA::getStateNumber(Set const& s) const 2 { 3 for (StateNumber i = 0; i < numberOfStates; ++i) 4 if (state[i]->getNFAStateSet() == s) 5 return i; 6 return errorState; 7 } This code is used in ¶153.
  56. 56. 6.2. Definitions 43 DFA class method definitions 143 ≡ 1 DFA::DFA(NFA const& nfa) : 2 numberOfStates(0), 3 state(maximumNumberOfStates), 4 language(nfa.getLanguage()) 5 { 6 Set nfaStates; 7 nfaStates += 0; 8 nfaStates = nfa.getEpsilonClosure(nfaStates); 9 state[0] = new DFAState(0, nfa, nfaStates); 10 11 numberOfStates = 1; 12 13 for (StateNumber i = 0; i < numberOfStates; ++i) { cerr << "."; DFAState* const current = state[i]; assert(current != 0); 14 15 16 17 18 19 for (uint8_t c = 0; c < maximumCharacter; ++c) { Set moveSet; moveSet = nfa.getMoveSet(current->getNFAStateSet(), c); moveSet = nfa.getEpsilonClosure(moveSet); 20 21 22 23 24 25 StateNumber nextState = errorState; if (moveSet != 0) { nextState = getStateNumber(moveSet); if (nextState == errorState) { assert(numberOfStates < maximumNumberOfStates); state[numberOfStates] = new DFAState(numberOfStates, nfa, moveSet); nextState = numberOfStates; numberOfStates += 1; } current->addTransition(c, nextState); } 26 27 28 29 30 31 32 33 34 35 36 37 38 39 } 40 } 41 42 } This code is used in ¶153. ¶143
  57. 57. Chapter 6. DFA State 44 ¶144 DFA class method definitions 144 ≡ 1 DFA::~DFA() 2 { 3 for (StateNumber i = 0; i < numberOfStates; ++i) 4 delete state[i]; 5 } This code is used in ¶153. ¶145 DFA class method definitions 145 ≡ 1 DFAState const& DFA::operator[](StateNumber s) const 2 { 3 assert(s < numberOfStates); 4 return *state[s]; 5 } This code is used in ¶153. ¶146 DFA class method definitions 146 ≡ 1 ostream& DFA::put(ostream& s) const 2 { 3 for (StateNumber i = 0; i < numberOfStates; ++i) 4 cout << *state[i]; 5 return s; 6 } This code is used in ¶153.
  58. 58. 6.2. Definitions DFA class method definitions 147 ≡ 1 StateNumber DFA::initializeGroups( 2 vector<Set>& group, 3 vector<StateNumber>& inGroup) const 4 { 5 StateNumber numberOfGroups = 0; 6 for (StateNumber i = 0; i < numberOfStates; ++i) 7 { 8 bool found = false; 9 StateNumber j; 10 for (j = 0; j < i; ++j) 11 { 12 if (state[i]->getAction() == state[j]->getAction() 13 && state[i]->getToken() == state[j]->getToken()) 14 { 15 found = true; 16 break; 17 } 18 } 19 if (found) 20 { 21 group[inGroup[j]] += i; 22 inGroup[i] = inGroup[j]; 23 } 24 else 25 { 26 assert(numberOfGroups < numberOfStates); 27 group[numberOfGroups] += i; 28 inGroup[i] = numberOfGroups; 29 numberOfGroups += 1; 30 } 31 } 32 return numberOfGroups; 33 } This code is used in ¶153. 45 ¶147
  59. 59. Chapter 6. DFA State 46 ¶148 DFA class method definitions 148 ≡ 1 DFA& DFA::rebuild(StateNumber numberOfGroups, 2 vector<Set> const& group, 3 vector<StateNumber> const& inGroup) 4 { 5 vector<DFAState*> oldState(state); 6 state.resize(numberOfGroups); 7 for (StateNumber i = 0; i < numberOfGroups; ++i) { assert(group[i] != 0); Set::const_iterator p(group[i].begin()), null; state[i] = oldState[*p]; state[i]->setNumber(i); for (uint8_t c = 0; c < maximumCharacter; ++c) { StateNumber const next = state[i]->getNextState(c); if (next != errorState) state[i]->addTransition(c, inGroup[next]); } for (++p; p != null; ++p) delete oldState[*p]; } numberOfStates = numberOfGroups; return *this; 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 } This code is used in ¶153.
  60. 60. 6.2. Definitions DFA class method definitions 149 ≡ 1 void DFA::minimize() 2 { 3 vector<Set> group(numberOfStates); 4 vector<StateNumber> inGroup(numberOfStates); 5 StateNumber numberOfGroups = initializeGroups(group, inGroup); 6 7 bool done = false; while (!done) { done = true; for (StateNumber i = 0; i < numberOfGroups; ++i) { if (group[i].getSize() > 1) { Set newGroup; Set::const_iterator p(group[i].begin()), null; StateNumber const first = *p; for (++p; p != null; ++p) { StateNumber const next = *p; for (uint8_t c = 0; c < maximumCharacter; ++c) { StateNumber const firstGoto = state[first]->getNextState(c); StateNumber const nextGoto = state[next]->getNextState(c); if (firstGoto != nextGoto && (firstGoto == errorState || nextGoto == errorState || inGroup[firstGoto] != inGroup[nextGoto])) { group[i] -= next; newGroup += next; inGroup[next] = numberOfGroups; break; } } } if (newGroup != 0) { assert(numberOfGroups < numberOfStates); group[numberOfGroups] = newGroup; numberOfGroups += 1; done = false; } } } } rebuild(numberOfGroups, group, inGroup); 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 } This code is used in ¶153. 47 ¶149
  61. 61. Chapter 6. DFA State 48 DFA class method definitions 150 ≡ 1 bool DFA::isColumnEquivalent(char c1, char c2) const 2 { 3 for (StateNumber s = 0; s < numberOfStates; ++s) 4 if (state[s]->getNextState(c1) != 5 state[s]->getNextState(c2)) 6 return false; 7 return true; 8 } ¶150 This code is used in ¶153. DFA class method definitions 151 ≡ 1 bool DFA::isRowEquivalent(StateNumber s1, StateNumber s2) const 2 { 3 for (uint8_t c = 0; c < maximumCharacter; ++c) 4 if (state[s1]->getNextState(c) != 5 state[s2]->getNextState(c)) 6 return false; 7 return true; 8 } ¶151 This code is used in ¶153.   6.3 ¶152 Files File: Lexis/DFA.h 152 ≡ 1 #if !defined(_Lexis_DFA_h) 2 #define _Lexis_DFA_h 3 4 5 6 7 8 9 #include #include #include #include #include #include <iostream> <string> "Lexis/State.h" "Lexis/NFA.h" "Lexis/DFAState.h" "Toolbox/Set.h" 10 11 using namespace Toolbox; 12 13 14 15 16 17 namespace Lexis { DFA class declaration 134 DFA class inline method definitions } 18 19 #endif 139–141
  62. 62. 6.3. Files 49 File: Lexis/DFA.cc 153 ≡ 1 #include <iostream> 2 #include "Lexis/DFA.h" 3 #include "Toolbox/EscapeSequence.h" 4 5 6 7 8 namespace Lexis { DFA class method definitions } 142–151 ¶153
  63. 63. 50 Chapter 6. DFA State
  64. 64. Chapter 7 CompressedDFA Class   7.1 Declarations CompressedDFA class declaration 154 ≡ 1 class CompressedDFA 2 { 3 CompressedDFA class field declarations 155 4 CompressedDFA class method declarations 156,157 5 }; ¶154 This code is used in ¶173. CompressedDFA class field declarations 155 ≡ 1 private: TableEntry numberOfStates; 2 private: TableEntry numberOfRows; 3 private: TableEntry numberOfColumns; 4 private: TableEntry numberOfTokens; 5 private: vector<TableEntry> rowMap; 6 private: vector<TableEntry> columnMap; 7 private: vector<Anchor> anchorTable; 8 private: vector<Action> actionTable; 9 private: vector<TableEntry> transitionTable; 10 private: vector<string> tokenTable; 11 private: vector<string> tokenList; 12 private: string language; ¶155 This code is used in ¶154. CompressedDFA class method declarations 156 ≡ 1 private: ostream& putMap(ostream&, char const*, TableEntry, 2 vector<TableEntry> const&) const; 3 private: ostream& putTransitionTable(ostream&) const; 4 private: ostream& putAnchorTable(ostream&) const; 5 private: ostream& putActionTable(ostream&) const; 6 private: ostream& putTokenList(ostream&) const; 7 private: ostream& putTokenTable(ostream&) const; This code is used in ¶154. ¶156
  65. 65. Chapter 7. CompressedDFA Class 52 ¶157 CompressedDFA class method declarations 157 ≡ 1 public: explicit CompressedDFA(DFA const&); 2 public: ~CompressedDFA(); 3 public: ostream& putTokens(ostream&) const; 4 public: ostream& putTables(ostream&) const; This code is used in ¶154.   7.2 ¶158 Definitions CompressedDFA class method definitions 158 ≡ 1 CompressedDFA::CompressedDFA(DFA const& dfa) : 2 numberOfStates(dfa.getNumberOfStates()), 3 numberOfRows(0), 4 numberOfColumns(0), 5 numberOfTokens(0), 6 rowMap(dfa.getNumberOfStates()), 7 columnMap(maximumCharacter), 8 anchorTable(dfa.getNumberOfStates()), 9 actionTable(dfa.getNumberOfStates()), 10 transitionTable(0), 11 tokenTable(dfa.getNumberOfStates()), 12 tokenList(dfa.getNumberOfStates()), 13 language(dfa.getLanguage()) 14 { 15 for (uint8_t c = 0; c < maximumCharacter; ++c) 16 columnMap[c] = maximumCharacter; 17 construct construct construct construct construct 18 19 20 21 22 23 columnTable and columnMap 159 rowTable and rowMap 160 anchorTable, actionTable and tokenTable tokenList 162 transitionTable 163 } This code is used in ¶174. 161
  66. 66. 7.2. Definitions construct columnTable and columnMap 159 ≡ 1 vector<TableEntry> columnTable(maximumCharacter); 2 for (uint8_t c = 0; c < maximumCharacter; ++c) 3 { 4 if (columnMap[c] == maximumCharacter) 5 { 6 columnTable[numberOfColumns] = c; 7 columnMap[c] = numberOfColumns; 8 for (uint8_t i = c + 1; i < maximumCharacter; ++i) 9 { 10 if (columnMap[i] == maximumCharacter && 11 dfa.isColumnEquivalent(c, i)) 12 columnMap[i] = numberOfColumns; 13 } 14 numberOfColumns += 1; 15 } 16 } 53 ¶159 This code is used in ¶158. construct rowTable and rowMap 160 ≡ 1 vector<TableEntry> rowTable(numberOfStates); 2 for (StateNumber s = 0; s < numberOfStates; ++s) 3 rowMap[s] = numberOfStates; 4 for (StateNumber s = 0; s < numberOfStates; ++s) 5 { 6 if (rowMap[s] == numberOfStates) 7 { 8 rowTable[numberOfRows] = s; 9 rowMap[s] = numberOfRows; 10 for (StateNumber i = s + 1; 11 i < numberOfStates; ++i) 12 { 13 if (rowMap[i] == numberOfStates && 14 dfa.isRowEquivalent(s, i)) 15 rowMap[i] = numberOfRows; 16 } 17 numberOfRows += 1; 18 } 19 } This code is used in ¶158. ¶160
  67. 67. Chapter 7. CompressedDFA Class 54 ¶161 construct anchorTable, actionTable and tokenTable 161 ≡ 1 for (StateNumber s = 0; s < numberOfStates; ++s) 2 { 3 anchorTable[s] = dfa[s].getAnchor(); 4 actionTable[s] = dfa[s].getAction(); 5 if (actionTable[s] == accept) 6 tokenTable[s] = dfa[s].getToken(); 7 else 8 tokenTable[s] = "end_of_file"; 9 } This code is used in ¶158. ¶162 construct tokenList 162 ≡ 1 tokenList[numberOfTokens++] = "_epsilon"; 2 tokenList[numberOfTokens++] = "end_of_file"; 3 for (StateNumber s = 0; s < numberOfStates; ++s) 4 { 5 TableEntry i; 6 for (i = 0; i < numberOfTokens; ++i) 7 if (tokenList[i] == tokenTable[s]) 8 break; 9 if (i == numberOfTokens) 10 { 11 tokenList[numberOfTokens] = tokenTable[s]; 12 numberOfTokens += 1; 13 } 14 } This code is used in ¶158. ¶163 construct transitionTable 163 ≡ 1 transitionTable.resize(numberOfRows * numberOfColumns); 2 for (TableEntry row = 0; row < numberOfRows; ++row) 3 for (TableEntry col = 0; 4 col < numberOfColumns; ++col) 5 transitionTable[row * numberOfColumns + col] = 6 dfa[rowTable[row]].getNextState(columnTable[col]); This code is used in ¶158. ¶164 CompressedDFA class method definitions 164 ≡ 1 CompressedDFA::~CompressedDFA() 2 {} This code is used in ¶174.
  68. 68. 7.2. Definitions CompressedDFA class method definitions 165 ≡ 1 ostream& CompressedDFA::putMap(ostream& stream, 2 char const* label, TableEntry size, 3 vector<TableEntry> const& map) const 4 { 5 stream << "TableEntry const LexisTables::" 6 << label << "[" << static_cast<uint16_t>(size) << "] =n"; 7 stream << "{n"; 8 stream << " "; 9 for (uint16_t i = 0; i < size; ++i) 10 { 11 stream << static_cast<uint16_t>(map[i]); 12 if (i < size - 1U) 13 stream << ", "; 14 if (i % 10 == 9) 15 { 16 stream << "n"; 17 stream << " "; 18 } 19 } 20 stream << "n"; 21 stream << "};n"; 22 return stream; 23 } 55 ¶165 This code is used in ¶174. CompressedDFA class method definitions 166 ≡ 1 ostream& CompressedDFA::putActionTable(ostream& stream) const 2 { 3 stream << "Action const LexisTables::actionTable[" 4 << static_cast<uint16_t>(numberOfStates) << "] =n"; 5 stream << "{n"; 6 stream << " "; 7 for (uint16_t i = 0; i < numberOfStates; ++i) 8 { 9 stream << actionTable[i]; 10 if (i < numberOfStates - 1U) 11 stream << ", "; 12 if (i % 4 == 3) 13 { 14 stream << "n"; 15 stream << " "; 16 } 17 } 18 stream << "n"; 19 stream << "};n"; 20 return stream; 21 } This code is used in ¶174. ¶166
  69. 69. Chapter 7. CompressedDFA Class 56 ¶167 CompressedDFA class method definitions 167 ≡ 1 ostream& CompressedDFA::putAnchorTable(ostream& stream) const 2 { 3 stream << "Anchor const LexisTables::anchorTable[" 4 << static_cast<uint16_t>(numberOfStates) << "] =n"; 5 stream << "{n"; 6 stream << " "; 7 for (uint16_t i = 0; i < numberOfStates; ++i) 8 { 9 stream << anchorTable[i]; 10 if (i < numberOfStates - 1U) 11 stream << ", "; 12 if (i % 4 == 3) 13 { 14 stream << "n"; 15 stream << " "; 16 } 17 } 18 stream << "n"; 19 stream << "};n"; 20 return stream; 21 } This code is used in ¶174. ¶168 CompressedDFA class method definitions 168 ≡ 1 ostream& CompressedDFA::putTokenList(ostream& stream) const 2 { 3 stream << "TableEntry const LexisTables::numberOfTokens = " 4 << static_cast<uint16_t>(numberOfTokens) << ";n"; 5 stream << "LexisTables::TokenName const LexisTables::nameTable[" 6 << static_cast<uint16_t>(numberOfTokens) << "] =n"; 7 stream << "{n"; 8 for (TableEntry i = 0; i < numberOfTokens; ++i) 9 { 10 stream << " { " << tokenList[i] << "," 11 << " "" << tokenList[i] << "" }"; 12 if (i < numberOfTokens - 1U) 13 stream << ",n"; 14 } 15 stream << "n"; 16 stream << "};n"; 17 return stream; 18 } This code is used in ¶174.
  70. 70. 7.2. Definitions CompressedDFA class method definitions 169 ≡ 1 ostream& CompressedDFA::putTokenTable(ostream& stream) const 2 { 3 stream << "LexisTables::TokenType const LexisTables::tokenTable[" 4 << static_cast<uint16_t>(numberOfStates) << "] =n"; 5 stream << "{n"; 6 stream << " "; 7 for (uint16_t i = 0; i < numberOfStates; ++i) 8 { 9 stream << tokenTable[i]; 10 if (i < numberOfStates - 1U) 11 stream << ", "; 12 if (i % 2 == 1) 13 { 14 stream << "n"; 15 stream << " "; 16 } 17 } 18 stream << "n"; 19 stream << "};n"; 20 return stream; 21 } This code is used in ¶174. 57 ¶169
  71. 71. Chapter 7. CompressedDFA Class 58 ¶170 CompressedDFA class method definitions 170 ≡ 1 ostream& CompressedDFA::putTransitionTable(ostream& stream) const 2 { 3 stream << "TableEntry const LexisTables::transitionTable[" 4 << static_cast<uint16_t>(numberOfRows * numberOfColumns) 5 << "] =n"; 6 stream << "{n"; 7 for (TableEntry row = 0; row < numberOfRows; ++row) 8 { 9 stream << " // row " << static_cast<uint16_t>(row) << "n"; 10 stream << " "; 11 for (TableEntry col = 0; col < numberOfColumns; ++col) 12 { 13 stream << static_cast<uint16_t>(transitionTable[ 14 row * numberOfColumns + col]); 15 if (col < numberOfColumns - 1U) 16 stream << ", "; 17 if (col % 10 == 9) 18 { 19 stream << "n"; 20 stream << " "; 21 } 22 } 23 if (row < numberOfRows - 1U) 24 stream << ","; 25 stream << "n"; 26 } 27 stream << "n"; 28 stream << "};n"; 29 return stream; 30 } This code is used in ¶174.
  72. 72. 7.2. Definitions CompressedDFA class method definitions 171 ≡ 1 ostream& CompressedDFA::putTokens(ostream& stream) const 2 { 3 stream << 4 "#if !defined(lexistab_h)n" 5 "#define lexistab_hn" 6 "n" 7 "#include "Lexis/Lexis.h"n" 8 "n" 9 "namespace " << language << "n" 10 "{n" 11 "using namespace Lexis;n" 12 "using Lexis::advance;n" 13 "class LexisTablesn" 14 "{n" 15 "public:n" 16 " enum TokenTypen" 17 " {n"; 18 for (TableEntry i = 0; i < numberOfTokens; ++i) 19 { 20 stream << "t" << tokenList[i] 21 << " = " << static_cast<uint16_t>(i); 22 if (i < numberOfTokens - 1U) 23 stream << ",n"; 24 } 25 stream << 26 "n" 27 " };n" 28 " struct TokenNamen" 29 " {n" 30 "tTokenType type;n" 31 "tchar const* name;n" 32 " };n" 33 " static uint8_t const numberOfTokens;n" 34 " static TokenName const nameTable[];n" 35 " static TableEntry const numberOfStates;n" 36 " static TableEntry const numberOfRows;n" 37 " static TableEntry const numberOfColumns;n" 38 " static TableEntry const rowMap[];n" 39 " static TableEntry const columnMap[];n" 40 " static TableEntry const transitionTable[];n" 41 " static Anchor const anchorTable[];n" 42 " static Action const actionTable[];n" 43 " static TokenType const tokenTable[];n" 44 "};n" 45 "}n" 46 "n" 47 "#endifn"; 48 return stream; 49 } This code is used in ¶174. 59 ¶171
  73. 73. Chapter 7. CompressedDFA Class 60 CompressedDFA class method definitions 172 ≡ 1 ostream& CompressedDFA::putTables(ostream& stream) const 2 { 3 stream << 4 "#include "lexistab.h"n" 5 "n" 6 "namespace " << language << "n" 7 "{n"; 8 putTokenList(stream); 9 stream << "TableEntry const LexisTables::numberOfStates = " 10 << static_cast<uint16_t>(numberOfStates) << ";n"; 11 stream << "TableEntry const LexisTables::numberOfColumns = " 12 << static_cast<uint16_t>(numberOfColumns) << ";n"; 13 putMap(stream, "columnMap", maximumCharacter, columnMap); 14 stream << "TableEntry const LexisTables::numberOfRows = " 15 << static_cast<uint16_t>(numberOfRows) << ";n"; 16 putMap(stream, "rowMap", numberOfStates, rowMap); 17 putTransitionTable(stream); 18 putAnchorTable(stream); 19 putActionTable(stream); 20 putTokenTable(stream); 21 stream << "}n"; 22 return stream; 23 } ¶172 This code is used in ¶174.   7.3 ¶173 Files File: Lexis/CompressedDFA.h 173 ≡ 1 #if !defined(_Lexis_Compressed_h) 2 #define _Lexis_Compressed_h 3 4 5 6 7 #include #include #include #include <iostream> <vector> <string> "Lexis/DFA.h" 8 9 10 11 12 namespace Lexis { CompressedDFA class declaration } 13 14 #endif 154
  74. 74. 7.3. Files 61 File: Lexis/CompressedDFA.cc 174 ≡ 1 #include <iostream> 2 #include "Lexis/CompressedDFA.h" ¶174 3 4 5 6 7 namespace Lexis { CompressedDFA class method definitions } 158,164–172
  75. 75. 62 Chapter 7. CompressedDFA Class
  76. 76. Chapter 8 Lexical Analyzer Classes   8.1 8.1.1 Common Types and Constants Type Declarations Lexis type declarations 175 ≡ 1 typedef uint16_t StateNumber; ¶175 This code is used in ¶235. Lexis type declarations 176 ≡ 1 typedef uint8_t TableEntry; ¶176 This code is used in ¶235. 8.1.2 Constants Lexis constants 177 ≡ 1 StateNumber const errorState = 255; ¶177 This code is used in ¶235. Lexis constants 178 ≡ 1 TableEntry const maximumCharacter = 128; This code is used in ¶235. ¶178
  77. 77. Chapter 8. Lexical Analyzer Classes 64 8.1.3 Anchor Enumeration Anchor enumeration declaration 1 enum Anchor 2 { 3 none = 0, 4 head = 1, 5 tail = 2, 6 both = 3 7 }; 179 ≡ This code is used in ¶235. ¶180 Anchor enumeration inline method definitions 180 ≡ 1 inline ostream& operator<<(ostream& s, Anchor a) 2 { 3 switch (a) 4 { 5 case none: s << "none"; break; 6 case head: s << "head"; break; 7 case tail: s << "tail"; break; 8 case both: s << "both"; break; 9 } 10 return s; 11 } This code is used in ¶235. 8.1.4 ¶181 Action Enumeration Action enumeration declaration 181 ≡ 1 enum Action 2 { 3 advance = 0, 4 accept = 1, 5 discard = 3 // We want (discard & accept) != 0. 6 }; This code is used in ¶235. ¶182 Action enumeration inline method definitions 182 ≡ 1 inline ostream& operator<<(ostream& s, Action a) 2 { 3 switch (a) 4 { 5 case advance: s << "advance"; break; 6 case accept: s << "accept"; break; 7 case discard: s << "discard"; break; 8 } 9 return s; 10 } This code is used in ¶235. ¶179
  78. 78. 8.2. Input Class Template 65   8.2 8.2.1 Input Class Template Declarations Input class template declaration 183 ≡ 1 template<typename T> 2 class Input 3 { 4 Input class template type declarations 184 5 Input constants 185 6 Input class template field declarations 186 7 Input class template method declarations 187,188 8 }; ¶183 This code is used in ¶235. Input class template type declarations 1 public: typedef T Token; 184 ≡ ¶184 This code is used in ¶183. Input constants 185 ≡ 1 private: enum Constants 2 { 3 maximumLexemeLength = 1024, 4 bufferSize = 2 * maximumLexemeLength 5 }; ¶185 This code is used in ¶183. Input class template field declarations 186 ≡ 1 private: istream& inputStream; 2 private: char buffer[bufferSize]; 3 private: char* nextInput; 4 private: char* endOfInput; 5 private: char* const endOfBuffer; 6 private: char* startOfLexeme; 7 private: char* endOfLexeme; 8 private: bool endOfFileRead; 9 private: uint16_t currentLineNumber; 10 private: uint16_t startMarkLineNumber; 11 private: uint16_t endMarkLineNumber; ¶186 This code is used in ¶183. Input class template method declarations 187 ≡ 1 private: void checkAssertions() const; 2 private: int16_t advance(); This code is used in ¶183. ¶187
  79. 79. Chapter 8. Lexical Analyzer Classes 66 ¶188 Input class template method declarations 188 ≡ 1 public: Input(istream&); 2 public: operator bool(); 3 public: char operator*(); 4 public: Input& operator++(); 5 public: Input& operator--(); 6 public: void markStart(); 7 public: void markEnd(); 8 public: void returnToStartMark(); 9 public: void returnToEndMark(); 10 public: void advanceStartMark(); 11 public: uint16_t getLineNumber() const; 12 public: string getLexeme() const; This code is used in ¶183. 8.2.2 ¶189 Definitions Input class template inline method definitions 189 ≡ 1 template <typename T> 2 inline uint16_t Input<T>::getLineNumber() const 3 { return startMarkLineNumber; } This code is used in ¶235. ¶190 Input class template method definitions 190 ≡ 1 template <typename T> 2 Input<T>::Input(istream& s) : 3 inputStream(s), 4 nextInput(buffer), 5 endOfInput(buffer), 6 endOfBuffer(buffer + bufferSize), 7 startOfLexeme(buffer), 8 endOfLexeme(buffer), 9 endOfFileRead(false), 10 currentLineNumber(0), 11 startMarkLineNumber(0), 12 endMarkLineNumber(0) 13 { 14 assert(endOfInput < endOfBuffer); 15 *endOfInput++ = ’n’; 16 checkAssertions(); 17 } This code is used in ¶236.
  80. 80. 8.2. Input Class Template Input class template method definitions 191 ≡ 1 template <typename T> 2 void Input<T>::checkAssertions() const 3 { 4 assert(buffer <= startOfLexeme); 5 assert(startOfLexeme <= endOfLexeme); 6 assert(endOfLexeme <= nextInput); 7 assert(nextInput <= endOfInput); 8 assert(endOfInput <= endOfBuffer); 9 assert(nextInput - startOfLexeme <= maximumLexemeLength); 10 } 67 ¶191 This code is used in ¶236. Input class template method definitions 192 ≡ 1 template <typename T> 2 int16_t Input<T>::advance() 3 { 4 if (nextInput == endOfInput && !endOfFileRead) 5 { 6 if (endOfInput == endOfBuffer) 7 { 8 uint32_t const shift = startOfLexeme - buffer; 9 char* dst = buffer; 10 char* src = startOfLexeme; 11 while (src < endOfInput) 12 *dst++ = *src++; 13 startOfLexeme -= shift; 14 endOfLexeme -= shift; 15 endOfInput -= shift; 16 nextInput -= shift; 17 } 18 assert(endOfInput < endOfBuffer); 19 int const c = inputStream.get(); 20 if (c == EOF) 21 endOfFileRead = true; 22 else 23 { 24 *endOfInput = c; 25 endOfInput += 1; 26 } 27 } 28 int16_t result = EOF; 29 if (nextInput < endOfInput) 30 result = *nextInput; 31 else 32 assert(endOfFileRead); 33 checkAssertions(); 34 return result; 35 } This code is used in ¶236. ¶192
  81. 81. Chapter 8. Lexical Analyzer Classes 68 ¶193 Input class template method definitions 193 ≡ 1 template <typename T> 2 Input<T>::operator bool() 3 { return advance() != EOF; } This code is used in ¶236. ¶194 Input class template method definitions 194 ≡ 1 template <typename T> 2 char Input<T>::operator*() 3 { 4 int16_t const c = advance(); 5 if (c != EOF) 6 return c; 7 else 8 return 0; 9 } This code is used in ¶236. ¶195 Input class template method definitions 195 ≡ 1 template <typename T> 2 Input<T>& Input<T>::operator++() 3 { 4 if (nextInput == endOfInput && !endOfFileRead) 5 advance(); 6 if (nextInput < endOfInput) 7 { 8 if (*nextInput == ’n’) 9 currentLineNumber += 1; 10 nextInput += 1; 11 if (nextInput - startOfLexeme > maximumLexemeLength) 12 { 13 if (*startOfLexeme == ’n’) 14 startMarkLineNumber += 1; 15 startOfLexeme += 1; 16 if (endOfLexeme < startOfLexeme) 17 { 18 if (*endOfLexeme == ’n’) 19 endMarkLineNumber += 1; 20 endOfLexeme += 1; 21 } 22 } 23 } 24 else 25 assert(endOfFileRead); 26 checkAssertions(); 27 return *this; 28 } This code is used in ¶236.
  82. 82. 8.2. Input Class Template Input class template method definitions 196 ≡ 1 template <typename T> 2 Input<T>& Input<T>::operator--() 3 { 4 if (nextInput > startOfLexeme) 5 { 6 nextInput -= 1; 7 if (*nextInput == ’n’) 8 currentLineNumber -= 1; 9 if (endOfLexeme > nextInput) 10 { 11 endOfLexeme = nextInput; 12 endMarkLineNumber = currentLineNumber; 13 } 14 } 15 checkAssertions(); 16 return *this; 17 } 69 ¶196 This code is used in ¶236. Input class template method definitions 197 ≡ 1 template <typename T> 2 void Input<T>::markStart() 3 { 4 startOfLexeme = nextInput; 5 endOfLexeme = nextInput; 6 startMarkLineNumber = currentLineNumber; 7 endMarkLineNumber = currentLineNumber; 8 checkAssertions(); 9 } ¶197 This code is used in ¶236. Input class template method definitions 198 ≡ 1 template <typename T> 2 void Input<T>::markEnd() 3 { 4 endOfLexeme = nextInput; 5 endMarkLineNumber = currentLineNumber; 6 checkAssertions(); 7 } This code is used in ¶236. ¶198
  83. 83. Chapter 8. Lexical Analyzer Classes 70 ¶199 Input class template method definitions 199 ≡ 1 template <typename T> 2 void Input<T>::returnToStartMark() 3 { 4 nextInput = startOfLexeme; 5 endOfLexeme = startOfLexeme; 6 currentLineNumber = startMarkLineNumber; 7 checkAssertions(); 8 } This code is used in ¶236. ¶200 Input class template method definitions 200 ≡ 1 template <typename T> 2 void Input<T>::returnToEndMark() 3 { 4 nextInput = endOfLexeme; 5 currentLineNumber = endMarkLineNumber; 6 checkAssertions(); 7 } This code is used in ¶236. ¶201 Input class template method definitions 201 ≡ 1 template <typename T> 2 void Input<T>::advanceStartMark() 3 { 4 if (startOfLexeme < endOfLexeme) 5 if (*startOfLexeme == ’n’) 6 startMarkLineNumber += 1; 7 startOfLexeme += 1; 8 checkAssertions(); 9 } This code is used in ¶236. ¶202 Input class template method definitions 202 ≡ 1 template <typename T> 2 string Input<T>::getLexeme() const 3 { 4 string result(startOfLexeme, endOfLexeme - startOfLexeme); 5 checkAssertions(); 6 return result; 7 } This code is used in ¶236.
  84. 84. 8.3. Token Class Template 71   8.3 8.3.1 Token Class Template Declarations Token class template declaration 203 ≡ 1 template <typename LT> 2 class Token 3 { 4 Token class template type declarations 204 5 Token class template field declarations 205 6 Token class template method declarations 206 7 }; ¶203 This code is used in ¶235. Token class template type declarations 204 ≡ 1 public: typedef LT Tables; 2 public: typedef TokenStream<Token> Stream; 3 public: typedef typename LT::TokenType Type; ¶204 This code is used in ¶203. Token class template field declarations 205 ≡ 1 private: Type type; 2 private: string lexeme; 3 private: uint16_t lineNumber; ¶205 This code is used in ¶203. Token class template method declarations 206 ≡ 1 public: Token(); 2 public: Token(Type, string const&, uint16_t = 0); 3 public: Token(Token const&); 4 public: Token& operator=(Token const&); 5 public: Type getType() const; 6 public: string const& getLexeme() const; 7 public: uint16_t getLineNumber() const; 8 public: string getName() const; 9 public: ostream& put(ostream& s) const; This code is used in ¶203. ¶206
  85. 85. Chapter 8. Lexical Analyzer Classes 72 8.3.2 Definitions Token class template inline method definitions 207 ≡ 1 template <typename LT> 2 inline Token<LT>::Type Token<LT>::getType() const 3 { return type; } This code is used in ¶235. ¶208 Token class template inline method definitions 208 ≡ 1 template <typename LT> 2 inline string const& Token<LT>::getLexeme() const 3 { return lexeme; } This code is used in ¶235. ¶209 Token class template inline method definitions 209 ≡ 1 template <typename LT> 2 inline uint16_t Token<LT>::getLineNumber() const 3 { return lineNumber; } This code is used in ¶235. ¶210 Token class template inline method definitions 210 ≡ 1 template <typename LT> 2 inline ostream& operator<<(ostream& stream, Token<LT> const& token) 3 { return token.put(stream); } This code is used in ¶235. ¶211 Token class template method definitions 1 template <typename LT> 2 Token<LT>::Token() : 3 type(LT::end_of_file), 4 lexeme(""), 5 lineNumber(0) 6 {} 211 ≡ This code is used in ¶236. ¶212 Token class template method definitions 212 ≡ 1 template <typename LT> 2 Token<LT>::Token(Token<LT>::Type t, string const& s, uint16_t l) : 3 type(t), 4 lexeme(s), 5 lineNumber(l) 6 { assert(0 <= type && type < LT::numberOfTokens); } This code is used in ¶236. ¶207
  86. 86. 8.3. Token Class Template Token class template method definitions 213 ≡ 1 template <typename LT> 2 Token<LT>::Token(Token<LT> const& t) : 3 type(t.type), 4 lexeme(t.lexeme), 5 lineNumber(t.lineNumber) 6 { assert(0 <= type && type < LT::numberOfTokens); } 73 ¶213 This code is used in ¶236. Token class template method definitions 214 ≡ 1 template <typename LT> 2 Token<LT>& Token<LT>::operator=(Token<LT> const& t) 3 { 4 type = t.type; 5 lexeme = t.lexeme; 6 lineNumber = t.lineNumber; 7 return *this; 8 } ¶214 This code is used in ¶236. Token class template method definitions 215 ≡ 1 template <typename LT> 2 string Token<LT>::getName() const 3 { 4 uint8_t i; 5 for (i = 0; i < LT::numberOfTokens; ++i) 6 if (LT::nameTable[i].type == type) 7 break; 8 assert(i < LT::numberOfTokens); 9 return LT::nameTable[i].name; 10 } ¶215 This code is used in ¶236. Token class template method definitions 216 ≡ 1 template <typename LT> 2 ostream& Token<LT>::put(ostream& s) const 3 { 4 s << "Token {" << getName(); 5 s << ", "" << EscapeSequence::encode(lexeme, ’"’) << """; 6 return s << ", " << lineNumber << "}"; 7 } This code is used in ¶236. ¶216
  87. 87. Chapter 8. Lexical Analyzer Classes 74   8.4 8.4.1 ¶217 TokenStream Class Template Declarations TokenStream class template forward declaration 1 template <typename T> 2 class TokenStream; 217 ≡ This code is used in ¶235. ¶218 TokenStream class template declaration 218 ≡ 1 template <typename T> 2 class TokenStream 3 { 4 TokenStream class template type declarations 219 5 TokenStream class template field declarations 220 6 TokenStream class template method declarations 221,222 7 }; This code is used in ¶235. ¶219 TokenStream class template type declarations 219 ≡ 1 public: typedef T Token; 2 public: typedef typename T::Tables Tables; This code is used in ¶218. ¶220 TokenStream class template field declarations 1 private: Input<Token> inputStream; 220 ≡ This code is used in ¶218. ¶221 TokenStream class template method declarations 221 ≡ 1 private: TableEntry getTransition(TableEntry, char) const; 2 private: bool isAcceptState(TableEntry) const; 3 private: bool isAcceptOrDiscardState(TableEntry) const; 4 private: bool isTailAnchored(TableEntry) const; 5 private: bool isHeadAnchored(TableEntry) const; 6 private: typename Token::Type getType(TableEntry) const; This code is used in ¶218. ¶222 TokenStream class template method declarations 222 ≡ 1 public: TokenStream(istream&); 2 public: TokenStream& operator>>(Token&); This code is used in ¶218.
  88. 88. 8.4. TokenStream Class Template 8.4.2 ¶223 75 Definitions TokenStream class template inline method definitions 223 ≡ 1 template <typename T> 2 inline TokenStream<T>::TokenStream(istream& s) : 3 inputStream(s) 4 {} This code is used in ¶235. TokenStream class template method definitions 224 ≡ ¶224 1 template <typename T> 2 TableEntry TokenStream<T>::getTransition(TableEntry state, char c) const 3 { 4 assert(state < Tables::numberOfStates); 5 if (static_cast<uint8_t>(c) < maximumCharacter) 6 return Tables::transitionTable[ 7 Tables::rowMap[state] * Tables::numberOfColumns 8 + Tables::columnMap[c]]; 9 else 10 return errorState; 11 } This code is used in ¶236. TokenStream class template method definitions 225 ≡ 1 template <typename T> 2 bool TokenStream<T>::isAcceptState(TableEntry state) const 3 { 4 assert(state < Tables::numberOfStates); 5 return Tables::actionTable[state] == accept; 6 } ¶225 This code is used in ¶236. TokenStream class template method definitions 226 ≡ 1 template <typename T> 2 bool TokenStream<T>::isAcceptOrDiscardState(TableEntry state) const 3 { 4 assert(state < Tables::numberOfStates); 5 return (Tables::actionTable[state] & accept) != 0; 6 } ¶226 This code is used in ¶236. TokenStream class template method definitions 227 ≡ 1 template <typename T> 2 bool TokenStream<T>::isTailAnchored(TableEntry state) const 3 { 4 assert(state < Tables::numberOfStates); 5 return (Tables::anchorTable[state] & tail) != 0; 6 } This code is used in ¶236. ¶227
  89. 89. Chapter 8. Lexical Analyzer Classes 76 ¶228 TokenStream class template method definitions 228 ≡ 1 template <typename T> 2 bool TokenStream<T>::isHeadAnchored(TableEntry state) const 3 { 4 assert(state < Tables::numberOfStates); 5 return (Tables::anchorTable[state] & head) != 0; 6 } This code is used in ¶236. ¶229 TokenStream class template method definitions 229 ≡ 1 template <typename T> 2 typename T::Type TokenStream<T>::getType(TableEntry state) const 3 { 4 assert(state < Tables::numberOfStates); 5 return Tables::tokenTable[state]; 6 } This code is used in ¶236.
  90. 90. 8.4. TokenStream Class Template TokenStream class template method definitions 230 ≡ 1 template <typename T> 2 TokenStream<T>& TokenStream<T>::operator>>(T& result) 3 { 4 TableEntry currentState = 0; 5 TableEntry nextState = 0; 6 TableEntry acceptState = 0; 7 inputStream.markStart(); 8 for (;;) 9 { 10 get the next character and compute nextState 231 11 currentState = nextState; 12 if (currentState != errorState) 13 { 14 normal transition processing 232 15 } 16 else 17 { 18 if (acceptState != 0) 19 { 20 return the last acceptable token 233 21 } 22 else 23 { 24 lexical error recovery 234 25 } 26 currentState = 0; 27 acceptState = 0; 28 inputStream.markStart(); 29 } 30 } 31 } 77 ¶230 This code is used in ¶236. get the next character and compute nextState 231 ≡ 1 if (inputStream) 2 { 3 char const nextChar = *inputStream; 4 ++inputStream; 5 nextState = getTransition(currentState, nextChar); 6 } 7 else if (acceptState != 0) 8 nextState = errorState; 9 else 10 { 11 result = T(Tables::end_of_file, "", 12 inputStream.getLineNumber()); 13 return *this; 14 } This code is used in ¶230. ¶231
  91. 91. Chapter 8. Lexical Analyzer Classes 78 ¶232 normal transition processing 232 ≡ 1 if(isAcceptOrDiscardState(currentState)) 2 { 3 acceptState = currentState; 4 inputStream.markEnd(); 5 } This code is used in ¶230. ¶233 return the last acceptable token 233 ≡ 1 inputStream.returnToEndMark(); 2 if (isTailAnchored(acceptState)) 3 --inputStream; 4 if (isHeadAnchored(acceptState)) 5 inputStream.advanceStartMark(); 6 if (isAcceptState(acceptState)) 7 { 8 result = T(getType(acceptState), 9 inputStream.getLexeme(), 10 inputStream.getLineNumber()); 11 return *this; 12 } This code is used in ¶230. ¶234 lexical error recovery 234 ≡ 1 inputStream.returnToStartMark(); 2 char const nextChar = *inputStream; 3 cerr << "Lexis: Input error on line " << 4 inputStream.getLineNumber() << "."; 5 cerr << " Input character ’" << 6 EscapeSequence::encode(nextChar, ’’’) 7 << "’ ignored.n"; 8 ++inputStream; This code is used in ¶230.
  92. 92. 8.5. Files 79   8.5 Files File: Lexis/Lexis.h 235 ≡ 1 #if !defined(_Lexis_Lexis_h) 2 #define _Lexis_Lexis_h ¶235 3 4 5 6 #include <iostream> #include <string> #include <stdint.h> 7 8 9 10 11 namespace Lexis { Lexis type declarations Lexis constants 177,178 175,176 12 Anchor enumeration declaration 179 Anchor enumeration inline method definitions 182 Input class template declaration 183 Input class template inline method definitions 14 180 Action enumeration declaration 181 Action enumeration inline method definitions 13 189 15 16 17 18 19 20 21 TokenStream class template forward declaration 22 217 23 Token class template declaration 203 Token class template inline method definitions 24 25 207–210 26 TokenStream class template declaration 218 TokenStream class template inline method definitions 27 28 29 } 30 31 #include "Lexis/Lexis.cc" 32 33 #endif 223
  93. 93. Chapter 8. Lexical Analyzer Classes 80 ¶236 File: Lexis/Lexis.cc 236 ≡ 1 #if !defined(_Lexis_Lexis_cc) 2 #define _Lexis_Lexis_cc 3 4 5 6 7 #include #include #include #include <iostream.h> <string> "Lexis/Lexis.h" "Toolbox/EscapeSequence.h" 8 9 using namespace Toolbox; 10 11 12 13 14 15 16 namespace Lexis { Input class template method definitions 190–202 Token class template method definitions 211–216 TokenStream class template method definitions 224–230 } 17 18 #endif
  94. 94. Chapter 9. Lexis Main Program 82 Chapter 9 Lexis Main Program   9.1 ¶237 main Method Definition Lexis main method definition 237 ≡ 1 using Lexis::NFA; 2 using Lexis::DFA; 3 using Lexis::CompressedDFA; 4 using Lexis::Token; 5 using Lexis::TokenStream; 6 using Lexis::ParserData; 7 using Lexis::ItemAttributes; 8 using LexisTokens::LexisTables; 9 using LexisGrammar::GramatikaTables; 10 using LexisGrammar::Parser; 11 using Gramatika::Value; 12 using Toolbox::PathName; 13 14 15 16 string const lexExtension(".lex"); string const tokenFileName("lexistab.h"); string const tableFileName("lexistab.cc"); 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 int main(int argc, char* argv[]) { if (argc != 2) { cerr << "Usage: lexis file[.lex]" << endl; return 1; } read the Lexis input file 238 NFA* nfa = result.nfa; if (nfa == 0) cerr << "Null automaton." << endl; else { compute the Lexis tables 239 write the Lexis output files 240 } return 0; } This code is used in ¶241.
  95. 95. 9.1. main Method Definition read the Lexis input file 238 ≡ 1 cerr << "Reading input file:"; 2 PathName argument(argv[1]); 3 string lexFile; 83 ¶238 4 5 6 7 8 if (argument.getExtension() == lexExtension) lexFile = argument; else lexFile = argument + lexExtension; 9 10 11 12 13 14 15 16 17 18 19 20 cerr << " " << lexFile; ifstream input(lexFile.c_str()); if (!input) { cerr << "Can’t open " << lexFile << endl; return 1; } TokenStream<Token<LexisTables> > tin(input); Parser<ParserData,Token<LexisTables>,ItemAttributes,GramatikaTables> parser; Value<Token<LexisTables>,ItemAttributes> const result = parser.parse(tin); cerr << "." << endl; This code is used in ¶237. compute the Lexis tables 239 ≡ 1 cerr << "Computing tables:"; 2 //cout << *nfa; 3 DFA dfa(*nfa); 4 delete nfa; 5 //cout << dfa; 6 dfa.minimize(); 7 //cout << dfa; 8 CompressedDFA cdfa(dfa); 9 cerr << "." << endl; This code is used in ¶237. ¶239

×