This document contains lecture slides for CSE340 - Principles of Programming Languages. It introduces lexical analysis and discusses how it is used to break source code into individual tokens by identifying strings and words. The key steps of lexical analysis are described, including reading the text line by line, grouping characters into strings, and verifying strings as valid words to create a vector of tokens. Regular expressions and deterministic finite automata are also introduced as techniques for defining token rules. Students are assigned to review these topics for lexical analysis in the provided textbook.

1. CSE340 - Principles of
Programming Languages
Lecture 02:
Introduction
Javier Gonzalez-Sanchez
javiergs@asu.edu
BYENG M1-38
Office Hours: By appointment

2. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 2
Keywords
Level
Paradigm
Language
Analysis
Lexical
Input:
Symbols
Output:
Words
Syntax
Input:
Words
Output:
Sentences
Semantic
Input:
Sentences
Translate or
Execute

3. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 3
Analysis
int x = 5;
float y = "hello;
String @z = "9.5";
int x = cse340;
if ( x > 14) while (5 == 5) if (int a) a = 1;
x = x; for ( ; ; );
y = 13.45.0;
int me = 99999000001111222000000111111222223443483045830948;
while { x != 9} ();
int {x} = 10;

4. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 4
High-Level Languages
X,E,G,O,O
#e1,I,I,0,7
@
OPR 19, AX
STO x, AX
LIT 5, AX
OPR 21, AX
LOD #e1,AX
CAL 1, AX
OPR 0, AX
5
Virtual Machine
(interpreter)
// sorce code
int x;
int foo () {
read (x);
print (5);
}
main () {
foo ();
}
Lexer
Parser
Semantic Analyzer
Code Generation
01001010101000010
01010100101010010
10100100000011011
11010010110101111
00010010101010010
10101001010101011
Assembler
compilation execution

5. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 5
Keywords
Lexical
Alphabet Symbol
String Word
Token

6. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 6
Lexical Analysis
int x = 5;
float y = "hello;
String @z = "9.5";
int x = cse340;
if ( x > 14) while (5 == 5) if (int a) a = 1;
x = x; for ( ; ; );
y = 13.45.0;
int me = 99999000001111222000000111111222223443483045830948;
while { x != 9} ();
int {x} = 10;

7. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 7
Lexical Analysis | Steps
a) Read a text FILE line by line
b) For each LINE:
• Read character by character.
• Create sets of consecutive characters (STRING). Try to
group the bigger amount of characters as possible.
• Start a new set each time that you need. Take care of:
Whitespace, Delimiter, Operator, End of Line and others
special characters.
c) For each STRING: verify if it is a valid WORD.
d) Create a VECTOR and store the STRINGs and WORDs.

8. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 8
Lexical Analysis
int x = 5; float y = "hello;
String@z="9.5”;intx=cse340;if(x>
14) while
(5 == 5) if (int a) a = 1; x = x;
for ( ; ; );y = 13.45.0;int me
=99999000001111222000000111111222
223443483045830948;while { x !=
9} ();int {x} = 10;

9. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 9
Lexical Analysis
int x = 5; float y = "hello;
String@z="9.5”;intx=cse340;if(x>
14) while
(5 == 5) if (int a) a = 1; x = x;
for ( ; ; );y = 13.45.0;int me
=99999000001111222000000111111222
223443483045830948;while { x !=
9} ();int {x} = 10;
”hello "world" bye"
# of STRINGS
9
12
3
18
12
2
6
12
3

10. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 10
Keywords
Lexical
Alphabet Symbol
String Word
Token Rules
Regular
Expression
Deterministic
Finite
Automata

11. Javier Gonzalez-Sanchez | CSE340 | Summer 2015 | 11
Homework
Review the following topics:
Regular Expressions (Text Book: Chapter 6)
and Deterministic Finite Automata

12. CSE340 - Principles of Programming Languages
Javier Gonzalez-Sanchez
javiergs@asu.edu
Summer 2015
Disclaimer. These slides can only be used as study material for the class CSE340 at ASU. They cannot be distributed or used for another purpose.