This document discusses compilers, tokenizers, and parsers. It defines a compiler as having two main components: a lexer (tokenizer) that reads input and generates tokens, and a parser that converts tokens into a structured data format. It describes how a tokenizer works by defining states, scanning for patterns, and returning a list of tokens. It recommends optimizations for tokenizers like using little memory, partial reading, and avoiding unnecessary function calls. Finally, it states that the parser analyzes the token stream and constructs an object-oriented tree structure, avoiding non-tail recursion to prevent hitting stack limits.

1. About tokens and lexemes Ben Scholzen Game Developer Gameforge Productions GmbH

3. Basic structure of a tokenizer and a parser

4. Where to optimize things for PHP

5. What about parser generators?

7. Create lots of function calls like lemon parsers in C

8. Are not working very performance-wise

9. Will eat up all your memory

11. Let's get started

13. Converts human-readable data into machine-readable data

15. Reads the input stream

16. Clears up the input data

18. Converts them into a data structure

19. What a compiler is and how it works Lexer Parser Tokens Document Stream Structure

24. The lexer (or tokenizer)

27. Example in PHP: $value = 5 * 7 ;

30. Reading char-by-char is too slow

31. Use the offset parameter

36. Going into practice

39. Throwing in a file

40. Preparing stuff

41. Base state

42. Operator state

43. Value state

44. Rounding it up

45. Some actual testing

47. [0]=>

48. array(2) {

49. [0]=>

50. string(8) "variable"

51. [1]=>

52. string(6) "$value"

53. }

54. [1]=>

55. array(2) {

56. [0]=>

57. string(8) "operator"

58. [1]=>

59. string(1) "="

60. }

61. [2]=>

62. array(2) {

63. [0]=>

64. string(6) "number"

65. [1]=>

66. string(1) "5"

68. array(2) {

69. [0]=>

70. string(8) "operator"

71. [1]=>

72. string(1) "*"

73. }

74. [4]=>

75. array(2) {

76. [0]=>

77. string(6) "number"

78. [1]=>

79. string(1) "7"

80. }

81. [5]=>

82. array(2) {

83. [0]=>

84. string(8) "operator"

85. [1]=>

86. string(1) ";"

87. }

88. }

89. The parser

91. Create an object-oriented tree-structure or interpret

94. Summary — Questions?

96. About tail-recursion in PHP: http://www.alternateinterior.com/2006/09/tail-recursion-in-php.html

97. My blog: http://www.dasprids.de

98. Rate this talk: http://joind.in/635

99. Follow me on twitter:

100. http://www.twitter.com/dasprid

101. Thank you!