Tokenization:
→ It will segment an input character sequence into tokens.
→ Tokens can be words, numbers, punctuations, and etc.
Tokenizer Types
→ Whitespace Tokenizer - Non whitespace sequences are identified as tokens
→ Simple Tokenizer - A character class tokenizer, sequences of the same character class are tokens
→ Learnable Tokenizer - A maximum entropy tokenizer, detects token boundaries based on probability model
Tokenization:
→ It will segment an input character sequence into tokens.
→ Tokens can be words, numbers, punctuations, and etc.
Tokenizer Types
→ Whitespace Tokenizer - Non whitespace sequences are identified as tokens
→ Simple Tokenizer - A character class tokenizer, sequences of the same character class are tokens
→ Learnable Tokenizer - A maximum entropy tokenizer, detects token boundaries based on probability model
19th CENTURY SAN JOSE IN A BOTTLE presentation, 11 16-13tgilman
This presentation was delivered at the New Almaden Quicksilver Mining Museum on November 16, 2013. It features a subset of content from the award winning book, 19TH CENTURY SAN JOSE IN A BOTTLE, as well as general interest material not included in the book.
I speak on this topic at local museums, libraries, schools, clubs, and events throughout the south bay. If you are interested in having me speak at an upcoming event, I can be contacted at tobingilman@hotmail.com.
If you would like to purchase the book, it is available in hardcover at:
http://www.blurb.com/b/4555302-19th-century-san-jose
and in softcover at:
http://www.blurb.com/b/4555151-19th-century-san-jose
19th CENTURY SAN JOSE IN A BOTTLE presentation, 11 16-13tgilman
This presentation was delivered at the New Almaden Quicksilver Mining Museum on November 16, 2013. It features a subset of content from the award winning book, 19TH CENTURY SAN JOSE IN A BOTTLE, as well as general interest material not included in the book.
I speak on this topic at local museums, libraries, schools, clubs, and events throughout the south bay. If you are interested in having me speak at an upcoming event, I can be contacted at tobingilman@hotmail.com.
If you would like to purchase the book, it is available in hardcover at:
http://www.blurb.com/b/4555302-19th-century-san-jose
and in softcover at:
http://www.blurb.com/b/4555151-19th-century-san-jose
2. Codes
• Barthes suggested that there are one or more
codes we use to identify texts.
• These texts may be “Open” (unraveled in a lot
of different ways) or “Closed” (there is only
one obvious thread to pull on).
4. Meanings of codes
• Hermeneutic/enigma code:
This code refers to a mystery in a text. Clues are
dropped but no clear answer is given. Enigma’s
during a narrative usually wants them to know
more, but unanswered enigmas often frustrate
audiences.
5. Proairetic/ action code
• This code contains sequential elements of
action in a text. For an example, a gunslinger
would draw his gun on an adversary and we
would wonder what the resolution of this
action would be.
6. Semantic code
• This code refers to parts within the text that
suggests or refers to additional meaning.
Essentially it involves a suggested additional
meaning by using connotation.
7. Symbolic Code
• This code is about symbolism in a text. It
exercises contrasts to create greater meaning,
drama and tension.
8. Referential code
• This code refers to anything that requires an
external body of knowledge, such as scientific,
historical or cultural knowledge.
