Unicode 101


Published on

An overview of character sets and encodings, scripts and languages, display issues, and how to deal with these in programming.

Published in: Technology
1 Comment
  • Great presentation!; (SlideShare seems to have eaten a couple of characters in slide 5…)
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Unicode 101

  1. 1. Unicode 101Bouvet søkekollokvie, 2011-10-26Lars Marius Garshol, <larsga@bouvet.no>http://twitter.com/larsga1
  2. 2. Agenda• Scripts and languages – background for what follows• Character encodings – basic concepts• Unicode – it‟s bigger than you think• Programming – some practical lessons2
  3. 3. Scripts and languagesA quick intro to grammatology3
  4. 4. The beginning of writing Chinese writing Egyptian hieroglyphics Sumerian cuneiform4000 BCE 3000 BCE 2000 BCE 1000 BCE 4
  5. 5. Logographic scripts• All of these first scripts are logographic – sometimes called “ideographic”, too• One character – one word – = mountain – = mouth / door – 門 = gate• Compounds for complicated concepts – 問 = question (mouth inside gate) – = hotel (literally: alcohol shop) – 日本 = Japan (literally: sun root)5
  6. 6. Simplified Chinese• In the 1950s and 60s the Communist Chinese government simplified the shapes of many characters – these modified characters are used in mainland China – Japan and Taiwan still use the original shapes Character Traditional Simplified Gate 門 门 Country 國 国 Vehicle 車 车 East 東 东6
  7. 7. Logographic scripts (2)Good Bad• Iconic and striking • Hard to learn• Compact • Hard to write on computers• Language-independent (to • Works poorly for inflected some degree) languages • Sorting is hard7
  8. 8. The next step: abjads• The Egyptians later developed a script called “hieratic” – meaning “priestly writing” – oldest known example from 1600 BCE, but must be older – actual origin kind of obscure• It is an abjad – that is, an alphabet with only consonant signs – (and some logographic elements)• A precursor to our own alphabet8
  9. 9. Why only consonants?• In Semitic languages everything revolves around “word roots” – these consist of three consonants – many, many words can be derived from the same root in a systematic way – (the same applies to Egyptian)• Example – s l m = peace – salaam = peace (related to Hebrew shalom) – islam = to have peace – moslem = one who has peace• Abjads therefore work well for Semites – and not quite as well for others9
  10. 10. Abjad family treeMuch simplified. This family Hieratichas many, many more scripts. Proto-Sinaitic Ethiopic Phoenician Greek Aramaic Kharoshthi Arabic Hebrew10
  11. 11. The invention of the alphabet• The Greeks didn‟t much like not having signs for vowels – so they invented them, thus giving us the alphabet• Salaam – in Arabic: (m l s) – in Greek: σαλαμ (s a l a m)• Grammatologists consider only scripts with both consonant and vowel signs alphabets – thus, there is no ”Arabic alphabet” or ”Chinese alphabet”11
  12. 12. Alphabet family treeAgain much simplified. Greek Etruscan Latin Armenian Cyrillic Georgian12
  13. 13. Alphabetic order13
  14. 14. ABG?• The Etruscans had no g sound, so they pronounced the third character k – the Romans inherited this• Thus, in Latin “C” was used for both k and g – this left Spurius Carvilius Ruga with a bit of a problem – he had to write his surname as “Ruca” – (Latin “ruca”: to fart) – so he invented “G” (C with a stroke)14
  15. 15. Backtracking Hieratic Proto-Sinaitic Ethiopic Phoenician Greek Aramaic Kharoshthi Arabic Hebrew15
  16. 16. Kharoshthi• A script developed for writing Sanskrit in India – 4th century BCE, or thereabouts• Sanskrit is Indo-European, like Greek – hence, the absence of vowels is a major problem – however, the Indians came up with a different solution• This is an abugida – Ethiopic, too16
  17. 17. The descendants of Kharoshthi• The Indian subcontinent has a large number of scripts – all are descended from Kharoshthi and follow the same basic model• These scripts also spread into Indo-China and beyond to Indonesia – Thai, Khmer, Javanese, Tibetan, etc etc• Total number of users must approach 2 billion17
  18. 18. Back to China Chinese Japanese Korean Vietnamese• Japanese is an inflected language – therefore developed extra characters for writing grammatical endings• The Vietnamese later moved on to Latin characters under French influence• The Koreans did something completely different...• Leaving out a number of minor ethnic scripts here18
  19. 19. Hangul• On October 9 1446, king Sejong announced the introduction of a new script: Hangul – it was designed to be easier to learn than Chinese characters – it was also designed to be a better fit to the Korean language• Hangul is like an alphabet, but better – wikipedia: Numerous linguists have praised Hangul for its featural design, describing it as "remarkable", "the most perfect phonetic system devised", and "brilliant, so deliberately does it fit the language like a glove.”19
  20. 20. Hangul design• Vowels have different shapes from consonants – these follow the shape of the vocal cords when speaking the vowel• Consonants have a different system Follows Chinese convention of equal-sized boxes for all characters. Each box contains hangul letters for one syllable.20
  21. 21. Syllabaries• Like abugidas, but unsystematic – that is, every letter combination must be learned by rote21
  22. 22. 1800 – present• Cover this?22
  23. 23. Kinds of scripts abjad featural alphabet abugida syllabary logographic feature letter syllable word what basic shapes correspond to23
  24. 24. Text directions• LTR top-down • Bottom up, left to right – Latin, Greek, Cyrillic, ... – Some minor Indonesian• RTL top-down abugidas – Arabic, Hebrew, Syriac, ... • Bottom up, right to left• Top to bottom, left to right – one minor Moslem Chinese abjad – Monglian, Uighur, Buryat, ... • Upwards boustrophedon• Top to bottom, right to left – Ogham (ancient Irish runes) – Nushu, Rong24
  25. 25. Combining characters• In Arabic, the same letter can have up to four different shapes – depending on its position in the word25
  26. 26. Character encodingBasics26
  27. 27. Two key concepts• Character set – a function number -> character – usually with a limited, fixed number of characters• Character encoding – a mapping from a bit stream to a sequence of numbers – the numbers, of course, refer to characters in some character set• Example – UTF-8 is an encoding for Unicode – UTF-16 is another27
  28. 28. Kinds of encodings• Single-byte – each byte is a number 0-255. end of story – ISO 8859-x• Double-byte – each word (2 bytes) is a number 0-65536. end of story – UCS-2• Variable length – more complex rules (UTF-8)• Escape code-based – uses escape codes to change between different modes – ISO 202228
  29. 29. ASCII• The mother of all character sets – 7-bit• Nearly all character sets today are ASCII subsets• The exception is EBCDIC – mostly used by IBM mainframes – also a terrible design29
  30. 30. Code pages• Primitive character set solution on IBM PCs• Basically, changing the code page would change the system font – 65 would always be „A‟ – 216 could be „Ø‟, Cyrillic, Greek, ... depending on code page• Essentially, swapping code page would change the contents of all text files... – awful for text processing software30
  31. 31. ISO 8859-x series• Lower 128 characters is ASCII• Higher 128 characters are language-specific• Now obsolete, thanks to Unicode• Microsoft have their own extensions – Windows-12xx, add extra characters where 8859 have obsolete control codes 1 Western Europe 5 Cyrillic 9 Turkish 1 Baltic 3 2 Central Europe 6 Arabic 10 Nordic (Sami) 1 Celtic 4 3 South Europe 7 Greek 11 Thai 1 Latin-1 5 ++31 4 North Europe 8 Hebrew 12 Doesn’t exist 1 Latin-3
  32. 32. The Far East• Generally, one character set per country – Japan JIS X 0208 – Korea KS X 1001 (and 1003) – China GB 2312 – Taiwan ???• Combined with different character encodings – ISO 2022 (-JP, -KR, -CN) – EUC (-JP, -KR, -TW)• Additional variants – Shift-JIS (Japanese, from Microsoft) – Big5 (Taiwanese) – ...32
  33. 33. China• Doesn‟t want to use Unicode• Instead introduced GB 18030 – takes GB 2312, then adds Unicode after the GB 2312 part – requires a mapping table for lower part – higher part can be mapped to Unicode algorithmically33
  34. 34. VISCII• Vietnamese character set – tries hard to maintain ASCII compatibility, but there are just too many Vietnamese characters...34
  35. 35. Unicode35
  36. 36. Unicode• The character set to end all character sets• Before, there was at least one character set for each script – generally, it would have Latin plus one more script – software therefore had to support many different internal representations of text• Now, Unicode supports every character that‟s ever appeared in a character set anywhere – therefore, it‟s the only character set you need36
  37. 37. Origin• 1987 – Engineers at Xerox and Apple discuss the possibility of a universal character set – they investigate, and decide it‟s feasible• 1988 – tentative proposal for a 16-bit character set• 1989 – Unicode Working Group set up – all of ISO 8859 added• 1990 – many more people join – Chinese characters added• 1991 – Unicode Consortium founded – Unicode 1.1 released• 1992 – ISO 10646 killed off, and replaced by Unicode37
  38. 38. Design goals• Universal – should be the only character set ever needed• Semantics – characters should have well-defined semantics – Ø≠∅• Dynamic composition – characters can be composed dynamically• Convertibility – every character in an existing character set, must have a single corresponding character in Unicode38
  39. 39. Structure• Originally intended to be 16-bit – 0x000 – 0xFFFF – explicit rationale: enough to encode all characters in daily use – implicit: not excessive use of space• Unfortunately, this is not nearly enough – decided to expand it in 1996 – keep the 16-bit structure – original range becomes Basic Multilingual Plane – each stretch of 0xFFFF characters is a plane – 17 planes (0-16) in all39
  40. 40. Contents• As of Unicode 6.0 – more than 109,000 characters – 93 different scripts40
  41. 41. The planes• Plane 0: BMP – Nearly the only one needed• Plane 1: Supplementary Multilingual Plane – mostly historical scripts and weird symbols• Plane 2: Supplementary Ideographic Plane – historical Chinese characters• Panes 3: Tertiary Ideographic Plane – not in use, reserved for ancient Chinese characters• Planes 4-13: Unused• Plane 14: Supplementary Special-purpose Plane• Planes 15-16: Private Use Area41
  42. 42. Basic Multilingual Plane 0xxx 1xxx 2xxx 3xxx 4xxx 5xxx 6xxx 7xxx 8xxx 9xxx Axxx Bxxx Cxxx Dxxx Exxx Fxxx CJKx1FF Latin Syll. Yix3FF Sym Hang Sym ul Priva use bols tex5FFx7FF Hang Hang Priva Scrip use te CJK CJK CJK CJK CJK CJK ul ul ts Braillex9FF CJK Scrip tsxBFF Surro gates StuffxDFF Hang ul LatinxFFF Misc 42
  43. 43. Plane 1 0xxx 1xxx 2xxx 3xxx 4xxx 5xxx 6xxx 7xxx 8xxx 9xxx Axxx Bxxx Cxxx Dxxx Exxx Fxxxx1FF Unde Large ciph Asianx3FF Conl LTR Indic angx5FF Hiero glyph icsx7FF Hiero Hiero Large Large Large Large Notat Notat Notat glyph glyph Asian Asian Asian Asian ional ional ional ics icsx9FF North Am. NearxBFF east Afric RTL anxDFF Sum erian UndexFFF ciph43
  44. 44. Encodings• UCS-2 – from ISO 10646 – two bytes per character – can only encode the BMP• UCS-4 – also from ISO 10646 – four bytes per character – can encode the whole thing• UTF-32 – same as UCS-444
  45. 45. UTF-16• Like UCS-2 – but extended with a trick to cover the full set• Surrogates – a block of code points set aside specifically for UTF- 16• Each non-BMP character is written as two surrogates, one low and one high – first 10 bits (0-03FF) added to D800 = first two bytes – next 10 bits added to DC00 = next two bytes• So, the characters become: – 1101 10xx xxxx xxxx 1101 11xx xxxx xxxx45
  46. 46. UTF-8• Cleverly designed variable-length encoding• ASCII is encoded as ASCII• Can encode all of Unicode as 4 bytes – whether more or less compact than UTF-16 depends on the text being coded – for files UTF-8 is usually far more compact46
  47. 47. UTF-8• Far and away most used Unicode encoding – because of compatibility with ASCII• Easy to recognize bit patterns – Lett Ã¥ kjenne igjen = UTF-8 interpreted as 8859-1 – æøå = æøå – ÆØÅ = ÆØÅ47
  48. 48. Reading UTF-8 as UTF-16• Two bytes get treated as a single character – effectively turns it into a random character As UTF-8 As UTF-1648
  49. 49. Han unification• Not only are there many Chinese characters – there are different variants of each character – differences between China (traditional & simplified), Japan, and Korea (also Vietnam)• Unicode has decided to encode these only once – different renderings are considered visual differences only• This is quite unpopular in the Far East – particularly in Japan49
  50. 50. Too many characters• Latin characters have a nearly infinite number of variants: a á à â ã ä å ā ą ă ậ ẩ ạ aː ȧ ...• New ones pop up all the time – these can‟t all be encoded50
  51. 51. Solution: combining characters• What if I needed Z with stroke, cedilla, and umlaut?• Simple, encode as – U+01B5 LATIN CAPITAL LETTER Z WITH STROKE – U+0327 COMBINING CEDILLA – U+0308 COMBINING DIAERESIS• Norwegian å should actually be written – a + combining ring51
  52. 52. Many ways to write a character• Unicode has inherited precomposed characters (like å) from older character sets – these are all included, for ease of roundtripping• Unicode normalization provides ways of streamlining this – unfortunately, it‟s complex, with numerous different normalization forms – won‟t go further into it52
  53. 53. Unicode Character Database• Unicode contains more than just the characters – there is a whole database of characters with many fields• It contains things like – names for each character – decomposition mappings – deprecation mappings – case mappings – breakdown into blocks – category for each character – numeric value – what script the character belongs to – ...53
  54. 54. Uses for UCD• Matching in regular expressions – by character category – by script – ...• Upper- and lower-casing of strings – beware, this is complicated...• Stripping accents – use decomposition mappings in UCD• ...54
  55. 55. More stuff in the Unicode standard• Guidance on upper/lower-casing – tricky, because there are national variations• Unicode Normalization• Sorting algorithm• Regular expression guidelines• Line breaking algorithm• Bidirectional text display (Arabic)55
  56. 56. ProgrammingDealing with characters56
  57. 57. The key principle• One internal representation for text – used everywhere, with no deviations – text from outside must always be converted• Modern programming languages enforce this – char/String vs byte (in Java) – Stream vs Reader/Writer (also Java)• Older languages do not – C char has no defined representation57
  58. 58. In an ideal world• The internal encoding of strings should not be visible – they should just be sequences of Unicode characters• In practice, this turns out to be difficult – string.charAt(): what should this return? – string.length(): what should this return? – etc58
  59. 59. Internal encodings• C strings are byte arrays• C++ bytes, UTF-16, or UTF-32• Java UTF-16• .NET UTF-16• Python UTF-16• Ruby ???• JavaScript UTF-16 or UCS-2 (poorly defined)• PHP strings are byte arrays1)• Perl UTF-8 1) http://kore-nordmann.de/blog/php_charset_encoding_FAQ.html#which-charset-59 encoding-do-strings-have-in-php
  60. 60. Java• String.charAt(int ix) – returns the UTF-16 code unit (word) at that index• String.codePointAt(int ix) – like charAt, but if it‟s a high surrogate, returns code point by combining with charAt(ix + 1)• String.length() – returns the number of UTF-16 code units in the string representation Few programming languages document the behaviour of the String class well enough for this to be clear...60
  61. 61. How long is a string?String str = "u01B5u0327u0308";System.out.println(str.length()); Output is 3, even though there is just a single, combined character.61
  62. 62. Find the bug Q: What encoding are we reading? A: We have no idea.import java.io.*;public class Cat { public static void main(String[] argv) throws IOException { BufferedReader in = new BufferedReader(new FileReader(argv[0])); String line = in.readLine(); while (line != null) { System.out.println(line); line = in.readLine(); } }}62
  63. 63. How to solve• Either find some way to auto-detect encoding – requires you to know the syntax of the file – and that syntax to have auto-detect rules• Or find some way for the user to specify the encoding – for example a command-line parameter63
  64. 64. Find the bug, 2 Q: How do we know this is UTF-8? A: We don’t.public HttpResponse get(String request) throws IOException { InputStream responseContent = null; HttpGet httpGet = new HttpGet(request); HttpResponse response = new DefaultHttpClient().execute(httpGet); responseContent = response.getEntity().getContent(); return new HttpResponse(response.getStatusCode(), response.getStatusLine().getReasonPhrase(), read(new InputStreamReader(responseContent, "UTF-8"))); }64
  65. 65. How to solve• Need to look at the Content-type – “Content-type: text/plain; charset=iso-8859-1”• Occasionally, it can be even harder – MIME-type specific rules for deciding charset if not specified in request• Best solution (for a generic Get class) is to return the stream (not the reader) – and provide enough info for clients to figure out the encoding65
  66. 66. URI vs IRI• Originally, the character encoding of URIs was not defined – characters must be ASCII, or %-escaped – however, character set of %-escapes not defined – this was RFC 2396• Then, RFC 3986 defined %-escapes as being UTF-8 – explicit characters must still be ASCII only• RFC 3987 introduced IRIs – here, everything is UTF-8 – non-ASCII characters do not need to be escaped66
  67. 67. How to distinguish IRIs and URIs• Well, uh, you can‟t, really... – if there are non-ASCII characters it‟s probably an IRI• Specifications can decide to support IRIs – for example, in XTM it‟s all IRIs• So the context can tell you, in some cases If it sounds like a mess, that’s because it is...67
  68. 68. XML and Unicode• One of the good things about XML is that it gets Unicode right – text coming out of an XML parser is always Unicode – unless the author has made a stupid mistake, there will be no character encoding problems• Does this via – syntax for declaring encoding in the document, – careful, explicit rules for detecting encoding, – escape syntax for Unicode characters, and – well-designed rules for Unicode characters in identifiers and elsewhere68
  69. 69. Representing characters in XML• Don‟t ever use entity references – &aring; and similar are the spawn of the devil – they require the DOCTYPE to be downloaded• Use UTF-8 as the character encoding – and declare it in the <?xml, or omit entirely – this way all characters can be expressed directly• If you absolutely must use stupid tricks, use &#XXXX; character references – it‟s better to avoid these, but in special cases (human authoring) they can be useful69