Unicode character set, utf8, utf16, utf32

1. Unicode

2. Character Set vs Encoding
● Character Set
A, B, C, D …
● Encoding
D = 0100100

3. ASCII
● Simple Encoding
A = 0100 0001
B = 0100 0010
a = 0110 0001
b = 0110 0010

4. Small Character Set

5. Code Pages

6. Unicode Character Set
A, B, C …
Ã, ß, Ĉ …
А, Б, В …
‫,ך‬ ऋ, ఋ ...
☺, ☻

7. Unicode Goals
● Universal – should be the only character set
ever needed
● Semantics – all characters must have well
defined semantics
● Originally intended to be 16 bits

8. Unicode Encoding
● Originally designed to be 16 bits per character
● UTF-8
● UTF-16
● UTF-32

9. UTF-32
Everything is 4 bytes:
0001 0110 0011 1001 0001 0110 0011 1001
0000 0000 0000 0000 0000 0000 0100 0001

10. UTF-8
Everything is as small as we can make it:
A = 0100 0001
à = 1100 0100 1001 1010
● 0xxx xxxx 7 bits
● 110x xxxx 10xx xxxx 11 bits
● 1110 xxxx 10xx xxxx 10xx xxxx 16 bits
● 1111 110x (10xx xxxx) x 5 31 bits

11. UTF-16
Everything is 2 bytes, except when it is 4 bytes.
0000 0000 0010 0100
1101 1000 0000 0001 1101 1100 0011 0111

12. 32 > 16 > 8
The letter A:
● UTF-8: 0100 0001
● UTF-16: 0000 0000 0100 0001
● UTF-32: 0000 0000 0000 0000
0000 0000 0100 0001

13. 32 > 8 > 16 ???
The letter ☻:
● UTF-8: 0111 0010 1001 1000
1011 1011
● UTF-16: 0010 0110 0011 1011
● UTF-32: 0000 0000 0000 0000
0010 0110 0011 1011

14. Sorting Unicode Characters
● 0100 0010 > 0100 0001
B > A
● 0x263B > 0x262D
☻> ☭
● Unicode Collation
● Custom Collation

15. Internal Encoding
● C – byte arrays
● C++ - byte
● Java – UTF-16
● .NET – UTF-16
● JavaScript – UTF-16
● Perl – byte

16. POM