This document discusses string encodings and how they have evolved over time. It begins with ASCII and the introduction of multibyte encodings to support more characters. It then covers Unicode and how it represents characters, and different encoding formats like UTF-8. It discusses issues like forcing encodings, transcoding between encodings, and ensuring encoding compatibility when combining strings. It provides examples of iterating over string elements and recommends declaring encodings to avoid problems.

1. STRING ENCODINGS

2. I’m still
@malditogeek

3. We’re hiring!

4. STRING ENCODINGS

7. FIRST THERE WAS...
ASCII

9. BUT WHAT ABOUT
THE 8TH BIT?

10. UH OH!

11. BUT MY
LANGUAGE
HAS 2000
CHARACTERS

12. WELCOME TO MULTIBYTE
ENCODINGS

13. UNICODE

14. CHARACTER SETS
vs
ENCODINGS

15. CHARACTER SETS
U+0061 = 97 = a

16. ENCODINGS
" abc ".encode("UTF-32BE")
"x00x00x00ax00x00x00bx00x00x00c"

17. UTF-8

18. RUBY 1.8
STRINGS ARE JUST
BYTE ARRAYS!

19. $KCODE=’u’

20. RUBY 1.9

21. M17N

22. STRING EXAMPLES
e = "é"
e.encoding.name # => UTF-8
e.size # => 1
e.bytesize # => 2

23. FORCING ENCODINGS
x.encoding.name #=> ISO-8859-1
x.bytesize #=> 6
x.valid_encoding? #=> true
x.force_encoding("UTF-8")
x.encoding.name #=> UTF-8
x.bytesize #=> 6
x.valid_encoding? #=> false
x =~ /x/ #=>
invalid byte sequence in UTF-8

24. TRANSCODING
x.encoding.name #=> ISO-8859-1
x.bytesize #=> 6
x.valid_encoding? #=> true
x.encode!("UTF-8")
x.encoding.name #=> UTF-8
x.bytesize #=> 12
x.valid_encoding? #=> true

25. COMPATIBILITY
if Encoding.compatible?(ascii_string, utf8_string)
new_string = ascii_string + utf8_string
new_string.encoding.name #=> UTF-8
end

26. ITERATION
x.bytes
x.each_byte {|b| puts b }
x.codepoints
x.each_codepoint {|c| puts c }
x.chars
x.each_char {|c| puts c }
x.lines
x.each_line {|l| puts l }

27. SOURCE ENCODING
# encoding: UTF-8

28. INTERNAL ENCODING
> ruby -E :UTF-8
Encoding.default_internal = 'UTF-8'

29. EXTERNAL ENCODING
> ruby -E UTF-8:
Encoding.default_external = 'UTF-8'
File.open("file.txt", "w:UTF-8")

30. WHAT DO?
USE MAGIC COMMENTS
DECLARE IO ENCODINGS
CONVERT BEFORE COMPARISONS
UNICODE-UTILS GEM

31. C’EST FINI

32. We’re hiring!

33. Thanks!
http://blog.grayproductions.net/
articles/understanding_m17n