This document discusses character sets and the iconv library for converting between character sets. It provides examples of using iconv in PHP to: 1) convert between character sets; 2) transliterate characters that do not exist in the target character set; and 3) use the output buffer handler to automatically convert character sets for output.

1. Character sets and iconv This presentation is about character sets and the iconv library (with usage examples in PHP) By Daniel Rhodes of Warp Asylum http://www.warpasylum.co.uk

2. What is a character set? Mapping of character x in human language y is value z

3. Western European languages often use 8-bit ISO 8859-1

4. English possible in 7-bit ASCII!

5. Some languages have complex / numerous characters and need 2, 3 or even 4 bytes to represent one character!

6. So, many different character sets exist

7. More about character sets Even same language may have many different character sets

8. Character sets tend not to be compatible

9. So, conversion is necessary and useful

10. But Unicode is coming through as a modernising, unifying character set

11. Unicode is one HUGE character set that can be used to represent any character from any language!

12. Character sets? Who cares! Anglophones very lucky as everything seems to just work (even if in the background different character sets are interacting)

13. English is not the only language!

14. An app expecting character set x but getting y (or an incorrect character set conversion) will result in mojibake

15. Mojibake? What's that? A great Japanese word meaning garbled (bake) characters (moji)

16. Often encountered in Japanese computing with its two traditional character sets, Unicode and a separate character set for emails!

17. Shouldn't really happen at all in modern computing

18. But it still does, mostly due to lack of implementation knowledge

19. Mojibake in English A slight case of mojibake here, the pound symbols (£) have garbled

20. Mojibake in German More severe now, umlauted vowels ( ä, ö and ü ) have garbled

21. Mojibake in Japanese Ouch!

22. What is the iconv library? API to convert between character sets

23. Works on strings

24. Some support for transliteration (changing / substituting characters in source character set that don't exist in target character set)

25. Your implementation may vary, but a HUGE number of character sets are supported

26. Some iconv use cases Convert legacy character set ↔ Unicode

27. Convert backend ↔ frontend character sets

28. Convert file's character set for import / export

29. Transliterate to remove unwanted characters

30. Transliterate to make safe for URL / filename

31. Let's look at some iconv usage examples in PHP..

32. What is PHP's iconv extension? Interface to iconv library

33. See http://uk.php.net/manual/en/book.iconv.php

34. iconv library should be on your OS

35. If not, need to install it before using the PHP extension

36. See http://www.gnu.org/software/libiconv

37. iconv extension presence phpinfo() will look something like:

38. A few directives iconv.input_encoding – currently unused

39. iconv.output_encoding – for ob_iconv_handler() [iconv handler for PHP's output buffering]

40. iconv.internal_encoding – for ob_iconv_handler(), iconv_mime_*() and iconv's string utility functions (which are present from PHP 5)

41. First play

42. Basic usage iconv() is the conversion function

43. Pass it the input string's character set,

44. the desired output character set

45. and the input string

46. BUT within reason...

47. Within reason

48. Character mapping You might not get every character from set x present in set y

49. So what to do if character absent? Bomb out and return an empty string?

50. NO! iconv gives us a few options

51. Let's look at transliteration first...

52. First transliteration

53. Transliteration Append //TRANSLIT to output character set as passed to iconv()

54. Approximates characters not present in output character set with closest equivalent

55. Closest equivalent might simply be '?' for wildly different character sets

56. More realistic transliteration

57. Ignore option We can also append //IGNORE to the output character set as passed to inconv()

58. This will simply skip over any characters that are absent from the output character set

59. Ignore example

60. Transliterate and ignore You may (or may not!) be able to combine the //TRANSLIT and //IGNORE behaviours

61. This will transliterate transliteratable characters and ignore the rest

62. Action it by appending //TRANSLIT//IGNORE to the output character set as passed to iconv()

63. Output buffer handler We also get a handler for PHP's output buffering

64. Allows us to, for example, output everything to the browser as ISO-8859-1 though our PHP scripts etc are using UTF-8

65. An automatic way to convert character sets for output without necessarily touching anything internally

66. Let's take a look...

67. ob_iconv_handler

68. Utility functions As of PHP 5, we also get some non-conversion utility functions

69. iconv_strlen()

70. iconv_strpos(), iconv_strrpos()

71. iconv_substr()

72. Character equivalents of core strlen(), strpos(), strrpos() and substr() [which are really byte functions]

73. Quite trivial so we'll look only at one, iconv_strlen()...

74. iconv_strlen()

75. Food for thought Unicode is the character set of the future

76. PHP iconv extension uses sytem locale [setlocale()] for transliteration

77. PHP iconv extension issues a notice even when //IGNORE is used

78. iconv library has no mechanism for custom character maps

79. Summary iconv library can be accessed on the command line

80. But extension for PHP (and many other languages!)

81. Many character sets supported

82. All or nothing conversion or softer transliteration

83. Links Should be able to get a PHP source code pack from wherever you got this presentation

84. http://spin.atomicobject.com/2011/07/13/some-useful-iconv-functionality

85. http://developer.loftdigital.com/blog/php-utf-8-cheatsheet

86. http://blog.grayproductions.net/articles/encoding_conversion_with_iconv

87. http://czyborra.com/charsets/iso8859.html