This document discusses character sets and the iconv library for converting between character sets. It provides examples of using iconv in PHP to: 1) convert between character sets; 2) transliterate characters that do not exist in the target character set; and 3) use the output buffer handler to automatically convert character sets for output.

1. Character sets and iconv This presentation is about character sets and the iconv library (with usage examples in PHP) By Daniel Rhodes of Warp Asylum http://www.warpasylum.co.uk

3. Western European languages often use 8-bit ISO 8859-1

4. English possible in 7-bit ASCII!

5. Some languages have complex / numerous characters and need 2, 3 or even 4 bytes to represent one character!

6. So, many different character sets exist

8. Character sets tend not to be compatible

9. So, conversion is necessary and useful

10. But Unicode is coming through as a modernising, unifying character set

11. Unicode is one HUGE character set that can be used to represent any character from any language!

13. English is not the only language!

14. An app expecting character set x but getting y (or an incorrect character set conversion) will result in mojibake

16. Often encountered in Japanese computing with its two traditional character sets, Unicode and a separate character set for emails!

17. Shouldn't really happen at all in modern computing

18. But it still does, mostly due to lack of implementation knowledge

23. Works on strings

24. Some support for transliteration (changing / substituting characters in source character set that don't exist in target character set)

25. Your implementation may vary, but a HUGE number of character sets are supported

27. Convert backend ↔ frontend character sets

28. Convert file's character set for import / export

29. Transliterate to remove unwanted characters

30. Transliterate to make safe for URL / filename

31. Let's look at some iconv usage examples in PHP..

33. See http://uk.php.net/manual/en/book.iconv.php

34. iconv library should be on your OS

35. If not, need to install it before using the PHP extension

36. See http://www.gnu.org/software/libiconv

39. iconv.output_encoding – for ob_iconv_handler() [iconv handler for PHP's output buffering]

40. iconv.internal_encoding – for ob_iconv_handler(), iconv_mime_*() and iconv's string utility functions (which are present from PHP 5)

41. First play

43. Pass it the input string's character set,

44. the desired output character set

45. and the input string

46. BUT within reason...

47. Within reason

49. So what to do if character absent? Bomb out and return an empty string?

50. NO! iconv gives us a few options

51. Let's look at transliteration first...

52. First transliteration

54. Approximates characters not present in output character set with closest equivalent

55. Closest equivalent might simply be '?' for wildly different character sets

56. More realistic transliteration

58. This will simply skip over any characters that are absent from the output character set

59. Ignore example

61. This will transliterate transliteratable characters and ignore the rest

62. Action it by appending //TRANSLIT//IGNORE to the output character set as passed to iconv()

64. Allows us to, for example, output everything to the browser as ISO-8859-1 though our PHP scripts etc are using UTF-8

65. An automatic way to convert character sets for output without necessarily touching anything internally

66. Let's take a look...

67. ob_iconv_handler

69. iconv_strlen()

70. iconv_strpos(), iconv_strrpos()

71. iconv_substr()

72. Character equivalents of core strlen(), strpos(), strrpos() and substr() [which are really byte functions]

73. Quite trivial so we'll look only at one, iconv_strlen()...

74. iconv_strlen()

76. PHP iconv extension uses sytem locale [setlocale()] for transliteration

77. PHP iconv extension issues a notice even when //IGNORE is used

78. iconv library has no mechanism for custom character maps

80. But extension for PHP (and many other languages!)

81. Many character sets supported

82. All or nothing conversion or softer transliteration

84. http://spin.atomicobject.com/2011/07/13/some-useful-iconv-functionality

85. http://developer.loftdigital.com/blog/php-utf-8-cheatsheet

86. http://blog.grayproductions.net/articles/encoding_conversion_with_iconv

87. http://czyborra.com/charsets/iso8859.html