Email
Like
Liked ×
Save
Private Content
Embed
Loading embed code…
We have emailed the verification/download link to "".
Login to your email and click the link to download the file directly.

To request the link at a different email address, update it here. Close
Validation messages. Success message. Fail message.

Check your bulk/spam folders if you can't find our mail.
Loading

+ Follow

Character sets

by Ligaya Turmelle

1,328 views

Learn a bit about Character Sets. Aimed at the LAMP stack

Accessibility

View text version

Upload Details

Uploaded via SlideShare as Apple Keynote

Usage Rights

File a copyright complaint

Statistics

Likes: 2
Downloads: 0
Comments: 0
Embed Views: 0
Views on SlideShare: 1,328
Total Views: 1,328

\n
\n
This is to give you an idea of how these came about\n
Bullet1: 1844 - first time characters (latin alphabet and numbers) where given a set encoding (using dots and hashes). It used from 1 to 6 dots (or dashes) to symbolize different characters.\n\nReferences: \n - http://codeworks.gnomedia.com/archives/2003/programming/a-short-history-of-character-sets/\n - http://en.wikipedia.org/wiki/Character_Set\n - http://en.wikipedia.org/wiki/Extended_Binary_Coded_Decimal_Interchange_Code\n - http://en.wikipedia.org/wiki/ASCII\n - http://en.wikipedia.org/wiki/Baudot_code\n
Bullet2: 1870 (patented 1874) - 5 bit code with equal on and off intervals, which allowed telegraph transmission of the letters, punctuation and control signals. This was later modified in 1901 by Donald Murray. \n\nReferences: \n - http://codeworks.gnomedia.com/archives/2003/programming/a-short-history-of-character-sets/\n - http://en.wikipedia.org/wiki/Character_Set\n - http://en.wikipedia.org/wiki/Extended_Binary_Coded_Decimal_Interchange_Code\n - http://en.wikipedia.org/wiki/ASCII\n - http://en.wikipedia.org/wiki/Baudot_code\n
Bullet3: Around 1930 the International Telegraph and Telephone Consultative Committee (French abbreviation of CCITT) used the Baudot/Murray code as the basis for the ITA2 standard (International Telegraphy Alphabet #2).\n\nReferences: \n - http://codeworks.gnomedia.com/archives/2003/programming/a-short-history-of-character-sets/\n - http://en.wikipedia.org/wiki/Character_Set\n - http://en.wikipedia.org/wiki/Extended_Binary_Coded_Decimal_Interchange_Code\n - http://en.wikipedia.org/wiki/ASCII\n - http://en.wikipedia.org/wiki/Baudot_code\n
Bullet4: ASCII (American Standard Code for Information Interchange) was “invented” by the American Standards Association in 1963. ASCII uses a 7 bit encoding scheme.\n\nEBCDIC (Extended Binary Coded Decimal Interchange Code) was also “invented” in 1963 and is an 8 bit encoding system that was most commonly found on IBM systems.\n\nReferences: \n - http://codeworks.gnomedia.com/archives/2003/programming/a-short-history-of-character-sets/\n - http://en.wikipedia.org/wiki/Character_Set\n - http://en.wikipedia.org/wiki/Extended_Binary_Coded_Decimal_Interchange_Code\n - http://en.wikipedia.org/wiki/ASCII\n - http://en.wikipedia.org/wiki/Baudot_code\n
\n
\n
Suppose that we have an alphabet with four letters: “Q”, “Y”, “q”, “y”. (there is no logic to the letters other then that I like them so don’t try to place any on it.) \n\n
Suppose that we have an alphabet with four letters: “Q”, “Y”, “q”, “y”. (there is no logic to the letters other then that I like them so don’t try to place any on it.) \n\nThese are the symbols of our alphabet.\n\n
We then give each letter a value: “Q” = 1, “Y” = 2, “q” = 3, “y” = 4. \n\n\n
So now the letter “Q” is a symbol, the number 1 is the encoding for the “Q” symbol.\n\nThe combination of all four letters and their encodings is a character set.\n\nReferences:\n- http://dev.mysql.com/doc/refman/5.0/en/charset-general.html\n\n
whole alphabets, symbols, punctuation marks... or Eastern writing systems.\n
\n
\n
So lets go back to our previous encoding example, \n
So lets go back to our previous encoding example, if we were to order the alphabet based on the encoding we now see that lower case q is greater then upper case Y. This is a collation.\n\nA collation is a set of rules, or in this case one rule, for ordering the character set.\n\nOur example here is very simplistic. Now think if you want to make the upper and lower case letters the same (case insensitive), or if we have accent characters in our alphabet, or multiple character mappings (“Ö” = “OE”), or eastern writing systems with thousands of characters... Things can get very complex very quickly on how we want to order the character set. \n
\n
When I was talking to my colleagues about my writing this presentation, I jokingly asked them if I could just get away with a single slide that just had “Use UTF8” on it. After some discussion amongst ourselves, we finally agreed on this slide to be the one single thing you should take away from the talk. It is not overly profound - but does sum up our experiences rather nicely.\n\nUse latin1 / ISO 8859-1 for everything if you can. We say this because it is fixed format, commonly used as a default for most things and can generally speaking be handled quickly and easily.\n\nIf however you are working with a language that does not work well with latin1 / ISO 8859-1, don’t try to be smart and use a specific character set for it - just use UTF8.\n
\n
UTF8 can represent every Unicode character and is backwards compatible with ASCII. Multi-byte refers to it using from 1-4 8 bit bytes as needed. \n\nHow many people already know what Unicode is? \n\nIf there is someone who does not know what Unicode is:\nGoogle Dictionary Definition: \nAn international encoding standard for use with different languages and scripts, by which each letter, digit, or symbol is assigned a unique numeric value that applies across different platforms and programs\n\nFrom the Unicode.org site: Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language.\n\nReferences:\nhttp://unicode.org/standard/WhatIsUnicode.html\nhttp://en.wikipedia.org/wiki/UTF-8\n
To make your life easy...\n
I mean this in the most literal sense. What is the character set of the input device? The HTML Form (accept-charset) and/or the web page for example? The output device that they are looking at it on? The HTML page or the VI display for example. What about intermediate steps? PHP, MySQL, batch scripts that may manipulate it, exporting scripts, cron scripts, etc.?\n\nThe more consistent you can be across everything, the less likely things will be bungled up along the way.\n
Do not assume that what ever part will just “know” what character set things are. If you can explicitly state it or set it - do. Default settings may be wildly different then what you think. Ex: MySQL - who can tell me what the default character set is for it? latin1 - What about the default collation? It is latin1_swedish_ci\n\nAgain if you are consistent with the character set you use this can almost be a set it and forget it type of thing. Ex: Apache in the section "AddDefaultCharset utf8"\n
\n
- A header on the web page\n\n
- from the apache manual:\nspecifies a default value for the media type charset parameter (the name of a character encoding) to be added to a response if and only if the response's content-type is either text/plain or text/html. This should override any charset specified in the body of the response via a META element, though the exact behavior is often dependent on the user's client configuration\n\n
- everyone but IE supports accept-charset - specifies the character-sets the server can handle for form-data.\n
- If you want to learn a bit about PHP and Unicode check out Andrei Zmievski’s PHP Community Conference presentation - The good, the bad and the ugly.\n\n
- mbstring provides multibyte specific string functions that help you deal with multibyte encodings in PHP. Also mbstring handles character encoding conversion between the possible encoding pairs. \n- mbstring is designed to handle Unicode-based encodings such as UTF-8 and UCS-2 and many single-byte encodings for convenience\n- mbstring has a host of ini settings \n\n
- PHP always outputs a character encoding by default in the Content-type: header. This can be set in the php.ini file.\n\n
- SET NAMES when you first connect to the MySQL database, indicates what character set the client will use to send SQL statements to the server. It also specifies the character set that the server should use for sending results back to the client.\n
- Most prefer to set it and forget it if they can\n- Server, database, table and column\n- SET NAMES ‘utf8’ - sets the charset in the communication between the client program and the MySQL server.\n
- Most prefer to set it and forget it if they can\n- Server, database, table and column\n- SET NAMES ‘utf8’ - sets the charset in the communication between the client program and the MySQL server.\n
- SET NAMES ‘utf8’ - sets the charset in the communication between the client program and the MySQL server.\n
Default server character set is latin1\nDefault collation for the server is latin1-swedish_ci\nThese built in defaults can be changed in the option file.\nDefaults are used if a DB charset and collation are not set in the CREATE statement.\n\nThe only way to change the *built in defaults* - is by recompiling (./configure --with-charset=X --with-collation=Y) \n\n
- This sets the default values for the tables in the database.\n - also used with LOAD DATA INFILE\n - character set + collation - then the values given are used.\n - character set only - the default collation for that character set will be used. To find out what the default collation for that character set is - use SHOW COLLATION\n - collation only - the character set associated with that collation is used.\n- neither are given - server level values are used.\n
Can you set these at the individual table and column level - sure.\n\nMis-matching character sets/collations translates into full table scans with conversions being done on *each* row.\n
character_set_client = the character set for the statements that arrive from the client\ncharacter_set_results = the character set used for the result set or the error message\ncharacter_set_connection = character set used for literals or number to string conversions\n\n
character sets can have multiple collations. Depending upon what you want to do in terms of sorting and comparison, you want to pick the right one. MySQL collation names follow a convention to help make it easier for you to know what you are picking.\n - ci - case insensitive\n - cs - case sensitive\n - bin - binary collation. based on the characters binary code values.\n\nFor those with more questions on _bin and binary strings, they want to check out http://dev.mysql.com/doc/refman/5.0/en/charset-binary-collations.html\n
show and example ordering by latin1_general_ci and another by latin1_general_cs to highlight that the collation has a real effect in things.\n
\n
\n
\n
\n

2 Likes

dapoayoola 5 months ago
mury89 Tags encoding character encoding 1 year ago

Character sets Presentation Transcript

Character Sets Suck
About Us• Ligaya Turmelle • Raymond DeRoo• Senior Technical • Data Architect Support Engineer• MySQL Now (AKA • Nimbuzz BV Oracle) • .15 decades• ~3 years
A very brief history of character sets
Character sets• Earliest form was Morse Code.• Baudot Code came next• ITA2• ASCII and EBCIDIC then come along in 1963
Character sets• Earliest form was Morse Code.• Baudot Code came next• ITA2• ASCII and EBCIDIC then come along in 1963
Character sets• Earliest form was Morse Code.• Baudot Code came next• ITA2• ASCII and EBCIDIC then come along in 1963
Character sets• Earliest form was Morse Code.• Baudot Code came next• ITA2• ASCII and EBCDIC then come along in 1963
What exactly is a Character Set?
it is a set ofsymbols and encodings
ExampleQ Y q y
ExampleQ Y q y = the symbols
ExampleQ Y q y = the symbols Q=1 Y=2 q=3 y=4
ExampleQ Y q y = the symbols Q=1 Y=2 is the encoding q=3 y=4
In real life mostcharacter sets have many characters
Now that we know abit about charactersets - what the heck is a collation?
Collations dealwith how we orderthe character set
CollationsQ=1Y=2q=3y=4
CollationsQ=1Y=2q=3 Q<Y<q<yy=4
So The answer - or at least OUR answer - to Character Sets Sucking is..
Use Latin1 if youcan - otherwise just use UTF8
What is UTF8?
a multi-bytecharacter encoding for the Unicode character set.
Handling Character Sets
Be consistent
Check everything along the way
Some specific Settings to consider
For the webpage• Content-Type: text/html; charset=utf-8• Apache virtual host setting • AddDefaultCharset utf-8• web form - accept-charset
For the webpage• Content-Type: text/html; charset=utf-8• Apache virtual host setting • AddDefaultCharset utf-8• web form - accept-charset
For the webpage• Content-Type: text/html; charset=utf-8• Apache virtual host setting • AddDefaultCharset utf-8• web form - accept-charset
PHP• Not here to discuss Unicode and PHP6• mbstring• default_charset• SET NAMES
PHP• Not here to discuss Unicode and PHP6• mbstring• default_charset• SET NAMES
PHP• Not here to discuss Unicode and PHP6• mbstring• default_charset• SET NAMES
PHP• Not here to discuss Unicode and PHP6• mbstring• default_charset• SET NAMES
MySQL• Has ﬁne grained control if you really want it • Can handle character set *and* collation settings at 4 levels within MySQL• Plus at the client/server connection level
MySQL• Has ﬁne grained control if you really want it • Can handle character set *and* collation settings at 4 levels within MySQL• Plus at the client/server connection level
MySQL• Has ﬁne grained control if you really want it • Can handle character set *and* collation settings at 4 levels within MySQL• Plus at the client/server connection level
MySQL (server level)• --character-set-server• --collation-server
MySQL (Database level)• Can be set in the CREATE/ALTER DATABASE statement CREATE DATABASE test CHARACTER SET utf8 COLLATE utf8_general_ci;
Character set table and columns• Save yourself the headache • don’t mix and match the values for these unless you must.• Mis-matches between different table and column character sets in your queries can kill performance
SET NAMES• Again - save the headache and just always send it.• Sets character set that the client will use to send SQL to the server.• Equivalent to: SET character_set_client=X; SET character_set_results=X SET character_set_connection= X
Collation names Explained• Conventions used in MySQL: • _ci • _cs • _bin
So to wrap this all up...
Use Latin1 if youcan - otherwise just use UTF8
Questions?
References - http://codeworks.gnomedia.com/archives/2003/programming/a-short-history-of-character-sets/ - http://en.wikipedia.org/wiki/Character_Set - http://en.wikipedia.org/wiki/Extended_Binary_Coded_Decimal_Interchange_Code - http://en.wikipedia.org/wiki/ASCII - http://en.wikipedia.org/wiki/Baudot_code - http://dev.mysql.com/doc/refman/5.0/en/charset-general.html - http://unicode.org/standard/WhatIsUnicode.html - http://en.wikipedia.org/wiki/UTF-8 - http://httpd.apache.org/docs/2.0/mod/core.html - http://www.w3schools.com/TAGS/att_form_accept_charset.asp - http://us3.php.net/mbstring - http://php.net/manual/en/ini.core.php - http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html - http://dev.mysql.com/doc/refman/5.0/en/charset-server.html - http://dev.mysql.com/doc/refman/5.0/en/charset-database.html

Character sets

by Ligaya Turmelle

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Character sets Presentation Transcript