Fundamental Unicode in Perl

Fundamental Unicode Nick Patch

What Is a Character? “The smallest component of written language that has semantic value;refers to the abstract meaning and/or shape, rather than a specific shape.” —The Unicode Consortium

What Is a Glyph? Glyphs are visualrepresentations of characters.

What Is a Glyph? Glyphs are visualrepresentations of characters.Fonts are collections of glyphs.

What Is a Glyph? Glyphs are visual representations of characters. Fonts are collections of glyphs.There may be many different glyphs for the same character.

What Is a Glyph? Glyphs are visual representations of characters. Fonts are collections of glyphs.There may be many different glyphs for the same character.This talk is not about fonts or glyphs.

Lettersabcπ‫ث‬й

Numbers123๓໓३

Punctuation ./?「«»」

Symbols™©≠☺☠

Control Characters CARRIAGE RETURN NO-BREAK SPACECOMBINING GRAPHEME JOINER RIGHT-TO-LEFT MARK

Character Set Many people use “character set” to mean one or more of these: Character Code Character Encoding Character RepertoireWhich makes for a confusing situation.

Character Code A defined mapping of characters to numbers. A ⇒ 41 B ⇒ 42 C ⇒ 43Each value in a character code is called a code point.

Character Encoding An algorithm to convertcode points to a digital form for ease of transmitting or storing data. 41 (A) ⇒ 1000001 42 (B) ⇒ 1000010 43 (C) ⇒ 1000011

Character Repertoire A character repertoire is a collection of distinct characters.Character codes, keyboards, and written languages all havewell-defined character repertoires.

Character Codes & Encodings ASCII character code: 128 code points character encoding: 7 bits each

Character Codes & Encodings ASCII character code: 128 code points character encoding: 7 bits each Latin 1 (ISO-8859-1) character code: 256 code pointscharacter encoding: 8 bits (1 byte) each

Character Codes & Encodings Unicode (character code)1,112,064 code points (110,000+ defined)

Character Codes & Encodings Unicode (character code)1,112,064 code points (110,000+ defined) character encodings: UTF-8 — 1 to 4 bytes each UTF-16 — 2 or 4 bytes each UTF-32 — 4 bytes each

Code Points A U+0041LATIN CAPITAL LETTER A ໓ U+0ED3 LAO DIGIT THREE U+1F4A9 PILE OF POO

Code Points Some code points have precomposed diacritics. ȫ U+022B LATIN SMALL LETTER OWITH DIAERESIS AND MACRON

Code PointsOther characters must be composed from multiple code points using “combing characters.” n̈ U+006E LATIN SMALL LETTER N U+0308 COMBINING DIAERESIS

Grapheme ClustersAny series of code points that are composed into a single user-perceived character. Informally known as “graphemes.” A (U+0041) n̥̈ (U+006E U+0308 U+0325) CRLF (U+000D U+000A)

Time for some… U+1F42ADROMEDARY CAMEL

String constants ... TIMTOWTDI# ¡jalapeño!say "x{A1}jalapex{D1}o!";

String constants ... TIMTOWTDI# ¡jalapeño!say "x{A1}jalapex{D1}o!";use v5.12;say "N{U+00A1}jalapeN{U+00D1}o!";

String constants ... TIMTOWTDIuse charnames qw( :full );say "N{INVERTED EXCLAMATIONMARK}jalapeN{LATIN SMALL LETTER N WITHTILDE}o!";

String constants ... TIMTOWTDIuse charnames qw( :full );say "N{INVERTED EXCLAMATIONMARK}jalapeN{LATIN SMALL LETTER N WITHTILDE}o!";use utf8;say ¡jalapeño!;

String constants ... POD=encoding UTF-8=head1 ¡jalapeño!

I/OUTF-8 encoded input ⇩ decode ⇩ Perl Unicode string ⇩ encode ⇩UTF-8 encoded output

I/Oopen my $fh, <:encoding(UTF-8), $filename;open my $fh, >:encoding(UTF-8), $filename;

I/Oopen my $fh, <:encoding(UTF-8), $filename;open my $fh, >:encoding(UTF-8), $filename;binmode $fh, :encoding(UTF-8);binmode STDIN, :encoding(UTF-8);

I/Ouse open qw( :encoding(UTF-8) );open my $fh, <, $filename;

I/Ouse open qw( :encoding(UTF-8) );open my $fh, <, $filename;# :std for STDIN, STDOUT, STDERRuse open qw( :encoding(UTF-8) :std );

I/Ouse open qw( :encoding(UTF-8) );open my $fh, <, $filename;# :std for STDIN, STDOUT, STDERRuse open qw( :encoding(UTF-8) :std );# CPAN module to enable everything UTF-8use utf8::all;

Explicit Encoding & Decodinguse Encode;my $internal = decode(UTF-8, $input);my $output = encode(UTF-8, $internal);

String LengthLet’s use this grapheme cluster as the string in our next example: ю́ U+044E CYRILLIC SMALL LETTER YU U+0301 COMBINING ACUTE ACCENT

String Length# UTF-8 encoded: D1 8E CC 81say length $encoded_grapheme; # 4

String Length# UTF-8 encoded: D1 8E CC 81say length $encoded_grapheme; # 4use Encode;# Unicode string: 044E 0301my $grapheme = decode(UTF-8, $encoded);say length $grapheme; # 2

String Length# UTF-8 encoded: D1 8E CC 81say length $encoded_grapheme; # 4use Encode;# Unicode string: 044E 0301my $grapheme = decode(UTF-8, $encoded);say length $grapheme; # 2my $length = () = $grapheme =~ /X/g;say $length; # 1

String Length# sort of complex for a simple length, eh?my $length = () = $str =~ /X/g;say $length;

String Length# sort of complex for a simple length, eh?my $length = () = $str =~ /X/g;say $length;# and tricky depending on the contextsay scalar( () = $str =~ /X/g );

String Length# sort of complex for a simple length, eh?my $length = () = $str =~ /X/g;say $length;# and tricky depending on the contextsay scalar( () = $str =~ /X/g );# a little better$length++ while $str =~ /X/g;say $length;

String Length# an alternative approachuse Unicode::GCString;say Unicode::GCString->new($str)->length;

String Length# an alternative approachuse Unicode::GCString;say Unicode::GCString->new($str)->length;# and yet another (Warning: I wrote it!)use Unicode::Util qw( grapheme_length );say grapheme_length($str);

CollationStandard ordering of stringsfor comparison and sorting. sort @names $a cmp $b $x gt $y $foo eq $bar

CollationPerl provides a collation algorithm based on code points.

CollationPerl provides a collation algorithm based on code points.@words = qw( Äpfel durian Xerxes ) sort @words # Xerxes durian Äpfel

CollationPerl provides a collation algorithm based on code points.@words = qw( Äpfel durian Xerxes ) sort @words # Xerxes durian Äpfel sort { lc $a cmp lc $b } @words # durian Xerxes Äpfel

CollationUnicode Collation Algorithm (UCA) providescollation based on natural language usage.

CollationUnicode Collation Algorithm (UCA) providescollation based on natural language usage.use Unicode::Collate;my $collator = Unicode::Collate->new;$collator->sort(@words);# Äpfel durian Xerxes

CollationUnicode Collation Algorithm (UCA) providescollation based on natural language usage. $collator->sort(@names) $collator->cmp($a, $b) $collator->gt($x, $y) $collator->eq($foo, $bar)

CollationUCA also provides locale-specific collations for different languages.

CollationUCA also provides locale-specific collations for different languages.use Unicode::Collate::Locale;my $kolator = Unicode::Collate::Locale->new( locale => pl # Polish);

NormalizationUnicode has 4 normalization forms. The most important are: NFD: Normalization Form Canonical Decomposition NFC: Normalization Form Canonical Composition

Normalizationuse Unicode::Normalize;# NFD can be helpful on input$str = NFD($input);# NFC is recommended on output$output = NFC($str);

NormalizationUTF-8 encoded input ⇩ decode ⇩ NFD ⇩ Perl Unicode string ⇩ NFC ⇩ encode ⇩UTF-8 encoded output

Unicode SemanticsBy default, unfortunately, strings and regexes are not guaranteed to use Unicode semantics. This is known as “The Unicode Bug.” There are a few ways to fix this:

Unicode SemanticsBy default, unfortunately, strings and regexes are not guaranteed to use Unicode semantics. This is known as “The Unicode Bug.” There are a few ways to fix this:utf8::upgrade($str);

Unicode SemanticsBy default, unfortunately, strings and regexes are not guaranteed to use Unicode semantics. This is known as “The Unicode Bug.” There are a few ways to fix this:utf8::upgrade($str);use v5.12;

Unicode SemanticsBy default, unfortunately, strings and regexes are not guaranteed to use Unicode semantics. This is known as “The Unicode Bug.” There are a few ways to fix this:utf8::upgrade($str);use v5.12;use feature unicode_strings;

UTF-8 vs. utf8 vs. :utf8 You’ll see the “utf8” encoding used frequently in Perl.“utf8” follows the UTF-8 standard very loosely and allows many errors in your data without warnings. By default, use “UTF-8” instead.

UTF-8 vs. utf8 vs. :utf8# utf8 is Perls internal encoding formmy $internal = decode(utf8, $input);# UTF-8 is the official UTF-8 encodingmy $internal = decode(UTF-8, $input);

UTF-8 vs. utf8 vs. :utf8# utf8 is Perls internal encoding formmy $internal = decode(utf8, $input);# UTF-8 is the official UTF-8 encodingmy $internal = decode(UTF-8, $input);# insecure! no encoding validation at allopen my $fh, <:utf8, $filename;# proper UTF-8 validationopen my $fh, <:encoding(UTF-8), $filename;

Questions?Slides will be posted to: @nickpatch

http://lanyrd.com	45
https://twitter.com	3
http://lanyrd.dev	2
http://us-w1.rockmelt.com	1

Fundamental Unicode in Perl

by Nick Patch

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 51

Statistics

Fundamental Unicode in Perl Presentation Transcript