Chirp 2010: Twitter International

Twitter International

Twitter International by Matt Sanford

Agenda: * Who am I * Some business-y talk about popularity outside of the US What’s on Tap * Some quick notes on our translation process * Technical details on what’s hard about non-English text • Hashtag for Questions: #chirpintl • Who’s this guy? • Twitter’s popularity outside of the US • Twitter’s Current & Future Translation Tools • Non-English Tweet Handling • Extraction and Auto-linking with Twitter Text • Character Counting • Invalid Tweet Text

Matt Sanford / @mzsanford • Joined Twitter from Summize (Twitter Search) • Worked on Search and Platform Short bio slide. Helpful when it comes to Q&A time. • Search by language, search refresh bar • Original OAuth implementer at Twitter • Now tech lead of the International team • Working on translation tools and non-US features • Standardized character counting • Author of Open Source Twitter Text libraries

Before I cover any technical details I wanted to give a little information on why people using the Twitter Platform should be interested in International The best way to do that is with numbers … International Business. Why Bother With International?

International: 60% & Growing 100% 75% 50% 25% 0% June 2009 September 2009 December 2009 March 2010 Bam. 60% of all Twitter A big part of this is Japan, where accounts are non-US we’re quite popular. … We crossed the 50% Another big part was the new mark September of translation efforts we launched. 2009 Spanish especially has been well received.

Attendees vs. Users Non-US 17% US International US 83% Chirp Attendees Twitter Accounts

A good example of Twitter International is Chile. Translating didn’t create an explosion in Twitter usage. What created an explosion was a need for faster information. Case Study: Chile We’re There When People Need Us.

Twitter Signups in Chile We’re There When People Need Us. Fenruary 21st February 24th February 27th March 2nd

Twitter Signups in Chile We’re There When People Need Us. Fenruary 21st February 24th February 27th March 2nd Urgent. In en Constitución apareció IVAN LARA DE URGENTEConstitucion an eight-year old boy named 8 Ivan Lara showed ABANDONADO en esa ciudad...busca AÑOS QUE ESTÁ up alone. He's looking for his family parientes en todo Chile favor copiar y pegar 10:50 AM Mar 2nd via web

As opposed to the event inﬂection we saw in Chile, in Japan we’ve seen long term, sustained growth. We’ve also been dedicating resources to some local- speciﬁc features. Case Study: Japan Not Godzilla Big, But We’re Working On It.

Daily Tweeters in Japan More Users Are Good. More Engaged Users Are Better. July ‘09 October ‘09 January ‘10 April ‘10

Japanese Mobile Follow Me Take Advantage of Existing Behavior

Japanese Mobile Follow Me Take Advantage of Existing Behavior Photo: ﬂickr.com/cogdog

Japanese Mobile Follow Me Take Advantage of Existing Behavior Photo: ﬂickr.com/cogdog Photo: ﬂickr.com/netwalkerz

Since translation is a big part of what we’re working on I want to cover that a little bit. Like all Twitter features we rely on user need to help deﬁne what we do. We could have paid translators but we felt like having user’s participate in the process was important. Translation Tool That led us to our current crow-sourcing model … Present & Future

2,600 Participating Translators And we plan more than double that number very soon when we send out more invites.

3,500 Strings to Translate

3,600 Strings to Translate

480,000 Translations Staggering passion and participation from the community.

On context: Point out the Post-slide note: We’ll be arrow versus the list- rolling out changes very view of other sites. Also: soon that focus on Translation Tools suggestions consensus over new translations. On deploy: unaided today • Volunteer crowd-sourcing • Augmented by in-house people • Built-in to twitter.com • Provides context during translation • Significantly higher quality • Social game dynamics • Database backed and heavily cached • Edits are launched in ~2 hours • Multiple levels of voting • Helps prevent abuse • Built-in proofing system

Translation Tools tomorrow • We’ve released some common terms on the API wiki • So you can benefit from our translation work • To help with consistency across clients • We’re hoping to provide even more data in the future • More languages. More strings. More ease. • New translation UI changes coming soon On releasing translations: We made this a goal and covered it in the translation agreement. Let me know after this talk what would help you.

Up until now we’ve covered more general Twitter topics. Now we’re going to talk about some of the more complicated topics. Most international issues boil down to things you think are simple turning out to be deceptively hard to get right. Things like: * Parsing t weets (and what’s so hard about it) * Counting characters (and why it’s not that simple) * Tweet text that we cannot accept (today) Engineering Topics Yeah, It’s Complicated.

Twitter Text Libraries Rather than re-implement these common features we recommend using the Open Source libraries we help maintain. • Provides extraction and auto-linking If you’re not using Ruby or • @user, @user/list, #hashtag, URLs Java: We provide a cross- language test suite so you can implement the same • Open Source* rules in another language. • Available in Ruby and Java from Twitter • Conformance Testing Data • Modeled after the Unicode conformance suite • YAML description of test cases for any language • Assurance that you meet the same standards • Many non-English test cases * http://twitter.com/about/opensource and on github

Twitter Text: Japanese Linking Issues not encountered in English: • Additional punctuation characters Quick tour of the issues • s in many languages ignores U+3000 (‘ ’) the Twitter Text libraries handle in Japanese that many previous libraries didn’t handle. • Full-width punctuation forms: • @ versus The lack of word spaces is a fundamental • # versus issue when it comes to parsing Tweets. • No spaces between words

Twitter Text: Japanese Linking Issues not encountered in English: • Additional punctuation characters Quick tour of the issues • s in many languages ignores U+3000 (‘ ’) the Twitter Text libraries handle in Japanese that many previous libraries didn’t handle. • Full-width punctuation forms: • @ versus The lack of word spaces is a fundamental • # versus issue when it comes to parsing Tweets. • No spaces between words My homepage is http://twitter.com http://twitter.com

Character counting Unicode FTW!

Character counting Unicode FTW! Don’t count bytes UTF-8: 0xE5 0x91 0xB3 (3 bytes) UTF-16: 0x54 0x73 (2 bytes) U+5473 Human: 1 character

Character counting Unicode FTW! Don’t count bytes UTF-8: 0xE5 0x91 0xB3 (3 bytes) UTF-16: 0x54 0x73 (2 bytes) U+5473 Human: 1 character Don’t even count Unicode code points e + U+0065 U+0301 =é {U+0065, U+0301} OR é U+00E9

Character counting Unicode FTW! Don’t count bytes UTF-8: 0xE5 0x91 0xB3 (3 bytes) UTF-16: 0x54 0x73 (2 bytes) U+5473 Human: 1 character Don’t even count Unicode code points e + U+0065 U+0301 =é {U+0065, U+0301} OR é U+00E9 We count the shortest representation* * Unicode NFC form. See: http://unicode.org/reports/tr15/

Invalid Tweet Text Slide on characters that Twitter does not allow in a Tweet. We purposely disallow those that have no meaning in the context of a Tweet, or that For a variety of reasons have security implications. We also have a technical limitation in MySQL that disallows certain characters. It’s ﬁxed in MySQL 6 but we’ll be moving to Disallowed on Purpose Cassandra. • Byte order Marks (not needed since we only accept UTF-8): U+FFFE & U+FEFF • Reserved Unicode Special: U+FFFF • Directional Change Characters (they allow complicated phishing attacks)*: U +202A, U+202B, U+202C, U+202D & U+202E Disallowed Due to Technical Limitations • Characters outside of the Basic Multilingual Plane (BMP) • That means all Unicode code points above U+FFFF • Some Unicode 5 Kanji, Many ancient writing systems and things like musical symbols. • We’re actively working on the move from MySQL to Cassandra, which will solve this. * Unicode Security Considerations: http://www.unicode.org/reports/tr36

Questions & Answers Here To Help.

http://www.labnol.org	32
http://www.slideshare.net	14
http://coderwall.com	2
http://paper.li	1

Chirp 2010: Twitter International

by Matt Sanford on Apr 16, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 49

Statistics

Chirp 2010: Twitter International — Presentation Transcript