@IndeedEng: Tokens and Millicents - technical challenges in launching Indeed around the world

2,186 views

Published on

This talk was held on Wednesday, June 25, 2014

Engineering a product to serve jobseekers around the world requires solving a diverse set of technical challenges. In this talk, we will delve deeper into some of those technical challenges we addressed to make our product succeed internationally. We will describe how language detection, text segmentation and stemming helped improve the relevance of our search results. We will also share how we’ve had to evolve our sponsored auction and billing systems to handle multiple currencies.

Watch on YouTube: https://www.youtube.com/watch?v=JMVEmzkh7II

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

@IndeedEng: Tokens and Millicents - technical challenges in launching Indeed around the world

  1. 1. go.indeed.com/IndeedEngTalks
  2. 2. Tokens and Millicents Technical challenges in launching Indeed around the world
  3. 3. Engineering Director Dan Heller
  4. 4. We help people get jobs.
  5. 5. what where job title, keywords or company name city, state or zip code software Find Jobsaustin
  6. 6. was wo job title, keywords or company name city, state or zip code produktionshelfer Jobs findenmünchen
  7. 7. キーワード 勤務地 職種、キーワード、会社名など 都道府県名または市区町村名 登録栄養士 求人検索大阪
  8. 8. Αθήνα τι που τίτλος θέσης εργασίας, λέξεις-κλειδιά ή όνομα εταιρείας πόλη ή πολιτεία βοηθός λογιστή Εύρεση θέσεων εργασίας
  9. 9. Software Engineer Preetha Appan
  10. 10. Precision and Recall
  11. 11. ALL JOBS Relevant Jobs Returned Jobs
  12. 12. Precision: Positive Predictive Value # Returned and relevant # Returned
  13. 13. Precision Job seeker searches for “architect” usually means “building architect”
  14. 14. Precision Job seeker searches for “architect” 10 jobs returned: 8 building architect jobs 2 software architect jobs
  15. 15. Precision Job seeker searches for “architect” 10 jobs returned: 8 building architect jobs Relevant 2 software architect jobs Not Relevant Precision: 8 / 10
  16. 16. Recall: Specificity # Returned and relevant # Relevant
  17. 17. Recall Job seeker searches for “hr” Jobs that mention “hr” or “human resources” are both relevant to the job seeker.
  18. 18. Recall Job seeker searches for “hr” 10 jobs are relevant: 7 hr jobs 3 human resources jobs
  19. 19. Recall Job seeker searches for “hr” 10 jobs are relevant: 7 hr jobs Returned 3 human resources jobs Not Returned Recall: 7 / 10
  20. 20. Improving Recall in Job Search
  21. 21. Senior Software Engineer - Search Indeed - Austin, TX Indeed.com is seeking a Senior Software Engineer responsible for the information retrieval system that powers Indeed’s job search website. If you are an engineer who's passionate about building innovative products... Job Description - English
  22. 22. Senior Software Engineer - Search Indeed - Austin, TX Indeed.com is seeking a Senior Software Engineer responsible for the information retrieval system that powers Indeed’s job search website. If you are an engineer who's passionate about building innovative products... Tokenization
  23. 23. Inverted Index ● Like index in the back of a book ● words = tokens, page numbers = doc ids
  24. 24. Inverted Index Token Job A Job B Job C assistant ✔ developer ✔ engineer ✔ lawyer ✔ ✔ paralegal ✔ ✔ retrieval ✔
  25. 25. Inverted Indexes Allow you to: ● Quickly find all documents containing a token ● Perform boolean queries, e.g “java AND developer”
  26. 26. Apache Lucene Open source inverted index implementation Fast, widely used
  27. 27. Tokenization with Lucene StandardAnalyzer ● Uses space and punctuation to determine token boundaries
  28. 28. StandardAnalyzer - problems ● C++, C# → C ● O’Reilly → O, Reilly
  29. 29. Tokenization with Lucene JobAnalyzer ● forked StandardAnalyzer ● Modified it to make it work for jobs
  30. 30. Secrétaire Saclay Au sein de la direction de la Qualité et de l'Environnement (DQE) vous seconderez la secrétaire-assistante. Vos principales missions seront : - organisation de réunions - l'accueil téléphonique - la gestion des missions .. Job Description - French
  31. 31. Chinese Japanese Korean (CJK)
  32. 32. Job Description - Chinese 岗位描述: 1、全厂电气设备的日常检查、记录,在操作工或 主操的指导下进行工艺操作. 2、现场液体充装,现场充装安全的管理. 3、负责现场工作环境的整洁. ...
  33. 33. Job Description - Japanese ちょっと想像してみてください。 ご近所のサーティワンにあなたが企画開発
  34. 34. Kanji Job Description - Japanese ちょっと想像してみてください。 ご近所のサーティワンにあなたが企画開発
  35. 35. Kanji Job Description - Japanese ちょっと想像してみてください。 ご近所のサーティワンにあなたが企画開発 Hiragana
  36. 36. Job Description - Japanese ちょっと想像してみてください。 ご近所のサーティワンにあなたが企画開発 Kanji Hiragana Katakana
  37. 37. Chinese using JobAnalyzer 全厂电气设备的日常检查、记录, 在操作工或主操的指导下进行工艺操作.
  38. 38. 全厂电气设备的日常检查、记录, 在操作工或主操的指导下进行工艺操作. Chinese using JobAnalyzer
  39. 39. 全厂电气设备的日常检查、记录, 在操作工或主操的指导下进行工艺操作. “Daily inspection of electrical equipment plant-wide” Chinese using JobAnalyzer
  40. 40. JobAnalyzer in CJK = Poor recall
  41. 41. CJKAnalyzer - bigrams
  42. 42. 医療事務兼検査助手
  43. 43. 医療事務兼検査助手 medical
  44. 44. 医療事務兼検査助手 ????
  45. 45. 医療事務兼検査助手 affairs
  46. 46. Use bigram tokenizer on query
  47. 47. “東京都” Tokyo prefecture
  48. 48. “東京都” Tokyo prefecture 東京都東京都
  49. 49. 東京都 “東京都” Tokyo prefecture 東京都 Tokyo
  50. 50. 東京都東京都 “東京都” Tokyo prefecture Tokyo Kyoto
  51. 51. Bigram tokenizer Drawbacks ● Poor precision
  52. 52. Bigram tokenizer Drawbacks ● Poor precision ● Too many terms
  53. 53. Properly tokenize CJK
  54. 54. Accent and gender normalization ● secrétaire, secretaire
  55. 55. Accent and gender normalization ● secrétaire, secretaire ● vendeur, vendeuse
  56. 56. Accent and gender normalization ● secrétaire, secretaire ● vendeur, vendeuse ● promotor@s
  57. 57. Language Detection
  58. 58. Language Detection options ● HTTP Content-Language response header ○ Most sites don’t provide this header ○ May not be accurate
  59. 59. Language Detection - ICU4J ● ICU4J’s CharsetDetector ○ Works well for languages with single byte encoded characters ○ Detect that language is one of Danish, Dutch, English, French, German, Italian, Portuguese, Swedish
  60. 60. Naive Bayesian classifier ● Features - words ● Strong independence assumption ● Class label - language
  61. 61. Naive Bayesian Language detector Hand labelled training data in each language
  62. 62. Naive Bayesian Language detector For each language, calculate P(wi ϵ Lj ) ● P(“experience” ϵ en) = 0.85
  63. 63. Naive Bayesian Language detector P(w1 ϵ Lj ) * P(w2 ϵ Lj ) * P(w3 ϵ Lj )*..
  64. 64. Using Unicode Blocks
  65. 65. Thai min max
  66. 66. min max Greek
  67. 67. ● 100% accurate ● Used in: ○ Thai ○ Greek ○ Korean ○ Hebrew Using Unicode Blocks
  68. 68. CJ language detection ● Strongly weight Hiragana and Katakana ● Some characters (Kanji) common between Chinese and Japanese ● p(卒 ϵ ja) = 0.99 p(卒 ϵ zh) = 0.000001
  69. 69. Language Results ● Did cross validation on hand labeled testing data ● 99% accurate for text > 30 characters ○ Average job description is 200 characters ● Fast - 0.6ms per job
  70. 70. Other language detectors Google - https://code.google.com/p/cld2/
  71. 71. CJK Tokenization
  72. 72. CJK tokenizers ● Dictionary-based ● Statistical model & dictionary
  73. 73. Dictionary-based tokenizers ● Dictionary of words in language ● Scan input sentence, return all possible tokenizations
  74. 74. Context matters
  75. 75. 北京大学生前来应聘
  76. 76. 北京 大学生前来应聘 Beijing
  77. 77. 北京 大学生 前来应聘 Beijing college students
  78. 78. 北京 大学生 前来 应聘 Beijing college students come to
  79. 79. 北京 大学生 前来 应聘 Beijing college students come to apply jobs
  80. 80. 北京 大学生 前来 应聘 Beijing college students come to apply jobs 北京大学生前来应聘
  81. 81. 北京 大学生 前来 应聘 Beijing college students come to apply jobs 北京大学 生前来应聘 Peking University
  82. 82. 北京 大学生 前来 应聘 Beijing college students come to apply jobs 北京大学 生前 来应聘 Peking University before death
  83. 83. 北京 大学生 前来 应聘 Beijing college students come to apply jobs 北京大学 生前 来应聘 Peking University before death come to apply jobs
  84. 84. 北京 大学生 前来 应聘 Beijing college students come to apply jobs 北京大学 生前 来应聘 Peking University before death come to apply jobs
  85. 85. Hidden Markov Model
  86. 86. 北京大学 Peking University 中国 China 生前 before death
  87. 87. 北京大学生前来应聘 北京大学 Peking University 生前 before death 北京 Beijing 大学生 college student
  88. 88. ✔ ✘ 北京大学 Peking University 生前 before death 北京 Beijing 大学生 college student 北京大学生前来应聘
  89. 89. CJK tokenizers ● Chinese - Imdict ● Japanese - Sen ● Korean - LuceneKorean
  90. 90. Chinese tokenization http://nlp.stanford.edu/projects/chinese-nlp.shtml
  91. 91. ● Different rules per language around ○ Gender ○ Plurals ○ Collation More recall challenges
  92. 92. Apply language specific rules to transform words to canonical form Use detected language
  93. 93. Stemming
  94. 94. What is stemming? the process of turning multiple variations of a word into a single equivalent root
  95. 95. Stemming examples ● driver, drivers → driver ● secretaire, secrétaire → secretaire ● vendeur, vendeuse → vendeur
  96. 96. Why stemming matters ● Return all possible relevant jobs given the user’s query, not just exact matches
  97. 97. Stemming - Lucene Analyzers ● Do stemming before adding to inverted index ● Examples ○ PorterStemFilter ○ SnowballAnalyzer ○ EnglishMinimalStemmer
  98. 98. Inverted Index Job A: Directrice de Documentaires Job B: Directeur de production Token Job A Job B de ✔ ✔ directeur ✔ ✔ documentaires ✔ production ✔
  99. 99. Search with stemming tokenizers ● At search time, use the same analyzer on the query ○ “directrice” → “directeur” ● Search for “directrice” returns both jobs
  100. 100. Modifying stem rules require full index rebuild ● If roots have changed need to re- process all jobs
  101. 101. Token Job A Job B de ✔ ✔ directeur ✔ ✔ documentaires ✔ production ✔
  102. 102. Drawbacks ● Loss of precise information ○ “Directrice” search should return exact match only
  103. 103. Decouple stemming from indexing
  104. 104. Term Expansion Maps
  105. 105. Term Expansion Maps ● Map from String->List<String> ● Key is root, values are tokens that stem to that root ● driver → driver, drivers ● vendeur → vendeur, vendeuse
  106. 106. Stemmer interface ● One method ● String stem(String token) ● Many implementations ● EnglishStemmer ● FrenchStemmer ● GermanStemmer ● SpanishStemmer
  107. 107. Building term expansion map for each language for each term in language root = Stemmer.stem(term) termMap[root].append(term) ● Takes ~1.5 minutes on index with 2 million tokens and 18 languages
  108. 108. Using term expansion map
  109. 109. Job A: Directrice de documentaires Job B: Directeur de production Token Job A Job B de ✔ ✔ directeur ✔ ✔ documentaires ✔ production ✔
  110. 110. Job A: Directrice de documentaires Job B: Directeur de production Token Job A Job B de ✔ ✔ directeur ✔ directrice ✔ documentaires ✔ production ✔
  111. 111. Search Service “directrice”
  112. 112. “directrice” “directeur” French Stemmer
  113. 113. “directrice” “directeur” Term Expansion Map French Stemmer Query Rewriter
  114. 114. “directrice” “directeur” Term Expansion Map French Stemmer Query Rewriter “directrice” OR “directeur”
  115. 115. Job A: Directrice de documentaires Job B: Directeur de production Token Job A Job B de ✔ ✔ directeur ✔ directrice ✔ documentaires ✔ production ✔
  116. 116. Benefits ● Modifying stem rules don’t require index rebuilds ○ Takes minutes on index with millions of jobs ○ Had flexibility to iteratively implement stemming rules as we come across different use cases
  117. 117. Benefits ● Precise information ○ “directrice” search query returns exact match only
  118. 118. Code deploy to change rules or add languages
  119. 119. 49 team members 26 nationalities 18 languages
  120. 120. Scale Stemming ● Indeed continued international expansion ● Needed stemming to scale without code deploys and coordination between developers and country managers.
  121. 121. Goal ● Efficient ○ Store term expansion maps efficiently ○ Search time as fast as possible
  122. 122. Goal ● Generic ○ identify patterns common to all languages. ■ ies→y in English, se→r in French
  123. 123. Goal ● Comprehensive ○ Support all use cases we care about: ■ plurals ■ synonyms ■ abbreviations ■ accent collation ■ gender suffixes
  124. 124. Goal ● Scalable ○ Adding a new language shouldn’t need a code deploy
  125. 125. Rule driven stemming one stemmer. all languages.
  126. 126. What is a stemming rule ● Rules transform tokens into their root form
  127. 127. Rule attributes ● Rules have “from” (origin) and “to” (replacement)
  128. 128. Rule attributes ● Rules have a type ○ Types define exactly how the text transformation happens
  129. 129. Rule type - exact ● Change origin to replacement when its an exact match
  130. 130. Exact rule English sr→senior attorney→lawyer Italian colf→domestica Dutch leraar→docent
  131. 131. Rule type - substring ● Change all occurrences of origin to replacement
  132. 132. Substring rule English - é→e résumé → resume café → cafe German - ä → a verkäufer → verkaufer French - ô→o hôtesse → hotesse
  133. 133. Rule type - suffix ● Change origin to replacement if it matches at the end of token
  134. 134. Suffix Rule - English ● ies→y ○ families → family ○ policies → policy ● s→’’ ○ nurses → nurse ○ drivers → driver
  135. 135. Suffix Rule - French ● euse→eur ○ serveuse→serveur ● ienne→ien ○ gardienne→ gardien
  136. 136. Rules are ordered
  137. 137. Order matters Stem “families” Rules ● s→’’ ● ies→y Apply s→’’
  138. 138. Order matters Stem “families” Rules ● s→’’ ● ies→y Apply s→’’ ● families → familie
  139. 139. Stem “families” Rules ● s→’’ ● ies→y Apply s→’’ ● families → familie Order matters ✘
  140. 140. Rules can be marked as terminal ● No more rules applied after terminal rule
  141. 141. Prevent over-stemming ● s → ‘’ can cause this → thi ● Min Length - special terminal rule ● Usually set to anywhere from 3 to 5
  142. 142. Babelfish: Stem rule editor ● Webapp to edit and publish rules ● Rules interpreted by generic stemmer ● 27 languages
  143. 143. Stem rule editor
  144. 144. Stem rule editor
  145. 145. Stem rule editor
  146. 146. Ability to audit rules
  147. 147. directrices directrice suffix rule “s” → “” directeur suffix rule “trice” → “teur” ingénieur ingenieur substring rule “é” → “e”
  148. 148. Job Seekers Stem Rule Editor EN s → ‘’, ces → y, … FR e → é, u → ù, … Jobs Index Builder Term Expansion Map sale → sale, sales policy → policy, policies Search Service Country Managers query results
  149. 149. Term expansion map storage ● Custom serialization format ○ Store string array as UTF8 bytes and offsets ○ Front encoding for additional compression ● 2X smaller than using Java native serialization
  150. 150. Comprehensive ● Gender ● Accents ● Plurals ● Synonyms
  151. 151. Scalable 27 languages use stemming rules Re-used language detection and stemming libraries in resume search
  152. 152. Efficient ● Term expansion map in Europe index has 2 million terms in 18 languages - 60MB on disk ● Building term expansion maps takes ~ 1.5 minutes ● Doing boolean query for stemming adds ~5ms to median search time (~35ms)
  153. 153. Stemming helps job seekers Searches that return no jobs reduced by 60% with stemming 3% to 5% more clicks
  154. 154. Multi-currency Sponsored Jobs
  155. 155. Sponsored Jobs at Indeed Real-time auction used to determine Sponsored Job impressions
  156. 156. Sponsored Jobs at Indeed Real-time auction used to determine Sponsored Job impressions Auction winner based on expected value
  157. 157. Expected Value = Bid x eCTR Expected Click-Through Rate*
  158. 158. Expected Value = Bid x eCTR Expected Click-Through Rate*
  159. 159. Expected Value = Bid x eCTR Expected Click-Through Rate*
  160. 160. Job Bid A $3.00 B $2.00 C $1.00
  161. 161. Job Bid eCTR A $3.00 5% B $2.00 10% C $1.00 8%
  162. 162. Job Bid x eCTR = Value A $3.00 5% $0.15 B $2.00 10% $0.20 C $1.00 8% $0.08
  163. 163. Job Bid x eCTR = Value → Rank A $3.00 5% $0.15 2 B $2.00 10% $0.20 1 C $1.00 8% $0.08 3
  164. 164. Job Bid x eCTR = Value → Rank B $2.00 10% $0.20 1 A $3.00 5% $0.15 2
  165. 165. Job Bid x eCTR = Value → Rank B $2.00 10% $0.20 1 A $3.00 5% $0.15 2 B could win the auction with a lower bid...
  166. 166. B could win the auction with a lower bid... …only charge what’s needed to win! Job Bid x eCTR = Value → Rank B $2.00 10% $0.20 1 A $3.00 5% $0.15 2
  167. 167. B could win the auction with a lower bid... …only charge what’s needed to win! Job Bid x eCTR = Value → Rank B $2.00 10% $0.20 1 A $3.00 5% $0.15 2 $1.50 x 10% = $0.15
  168. 168. B could win the auction with a lower bid... …only charge what’s needed to win! Cost = $1.51 Job Bid x eCTR = Value → Rank B $2.00 10% $0.20 1 A $3.00 5% $0.15 2
  169. 169. B could win the auction with a lower bid... …only charge what’s needed to win! Cost = $1.51 Job Bid x eCTR = Value → Rank B $2.00 10% $0.20 1 A $3.00 5% $0.15 2
  170. 170. B could win the auction with a lower bid... …only charge what’s needed to win! Cost = $1.51 Job Bid x eCTR = Value → Rank B $2.00 10% $0.20 1 A $3.00 5% $0.15 2
  171. 171. Sponsored Jobs at Indeed “Generalized Second Price Auction”
  172. 172. Sponsored Jobs at Indeed “Generalized Second Price Auction” ● Fair for employers ● Ensures sponsored results are relevant and useful for job seekers
  173. 173. Sponsored Jobs at Indeed Employers set their bid & budget
  174. 174. Sponsored Jobs at Indeed Employers set their bid & budget employer_id int(10) unsigned, bid decimal(10,2) unsigned, daily_budget decimal(10,2) unsigned,
  175. 175. Sponsored Jobs at Indeed A builder process creates read-optimized data structures for the auction system
  176. 176. On search results page, execute auction to determine sponsored impressions Sponsored Jobs at Indeed
  177. 177. Sponsored Jobs at Indeed When job seeker clicks on sponsored result, log information from the auction employerId jobId bid cost …
  178. 178. Sponsored Jobs at Indeed Process click logs to update budgets and charge employers
  179. 179. Sponsored Jobs at Indeed Process click logs to update budgets and charge employers Apply business rules during click processing: ● Fraud detection ● Duplicate click detection
  180. 180. SJ outside the US Non-US employers wanted their jobs in sponsored results...
  181. 181. SJ outside the US Non-US employers wanted their jobs in sponsored results... ...but they don’t have US Dollars
  182. 182. SJ outside the US v1: Use credit cards Credit card company convert charges to employer’s currency
  183. 183. SJ outside the US Credit Cards + No changes needed
  184. 184. SJ outside the US Credit Cards + No changes needed - Bad UX for employers
  185. 185. SJ outside the US Credit Cards + No changes needed - Bad UX for employers - Disadvantaged exchange rates
  186. 186. SJ outside the US Credit Cards + No changes needed - Bad UX for employers - Disadvantaged exchange rates - Employers bear currency risk
  187. 187. Credit Cards: Currency Risk Desired Daily Budget: CA $100.00 Exchange rate on Jan 1: 0.9351 Set Daily Budget to: $93.51
  188. 188. Credit Cards: Currency Risk Desired Daily Budget: CA $100.00 Exchange rate on Jan 1: 0.9351 Set Daily Budget to: $93.51 Exchange rate on Jan 31: 0.8970 Effective Daily Budget: CA $104.25
  189. 189. Credit Cards: Currency Risk +4.25% Desired Daily Budget: CA $100.00 Exchange rate on Jan 1: 0.9351 Set Daily Budget to: $93.51 Exchange rate on Jan 31: 0.8970 Effective Daily Budget: CA $104.25
  190. 190. Multi-currency Sponsored Jobs Auction
  191. 191. Multi-currency SJ Employers can set bids and budgets in preferred currency Canadian Dollars CAD Australian Dollars AUD Japanese Yen JPY Euro EUR British Pounds GBP Swiss Francs CHF
  192. 192. Multi-currency SJ Single auction for all employers using any currency
  193. 193. Multi-currency SJ Fair exchange rates for employers
  194. 194. Multi-currency SJ Transparent and repeatable calculations
  195. 195. Multi-currency SJ Create a new “pseudo-currency” for use within the auction: millicent
  196. 196. Millicents Exchange rate between USD and millicents is fixed: $0.01 == 1000 millicents $1.00 == 105 millicents
  197. 197. Millicents Exchange rates between other currencies and millicents can vary over time: €1.00 == 136,170 millicents ¥100 == 98,350 millicents
  198. 198. Millicents Provide enough granularity to differentiate similar values in different currencies
  199. 199. Millicents Provide enough granularity to differentiate similar values in different currencies All of these are about $1.00 (USD): £0.60 (GBP) €0.73 (EUR) ¥102 (JPY)
  200. 200. Millicents Provide enough granularity to differentiate similar values in different currencies All of these are about $1.00 (USD): £0.60 (GBP) €0.73 (EUR) Which is larger? ¥102 (JPY)
  201. 201. Millicents Converting to USD doesn’t help USD: $1.00 → $1.00 GBP: £0.60 → $1.00 EUR: €0.73 → $1.00 JPY: ¥102 → $1.00
  202. 202. Millicents Millicents provide granularity to rank values USD: $1.00 → 100000 mc GBP: £0.60 → 100450 mc EUR: €0.73 → 99519 mc JPY: ¥102 → 100317 mc
  203. 203. Millicents 32 bit signed values $21,474 USD equivalent 64 bit signed values $9.2 trillion USD equivalent
  204. 204. Local Currency Values Values in specific currency are represented with currency code and an integer Integer represents “minor unit”, depends on the currency type: (USD, 543) == $5.43 (EUR, 543) == €5.43 (JPY, 543) == ¥543
  205. 205. Local Currency Values For each currency, preferable that the “minor unit” is roughly equal to $0.01 USD ● Exchange rate representation ● Fairness in auction competition
  206. 206. Local Currency Values 32 bit signed values $21 million USD (and others) ¥2.1 billion JPY 64 bit signed values $90 quadrillion USD (and others) ¥9 quintillion JPY
  207. 207. Multi-currency SJ Change bid and budget representations to use [currency, integer]
  208. 208. Multi-currency SJ Create process to retrieve and record exchange rates every day
  209. 209. Multi-currency SJ Auction builder process converts bids to millicents, saves the exchange rate used
  210. 210. Multi-currency SJ Execute auction in millicents
  211. 211. Multi-currency SJ Record results in millicents & local currency
  212. 212. Multi-currency SJ Add multi-currency data to click logs: employerId jobId bid cost ... employerId jobId currency exchangeRate bidInCurrency bidMillicents costMillicents ...
  213. 213. Multi-currency SJ During click processing, convert auction cost (in millicents) back to employer’s currency using same exchange rate costInMillicents currency exchangeRate → costInCurrency
  214. 214. “How much revenue did we make today?” $1,000
  215. 215. “How much revenue did we make today?” $1,000 $548 USD €273 EUR ¥8,253 JPY
  216. 216. “How much revenue did we make today?” $1,000 $548 USD €273 EUR ¥8,253 JPY 100,000,000 mc
  217. 217. Revenue Reporting If the auction millicent cost is used, there could be errors!
  218. 218. Revenue Reporting If the auction millicent cost is used, there could be errors! Millicent Cost: 53,826 millicents Euro Cost: €0.39483
  219. 219. Revenue Reporting If the auction millicent cost is used, there could be errors! Millicent Cost: 53,826 millicents Euro Cost: €0.39483
  220. 220. Revenue Reporting If the auction millicent cost is used, there could be errors! Millicent Cost: 53,826 millicents Euro Cost: €0.39 Actual Millicent Cost: 53,168 millicents
  221. 221. Revenue Reporting If the auction millicent cost is used, there could be errors! Millicent Cost: 53,826 millicents Euro Cost: €0.39 Actual Millicent Cost: 53,168 millicents 1.2% difference!
  222. 222. Active Non-US Employers Great Britain Japan Canada
  223. 223. 30% of Sponsored clicks non-USD
  224. 224. 2004
  225. 225. 2014
  226. 226. International Success United Kingdom 1.) Indeed 2.) Reed 3.) Totaljobs France 1.) Indeed 2.) Cadremploi 3.) Monster Netherlands 1.) Indeed 2.) NVB 3.) Monsterboard Canda 1.) Indeed 2.) Workopolis 3.) Monster Italy 1.) Indeed 2.) Infojobs 3.) Jobrapido Brazil 1.) Indeed 2.) Catho 3.) Infojobs Japan 1.) Rikunabi 2.) Indeed 3.) Rikunabi Next Australia 1.) Seek 2.) Indeed 3.) Careerone India 1.) Naukri 2.) Timesjobs 3.) Indeed
  227. 227. Next @IndeedEng Talk August 27th, 2014 http://engineering.indeed.com/talks https://twitter.com/IndeedEng

×