This document proposes a crowdsourcing-based mobile image translation and knowledge sharing service. It describes preliminary studies conducted to test the feasibility of using human translators to translate images captured by mobile devices into text that is then translated into English. The studies found that human translators were better able to understand unclear questions from images and provide additional context compared to traditional optical character recognition and machine translation. However, the studies also revealed challenges around communication between requesters and translators and quality control. The document concludes by discussing ways to improve the service, such as dynamic task allocation and incentive structures.

1. A Crowdsourcing Based Mobile
Image Translation and
Knowledge Sharing Service
Yefeng Liu, Vili Lehdonvirta1, Mieke Kleppe2, Todorka
Alexandrova, Hiroaki Kimura, Tatsuo Nakajima
Department of Computer Science
Waseda University, Tokyo, Japan
1Helsinki Institute for Information Technology
2Eindhoven University of Technology
yefeng@dcl.info.waseda.ac.jp

2. Outline
• Introduction
• Human Mobile Image Translation
• Preliminary Study
• Discussion
• Future Directions
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 2

3. Introduction
“...I can’t wear tie
here?? Should I
take off my tie?..”
A menu board outside a restaurant, Tokyo
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 3

4. Real World Problem
• Digital pocket translators or online
translation services are useless if you
donʼt know how to input the characters.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 4

5. (Typical) Mobile Image Translation
Image OCR MT English
Text Optical Character Recognition Machine Translation Text
Poor
Irregular fonts or formats, handwriting, etc. performance
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 5

6. Our Solution:
Human Mobile Image Translation
Image Text
Translator English
Text
Question of Outsourcing Community
the image
Crowdsourcing
• Better quality in text recognition and translation
• Human worker can provide richer interpretations and responses in
addition to literal answers.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 6

7. Image Based Translator + Mobile Q&A
• NOT only a translator
• But also a knowledge broker that allows users to share high level
information pertinent to the situation at hand, e.g.
Q: “what’s the
• advice difference between 1
and 2 in my electricity
• explanations bill?”
• instructions
A: “1is basic charge,
• suggestions 2 is additional fee ”
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 7

8. Basic work-flow overview
English
Kanji
Open call Scoring
etc.
Requester Best
Translators Requester
answer
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 8

9. Comparison
Expert/Social
Translation Mobility Quality
Search
(OCR based)
Mobile Image Yes No Yes So-so
Translation
Human-based
Search Yes & No Yes No Good
Proposed
Solution Yes Yes Yes Good
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 9

10. Preliminary study
• A preliminary study and design research aims to
• verify the feasibility of the design
• identify real user requirements and design
issues
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 10

11. Preliminary study - Method
Collected pictures/questions from potential users
Fifteen characteristic cases were selected from the collected images
Interviewed the requesters what kind of answers they were expecting
Assigned questions to invited translators
Interviewed translators for their feedbacks
Compared the results with the requesters’ expectations
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 11

12. Preliminary Study Cases - Example
“...how long do I have to wait?”
Information in the picture is insufficient for collecting their feedback.
we interviewed the translator
to answer this question.
However, most of the repliers can still
suggest an approximate waiting time
according to their life experiences.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 12

13. Preliminary Study Cases - Example (2)
“What are the events between 5th
and 8th?”
we interviewed the translator for collecting their feedback.
Poor question text.
Some translators misunderstood the
question, thus provided useless answers.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 13

14. Preliminary Study - Implication
1. Communication between requester and worker.
Better communication Better understanding Better result
we interviewed the translator for collecting their feedback.
2. Question/Answer style
• Short, but clear (e.g clarify to what level of details is wanted);
• Question with choices is better;
• Asking for links (of image/web page/etc) is a good way to lower the
difficulty and faster the response time.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 14

15. Discussion (1)
1. Quality of outcome
Misunderstanding between requester and
worker strongly affects quality of outcome.
- Workers often are not native English speakers.
- Requesters may use unclear or too complicated English.
- People always make mistakes.
- Malicious replies.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 15

16. Discussion (2)
Kanji English
Scoring
open
call
Best
Requester Translators Proofreaders Requester answer
An additional proofreading phase.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 16

17. Discussion (3)
2. Different user types (user requirements)
Client Users
Short-term stay Long-term stay
Need immediate Waitable Need immediate Waitable
answer answer
A B C D
may have different preference on the accuracy vs. timeliness trade-off
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 17

18. Future Directions (1)
1. Dynamical task allocation with real time requirement
• Task is better be assigned to worker who is:
i. capable for the task
ii. available for the task
Not only about if the worker is free, but also involves
other factors like expertise, properties of question, etc.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 18

19. Future Directions (2)
2. Motivation and Incentive
Social and Intrinsic incentive: game play
A location-based mobile game is designed
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 19

20. Conclusion
• Conclusion
• Human-based mobile image translation system
• Preliminary study
• Findings and future directions
• Current Status
Preliminary Prototype Early Usability/On
Design Redesign Redesign
Study Implementation Test field study
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 20

21. Thank you for your attention!
Yefeng Liu, PhD candidate
yefeng@dcl.info.waseda.ac.jp
Distributed & Ubiquitous Computing Lab.
Depart. of Computer Science, Waseda University
http://www.dcl.info.waseda.ac.jp/

22. “keywords” style answer is preferred
a). “Pork, spicy, famous chinese food”
we interviewed the translator for collecting their feedback.
b). “Twice cooked pork (huiguo rou)”
- meaningless if don’t know the name
- Many translators use English as 2nd or 3rd language, they often
face the problem of being unable to explain in long sentence.
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 22

23. we interviewed the translator for collecting their feedback.
“what’re these two? can you provide links of
“is it a show or training course?” pics”
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 23

24. we interviewed the translator for collecting their feedback.
I wanna buy the ticket for swim! what divination result I got here?
A Crowdsourcing Based Mobile Image Translation & Knowledge Sharing Service, MUM 2010, Limassol, Cyprus 24