Slides from a presentation on Ubiquity at the workshop on Information Access in a Multilingual World at ACM SIGIR 2009 in Boston. Full paper available at http://mitcho.com/academic/ .

1. ubiquity:
designing a multilingual
natural language interface
mitcho
Michael Yoshitaka Erlewine
2009 SIGIR Workshop on Information
Access in a Multilingual World
Boston, July 23rd, 2009

2. ubiquity:
designing a multilingual natural language interface
• focus on multilingual delivery and interfaces
• reconsidering the text-based interface
• localizing ubiquity

3. ubiquity:
designing a multilingual natural language interface
• focus on multilingual delivery and interfaces
• reconsidering the text-based interface
• localizing ubiquity

4. USER INFORMATION
INTERFACE

5. USER INFORMATION
English
Deutsch
Español
Français
INTERFACE
Português
Русский
Italiano
Polski
Türkçe

6. USER INFORMATION
INTERFACE

7. USER INFORMATION
English
Deutsch
Español
Français
INTERFACE
Português
Русский
Italiano
Polski
Türkçe

8. USER INFORMATION
English
Deutsch
Español
Français
INTERFACE
Português
Русский
Italiano
Polski
Türkçe
this ta
lk

9. EQUIPOTENT MULTILINGUAL INTERFACES
• not just translating content or localizing interfaces
• ensuring equal access to information
• ensuring equal generativity and potential of the
platform

10. EQUIPOTENT MULTILINGUAL INTERFACES
• not just translating content or localizing interfaces
• ensuring equal access to information
• ensuring equal generativity and potential of the
platform

11. EQUIPOTENT MULTILINGUAL INTERFACES
• not just translating content or localizing interfaces
• ensuring equal access to information
• ensuring equal generativity and potential of the
platform

12. “We create communities of people
involved in making the Internet
experience better for all of us... The
Internet is a global public resource that
must remain open and accessible.”
—Mozilla Manifesto

14. Afrikaans English (US) Gaeilge Português (Europeu)
Shqip Esperanto Italiano ਪ"ਜਾਬੀ
‫ﻋﺮﺑﻲ‬ Eesti keel română
অসমীয়া suomi Русский
Kannada
Euskara Français српски
Беларуская Frysk සිංහල
Kurdî
বাংলা (ভারত) Galego slovenčina
Latviešu
Български ქართული slovensko
lietuvių kalba
català Deutsch Español (de Argentina)
Македонски
( ) Ελληνικά Español (de España)
मराठी
ગ"જરાતી Svenska
( )
Norsk bokmål Telugu
Hrvatski ‫עברית‬ Norsk nynorsk ไทย
Čeština !ह#दी (भारत) occitan (lengadocian) Українська
Dansk Magyar ‫ﻓﺎﺭﺳﯽ‬ Tiếng Việt
Nederlands íslenska Polski Cymraeg
English (British) Bahasa Indonesia Português (do Brasil)

15. ubiquity:
designing a multilingual natural language interface
• focus on multilingual delivery and interfaces
• reconsidering the text-based interface
• localizing ubiquity

16. BUILDING A SCALABLE INTERFACE PARADIGM

17. BUILDING A SCALABLE INTERFACE PARADIGM
The web is offering more and more
functionality and API’s...

18. BUILDING A SCALABLE INTERFACE PARADIGM
The web is offering more and more
functionality and API’s...
...but how do you use them?

19. BUILDING A SCALABLE INTERFACE PARADIGM
The web is offering more and more
functionality and API’s...
...but how do you use them?
• Following the GUI paradigm...

20. BUILDING A SCALABLE INTERFACE PARADIGM
The web is offering more and more
functionality and API’s...
...but how do you use them?
• Following the GUI paradigm...
• add more buttons?

21. BUILDING A SCALABLE INTERFACE PARADIGM
The web is offering more and more
functionality and API’s...
...but how do you use them?
• Following the GUI paradigm...
• add more buttons?
• add more menus?

22. CC-BY-SA: flickr.com/photos/daveduarte/2678853923/

23. ACM interactions 2008

24. CC-BY: flickr.com/photos/striatic/1629254/

26. ?

28. Aza Raskin on
text-based interfaces
“...maybe this confusion isn’t the fault of
command-line interfaces in general; maybe
it’s just the command lines we’re used to. If
commands were memorable, and their syntax
forgiving, perhaps we wouldn’t be so scared
to reconsider these interface paradigms.”
CC-BY-SA: flickr.com/photos/
seanosh/3307290530/

29. Aza Raskin on
text-based interfaces
“...maybe this confusion isn’t the fault of
command-line interfaces in general; maybe
it’s just the command lines we’re used to. If
commands were memorable, and their syntax
forgiving, perhaps we wouldn’t be so scared
to reconsider these interface paradigms.”
CC-BY-SA: flickr.com/photos/
seanosh/3307290530/

30. a natural language
interface

31. ubiquity
ubiquity.mozilla.com

32. [demo 1]
inserting a map into email
(video not included in PDF)

33. without
ubiquity

34. without
ubiquity

35. without
ubiquity

36. without
ubiquity

37. [demo 1]
inserting a map into email:
one step with ubiquity
(video not included in PDF)

38. [demo 2]
translating and emailing web content
(video not included in PDF)

39. without
ubiquity

40. without
ubiquity

41. without
ubiquity

42. without
ubiquity

43. without
ubiquity

44. [demo 2]
translating and emailing web content:
fast and straightforward with ubiquity
(video not included in PDF)

45. goal:
how?

46. goal:
accomplish more
on the web
how?

47. goal:
accomplish more
on the web
how?
connect the web
with language

49. verbs*
*JavaScript

50. ubiquity:
designing a multilingual natural language interface
• focus on multilingual delivery and interfaces
• reconsidering the text-based interface
• localizing ubiquity

51. LOCALIZING UBIQUITY
• The goal: “natural syntax”
• Users must expect that not all
natural inputs are
understandable.
• But the parser must not add
restrictions which go against the
user’s natural syntax.

52. LOCALIZING UBIQUITY
• The goal: “natural syntax”
• Users must expect that not all
natural inputs are
understandable.
• But the parser must not add
restrictions which go against the
user’s natural syntax.

53. LOCALIZING UBIQUITY
• EX: The following must behave equivalently:
(Japanese, a scrambling language)
•
Tarō-ni ball-o nager=o
Tarō-dat ball-acc throw=imper
•

54. LOCALIZING UBIQUITY
• EX: The following must behave equivalently:
(Japanese)
• ... nager=o (imperative)
• ... nager=u (infinitive)
• ... nage=te (linking form)

55. LOCALIZING UBIQUITY
• Putting theoretical linguistic research and
insights into practice
• Writing individual language parsers is
difficult and time-intensive
• A Principles and Parameters model
• One universal parser with minimal
language settings

56. LOCALIZING UBIQUITY
• Putting theoretical linguistic research and
insights into practice
• Writing individual language parsers is
difficult and time-intensive
• A Principles and Parameters model
• One universal parser with minimal
language settings

57. LOCALIZING UBIQUITY
• Putting theoretical linguistic research and
insights into practice
• Writing individual language parsers is
difficult and time-intensive
• A Principles and Parameters model
• One universal parser with minimal
language settings

58. LOCALIZING UBIQUITY
• Putting theoretical linguistic research and
insights into practice
• Writing individual language parsers is
difficult and time-intensive
• A Principles and Parameters model
• One universal parser with minimal
language settings

59. parser (animation not included in PDF)

60. parser (animation not included in PDF)

61. parser (animation not included in PDF)

62. parser (animation not included in PDF)
N N V

63. parser (animation not included in PDF)
N N V
goal object

64. parser (animation not included in PDF)
N N V
goal object
V ( goal, object )

65. LOCALIZING UBIQUITY
Argument identification via semantic role
CmdUtils.CreateCommand({
names: ["email", "mail"],
arguments: [
{role: "object", nountype: noun_arb_text},
{role: "goal", nountype: noun_type_contact}
],
...

66. LOCALIZING UBIQUITY
Argument identification via semantic role
CmdUtils.CreateCommand({
names: ["email", "mail"],
arguments: [
{role: "object", nountype: noun_arb_text},
{role: "goal", nountype: noun_type_contact}
],
...

67. LOCALIZING UBIQUITY
Argument identification via semantic role

68. LOCALIZING UBIQUITY
Argument identification via semantic role

69. LOCALIZING UBIQUITY
Argument identification via semantic role
role English Japanese
object ø o
goal to ni, e
location at, on de

70. LOCALIZING UBIQUITY
Argument identification via semantic role
role English Japanese
object ø o
goal to ni, e
location at, on de

71. LOCALIZING UBIQUITY
Argument identification via semantic role
role English Japanese
object ø o
goal to ni, e
location at, on de

72. LOCALIZING UBIQUITY
Argument identification via semantic role
role English Japanese
object ø o
goal to ni, e
location at, on de

73. [demo 3]
ubiquity in Japanese: “hello to Spanish”
(video not included in PDF)

74. LOCALIZING UBIQUITY
• Support for various linguistic structures
• Equal support for verb-initial and
-final languages
• Smart argument-first suggestions

75. LOCALIZING UBIQUITY
Minimal language descriptions
branching: 'left',
usespaces: false,
joindelimiter: '',
anaphora: ["これ", "それ", "あれ"],
roles: [
{role: 'object', delimiter: 'を'},
{role: 'goal', delimiter: 'に'},
{role: 'source', delimiter: 'から'},
{role: 'position', delimiter: 'で'},
{role: 'position', delimiter: 'に'},
{role: 'instrument', delimiter: 'で'},
{role: 'alias', delimiter: 'として'},
]

76. LOCALIZING UBIQUITY
• An equipotent multilingual platform par excellence
• Equivalent underlying parse structures
• Commands via semantic role:
write once, use in any language
• Easy to add new languages

77. LOCALIZING UBIQUITY
• An equipotent multilingual platform par excellence
• Equivalent underlying parse structures
• Commands via semantic role:
write once, use in any language
• Easy to add new languages

78. LOCALIZING UBIQUITY
• An equipotent multilingual platform par excellence
• Equivalent underlying parse structures
• Commands via semantic role:
write once, use in any language
• Easy to add new languages

79. ubiquity 0.5
with Parser 2

80. ubiquity 0.5
with Parser 2
Català Italiano
Dansk
Language
settings: English Português
Español Svenska
Français

81. ubiquity 0.5
with Parser 2

82. ubiquity 0.5
with Parser 2
Català
Dansk
English
Português

83. ubiquity:
designing a multilingual natural language interface
• Conclusion
• broad and equal access to information requires
equipotent multilingual interfaces
• ubiquity: an equipotent platform par excellence
• challenging internationalization possible
through application of linguistic research

84. ubiquity:
designing a multilingual natural language interface
• Conclusion
• broad and equal access to information requires
equipotent multilingual interfaces
• ubiquity: an equipotent platform par excellence
• challenging internationalization possible
through application of linguistic research

85. ubiquity:
designing a multilingual natural language interface
• Conclusion
• broad and equal access to information requires
equipotent multilingual interfaces
• ubiquity: an equipotent platform par excellence
• challenging internationalization possible
through application of linguistic research

86. Thank you!
Questions?
mitcho
Michael Yoshitaka Erlewine
by-nc-sa
mitcho.com
@mitchoyoshitaka