Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

What is necessary for the next input method framework?

Input method framework is a software that has been used to input complex characters (e.g., Chinese characters, Hiragana, Hangul). Strictly speaking, the role of input method framework is bridging desktop applications and input method engines, which translate typed keys into complex characters. During this two decades, several input method frameworks including Kinput2, SCIM, UIM, Gcin, Fcitx, and IBus have been developed.

The situation surrounding text input method is changing. Firstly, new approaches such as software keyboard for touch screen and speech to text input are available on platforms other than Linux desktop.

Another change is the integration of input method into desktop application platforms. For example, IBus is now a part of GNOME desktop environment; Not only GNOME, Qt also include IBus support. Furthermore, Flatpak also uses a subset of IBus D-Bus interface for applications in Flatpak sandbox to communicate with an input method running on its host desktop.

Some people might think IBus is the defacto standard input method framework. However, quite many people prefer Fcitx or else due to the design issues of IBus. The latter people might think they are losing their freedom to select input method framework.

Now the speaker thinks that it is time to discuss the variety of input method frameworks is really necessary for the future Linux desktop environment. An input method framework itself does not provide much experience for users because its primary role is usually invisible to users. Thereby, how about bringing the war of input method framework and using our effort to improve input method engine, implementing the new approaches mentioned above, and supporting newer application platforms like Wayland?

In this talk, we would like to discuss what is necessary for the future input method framework by reviewing the design issues of IBus. The topics will be as the followings:

- Thin and high-level protocol with libraries avoiding code duplication
- The separation of responsibility between IBus daemon and plugins
- Importance of opened community

Note that the speaker is neither a developer of IBus nor GNOME. The attendees from GNOME community are welcome to improve this discussion.

What is necessary for the next input method framework?

  1. 1. 2018/08/11 1/23 What is necessary for the next input method framework Fuminobu TAKEYAMA (ftake) openSUSE.Asia Summit 2018, Taipei
  2. 2. 2018/08/11 2/23 About me ● TAKEYAMA, Fuminobu (武山 文信) – openSUSE ID: ftake – from Japan ● Community (weekend) developer – Maintainer of openSUSE M17N (Multilingualization) ● ● – Not a developer of any specific input method framework ● Communities – Organizer of Japan openSUSE User Group – openSUSE.Asia Summit 2017 Local Organizing Chair
  3. 3. 2018/08/11 3/23 What is an input method? ● A mechanism to input various characters – Complex characters used in Chinese, Japanese, Korean – Usually using several key strokes aoi あおい 青い
  4. 4. 2018/08/11 4/23 What is an input method framework (IMF) ? ● An interface between applications and input method engines (conversion software) – Also allows switching multiple input method engines – e.g., IBus, Fcitx, GCIN, SCIM, UIM, IIIMF, Kinput2, ... GTK App. Qt App. X App. IMF daemon *-qt *-xim*-gtk IMFGUI QtGtk X server Mozc libchewing3 *-mozc *-chewing Engines Different API for each GUI platform GTK, Qt IM module, XIM An unified API to implement engines
  5. 5. 2018/08/11 5/23 Input method frameworks on openSUSE ● openSUSE supports multiple IMFs ● Different IMF is used depending on the session locale by default – GCIN for zh_TW, zh_HK – Fcitx for zh_ CN, etc. – IBus for ja_JP
  6. 6. 2018/08/11 6/23 Recent changes in the API layer ● Qt5 contains ibus-qt (GTK+ 4 will also) ● Flatpak applications can communicate with IBus running on the host – using the IBus portal D-Bus interface GTK App. Qt App. X App. ibus-daemon ibus-qt ibus-ximibus-gtk QtGtk X server Mozc libchewing3 ibus-mozc ibus-chewing IBus D-Bus Interface Flatpak App. Flatpak portal Engines ... ... ibus-ui-gtk3ibus-ui-gtk3
  7. 7. 2018/08/11 7/23 But, Flatpak supports only IBus ● Issue #43 Add other im module for input method – 3 I'm still not convinced we want to support multiple input method frameworks By Matthias Clasen
  8. 8. 2018/08/11 8/23 Some people might think IBus is now the standard IMF? ● No – Quite many people prefer Fcitx or Gcin and hate IBus – By Googling “ibus”, several blog posts saying “how ibus is bad” are found – Due to the several design issues (mentioned later) ● Why so many IMFs have been developed? – Formerly, for more stability ● Let’s remind conversion engines were running on an application process – Now, usability, customizability, ...
  9. 9. 2018/08/11 9/23 Do we still need various IMFs? ● I think no if we have a thin and simple IMF – No major difference in the core part of today’s IMFs – “An interface between applications and engines” does not provide much user experience – GUI should be separated into engines and replaceable ● No more multiple IMF support cost ● It’s time to end the war of input method framework
  10. 10. 2018/08/11 10/23 So let’s discuss what is necessary for the next IMF? ● From the viewpoint of architecture/design ● By reviewing the problems of IBus (and Fcitx)
  11. 11. 2018/08/11 11/23 1. Separation of GUI from core IMF service 1/2 ● GUI of input method is now provided by IMF – Design of the GUI is a major cause of discontent ● A candidate window should be a part of an engine – e.g., IBus cannot show hints (word usage) on a candidate list ● Mozc replaces candidate window with its own in irregular way but it is broken on GNOME Wayland shell Candidate window of ibus-ui-gtk3 (with ibus-kkc) Candidate window of Mozc
  12. 12. 2018/08/11 12/23 1. Separation of GUI from core IMF service 2/2 ● For IBus, some major logic of IMF is implemented in GUI layer (status panel applications) – Switching keyboard layout, loading plugins, etc. ● What is worse, there are three GUI layer variations – ibus-ui-gtk3 – GNOME keyboard plugin – KDE Input method panel
  13. 13. 2018/08/11 13/23 2. Running without specific GUI libraries ● IBus depends on GTK+ – KDE input method panel still requires ibus-ui-gtk3 – Fcitx have less dependency ● Will be necessary for embedded – e.g., Automotive Grade Linux
  14. 14. 2018/08/11 14/23 3. Wayland support ● Need APIs between IM engine and Wayland compositor (server) – To get absolute position of text cursor – To place IM engine’s window to appropriate position
  15. 15. 2018/08/11 15/23 4. Redesigned keyboard layout switch 1/4 ● Both IBus and Fcitx support this for layouts such as Russian and Arabic – ASDF ⇔ ФЫВА By Nkoke, No modification, CC-BY-SA 3.0 A S D F
  16. 16. 2018/08/11 16/23 4. Redesigned keyboard layout switch 2/4 ● A layout is provided as an IBus engine – ibus-simple – Some engine like Anthy allow to specify a layout too
  17. 17. 2018/08/11 17/23 4. Redesigned keyboard layout switch 3/4 ● The following situations are not supported – To switch between JP and US layout with engines like Anthy, Mozc... – To use two keyboards with different layout (US, JP) Internal JP layout USB US layout
  18. 18. 2018/08/11 18/23 4. Redesigned keyboard layout switch 4/4 ● Switching pairs of an engine and a layout is necessary ● Anthy JP ● Anthy US ● Chewing US ● Simple RU – The same engine can be registered multiple times ● Hardware-specific layout settings – Internal Keyboard: JP – USB Keyboard: US
  19. 19. 2018/08/11 19/23 5. Support of text input without physical keyboard ● Integration with input and conversion is necessary – Should be implemented as an “engine” (plugin) ● Screen keyboard – Integrated conversion UI – Need API to show up ● Voice input – Converted texts are directly passed to applications ATOK input method on Android
  20. 20. 2018/08/11 20/23 6. Input mode v.s. input engine 1/2 ● IBus 1.5 is designed for CJK users to use two engines by switching Meta+Space – Conversion engine: Anthy, Mozc, Chewing, Pinyin – ibus-simple: for direct input (a-z, 0-9, ...) ● Problems – 半角/全角 key is not used to turn on/off ● Even though most Japanese use ● IBus need modifier keys to show its engine selector window while the keys are pressed – No shortcut to switch to direct input mode Turning of/off IBus was obsoleted
  21. 21. 2018/08/11 21/23 6. Input mode v.s. input engine 2/2 ● Workaround in IBus 1.5 for CJK – Use local mode of each engine and do not use ibus-simple (shortcut keys are provided by engines) ● “A” mode and “あ” mode in Anthy, Mozc ● “英” mode and “中” mode in Chewing ● What should this be? – Exposing each local mode as an engine like macOS X (Not sure this is the best) ● Direct (Simple) RU ● “A” (Mozc) US ● “あ” (Mozc) US ● “中” (Chewing) US Meta+Space Settings for Russian, Japanese and Traditional Chinese
  22. 22. 2018/08/11 22/23 7. Open community ● It is getting difficult to make a good solution without open discussion – An input method framework is involved with many parts of applications and platform ● Not only GNOME – If the design changes in IBus 1.5 was discussed openly, we could have resolved some problems before the release
  23. 23. 2018/08/11 23/23 Conclusion ● I think it is possible to make a standard IMF – By separating into thin core service and various engine plugins ● Let’s use our effort not for multiple IMF support but for – New input mechanism such as screen keyboard and voice input – Wayland integration – Resolving application specific issues ● Cursor position problem ● Electron applications bug in Flatpak