Successfully reported this slideshow.

Easy contributable internationalization process with Sphinx @ PyCon APAC 2016

2

Share

•
Aug. 14, 2016
• 2 likes • 2,854 views
Upcoming SlideShare
ドキュメントを作りたくなってしまう魔法のツール「Sphinx」
ドキュメントを作りたくなってしまう魔法のツール「Sphinx」
Loading in …3
×
1 of 66
1 of 66

Easy contributable internationalization process with Sphinx @ PyCon APAC 2016

Aug. 14, 2016
• 2 likes • 2,854 views

2

Share

Download to read offline

Description

Sphinx can extract paragraphs from sphinx document and store them into gettext format translation catalog files.

Gettext format translation catalog is easy to translate from one language to other languages.
Also Sphinx support internationalization by using such catalog files.
You can use your favorite editors or services to translate your sphinx docs.

In this slide, I'll explain 3 things; (1) entire process to translate sphinx docs. (2) automation mechanism for the process. (3) tips for writing docs and translating.

Transcript

  1. 1. Takayuki Shimizukawa Sphinx co-maintainer Sphinx-users.jp 1
  2. 2. 안녕하세요. annyeonghaseyo. 나는시미즈카와 입니다. naneun shimizukawa imnida. 2
  3. 3. こんにちは Kon-Nichi-Wa 3
  4. 4. Who am I @shimizukawa  Python developer since 2004  Sphinx co-maintainer  Sphinx-users.jp organizer  PyCon JP committee  BePROUD co, ltd. member 4
  5. 5. Agenda 1. Sphinx introduction & Setup for i18n 2. A easy way and Non easy ways to translate 3. Sphinx i18n feature 4. Automated translation process with several services 5. Tips 6
  6. 6. 7
  7. 7. Question1 How many people do you use Open Source Software? 8
  8. 8. Question2 How many people may have contributed to the OSS? 9
  9. 9. Question3 Does the translation into familiar language contribute to other developers? 10
  10. 10. 11 Translator Product Author Translated Docs Readers English Docs Translators
  11. 11. Tools & Services Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 12
  12. 12. 13
  13. 13. What is Sphinx? Sphinx is a documentation generator Sphinx generates doc as several output formats from the reST text markup 14 1. Sphinx introduction & Setup for i18n Sphinx reSTreSTreStructuredText (reST) reST Parser HTML Builder ePub Builder LaTeX Builder texlive HTML theme Favorite Editor
  14. 14. The history of Sphinx (short ver.) 15 1. Sphinx introduction & Setup for i18n The father of Sphinx Too hard to maintenance ~2007 Easy to write Easy to maintenance 2007~
  15. 15. Sphinx Before and After Before  There was no standard ways to write documents  Sometime, we need converting markups into other formats After  Generate multiple output format from single source  Integrated html themes make read docs easier  API references can be integrated into narrative docs  Automated doc building and hosting by ReadTheDocs 16 1. Sphinx introduction & Setup for i18n
  16. 16. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 17 1. Sphinx introduction & Setup for i18n
  17. 17. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 18 1. Sphinx introduction & Setup for i18n
  18. 18. SPHINX docutils HTML Builder HTML theme (Jinja2) gettext Builder *.pot *.po I18N *.mo OmegaT Pootle Transifex Translation Tools, Services Favorite Editor Sphinx i18n feature (built-in) 19 1. Sphinx introduction & Setup for i18n reST Parser (directive / role) doctree (Intermediate) reSTreST reStructuredText (reST)
  19. 19. $ pip install sphinx Support tools for translation How to install Sphinx with i18n tools To build a sphinx document $ pip install sphinx-intl $ pip install transifex-client 20 1. Sphinx introduction & Setup for i18n
  20. 20. $ git clone https://github.com/shimizukawa/deepthought.git $ cd deepthought/doc $ make html ... Build finished. The HTML pages are in _build/html. "make html" command generates html files into _build/html. make html once 21 1. Sphinx introduction & Setup for i18n
  21. 21. Files & setup conf.py $ tree /path/to/deepthought +- deep_thought | +- __init__.py | +- api.py | +- calc.py | +- utils.py +- doc | +- _build/ | | +- html/ | +- _static/ | +- _template/ | +- conf.py | +- index.py | +- make.bat | +- Makefile +- setup.py 22 1. Sphinx introduction & Setup for i18n Document source Document build output Target library for doc1. ... 2. 3. language = 'ko' 4. locale_dirs = ['locale'] 5. doc/conf.py
  22. 22. 23
  23. 23. Easy contributable internationalization process with Sphinx 24
  24. 24. What is internationalization? 25 2. Easy contributable internationalization process with Sphinx I18N
  25. 25. Easy contributable internationalization process with Sphinx 26
  26. 26. What is easy contributable? 27 2. Easy contributable internationalization process with Sphinx
  27. 27. Not a easy way (example 1/3) The manual are provided only in the HTML files You can rewrite HTML files How to follow the original? 28 2. Easy contributable internationalization process with Sphinx
  28. 28. Not a easy way (example 2/3) The HTML manual are generated from reST files and docstrings in the source files  You can rewrite reST files and docstrings  How to follow the upstream? 29 2. Easy contributable internationalization process with Sphinx
  29. 29. Not a easy way (example 3/3) OK, we are engineers. we can use GIT!  Learn git  Learn GitHub  git clone to get the code  Translation (Yes! This is what I want to do!)  git commit  git pull  Sometimes you must resolve conflict  git push 30 2. Easy contributable internationalization process with Sphinx
  30. 30. Not easy vs easy Not a easy way Easy way 1. Learn git 2. Learn GitHub 3. git clone to get the code 4. Translation 5. git commit, pull, push 6. Resolve conflict 7. Build html your self 1. No git 2. No Github 3. No file 4. Translation 5. Update Automatically 6. No conflict 7. You can get a HTML output w/o hand-build. 31 2. Easy contributable internationalization process with Sphinx
  31. 31. 32
  32. 32. Two i18n features of Sphinx  Output pot files: from reST  Input po files: to generate translated HTML 33 3. Sphinx i18n feature reST pot reST html po
  33. 33. Translation flow  Generate pot  Translate pot into po  Generate Translated HTML 34 3. Sphinx i18n feature reST pot reST html po pot po Translator Please translate Translate! Thanks for your Contribution!
  34. 34. #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" msgid "For example:" msgstr "" 35 Output pot files 3. Sphinx i18n feature $ make gettext ... Build finished. The message catalogs are in _build/gettext. $ ls _build/gettext api.pot examples.pot generated.pot index.pot generated.pot reST pot
  35. 35. Preparing po files to translate doc +- _build/ | +- gettext/ | +- api.pot | +- examples.pot | +- generated.pot | +- index.pot +- locale/ doc +- _build/ +- locale/ +- ko/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/ Translatethem Translator pot po 36 3. Sphinx i18n feature
  36. 36. 37 Preparing po files to translate $ sphinx-intl update -p _build/gettext -l ko Create: locale/ko/LC_MESSAGES/api.po Create: locale/ko/LC_MESSAGES/examples.po Create: locale/ko/LC_MESSAGES/generated.po Create: locale/ko/LC_MESSAGES/index.po At first, sphinx-intl copy pot files and rename them pot po sphinx-intl $ make gettext $ sphinx-intl update -p _build/gettext -l ko Not Changed: locale/ko/LC_MESSAGES/api.po Updated: locale/ko/LC_MESSAGES/examples.po +3, -1 After change the document, sphinx-intl update differences 3. Sphinx i18n feature
  37. 37. Translate po files #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" generated.po pot po sphinx-intl Translator #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "JSON 형식의 :class:`str` 유형에 ``obj`` 를 직렬화." generated.po Translate by using Vim, Emacs, OmegaT, ... 38 3. Sphinx i18n feature
  38. 38. Input po files 39 3. Sphinx i18n feature reST html po Translated $ make html ... Build finished. The HTML pages are in _build/html.
  39. 39. Big picture 40 3. Sphinx i18n feature reST pot html po make gettext sphinx-intl Translator make html Doc Author Translation Catalog Translated catalog
  40. 40. 41
  41. 41. Entire process to translate sphinx docs 42 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog Translator Translator Translator Autor / Translator upload Translator clone Translator
  42. 42. To Be 43 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog upload Translators To Be Automated To Be Parallel clone
  43. 43. Translation tool types  Vim / Emacs (Editors)  Edit local files  Translation support plugins are available.  OmegaT (Translation Tools)  Edit local files.  Translation support features.  Transifex (Services)  Edit online  Translation support features. 44 4. Automated translation process with several services po Translators To Be Parallel
  44. 44. Be Parallel by using Transifex Transifex provides... As API: Upload pot Download po As Web console: Glossary Translation memory Recommendation Auto-translation 45 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download
  45. 45. Translation on Transifex web console 46 4. Automated translation process with several services Original Text Translated Text (you should keep reST syntax) Suggestions from Translation Memory (TM) Original (pot) Translation (po) Copy orig to translation Machine translation Save Review (if needed) Translators Parallel 1 2 4 3 5 6
  46. 46. To Be Automated 47 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download reST html make gettext make html Doc Author upload To Be Automated clone
  47. 47. To Be Automated 48 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
  48. 48. The procedure for automation 1. clone repository 2. make gettext 3. Upload pot 4. Download po 5. make html 6. Upload html 49 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
  49. 49. 1. pip install sphinx sphinx-intl transifex-client 2. git clone https://github.com/shimizukawa/deepthought.git 3. cd deepthought/doc 4. sphinx-intl create-transifexrc # create ~/.transifexrc 5. sphinx-intl create-txconfig # create .tx/config 6. make gettext 7. sphinx-intl -p _build/gettext update-txconfig-resources # update .tx/config 8. tx push -s # push pot files to transifex 9. tx pull -l ko # pull po files from transifex 10. make html SPHINXOPTS="-D language=ko" Shell commands for automation run.sh $ export SPHINXINTL_TRANSIFEX_USERNAME=mice $ export SPHINXINTL_TRANSIFEX_PASSWORD=42 $ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7 $ export SPHINXINTL_POT_DIR=_build/gettext $ run.sh 50 4. Automated translation process with several services
  50. 50. The drone.io 51 WebHook Deploy Clone repository Run shell script The drone.io is a continuous integration service. 4. Automated translation process with several services
  51. 51. GitHub + drone.io + S3 52 GitHub Amazon S3 Transifex 1. Clone repository 2. make gettext. 3. Upload pot 4. Download po 5. make html 6. Upload html 4. Automated translation process with several services 2 1 3
  52. 52. Be Automated 53 pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated Upload pot Download po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
  53. 53. Automated by drone.io 54 Upload potDownload po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
  54. 54. View from doc author  Doc Author doesn't require annoying procedure. Update Translation Source Doc Author Notify See Commit 55 4. Automated translation process with several services
  55. 55. View from doc translators  No git  No file  No conflict  Update Automatically  They can get a translated HTML output w/o hand-build. Translators Parallel See Translate Translated Pages 56 4. Automated translation process with several services
  56. 56. The entire automated process 57 Update Translation Source Doc Author Translators Parallel Notify See See Publish Translate Commit Download Translations Notify Automated 4. Automated translation process with several services
  57. 57. Summary Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 58 4. Automated translation process with several services
  58. 58. 59
  59. 59. msgid "Sphinx translates a set of reStructuredText_ source files into various output formats." msgstr "Sphinx translates a set of “ “`reStructuredText <http://docutils.sphinx-users.jp/web/rst.html>`_ “ “source files into various output formats." *.po How to handle URLs 60 5. TIPS Sphinx translates a set of reStructuredText_ source files into various output formats. .. _reStructuredText: http://docutils.sourceforge.net/rst.html *.rst Hyperlink Target  Hyperlink Target doesn’t appear in POT files.  If you want to change URLs into translation version, you can use Embedded URLs notation.
  60. 60. How to change language w/o rewriting conf.py  You can use SPHINXOPTS make option  It also works with other targets not only ’html’.  linkcheck target is useful if your translation contains URLs. It will find typos in translation text. 61 5. TIPS $ make linkcheck SPHINXOPTS="-D language=ko" $ make html SPHINXOPTS="-D language=ko"
  61. 61. You can control pot file resolution  Use gettext_compat  By default ‘gettext_compat = True’.  reST files under sub directories are collected into single POT file. 62 5. TIPS language = 'ko' locale_dirs = ['locale'] gettext_compat = False # generate .pot for each .rst doc/conf.py
  62. 62. Translatable sphinx html templete  Sphinx HTML template is Jinja2.  You can use “{% trans %}” template tag.  “make gettext” exports “trans” contents into sphinx.pot file files. 63 5. TIPS <p> <em>{%trans%}What users say:{%endtrans%}</em> </p> <p> {%trans%}“Cheers for a great tool that actually makes programmers <b>want</b> to write documentation!“{%endtrans%} </p> _templates/index.html
  63. 63. Example projects  Sphinx-1.4 doc for "ja" translation https://drone.io/bitbucket.org/shimizukawa/sphinx-doc15/admin  Docutils doc for "ja" translation https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin 64 5. Tips
  64. 64. Questions? @shimizukawa Grab me after the session :) I’ll be in SPRINT venue too! 65
  65. 65. Thanks :) 66

Editor's Notes

  • （ここの文字サイズ大きくする、マウスをレーザーポインタにする）
    Hi everyone. Thank you for coming my session.
    //Can you hear me? OK
    //Can you see the session title?
  • あにょん　はせよ
    なぬん　しみじゅかわ　いむにだ
  • こんにちは
  • I’m Takayuki Shimizukawa came from Japan.
    This is the second time of speaker at PyCon in Korea.
  • This is a capture image from PyCon APAC site.
    （クリック）
    Today I’ll give a talk relates Languages.
    Title is: Easy contributable document internationalization process with Sphinx.
  • This is Agenda of this session.

    1. Sphinx introduction & Setup for i18n.
    2. A easy way and Non easy ways to translate 3. Sphinx i18n feature 4. Automated translation process with several services
    5. Tips
  • Before proceeding to the subject, I want to ask you.
  • How many people do you use Open Source Software?
    -> 10, 20, ... Thanks.
    A-> Obviously, most people are using OSS.
    B-> Oh, It's surprising. Half of the people don't use OSS.
  • How many people may have contributed to the OSS?
    -> 1, 2, ... Thanks,
    A-> Well,... it's a very small number.
    B-> Oh, a lot of people have contributed. It's very good news.

    IMO, using OSS is the first contribution.
    Also we can say that introducing the OSS to other people is another contribution.
    So, almost (a lot of / a half of) people who have raised your hand to the first question is already contributing to the OSS.
  • Many of the OSS provides the document in English.
    How do you think, Does the translation into familiar language contribute to other developers? Yes?

    10, 20,... Thanks.
    A-> most people think it's a contribution, I agree it completely.
    B-> Looks not too many. I guess everyone can read English, right?

    In Japan, many programmers aren't familiar to English.
    In such situations, translated documents helps me when I introduce some library to other developers.
    Because of this, I occasionally translate documents.
  • I think, by your translation of a OSS document, more people will be able to use the OSS.
    （クリック）
    If more than one person can join such translation work easily, it would be great.
    In this session, I'll introduce how to make a mechanism which is easy to join as a translator.
  • In order to make the mechanism, we'll use such tools and services in this session.
    Tools; sphinx, docutils, sphinx-intl, transifex-client
    Services; Transifex, Drone.io.

  • First subject, I'll introduce the sphinx introduction and setup Sphinx for i18n.
  • What is Sphinx?
    Sphinx is a documentation generator.
    Sphinx generates doc as several output formats from reStructuredText markup that is an extensible.
    （ポインタでinputとoutputを指す）
  • This man, Georg Brandl is the father of Sphinx.
    （クリック）
    Until 2007, python official document was written by LaTeX.
    But, it's too hard to maintenance.
    Georg was trying to change such situation.
    （クリック）
    So then, he created Sphinx in 2007.
    Sphinx provides usability and maintainability of the Python official document.
  • Sphinx before and after.
    Before
    There was no standard ways to write documents. One of example is a Python official document. It was written by LaTeX and several some python scripts jungle.
    And, Sometime, we need converting markups into other formats

    Since sphinx has been released,
    * We can generate more multiple output format from single source.
    * Integrated html themes make read docs easier.
    * API references can be integrated into narrative docs.
    * Automated doc building and hosting by ReadTheDocs service.
  • Nowadays, sphinx has been used by these libraries and tools.
    Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

    And Non python library/tools also using Sphinx for them docs: Chef, CakePHP(2.x), MathJax, Selenium, Varnish
  • Nowadays, sphinx has been used by these libraries and tools.
    Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

    And Non python library/tools also using Sphinx for them docs: Chef, CakePHP(2.x), MathJax, Selenium, Varnish
  • Sphinx extend the i18n mechanism to provide translation feature.
    In this session, let's take a look the i18n feature.
  • How to install Sphinx with i18n tools.
    You need to install sphinx to build a sphinx document.
    Also you need to install sphinx-intl and transifex-client for translation.
    You should use newest packages.
  • At first, you should get source code whether from github or other place to translate it.
    In this case, the source code already has a sphinx doc directory.

    So, type "make html" in doc directory to generate html output.
    You can see the output in _build/html directory.
  • Now you can see the directories/files structure, like this.
    （ポインタ）
    * Target library for doc that is cloned from existent repository.
    * Document build output that was generated by "make html"
    * Document source that was generated by "sphinx-quickstart"
    （クリック）
    This is a conf.py file in the doc directory.
    You should add these 2 lines to the conf.py
    * Line-3rd: set a language code you want to use for output doc.
    * Line-4th: set a directory relative path from conf.py location. The locale directory will have translation catalog files.
    Now setup is over.
  • Well, it is the main subjects from here.

    Easy contributable internationalization process with Sphinx.
  • internationalization
  • What is internationalization? It's usually called I18N that I’ve already mentioned.
    The internationalization process is sometimes called translation or localization enablement without rewriting the source code from a technical point of view.

    "without rewriting the source code" is important.
  • Easy contributable.
  • What is easy contributable?
    Some software users or document readers have wanted to contribute by translation.
    But, if the preparation of the translation takes long time or requires many kind of skill, they are frustrated and give up.
    "Easy contributable" is the STATE that will be accepted the contributions immediately.
  • So, for the comparison, I'll introduce the three cases of difficult to contribute.

    First one.
    The manual are provided only in the HTML files.
    Of course, you can translate it.
    But, usually the html files are frequently updated.
    How do you follow the original changes to update your translations?

    This is the first case. I have no idea how to follow the original changes.
  • 2nd example.
    When the HTML manual are generated from reST files and docstrings in the source files,
    You can rewrite original files to translate it.

    But this approach has three problems: No1. You must be careful to maintain reStructuredText structure. No2. It's hard to divide translation tasks for a number of volunteer translators. No3. It's hard to follow the upstream document source that is frequently updated.
  • last example.
    OK, we are engineers. We can use git!
    （上を読む）
    There are many many steps to complete our translations.
    Additionally, in this case, the repository owner should allow write access permission for each translators.

    This is also not a easy way.
  • Well then, what is a easy way?
    My opinion is this. Left side is Not a easy way.
    Right side is A easy way.
    We need a easy way.
    The combination of the Sphinx i18n feature and some of the services give you a easy way.

    That's what is the goal of this session
    OK, let's move forward.
  • 3rd Subject. Sphinx i18n feature.
    I'll explain the detail of the feature.
  • Sphinx i18n feature is composed of two functions.

    First one is the generation pot files from reST files.
    pot file is a famous translation catalog format for gettext that is used for i18n, you know.

    Second one is the generation translated html files with using po files that is translated version of catalogs.
    It's the human work to prepare the translated po files from the pot files.
  • Like this, Converting the POT files into PO files is the job of humans.

    （クリック）In this flow, Author generate pot file at first part, and then, Author ask translators to translate them.
    （クリック）Translators receive the pot files from Author in some way, then translate them.
    （クリック）Once it has been finished, translators send back the po files to Author.

    Let's look a little more detail from first part.
  • Author can use "make gettext" command to generate pot files.

    "make gettext" command extracts text from reST files and python sources that referenced by autodoc.
    It means, you can translate them into any language without rewriting the original reST files and python source files.

    As you see, pot translation catalog makes easy to translate from one language to another languages. "msgid" line have a original sentence.
    "msgstr" line will have translated one.
  • Translated PO files should have as a right side structure.
    "locale" directory has “ko" and it has "LC_MESSAGES" directory, and it has translated PO files.
    PO files are copy of POT files with changed file's extension.
    If you want to translate it into other languages, like 'ja', you can create other directories under "locale" directory.

    When original POT files are updated by changing of original document, you should update PO files too.
    If you had done it manually, it is very hard.
    You can automated it by sphinx-intl.
  • At first, sphinx-intl copy pot files and rename them.
    When the document is changed, translators can use sphinx-intl to update PO differences.
    By same command.

    As you see, translators can finish the boring tasks by one command.
    sphinx-intl copy pot files and rename them on behalf of you.
    sphinx-intl also update differences for each po files when the document is changed.

    It's easy to use.
    As a consequence of introducing the sphinx-intl tool, translators can concentrate for translation.
  • This is a translation work.

    Translators can use their favorite editors or helpful tools/services easily that provide translation support features like a translation memory, recommending similar translation, glossary and auto-translation.
    Once the translation is complete, finally let's run the "make html" command.
  • Finally author and translators can run "make html" and can get translated output w/o editing reST and python code.
    This is a translated HTML sample.
    Internally, Sphinx parse reST file and replace them into translated one by using po file.
  • Entire process to translate sphinx docs is this.
    If you translate some sphinx document without using the i18n feature, you need to rewrite original document source files.
    The Sphinx i18n feature provides a easy way to translate a document as you seen.
  • So far, we've found the mechanism of i18n of Sphinx.
    In 4th Subject, Automated translation process with several services, we will continue to automate this mechanism.
  • Right now, we have seen an overall picture of the process.
    Actually, I have a thing that was not pictured here.
    （クリック）
    Yes, these commands such as "make blah blah" requires your hands to invoke.
    In other words, the translator must learn such steps, yet.
    Additionally, it would be difficult to translate in parallel.

    It's still not a easy way.
  • OK, Let's make it to be automation and parallelization.

    And then, the translator doesn't need to know the mechanism.
    And, make the translation work possible to parallelize.

    Let's take a look at from the parallelization of the translation work.
  • Once you get po files, you can use helpful tools/services easily that provide translation support features like a translation memory, recommending similar translation, glossary and auto-translation.

    In particular, the use of the online translation service is important for parallelization.
    I'd like to introduce the Transifex as a online translation service.
  • Transifex provides the following functions: Upload pot and download po files by API, glossary feature, translation memory, recommending translations, automatic translation by using Google / Microsoft API.
    Because you can do translation on the Web screen, you can translate them with other translators in parallel.
  • Translation on Transifex web console.
    1. You can select one untranslated message
    2. And Translate it by your self, or by machine translation
    3. If the message contains long module or method name, you can click the icon to copy original to translation box w/o typing.
    4. Sometimes, you are suggested translation messages from other translator, or Translation Memory. Translation Memory (called TM) pick up suggestions from similar translation in the project or its historical changes. For example, When Doc Author fix a typo, translation will be discarded. However, TM suggests previous translation that matches 99% with current original message.
    5. Once you finish the translation, you can save it.
    6. Optionally, reviewer can review it.
  • We have made translation parallelization.
    Next.
    Let's automate the central part.
  • The procedure you want to automate is made up of 6 steps.
    At first, we will review the 6 steps.
  • First step is, Clone repository to be translated.
    Step 2. make gettext to generate translation catalog.
    Step 3. Upload pot translation catalog source.
    Step 4. Download translated po files.
    Step 5. "make html" command to generate translated document.
    Step 6. Upload html to publish it

    That's all.
    Next, we will replace these all steps on the command line.
  • This is the command line that include the steps.

    Line1: install sphinx and relates tools
    Line2: clone repository
    Line4: create a transifex account file by using sphinx-intl that reference environment variables (USERNAME and PASSWORD).
    Line5: create a transifex config file by using sphinx-intl that also reference a environment variable (PROJECT_NAME)
    Line6: generate pot files by make gettext
    Line7: update resources information in transifex config file that also reference a environment variable (POT_DIR)
    Line8: push pot files to transifex
    Line9: pull translated po files from transifex
    Line10: make html to generate translated html with using translated catalogs.

    Now we replaced 5 steps on the command line, except HTML uploading part.
    I'll explain the part later.

    As you seen above, You can automate the process by this script.

    Rest of work is preparing a environment to run the script automatically.
    As one of the environment, I'll introduce the drone.io service.
  • The drone.io is a continuous integration service.
    Drone.io integrates seamlessly with Github, Bitbucket and Google Code to make setup fast and simple.
    Drone.io also integrates with Amazon, Heroku, Google AppEngine and more. For custom deployments you can use SSH and shell scripts.
    You can use the service for public project without charge.
  • In my case, I usually use Github and S3 integrations.
    Process of that is,
    （クリック）
    1. Push notification from GitHub
    2. That invoke the script you've seen
    3. Then, deploy html files onto S3 storage. (You need a publish html setting of S3)

    If you set WebHook notification URL to transifex hook setting, updating translations also invoke the drone.io script.
  • By using drone.io service, automation can be achieved.
    Upper-Left process is replaced with Right-Down one.
  • 1 WebHook from Github invokes the script.
    The script execute commands 2, 3, 4 and 5.
    Finally drone.io deployment feature will deploys HTML to AWS S3 storage.

    As this, we got the automated mechanism by using drone.io environment.
    Maybe you can achieve this with using Travis-CI or CirculeCI or some other CI services.
  • In summary, let's look at the results of the automation at each point of view as author and translators.

    At first, a point of view from the doc author.

    When author push to the repository, translation source of Transifex will be updated, and author can check the translated html in the site.
    Doc Author doesn't require annoying procedure.
  • The second is, a point of view from doc translators.
    They can translate the document in parallel with;
    No git, No file, No conflict.
    Translation source are updated Automatically
    They can get a translated HTML output w/o hand-build.

    It means that the translation contributors can focus on the translation.
  • By the mechanism of automation, both of doc author and translators will provide the maximum effect with a minimum of effort.
  • In order to make the mechanism, we'll use such tools and services in this session.
    Tools; sphinx, docutils, sphinx-intl, transifex-client
    Services; Transifex, Drone.io.

  • I'd like to introduce some of the tips.
  • How to handle URLs.
    By current implementation of Sphinx, hyperlink target in reST files doesn’t appear in POT files.
    It means you can’t change the URL in the hyperlink target.

    If you want to change URLs into translation version, you can use Embedded URLs notation (like above example).
  • How to change language w/o rewriting conf.py

    Because of you are not a owner of documentation, you can’t change the conf.py in the document.
    In such case, you can use SPHINXOPTS make option.
    It also works with other targets not only ’make html’.
    linkcheck target is useful if your translation contains URLs. It will find typos in translation text.
  • These 2 examples are using earlier techniques.
    • Technology

    Description

    Sphinx can extract paragraphs from sphinx document and store them into gettext format translation catalog files.

    Gettext format translation catalog is easy to translate from one language to other languages.
    Also Sphinx support internationalization by using such catalog files.
    You can use your favorite editors or services to translate your sphinx docs.

    In this slide, I'll explain 3 things; (1) entire process to translate sphinx docs. (2) automation mechanism for the process. (3) tips for writing docs and translating.

    Transcript

    1. 1. Takayuki Shimizukawa Sphinx co-maintainer Sphinx-users.jp 1
    2. 2. 안녕하세요. annyeonghaseyo. 나는시미즈카와 입니다. naneun shimizukawa imnida. 2
    3. 3. こんにちは Kon-Nichi-Wa 3
    4. 4. Who am I @shimizukawa  Python developer since 2004  Sphinx co-maintainer  Sphinx-users.jp organizer  PyCon JP committee  BePROUD co, ltd. member 4
    5. 5. Agenda 1. Sphinx introduction & Setup for i18n 2. A easy way and Non easy ways to translate 3. Sphinx i18n feature 4. Automated translation process with several services 5. Tips 6
    6. 6. 7
    7. 7. Question1 How many people do you use Open Source Software? 8
    8. 8. Question2 How many people may have contributed to the OSS? 9
    9. 9. Question3 Does the translation into familiar language contribute to other developers? 10
    10. 10. 11 Translator Product Author Translated Docs Readers English Docs Translators
    11. 11. Tools & Services Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 12
    12. 12. 13
    13. 13. What is Sphinx? Sphinx is a documentation generator Sphinx generates doc as several output formats from the reST text markup 14 1. Sphinx introduction & Setup for i18n Sphinx reSTreSTreStructuredText (reST) reST Parser HTML Builder ePub Builder LaTeX Builder texlive HTML theme Favorite Editor
    14. 14. The history of Sphinx (short ver.) 15 1. Sphinx introduction & Setup for i18n The father of Sphinx Too hard to maintenance ~2007 Easy to write Easy to maintenance 2007~
    15. 15. Sphinx Before and After Before  There was no standard ways to write documents  Sometime, we need converting markups into other formats After  Generate multiple output format from single source  Integrated html themes make read docs easier  API references can be integrated into narrative docs  Automated doc building and hosting by ReadTheDocs 16 1. Sphinx introduction & Setup for i18n
    16. 16. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 17 1. Sphinx introduction & Setup for i18n
    17. 17. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 18 1. Sphinx introduction & Setup for i18n
    18. 18. SPHINX docutils HTML Builder HTML theme (Jinja2) gettext Builder *.pot *.po I18N *.mo OmegaT Pootle Transifex Translation Tools, Services Favorite Editor Sphinx i18n feature (built-in) 19 1. Sphinx introduction & Setup for i18n reST Parser (directive / role) doctree (Intermediate) reSTreST reStructuredText (reST)
    19. 19. $ pip install sphinx Support tools for translation How to install Sphinx with i18n tools To build a sphinx document $ pip install sphinx-intl $ pip install transifex-client 20 1. Sphinx introduction & Setup for i18n
    20. 20. $ git clone https://github.com/shimizukawa/deepthought.git $ cd deepthought/doc $ make html ... Build finished. The HTML pages are in _build/html. "make html" command generates html files into _build/html. make html once 21 1. Sphinx introduction & Setup for i18n
    21. 21. Files & setup conf.py $ tree /path/to/deepthought +- deep_thought | +- __init__.py | +- api.py | +- calc.py | +- utils.py +- doc | +- _build/ | | +- html/ | +- _static/ | +- _template/ | +- conf.py | +- index.py | +- make.bat | +- Makefile +- setup.py 22 1. Sphinx introduction & Setup for i18n Document source Document build output Target library for doc1. ... 2. 3. language = 'ko' 4. locale_dirs = ['locale'] 5. doc/conf.py
    22. 22. 23
    23. 23. Easy contributable internationalization process with Sphinx 24
    24. 24. What is internationalization? 25 2. Easy contributable internationalization process with Sphinx I18N
    25. 25. Easy contributable internationalization process with Sphinx 26
    26. 26. What is easy contributable? 27 2. Easy contributable internationalization process with Sphinx
    27. 27. Not a easy way (example 1/3) The manual are provided only in the HTML files You can rewrite HTML files How to follow the original? 28 2. Easy contributable internationalization process with Sphinx
    28. 28. Not a easy way (example 2/3) The HTML manual are generated from reST files and docstrings in the source files  You can rewrite reST files and docstrings  How to follow the upstream? 29 2. Easy contributable internationalization process with Sphinx
    29. 29. Not a easy way (example 3/3) OK, we are engineers. we can use GIT!  Learn git  Learn GitHub  git clone to get the code  Translation (Yes! This is what I want to do!)  git commit  git pull  Sometimes you must resolve conflict  git push 30 2. Easy contributable internationalization process with Sphinx
    30. 30. Not easy vs easy Not a easy way Easy way 1. Learn git 2. Learn GitHub 3. git clone to get the code 4. Translation 5. git commit, pull, push 6. Resolve conflict 7. Build html your self 1. No git 2. No Github 3. No file 4. Translation 5. Update Automatically 6. No conflict 7. You can get a HTML output w/o hand-build. 31 2. Easy contributable internationalization process with Sphinx
    31. 31. 32
    32. 32. Two i18n features of Sphinx  Output pot files: from reST  Input po files: to generate translated HTML 33 3. Sphinx i18n feature reST pot reST html po
    33. 33. Translation flow  Generate pot  Translate pot into po  Generate Translated HTML 34 3. Sphinx i18n feature reST pot reST html po pot po Translator Please translate Translate! Thanks for your Contribution!
    34. 34. #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" msgid "For example:" msgstr "" 35 Output pot files 3. Sphinx i18n feature $ make gettext ... Build finished. The message catalogs are in _build/gettext. $ ls _build/gettext api.pot examples.pot generated.pot index.pot generated.pot reST pot
    35. 35. Preparing po files to translate doc +- _build/ | +- gettext/ | +- api.pot | +- examples.pot | +- generated.pot | +- index.pot +- locale/ doc +- _build/ +- locale/ +- ko/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/ Translatethem Translator pot po 36 3. Sphinx i18n feature
    36. 36. 37 Preparing po files to translate $ sphinx-intl update -p _build/gettext -l ko Create: locale/ko/LC_MESSAGES/api.po Create: locale/ko/LC_MESSAGES/examples.po Create: locale/ko/LC_MESSAGES/generated.po Create: locale/ko/LC_MESSAGES/index.po At first, sphinx-intl copy pot files and rename them pot po sphinx-intl $ make gettext $ sphinx-intl update -p _build/gettext -l ko Not Changed: locale/ko/LC_MESSAGES/api.po Updated: locale/ko/LC_MESSAGES/examples.po +3, -1 After change the document, sphinx-intl update differences 3. Sphinx i18n feature
    37. 37. Translate po files #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" generated.po pot po sphinx-intl Translator #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "JSON 형식의 :class:`str` 유형에 ``obj`` 를 직렬화." generated.po Translate by using Vim, Emacs, OmegaT, ... 38 3. Sphinx i18n feature
    38. 38. Input po files 39 3. Sphinx i18n feature reST html po Translated $ make html ... Build finished. The HTML pages are in _build/html.
    39. 39. Big picture 40 3. Sphinx i18n feature reST pot html po make gettext sphinx-intl Translator make html Doc Author Translation Catalog Translated catalog
    40. 40. 41
    41. 41. Entire process to translate sphinx docs 42 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog Translator Translator Translator Autor / Translator upload Translator clone Translator
    42. 42. To Be 43 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog upload Translators To Be Automated To Be Parallel clone
    43. 43. Translation tool types  Vim / Emacs (Editors)  Edit local files  Translation support plugins are available.  OmegaT (Translation Tools)  Edit local files.  Translation support features.  Transifex (Services)  Edit online  Translation support features. 44 4. Automated translation process with several services po Translators To Be Parallel
    44. 44. Be Parallel by using Transifex Transifex provides... As API: Upload pot Download po As Web console: Glossary Translation memory Recommendation Auto-translation 45 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download
    45. 45. Translation on Transifex web console 46 4. Automated translation process with several services Original Text Translated Text (you should keep reST syntax) Suggestions from Translation Memory (TM) Original (pot) Translation (po) Copy orig to translation Machine translation Save Review (if needed) Translators Parallel 1 2 4 3 5 6
    46. 46. To Be Automated 47 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download reST html make gettext make html Doc Author upload To Be Automated clone
    47. 47. To Be Automated 48 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
    48. 48. The procedure for automation 1. clone repository 2. make gettext 3. Upload pot 4. Download po 5. make html 6. Upload html 49 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
    49. 49. 1. pip install sphinx sphinx-intl transifex-client 2. git clone https://github.com/shimizukawa/deepthought.git 3. cd deepthought/doc 4. sphinx-intl create-transifexrc # create ~/.transifexrc 5. sphinx-intl create-txconfig # create .tx/config 6. make gettext 7. sphinx-intl -p _build/gettext update-txconfig-resources # update .tx/config 8. tx push -s # push pot files to transifex 9. tx pull -l ko # pull po files from transifex 10. make html SPHINXOPTS="-D language=ko" Shell commands for automation run.sh $ export SPHINXINTL_TRANSIFEX_USERNAME=mice $ export SPHINXINTL_TRANSIFEX_PASSWORD=42 $ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7 $ export SPHINXINTL_POT_DIR=_build/gettext $ run.sh 50 4. Automated translation process with several services
    50. 50. The drone.io 51 WebHook Deploy Clone repository Run shell script The drone.io is a continuous integration service. 4. Automated translation process with several services
    51. 51. GitHub + drone.io + S3 52 GitHub Amazon S3 Transifex 1. Clone repository 2. make gettext. 3. Upload pot 4. Download po 5. make html 6. Upload html 4. Automated translation process with several services 2 1 3
    52. 52. Be Automated 53 pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated Upload pot Download po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
    53. 53. Automated by drone.io 54 Upload potDownload po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
    54. 54. View from doc author  Doc Author doesn't require annoying procedure. Update Translation Source Doc Author Notify See Commit 55 4. Automated translation process with several services
    55. 55. View from doc translators  No git  No file  No conflict  Update Automatically  They can get a translated HTML output w/o hand-build. Translators Parallel See Translate Translated Pages 56 4. Automated translation process with several services
    56. 56. The entire automated process 57 Update Translation Source Doc Author Translators Parallel Notify See See Publish Translate Commit Download Translations Notify Automated 4. Automated translation process with several services
    57. 57. Summary Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 58 4. Automated translation process with several services
    58. 58. 59
    59. 59. msgid "Sphinx translates a set of reStructuredText_ source files into various output formats." msgstr "Sphinx translates a set of “ “`reStructuredText <http://docutils.sphinx-users.jp/web/rst.html>`_ “ “source files into various output formats." *.po How to handle URLs 60 5. TIPS Sphinx translates a set of reStructuredText_ source files into various output formats. .. _reStructuredText: http://docutils.sourceforge.net/rst.html *.rst Hyperlink Target  Hyperlink Target doesn’t appear in POT files.  If you want to change URLs into translation version, you can use Embedded URLs notation.
    60. 60. How to change language w/o rewriting conf.py  You can use SPHINXOPTS make option  It also works with other targets not only ’html’.  linkcheck target is useful if your translation contains URLs. It will find typos in translation text. 61 5. TIPS $ make linkcheck SPHINXOPTS="-D language=ko" $ make html SPHINXOPTS="-D language=ko"
    61. 61. You can control pot file resolution  Use gettext_compat  By default ‘gettext_compat = True’.  reST files under sub directories are collected into single POT file. 62 5. TIPS language = 'ko' locale_dirs = ['locale'] gettext_compat = False # generate .pot for each .rst doc/conf.py
    62. 62. Translatable sphinx html templete  Sphinx HTML template is Jinja2.  You can use “{% trans %}” template tag.  “make gettext” exports “trans” contents into sphinx.pot file files. 63 5. TIPS <p> <em>{%trans%}What users say:{%endtrans%}</em> </p> <p> {%trans%}“Cheers for a great tool that actually makes programmers <b>want</b> to write documentation!“{%endtrans%} </p> _templates/index.html
    63. 63. Example projects  Sphinx-1.4 doc for "ja" translation https://drone.io/bitbucket.org/shimizukawa/sphinx-doc15/admin  Docutils doc for "ja" translation https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin 64 5. Tips
    64. 64. Questions? @shimizukawa Grab me after the session :) I’ll be in SPRINT venue too! 65
    65. 65. Thanks :) 66

    Editor's Notes

  • （ここの文字サイズ大きくする、マウスをレーザーポインタにする）
    Hi everyone. Thank you for coming my session.
    //Can you hear me? OK
    //Can you see the session title?
  • あにょん　はせよ
    なぬん　しみじゅかわ　いむにだ
  • こんにちは
  • I’m Takayuki Shimizukawa came from Japan.
    This is the second time of speaker at PyCon in Korea.
  • This is a capture image from PyCon APAC site.
    （クリック）
    Today I’ll give a talk relates Languages.
    Title is: Easy contributable document internationalization process with Sphinx.
  • This is Agenda of this session.

    1. Sphinx introduction & Setup for i18n.
    2. A easy way and Non easy ways to translate 3. Sphinx i18n feature 4. Automated translation process with several services
    5. Tips
  • Before proceeding to the subject, I want to ask you.
  • How many people do you use Open Source Software?
    -> 10, 20, ... Thanks.
    A-> Obviously, most people are using OSS.
    B-> Oh, It's surprising. Half of the people don't use OSS.
  • How many people may have contributed to the OSS?
    -> 1, 2, ... Thanks,
    A-> Well,... it's a very small number.
    B-> Oh, a lot of people have contributed. It's very good news.

    IMO, using OSS is the first contribution.
    Also we can say that introducing the OSS to other people is another contribution.
    So, almost (a lot of / a half of) people who have raised your hand to the first question is already contributing to the OSS.
  • Many of the OSS provides the document in English.
    How do you think, Does the translation into familiar language contribute to other developers? Yes?

    10, 20,... Thanks.
    A-> most people think it's a contribution, I agree it completely.
    B-> Looks not too many. I guess everyone can read English, right?

    In Japan, many programmers aren't familiar to English.
    In such situations, translated documents helps me when I introduce some library to other developers.
    Because of this, I occasionally translate documents.
  • I think, by your translation of a OSS document, more people will be able to use the OSS.
    （クリック）
    If more than one person can join such translation work easily, it would be great.
    In this session, I'll introduce how to make a mechanism which is easy to join as a translator.
  • In order to make the mechanism, we'll use such tools and services in this session.
    Tools; sphinx, docutils, sphinx-intl, transifex-client
    Services; Transifex, Drone.io.

  • First subject, I'll introduce the sphinx introduction and setup Sphinx for i18n.
  • What is Sphinx?
    Sphinx is a documentation generator.
    Sphinx generates doc as several output formats from reStructuredText markup that is an extensible.
    （ポインタでinputとoutputを指す）
  • This man, Georg Brandl is the father of Sphinx.
    （クリック）
    Until 2007, python official document was written by LaTeX.
    But, it's too hard to maintenance.
    Georg was trying to change such situation.
    （クリック）
    So then, he created Sphinx in 2007.
    Sphinx provides usability and maintainability of the Python official document.
  • Sphinx before and after.
    Before
    There was no standard ways to write documents. One of example is a Python official document. It was written by LaTeX and several some python scripts jungle.
    And, Sometime, we need converting markups into other formats

    Since sphinx has been released,
    * We can generate more multiple output format from single source.
    * Integrated html themes make read docs easier.
    * API references can be integrated into narrative docs.
    * Automated doc building and hosting by ReadTheDocs service.
  • Nowadays, sphinx has been used by these libraries and tools.
    Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

    And Non python library/tools also using Sphinx for them docs: Chef, CakePHP(2.x), MathJax, Selenium, Varnish
  • Nowadays, sphinx has been used by these libraries and tools.
    Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

    And Non python library/tools also using Sphinx for them docs: Chef, CakePHP(2.x), MathJax, Selenium, Varnish
  • Sphinx extend the i18n mechanism to provide translation feature.
    In this session, let's take a look the i18n feature.
  • How to install Sphinx with i18n tools.
    You need to install sphinx to build a sphinx document.
    Also you need to install sphinx-intl and transifex-client for translation.
    You should use newest packages.
  • At first, you should get source code whether from github or other place to translate it.
    In this case, the source code already has a sphinx doc directory.

    So, type "make html" in doc directory to generate html output.
    You can see the output in _build/html directory.
  • Now you can see the directories/files structure, like this.
    （ポインタ）
    * Target library for doc that is cloned from existent repository.
    * Document build output that was generated by "make html"
    * Document source that was generated by "sphinx-quickstart"
    （クリック）
    This is a conf.py file in the doc directory.
    You should add these 2 lines to the conf.py
    * Line-3rd: set a language code you want to use for output doc.
    * Line-4th: set a directory relative path from conf.py location. The locale directory will have translation catalog files.
    Now setup is over.
  • Well, it is the main subjects from here.

    Easy contributable internationalization process with Sphinx.
  • internationalization
  • What is internationalization? It's usually called I18N that I’ve already mentioned.
    The internationalization process is sometimes called translation or localization enablement without rewriting the source code from a technical point of view.

    "without rewriting the source code" is important.
  • Easy contributable.
  • What is easy contributable?
    Some software users or document readers have wanted to contribute by translation.
    But, if the preparation of the translation takes long time or requires many kind of skill, they are frustrated and give up.
    "Easy contributable" is the STATE that will be accepted the contributions immediately.
  • So, for the comparison, I'll introduce the three cases of difficult to contribute.

    First one.
    The manual are provided only in the HTML files.
    Of course, you can translate it.
    But, usually the html files are frequently updated.
    How do you follow the original changes to update your translations?

    This is the first case. I have no idea how to follow the original changes.
  • 2nd example.
    When the HTML manual are generated from reST files and docstrings in the source files,
    You can rewrite original files to translate it.

    But this approach has three problems: No1. You must be careful to maintain reStructuredText structure. No2. It's hard to divide translation tasks for a number of volunteer translators. No3. It's hard to follow the upstream document source that is frequently updated.
  • last example.
    OK, we are engineers. We can use git!
    （上を読む）
    There are many many steps to complete our translations.
    Additionally, in this case, the repository owner should allow write access permission for each translators.

    This is also not a easy way.
  • Well then, what is a easy way?
    My opinion is this. Left side is Not a easy way.
    Right side is A easy way.
    We need a easy way.
    The combination of the Sphinx i18n feature and some of the services give you a easy way.

    That's what is the goal of this session
    OK, let's move forward.
  • 3rd Subject. Sphinx i18n feature.
    I'll explain the detail of the feature.
  • Sphinx i18n feature is composed of two functions.

    First one is the generation pot files from reST files.
    pot file is a famous translation catalog format for gettext that is used for i18n, you know.

    Second one is the generation translated html files with using po files that is translated version of catalogs.
    It's the human work to prepare the translated po files from the pot files.
  • Like this, Converting the POT files into PO files is the job of humans.

    （クリック）In this flow, Author generate pot file at first part, and then, Author ask translators to translate them.
    （クリック）Translators receive the pot files from Author in some way, then translate them.
    （クリック）Once it has been finished, translators send back the po files to Author.

    Let's look a little more detail from first part.
  • Author can use "make gettext" command to generate pot files.

    "make gettext" command extracts text from reST files and python sources that referenced by autodoc.
    It means, you can translate them into any language without rewriting the original reST files and python source files.

    As you see, pot translation catalog makes easy to translate from one language to another languages. "msgid" line have a original sentence.
    "msgstr" line will have translated one.
  • Translated PO files should have as a right side structure.
    "locale" directory has “ko" and it has "LC_MESSAGES" directory, and it has translated PO files.
    PO files are copy of POT files with changed file's extension.
    If you want to translate it into other languages, like 'ja', you can create other directories under "locale" directory.

    When original POT files are updated by changing of original document, you should update PO files too.
    If you had done it manually, it is very hard.
    You can automated it by sphinx-intl.
  • At first, sphinx-intl copy pot files and rename them.
    When the document is changed, translators can use sphinx-intl to update PO differences.
    By same command.

    As you see, translators can finish the boring tasks by one command.
    sphinx-intl copy pot files and rename them on behalf of you.
    sphinx-intl also update differences for each po files when the document is changed.

    It's easy to use.
    As a consequence of introducing the sphinx-intl tool, translators can concentrate for translation.
  • This is a translation work.

    Translators can use their favorite editors or helpful tools/services easily that provide translation support features like a translation memory, recommending similar translation, glossary and auto-translation.
    Once the translation is complete, finally let's run the "make html" command.
  • Finally author and translators can run "make html" and can get translated output w/o editing reST and python code.
    This is a translated HTML sample.
    Internally, Sphinx parse reST file and replace them into translated one by using po file.
  • Entire process to translate sphinx docs is this.
    If you translate some sphinx document without using the i18n feature, you need to rewrite original document source files.
    The Sphinx i18n feature provides a easy way to translate a document as you seen.
  • So far, we've found the mechanism of i18n of Sphinx.
    In 4th Subject, Automated translation process with several services, we will continue to automate this mechanism.
  • Right now, we have seen an overall picture of the process.
    Actually, I have a thing that was not pictured here.
    （クリック）
    Yes, these commands such as "make blah blah" requires your hands to invoke.
    In other words, the translator must learn such steps, yet.
    Additionally, it would be difficult to translate in parallel.

    It's still not a easy way.
  • OK, Let's make it to be automation and parallelization.

    And then, the translator doesn't need to know the mechanism.
    And, make the translation work possible to parallelize.

    Let's take a look at from the parallelization of the translation work.
  • Once you get po files, you can use helpful tools/services easily that provide translation support features like a translation memory, recommending similar translation, glossary and auto-translation.

    In particular, the use of the online translation service is important for parallelization.
    I'd like to introduce the Transifex as a online translation service.
  • Transifex provides the following functions: Upload pot and download po files by API, glossary feature, translation memory, recommending translations, automatic translation by using Google / Microsoft API.
    Because you can do translation on the Web screen, you can translate them with other translators in parallel.
  • Translation on Transifex web console.
    1. You can select one untranslated message
    2. And Translate it by your self, or by machine translation
    3. If the message contains long module or method name, you can click the icon to copy original to translation box w/o typing.
    4. Sometimes, you are suggested translation messages from other translator, or Translation Memory. Translation Memory (called TM) pick up suggestions from similar translation in the project or its historical changes. For example, When Doc Author fix a typo, translation will be discarded. However, TM suggests previous translation that matches 99% with current original message.
    5. Once you finish the translation, you can save it.
    6. Optionally, reviewer can review it.
  • We have made translation parallelization.
    Next.
    Let's automate the central part.
  • The procedure you want to automate is made up of 6 steps.
    At first, we will review the 6 steps.
  • First step is, Clone repository to be translated.
    Step 2. make gettext to generate translation catalog.
    Step 3. Upload pot translation catalog source.
    Step 4. Download translated po files.
    Step 5. "make html" command to generate translated document.
    Step 6. Upload html to publish it

    That's all.
    Next, we will replace these all steps on the command line.
  • This is the command line that include the steps.

    Line1: install sphinx and relates tools
    Line2: clone repository
    Line4: create a transifex account file by using sphinx-intl that reference environment variables (USERNAME and PASSWORD).
    Line5: create a transifex config file by using sphinx-intl that also reference a environment variable (PROJECT_NAME)
    Line6: generate pot files by make gettext
    Line7: update resources information in transifex config file that also reference a environment variable (POT_DIR)
    Line8: push pot files to transifex
    Line9: pull translated po files from transifex
    Line10: make html to generate translated html with using translated catalogs.

    Now we replaced 5 steps on the command line, except HTML uploading part.
    I'll explain the part later.

    As you seen above, You can automate the process by this script.

    Rest of work is preparing a environment to run the script automatically.
    As one of the environment, I'll introduce the drone.io service.
  • The drone.io is a continuous integration service.
    Drone.io integrates seamlessly with Github, Bitbucket and Google Code to make setup fast and simple.
    Drone.io also integrates with Amazon, Heroku, Google AppEngine and more. For custom deployments you can use SSH and shell scripts.
    You can use the service for public project without charge.
  • In my case, I usually use Github and S3 integrations.
    Process of that is,
    （クリック）
    1. Push notification from GitHub
    2. That invoke the script you've seen
    3. Then, deploy html files onto S3 storage. (You need a publish html setting of S3)

    If you set WebHook notification URL to transifex hook setting, updating translations also invoke the drone.io script.
  • By using drone.io service, automation can be achieved.
    Upper-Left process is replaced with Right-Down one.
  • 1 WebHook from Github invokes the script.
    The script execute commands 2, 3, 4 and 5.
    Finally drone.io deployment feature will deploys HTML to AWS S3 storage.

    As this, we got the automated mechanism by using drone.io environment.
    Maybe you can achieve this with using Travis-CI or CirculeCI or some other CI services.
  • In summary, let's look at the results of the automation at each point of view as author and translators.

    At first, a point of view from the doc author.

    When author push to the repository, translation source of Transifex will be updated, and author can check the translated html in the site.
    Doc Author doesn't require annoying procedure.
  • The second is, a point of view from doc translators.
    They can translate the document in parallel with;
    No git, No file, No conflict.
    Translation source are updated Automatically
    They can get a translated HTML output w/o hand-build.

    It means that the translation contributors can focus on the translation.
  • By the mechanism of automation, both of doc author and translators will provide the maximum effect with a minimum of effort.
  • In order to make the mechanism, we'll use such tools and services in this session.
    Tools; sphinx, docutils, sphinx-intl, transifex-client
    Services; Transifex, Drone.io.

  • I'd like to introduce some of the tips.
  • How to handle URLs.
    By current implementation of Sphinx, hyperlink target in reST files doesn’t appear in POT files.
    It means you can’t change the URL in the hyperlink target.

    If you want to change URLs into translation version, you can use Embedded URLs notation (like above example).
  • How to change language w/o rewriting conf.py

    Because of you are not a owner of documentation, you can’t change the conf.py in the document.
    In such case, you can use SPHINXOPTS make option.
    It also works with other targets not only ’make html’.
    linkcheck target is useful if your translation contains URLs. It will find typos in translation text.
  • These 2 examples are using earlier techniques.
    • Technology
    License: CC Attribution License

    Recommended

    More Related Content

    Slideshows for you

    RootedCON 2017 - Docker might not be your friend. Trojanizing Docker images
    Daniel Garcia (a.k.a cr0hn)
    Social Coding GitHub 2015
    Kenu, GwangNam Heo
    Welcome to Python
    Elena Williams
    5 minute intro to virtualenv
    amenasse
    Intro to-venv-py3
    Dhineshsunder ganapathi
    Intro to Jupyter Notebooks
    Francis Michael Bautista

    Viewers also liked

    素振りのススメ at Python入門者の集い
    Takayuki Shimizukawa
    仕事で使うちょっとしたコードをOSSとして開発メンテしていく - Django Redshift Backend の開発 - PyCon JP 2016
    Takayuki Shimizukawa
    JUS関西 Sphinxワークショップ@関西 Sphinx紹介
    Takayuki Shimizukawa
    世界のSphinx事情 @ SphinxCon JP 2015
    Takayuki Shimizukawa
    Sphinxで作る貢献しやすい ドキュメント翻訳の仕組み
    Takayuki Shimizukawa
    Pythonはどうやってlen関数で長さを手にいれているの？
    Takayuki Shimizukawa

    Similar to Easy contributable internationalization process with Sphinx @ PyCon APAC 2016

    Easy contributable internationalization process with Sphinx (PyCon APAC 2015 ...
    Takayuki Shimizukawa
    The Future of Dependency Management for Ruby
    Hiroshi SHIBATA
    Introduction to GitHub - New England VTUG Winter Warmer
    Jonathan Frappier
    Intro to Git, GitHub, and BitBucket
    Legal Services National Technology Assistance Project (LSNTAP)
    Lean Php Presentation
    Alan Pinstein
    Django Documentation
    Ying wei (Joe) Chou
    Efficient development workflows with composer
    nuppla
    The Future of Bundled Bundler
    Hiroshi SHIBATA
    Open source projects with python
    roskakori
    Faster and Smarter Development with Drupal Console
    FFW
    Symfony2 from the Trenches
    Jonathan Wage
    An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
    Steffen Gebert
    Packages, Releases, QGSkel
    Quintagroup
    Django
    Narcisse Siewe
    .NET Conf 2019 - Indexing and searching NuGet.org with Azure Functions and Se...
    Maarten Balliauw
    Upgrading from CFEngine2 to CFEngine3 - Webinar Slides
    CFEngine
    Into The Box 2019 - Keynote Day1
    Ortus Solutions, Corp
    Debian packaging howto
    Ding Zhou
    GitHub for Newbies - Boston VMware User Group
    Jonathan Frappier
    Road to sbt 1.0: Paved with server (2015 Amsterdam)
    Eugene Yokota

    More from Takayuki Shimizukawa

    Webアプリを並行開発する際のマイグレーション戦略
    Takayuki Shimizukawa
    『自走プログラマー』 が我々に必要だった理由
    Takayuki Shimizukawa
    エキスパートPythonプログラミング改訂3版の読みどころ
    Takayuki Shimizukawa
    RLSを用いたマルチテナント実装 for Django
    Takayuki Shimizukawa
    独学プログラマーのその後
    Takayuki Shimizukawa
    【修正版】Django + SQLAlchemy: シンプルWay
    Takayuki Shimizukawa
    Sphinx customization for OGP support at SphinxCon JP 2018
    Takayuki Shimizukawa
    Sphinx autodoc - automated api documentation - PyCon.KR 2015
    Takayuki Shimizukawa
    Easy contributable internationalization process with Sphinx @ pyconmy2015
    Takayuki Shimizukawa
    Sphinx autodoc - automated api documentation - PyCon.MY 2015
    Takayuki Shimizukawa
    Sphinx autodoc - automated API documentation (EuroPython 2015 in Bilbao)
    Takayuki Shimizukawa
    Easy contributable internationalization process with Sphinx @ pyconsg2015
    Takayuki Shimizukawa
    Sphinx autodoc - automated API documentation (PyCon APAC 2015 in Taiwan)
    Takayuki Shimizukawa
    PyPro2の読みどころ紹介：Python開発の過去と現在 - BPStudy93
    Takayuki Shimizukawa
    Pythonコミュニティ-紹介 @ LOCAL 2015
    Takayuki Shimizukawa
    PyPro2の読みどころ紹介：Python開発の過去と現在
    Takayuki Shimizukawa
    執筆中のPythonプロフェッショナルプログラミング第2版でsphinxを使っている話
    Takayuki Shimizukawa
    BPStudy#48 PyCon JP 2011 開催報告
    Takayuki Shimizukawa
    Sphinxを使って本を書こう #pyconjp 2012
    Takayuki Shimizukawa
    PYTHON PACKAGING （PyFes 2012.03 発表資料）
    Takayuki Shimizukawa

    Featured

    What to Upload to SlideShare
    SlideShare
    Be A Great Product Leader (Amplify, Oct 2019)
    Adam Nash
    Trillion Dollar Coach Book (Bill Campbell)
    Eric Schmidt
    APIdays Paris 2019 - Innovation @ scale, APIs as Digital Factories' New Machi...
    apidays
    A few thoughts on work life-balance
    Wim Vanderbauwhede
    Is vc still a thing final
    Mark Suster
    The GaryVee Content Model
    Gary Vaynerchuk
    Mammalian Brain Chemistry Explains Everything
    Loretta Breuning, PhD
    Blockchain + AI + Crypto Economics Are We Creating a Code Tsunami?
    Dinis Guarda
    The AI Rush
    Jean-Baptiste Dumont
    AI and Machine Learning Demystified by Carol Smith at Midwest UX 2017
    Carol Smith
    10 facts about jobs in the future
    Pew Research Center's Internet & American Life Project
    Harry Surden - Artificial Intelligence and Law Overview
    Harry Surden
    Inside Google's Numbers in 2017
    Rand Fishkin
    Pinot: Realtime Distributed OLAP datastore
    Kishore Gopalakrishna
    How to Become a Thought Leader in Your Niche
    Leslie Samuel
    Visual Design with Data
    Seth Familian
    Designing Teams for Emerging Challenges
    Aaron Irizarry
    UX, ethnography and possibilities: for Libraries, Museums and Archives
    Ned Potter
    Winners and Losers - All the (Russian) President's Men
    Ian Bremmer

    Related Books

    Free with a 30 day trial from Scribd

    See all
    Bezonomics: How Amazon Is Changing Our Lives and What the World's Best Companies Are Learning from It Brian Dumaine
    (4/5)
    Free
    No Filter: The Inside Story of Instagram Sarah Frier
    (4.5/5)
    Free
    Autonomy: The Quest to Build the Driverless Car—And How It Will Reshape Our World Lawrence D. Burns
    (5/5)
    Free
    Talk to Me: How Voice Computing Will Transform the Way We Live, Work, and Think James Vlahos
    (4/5)
    Free
    From Gutenberg to Google: The History of Our Future Tom Wheeler
    (3.5/5)
    Free
    SAM: One Robot, a Dozen Engineers, and the Race to Revolutionize the Way We Build Jonathan Waldman
    (5/5)
    Free
    The Future Is Faster Than You Think: How Converging Technologies Are Transforming Business, Industries, and Our Lives Peter H. Diamandis
    (4.5/5)
    Free
    So You Want to Start a Podcast: Finding Your Voice, Telling Your Story, and Building a Community That Will Listen Kristen Meinzer
    (3.5/5)
    Free
    Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Seth Stephens-Davidowitz
    (4.5/5)
    Free
    Life After Google: The Fall of Big Data and the Rise of the Blockchain Economy George Gilder
    (4/5)
    Free
    Future Presence: How Virtual Reality Is Changing Human Connection, Intimacy, and the Limits of Ordinary Life Peter Rubin
    (4/5)
    Free
    Live Work Work Work Die: A Journey into the Savage Heart of Silicon Valley Corey Pein
    (4.5/5)
    Free
    How to Survive a Robot Uprising: Tips on Defending Yourself Against the Coming Rebellion Daniel H. Wilson
    (4/5)
    Free
    Ninety Percent of Everything: Inside Shipping, the Invisible Industry That Puts Clothes on Your Back, Gas in Your Car, and Food on Your Plate Rose George
    (4/5)
    Free
    Carrying the Fire: 50th Anniversary Edition Michael Collins
    (4.5/5)
    Free
    The Basics of Bitcoins and Blockchains: An Introduction to Cryptocurrencies and the Technology that Powers Them (Cryptography, Derivatives Investments, Futures Trading, Digital Assets, NFT) Antony Lewis
    (4/5)
    Free

    Related Audiobooks

    Free with a 30 day trial from Scribd

    See all
    Einstein's Fridge: How the Difference Between Hot and Cold Explains the Universe Paul Sen
    (4.5/5)
    Free
    After Steve: How Apple Became a Trillion-Dollar Company and Lost its Soul Tripp Mickle
    (4.5/5)
    Free
    The Quiet Zone: Unraveling the Mystery of a Town Suspended in Silence Stephen Kurczy
    (4.5/5)
    Free
    The Wires of War: Technology and the Global Struggle for Power Jacob Helberg
    (4/5)
    Free
    System Error: Where Big Tech Went Wrong and How We Can Reboot Rob Reich
    (4.5/5)
    Free
    If Then: How the Simulmatics Corporation Invented the Future Jill Lepore
    (4.5/5)
    Free
    Liftoff: Elon Musk and the Desperate Early Days That Launched SpaceX Eric Berger
    (5/5)
    Free
    The Science of Time Travel: The Secrets Behind Time Machines, Time Loops, Alternate Realities, and More! Elizabeth Howell
    (3/5)
    Free
    A Brief History of Motion: From the Wheel, to the Car, to What Comes Next Tom Standage
    (4.5/5)
    Free
    An Ugly Truth: Inside Facebook’s Battle for Domination Sheera Frenkel
    (4.5/5)
    Free
    Bitcoin Billionaires: A True Story of Genius, Betrayal, and Redemption Ben Mezrich
    (4.5/5)
    Free
    The Players Ball: A Genius, a Con Man, and the Secret History of the Internet's Rise David Kushner
    (4.5/5)
    Free
    User Friendly: How the Hidden Rules of Design Are Changing the Way We Live, Work, and Play Cliff Kuang
    (4/5)
    Free
    Lean Out: The Truth About Women, Power, and the Workplace Marissa Orr
    (4.5/5)
    Free
    Blockchain: The Next Everything Stephen P Williams
    (4/5)
    Free
    Uncanny Valley: A Memoir Anna Wiener
    (4/5)
    Free

    ×