Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Easy contributable internationalization process with Sphinx @ PyCon APAC 2016

1,555 views

Published on

Sphinx can extract paragraphs from sphinx document and store them into gettext format translation catalog files.

Gettext format translation catalog is easy to translate from one language to other languages.
Also Sphinx support internationalization by using such catalog files.
You can use your favorite editors or services to translate your sphinx docs.

In this slide, I'll explain 3 things; (1) entire process to translate sphinx docs. (2) automation mechanism for the process. (3) tips for writing docs and translating.

Published in: Technology
  • Be the first to comment

Easy contributable internationalization process with Sphinx @ PyCon APAC 2016

  1. 1. Takayuki Shimizukawa Sphinx co-maintainer Sphinx-users.jp 1
  2. 2. 안녕하세요. annyeonghaseyo. 나는시미즈카와 입니다. naneun shimizukawa imnida. 2
  3. 3. こんにちは Kon-Nichi-Wa 3
  4. 4. Who am I @shimizukawa  Python developer since 2004  Sphinx co-maintainer  Sphinx-users.jp organizer  PyCon JP committee  BePROUD co, ltd. member 4
  5. 5. Agenda 1. Sphinx introduction & Setup for i18n 2. A easy way and Non easy ways to translate 3. Sphinx i18n feature 4. Automated translation process with several services 5. Tips 6
  6. 6. 7
  7. 7. Question1 How many people do you use Open Source Software? 8
  8. 8. Question2 How many people may have contributed to the OSS? 9
  9. 9. Question3 Does the translation into familiar language contribute to other developers? 10
  10. 10. 11 Translator Product Author Translated Docs Readers English Docs Translators
  11. 11. Tools & Services Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 12
  12. 12. 13
  13. 13. What is Sphinx? Sphinx is a documentation generator Sphinx generates doc as several output formats from the reST text markup 14 1. Sphinx introduction & Setup for i18n Sphinx reSTreSTreStructuredText (reST) reST Parser HTML Builder ePub Builder LaTeX Builder texlive HTML theme Favorite Editor
  14. 14. The history of Sphinx (short ver.) 15 1. Sphinx introduction & Setup for i18n The father of Sphinx Too hard to maintenance ~2007 Easy to write Easy to maintenance 2007~
  15. 15. Sphinx Before and After Before  There was no standard ways to write documents  Sometime, we need converting markups into other formats After  Generate multiple output format from single source  Integrated html themes make read docs easier  API references can be integrated into narrative docs  Automated doc building and hosting by ReadTheDocs 16 1. Sphinx introduction & Setup for i18n
  16. 16. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 17 1. Sphinx introduction & Setup for i18n
  17. 17. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 18 1. Sphinx introduction & Setup for i18n
  18. 18. SPHINX docutils HTML Builder HTML theme (Jinja2) gettext Builder *.pot *.po I18N *.mo OmegaT Pootle Transifex Translation Tools, Services Favorite Editor Sphinx i18n feature (built-in) 19 1. Sphinx introduction & Setup for i18n reST Parser (directive / role) doctree (Intermediate) reSTreST reStructuredText (reST)
  19. 19. $ pip install sphinx Support tools for translation How to install Sphinx with i18n tools To build a sphinx document $ pip install sphinx-intl $ pip install transifex-client 20 1. Sphinx introduction & Setup for i18n
  20. 20. $ git clone https://github.com/shimizukawa/deepthought.git $ cd deepthought/doc $ make html ... Build finished. The HTML pages are in _build/html. "make html" command generates html files into _build/html. make html once 21 1. Sphinx introduction & Setup for i18n
  21. 21. Files & setup conf.py $ tree /path/to/deepthought +- deep_thought | +- __init__.py | +- api.py | +- calc.py | +- utils.py +- doc | +- _build/ | | +- html/ | +- _static/ | +- _template/ | +- conf.py | +- index.py | +- make.bat | +- Makefile +- setup.py 22 1. Sphinx introduction & Setup for i18n Document source Document build output Target library for doc1. ... 2. 3. language = 'ko' 4. locale_dirs = ['locale'] 5. doc/conf.py
  22. 22. 23
  23. 23. Easy contributable internationalization process with Sphinx 24
  24. 24. What is internationalization? 25 2. Easy contributable internationalization process with Sphinx I18N
  25. 25. Easy contributable internationalization process with Sphinx 26
  26. 26. What is easy contributable? 27 2. Easy contributable internationalization process with Sphinx
  27. 27. Not a easy way (example 1/3) The manual are provided only in the HTML files You can rewrite HTML files How to follow the original? 28 2. Easy contributable internationalization process with Sphinx
  28. 28. Not a easy way (example 2/3) The HTML manual are generated from reST files and docstrings in the source files  You can rewrite reST files and docstrings  How to follow the upstream? 29 2. Easy contributable internationalization process with Sphinx
  29. 29. Not a easy way (example 3/3) OK, we are engineers. we can use GIT!  Learn git  Learn GitHub  git clone to get the code  Translation (Yes! This is what I want to do!)  git commit  git pull  Sometimes you must resolve conflict  git push 30 2. Easy contributable internationalization process with Sphinx
  30. 30. Not easy vs easy Not a easy way Easy way 1. Learn git 2. Learn GitHub 3. git clone to get the code 4. Translation 5. git commit, pull, push 6. Resolve conflict 7. Build html your self 1. No git 2. No Github 3. No file 4. Translation 5. Update Automatically 6. No conflict 7. You can get a HTML output w/o hand-build. 31 2. Easy contributable internationalization process with Sphinx
  31. 31. 32
  32. 32. Two i18n features of Sphinx  Output pot files: from reST  Input po files: to generate translated HTML 33 3. Sphinx i18n feature reST pot reST html po
  33. 33. Translation flow  Generate pot  Translate pot into po  Generate Translated HTML 34 3. Sphinx i18n feature reST pot reST html po pot po Translator Please translate Translate! Thanks for your Contribution!
  34. 34. #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" msgid "For example:" msgstr "" 35 Output pot files 3. Sphinx i18n feature $ make gettext ... Build finished. The message catalogs are in _build/gettext. $ ls _build/gettext api.pot examples.pot generated.pot index.pot generated.pot reST pot
  35. 35. Preparing po files to translate doc +- _build/ | +- gettext/ | +- api.pot | +- examples.pot | +- generated.pot | +- index.pot +- locale/ doc +- _build/ +- locale/ +- ko/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/ Translatethem Translator pot po 36 3. Sphinx i18n feature
  36. 36. 37 Preparing po files to translate $ sphinx-intl update -p _build/gettext -l ko Create: locale/ko/LC_MESSAGES/api.po Create: locale/ko/LC_MESSAGES/examples.po Create: locale/ko/LC_MESSAGES/generated.po Create: locale/ko/LC_MESSAGES/index.po At first, sphinx-intl copy pot files and rename them pot po sphinx-intl $ make gettext $ sphinx-intl update -p _build/gettext -l ko Not Changed: locale/ko/LC_MESSAGES/api.po Updated: locale/ko/LC_MESSAGES/examples.po +3, -1 After change the document, sphinx-intl update differences 3. Sphinx i18n feature
  37. 37. Translate po files #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" generated.po pot po sphinx-intl Translator #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "JSON 형식의 :class:`str` 유형에 ``obj`` 를 직렬화." generated.po Translate by using Vim, Emacs, OmegaT, ... 38 3. Sphinx i18n feature
  38. 38. Input po files 39 3. Sphinx i18n feature reST html po Translated $ make html ... Build finished. The HTML pages are in _build/html.
  39. 39. Big picture 40 3. Sphinx i18n feature reST pot html po make gettext sphinx-intl Translator make html Doc Author Translation Catalog Translated catalog
  40. 40. 41
  41. 41. Entire process to translate sphinx docs 42 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog Translator Translator Translator Autor / Translator upload Translator clone Translator
  42. 42. To Be 43 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog upload Translators To Be Automated To Be Parallel clone
  43. 43. Translation tool types  Vim / Emacs (Editors)  Edit local files  Translation support plugins are available.  OmegaT (Translation Tools)  Edit local files.  Translation support features.  Transifex (Services)  Edit online  Translation support features. 44 4. Automated translation process with several services po Translators To Be Parallel
  44. 44. Be Parallel by using Transifex Transifex provides... As API: Upload pot Download po As Web console: Glossary Translation memory Recommendation Auto-translation 45 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download
  45. 45. Translation on Transifex web console 46 4. Automated translation process with several services Original Text Translated Text (you should keep reST syntax) Suggestions from Translation Memory (TM) Original (pot) Translation (po) Copy orig to translation Machine translation Save Review (if needed) Translators Parallel 1 2 4 3 5 6
  46. 46. To Be Automated 47 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download reST html make gettext make html Doc Author upload To Be Automated clone
  47. 47. To Be Automated 48 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
  48. 48. The procedure for automation 1. clone repository 2. make gettext 3. Upload pot 4. Download po 5. make html 6. Upload html 49 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
  49. 49. 1. pip install sphinx sphinx-intl transifex-client 2. git clone https://github.com/shimizukawa/deepthought.git 3. cd deepthought/doc 4. sphinx-intl create-transifexrc # create ~/.transifexrc 5. sphinx-intl create-txconfig # create .tx/config 6. make gettext 7. sphinx-intl -p _build/gettext update-txconfig-resources # update .tx/config 8. tx push -s # push pot files to transifex 9. tx pull -l ko # pull po files from transifex 10. make html SPHINXOPTS="-D language=ko" Shell commands for automation run.sh $ export SPHINXINTL_TRANSIFEX_USERNAME=mice $ export SPHINXINTL_TRANSIFEX_PASSWORD=42 $ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7 $ export SPHINXINTL_POT_DIR=_build/gettext $ run.sh 50 4. Automated translation process with several services
  50. 50. The drone.io 51 WebHook Deploy Clone repository Run shell script The drone.io is a continuous integration service. 4. Automated translation process with several services
  51. 51. GitHub + drone.io + S3 52 GitHub Amazon S3 Transifex 1. Clone repository 2. make gettext. 3. Upload pot 4. Download po 5. make html 6. Upload html 4. Automated translation process with several services 2 1 3
  52. 52. Be Automated 53 pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated Upload pot Download po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
  53. 53. Automated by drone.io 54 Upload potDownload po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
  54. 54. View from doc author  Doc Author doesn't require annoying procedure. Update Translation Source Doc Author Notify See Commit 55 4. Automated translation process with several services
  55. 55. View from doc translators  No git  No file  No conflict  Update Automatically  They can get a translated HTML output w/o hand-build. Translators Parallel See Translate Translated Pages 56 4. Automated translation process with several services
  56. 56. The entire automated process 57 Update Translation Source Doc Author Translators Parallel Notify See See Publish Translate Commit Download Translations Notify Automated 4. Automated translation process with several services
  57. 57. Summary Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 58 4. Automated translation process with several services
  58. 58. 59
  59. 59. msgid "Sphinx translates a set of reStructuredText_ source files into various output formats." msgstr "Sphinx translates a set of “ “`reStructuredText <http://docutils.sphinx-users.jp/web/rst.html>`_ “ “source files into various output formats." *.po How to handle URLs 60 5. TIPS Sphinx translates a set of reStructuredText_ source files into various output formats. .. _reStructuredText: http://docutils.sourceforge.net/rst.html *.rst Hyperlink Target  Hyperlink Target doesn’t appear in POT files.  If you want to change URLs into translation version, you can use Embedded URLs notation.
  60. 60. How to change language w/o rewriting conf.py  You can use SPHINXOPTS make option  It also works with other targets not only ’html’.  linkcheck target is useful if your translation contains URLs. It will find typos in translation text. 61 5. TIPS $ make linkcheck SPHINXOPTS="-D language=ko" $ make html SPHINXOPTS="-D language=ko"
  61. 61. You can control pot file resolution  Use gettext_compat  By default ‘gettext_compat = True’.  reST files under sub directories are collected into single POT file. 62 5. TIPS language = 'ko' locale_dirs = ['locale'] gettext_compat = False # generate .pot for each .rst doc/conf.py
  62. 62. Translatable sphinx html templete  Sphinx HTML template is Jinja2.  You can use “{% trans %}” template tag.  “make gettext” exports “trans” contents into sphinx.pot file files. 63 5. TIPS <p> <em>{%trans%}What users say:{%endtrans%}</em> </p> <p> {%trans%}“Cheers for a great tool that actually makes programmers <b>want</b> to write documentation!“{%endtrans%} </p> _templates/index.html
  63. 63. Example projects  Sphinx-1.4 doc for "ja" translation https://drone.io/bitbucket.org/shimizukawa/sphinx-doc15/admin  Docutils doc for "ja" translation https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin 64 5. Tips
  64. 64. Questions? @shimizukawa Grab me after the session :) I’ll be in SPRINT venue too! 65
  65. 65. Thanks :) 66

×