Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Takayuki Shimizukawa
Sphinx co-maintainer
Sphinx-users.jp
1
Who am I
@shimizukawa
1. Sphinx co-maintainer
2. Sphinx-users.jp
3. PyCon JP committee
 BePROUD co, ltd.
2
http://pycon.jp/
3
Registration: Opened
Speaker Call for Proposals: until July 15th.
4
Agenda
1. Sphinx introduction & Setup for i18n
2. A easy way and Non easy ways to translate
3. Sphinx i18n feature
4. Auto...
6
Question1
How many people do you use
Open Source Software?
7
Question2
How many people may have
contributed to the OSS?
8
Question3
Does the translation into familiar
language contribute to other
developers?
9
10
Translator
Product
Author
Translated
Docs
Readers
English
Docs
Translators
Tools & Services
Tools
1. sphinx - documentation generator
2. docutils - base of sphinx
3. sphinx-intl - support utility f...
12
What is Sphinx?
Sphinx is a documentation generator
Sphinx generates doc as several output
formats from the reST text mark...
The history of Sphinx (short ver.)
14
1. Sphinx introduction & Setup for i18n
The father of
Sphinx
Too hard to
maintenance...
Many docs are written by Sphinx
For examples
 Python libraries/tools:
Python, Sphinx, Flask, Jinja2, Django, Pyramid,
SQL...
Relation between Sphinx and Docutils
 Sphinx based upon Docutils library
 Docutils/Sphinx supports reStructuredText
it c...
Comparing Docutils and Sphinx
Docutils Sphinx
1. Single page
2. Link to others with full name
3. Cross reference in a page...
SPHINX
docutils
HTML Builder HTML theme
(Jinja2)
gettext Builder
*.pot
*.po
I18N
*.mo
OmegaT
Pootle
Transifex
Translation ...
$ pip install sphinx
Support tools for translation
How to install Sphinx with i18n tools
To build a sphinx document
$ pip ...
$ git clone https://github.com/shimizukawa/deepthought.git
$ cd deepthought/doc
$ make html
...
Build finished. The HTML p...
Files & setup conf.py
$ tree /path/to/deepthought
+- deep_thought
| +- __init__.py
| +- api.py
| +- calc.py
| +- utils.py
...
22
What is
internationalization?
23
2. Easy contributable internationalization process with Sphinx
I18N
What is
easy contributable?
24
2. Easy contributable internationalization process with Sphinx
Not a easy way (example 1/3)
The manual are provided only in the
HTML files
Rewriting reST files
25
2. Easy contributable...
Not a easy way (example 2/3)
The manual are generated from
reST files and
docstrings in the source files
 Rewriting reST ...
Not a easy way (example 3/3)
OK, we are engineers. We can use git!
 Learn git
 Learn GitHub
 git clone to get the code
...
Not easy vs easy
Not a easy way Easy way
1. Learn git
2. Learn GitHub
3. git clone to get the code
4. Translation
5. git c...
29
Two i18n features of Sphinx
 Output pot files:
from reST
 Input po files:
to generate translated HTML
30
3. Sphinx i18n ...
Translation flow
 Generate pot
 Translate pot into po
 Generate Translated HTML
31
3. Sphinx i18n feature
reST pot
reST...
#: ../../../deep_thought/utils.py:docstring of
deep_thought.utils.dumps:1
msgid "Serialize ``obj`` to a JSON formatted :cl...
Preparing po files to translate
doc
+- _build/
| +- gettext/
| +- api.pot
| +- examples.pot
| +- generated.pot
| +- index....
34
Preparing po files to translate
$ sphinx-intl update -p _build/gettext -l zh_cn
Create: locale/zh_cn/LC_MESSAGES/api.po...
Translate po files
#: ../../../deep_thought/utils.py:docstring of
deep_thought.utils.dumps:1
msgid "Serialize ``obj`` to a...
Input po files
36
3. Sphinx i18n feature
reST
html
po
Translated
$ make html
...
Build finished. The HTML pages are in _bu...
Big picture
37
3. Sphinx i18n feature
reST pot
html
po
make gettext
sphinx-intl
Translator
make html
Doc Author
Translatio...
Demo
38
39
Entire process to translate sphinx docs
40
4. Automated translation process with several services
reST pot
html
po
make ge...
To Be
41
4. Automated translation process with several services
reST pot
html
po
make gettext
sphinx-intl
make html
Doc Au...
Translation tool types
 Vim / Emacs (Editors)
 Edit local files
 Translation support plugins are available.
 OmegaT (T...
Be Parallel by using Transifex
Transifex provides...
As API:
Upload pot
Download po
As Web console:
Glossary
Translati...
Translation on Transifex web console
44
4. Automated translation process with several services
Original Text
Translated Te...
To Be Automated
45
4. Automated translation process with several services
po
Translators
Parallel
pot
Upload
pot
Auto Upda...
To Be Automated
46
4. Automated translation process with several services
pot
Upload
sphinx-intl
transifex-client
po trans...
The procedure for automation
1. clone repository
2. make gettext
3. Upload pot
4. Download po
5. make html
6. Upload html
...
1. pip install sphinx sphinx-intl transifex-client==0.8
2. git clone https://github.com/shimizukawa/deepthought.git
3. cd ...
The drone.io
49
WebHook
Deploy
Clone repository
Run shell script
The drone.io is a continuous integration service.
4. Auto...
GitHub + drone.io + S3
50
GitHub
Amazon S3
Transifex 1. Clone repository
2. make gettext.
3. Upload pot
4. Download po
5. ...
Be Automated
51
pot
Upload sphinx-intl
transifex-client
po
transifex-client
Download
reST
html
make gettext
make html
uplo...
Automated by drone.io
52
Upload
potDownload
po
upload
WebHook
clone
1
5
6
make html
make gettext2
3
4
1
WebHook
Automated
...
View from doc author
 Doc Author doesn't require annoying procedure.
Update
Translation Source
Doc Author
Notify
See
Comm...
View from doc translators
 No git
 No file
 No conflict
 Update Automatically
 They can get a translated HTML output ...
The entire automated process
55
Update
Translation Source
Doc Author
Translators
Parallel
Notify
See
See
Publish
Translate...
Summary
Tools
1. sphinx - documentation generator
2. docutils - base of sphinx
3. sphinx-intl - support utility for sphinx...
57
TIP: Drone.io limits 15mins for a build
 Drone.io limits 15mins for a build.
 Install from wheel package may help you
58...
TRAP: Version of transifex-client
 transifex-client 0.11b3 is not stable (for me)
 Especially for Windows users
 If you...
 Drone.io only list-up
repositories which you
have admin permission.
 Prepare a empty
repository to create a
drone.io pr...
Examples
 Sphinx-1.3 doc for "ja" translation
https://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin
 Sphinx-1.4 ...
Questions?
@shimizukawa
Grab me after the session :)
62
Thanks :)
63
Upcoming SlideShare
Loading in …5
×

Easy contributable internationalization process with Sphinx @ pyconsg2015

1,469 views

Published on

Sphinx can extract paragraphs from sphinx document and store them into gettext format translation catalog files.
Gettext format translation catalog is easy to translate from one language to other languages.
Also Sphinx support internationalization by using such catalog files.
You can use your favorite editors or services to translate your sphinx docs.
In this session, I'll explain 3 things; (1) entire process to translate sphinx docs. (2) automation mechanism for the process. (3) tips, tricks and traps for writing docs and translating.

Published in: Technology
  • Be the first to comment

Easy contributable internationalization process with Sphinx @ pyconsg2015

  1. 1. Takayuki Shimizukawa Sphinx co-maintainer Sphinx-users.jp 1
  2. 2. Who am I @shimizukawa 1. Sphinx co-maintainer 2. Sphinx-users.jp 3. PyCon JP committee  BePROUD co, ltd. 2
  3. 3. http://pycon.jp/ 3
  4. 4. Registration: Opened Speaker Call for Proposals: until July 15th. 4
  5. 5. Agenda 1. Sphinx introduction & Setup for i18n 2. A easy way and Non easy ways to translate 3. Sphinx i18n feature 4. Automated translation process with several services 5. Tips, tricks, traps 5
  6. 6. 6
  7. 7. Question1 How many people do you use Open Source Software? 7
  8. 8. Question2 How many people may have contributed to the OSS? 8
  9. 9. Question3 Does the translation into familiar language contribute to other developers? 9
  10. 10. 10 Translator Product Author Translated Docs Readers English Docs Translators
  11. 11. Tools & Services Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 11
  12. 12. 12
  13. 13. What is Sphinx? Sphinx is a documentation generator Sphinx generates doc as several output formats from the reST text markup 13 1. Sphinx introduction & Setup for i18n Sphinx reSTreSTreStructuredText (reST) reST Parser HTML Builder ePub Builder LaTeX Builder texlive HTML theme Favorite Editor
  14. 14. The history of Sphinx (short ver.) 14 1. Sphinx introduction & Setup for i18n The father of Sphinx Too hard to maintenance ~2007 Easy to write Easy to maintenance 2007~
  15. 15. Many docs are written by Sphinx For examples  Python libraries/tools: Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …  Non python libraries/tools: Chef, CakePHP(2.x), MathJax, Selenium, Varnish 15 1. Sphinx introduction & Setup for i18n
  16. 16. Relation between Sphinx and Docutils  Sphinx based upon Docutils library  Docutils/Sphinx supports reStructuredText it called reST, it's an extensible markup (PEP287 / accepted at 2002) 16 1. Sphinx introduction & Setup for i18n Sphinx reST Parser HTML Builder ePub Builder LaTeX Builder HTML theme docutils
  17. 17. Comparing Docutils and Sphinx Docutils Sphinx 1. Single page 2. Link to others with full name 3. Cross reference in a page 4. Writers: html, latex, man, xml, ... 5. Standard reST markup specs 1. Multiple Pages 2. toctree to connect all pages under single tree structure 3. Cross reference over each other pages 4. Additional writers: html(w/ themes), pdf(latex), texinfo, epub, … 5. Additional markup specs: autodoc directive and so on 17 1. Sphinx introduction & Setup for i18n
  18. 18. SPHINX docutils HTML Builder HTML theme (Jinja2) gettext Builder *.pot *.po I18N *.mo OmegaT Pootle Transifex Translation Tools, Services Favorite Editor Sphinx i18n feature (built-in) 18 1. Sphinx introduction & Setup for i18n reST Parser (directive / role) doctree (Intermediate) reSTreST reStructuredText (reST)
  19. 19. $ pip install sphinx Support tools for translation How to install Sphinx with i18n tools To build a sphinx document $ pip install sphinx-intl $ pip install transifex-client=0.8 19 1. Sphinx introduction & Setup for i18n
  20. 20. $ git clone https://github.com/shimizukawa/deepthought.git $ cd deepthought/doc $ make html ... Build finished. The HTML pages are in _build/html. "make html" command generates html files into _build/html. make html once 20 1. Sphinx introduction & Setup for i18n
  21. 21. Files & setup conf.py $ tree /path/to/deepthought +- deep_thought | +- __init__.py | +- api.py | +- calc.py | +- utils.py +- doc | +- _build/ | | +- html/ | +- _static/ | +- _template/ | +- conf.py | +- index.py | +- make.bat | +- Makefile +- setup.py 21 1. Sphinx introduction & Setup for i18n Document source Document build output Target library for doc1. ... 2. 3. language = 'ma' 4. locale_dirs = ['locale'] 5. doc/conf.py
  22. 22. 22
  23. 23. What is internationalization? 23 2. Easy contributable internationalization process with Sphinx I18N
  24. 24. What is easy contributable? 24 2. Easy contributable internationalization process with Sphinx
  25. 25. Not a easy way (example 1/3) The manual are provided only in the HTML files Rewriting reST files 25 2. Easy contributable internationalization process with Sphinx
  26. 26. Not a easy way (example 2/3) The manual are generated from reST files and docstrings in the source files  Rewriting reST files  Rewriting docstrins in the source files. 26 2. Easy contributable internationalization process with Sphinx
  27. 27. Not a easy way (example 3/3) OK, we are engineers. We can use git!  Learn git  Learn GitHub  git clone to get the code  Translation (Yes! This is what I want to do!)  git commit  git pull  Sometimes resolve conflict  git push 27 2. Easy contributable internationalization process with Sphinx
  28. 28. Not easy vs easy Not a easy way Easy way 1. Learn git 2. Learn GitHub 3. git clone to get the code 4. Translation 5. git commit, pull, push 6. Resolve conflict 7. Build html your self 1. No git 2. No Github 3. No file 4. Translation 5. Update Automatically 6. No conflict 7. You can get a HTML output w/o hand-build. 28 2. Easy contributable internationalization process with Sphinx
  29. 29. 29
  30. 30. Two i18n features of Sphinx  Output pot files: from reST  Input po files: to generate translated HTML 30 3. Sphinx i18n feature reST pot reST html po
  31. 31. Translation flow  Generate pot  Translate pot into po  Generate Translated HTML 31 3. Sphinx i18n feature reST pot reST html po pot po Translator Please translate Translate! Thanks for your Contribution!
  32. 32. #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" msgid "For example:" msgstr "" 32 Output pot files 3. Sphinx i18n feature $ make gettext ... Build finished. The message catalogs are in _build/gettext. $ ls _build/gettext api.pot examples.pot generated.pot index.pot generated.pot reST pot
  33. 33. Preparing po files to translate doc +- _build/ | +- gettext/ | +- api.pot | +- examples.pot | +- generated.pot | +- index.pot +- locale/ doc +- _build/ +- locale/ +- zh_cn/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/ Translatethem Translator pot po 33 3. Sphinx i18n feature
  34. 34. 34 Preparing po files to translate $ sphinx-intl update -p _build/gettext -l zh_cn Create: locale/zh_cn/LC_MESSAGES/api.po Create: locale/zh_cn/LC_MESSAGES/examples.po Create: locale/zh_cn/LC_MESSAGES/generated.po Create: locale/zh_cn/LC_MESSAGES/index.po At first, sphinx-intl copy pot files and rename them pot po sphinx-intl $ make gettext $ sphinx-intl update -p _build/gettext -l zh_cn Not Changed: locale/zh_cn/LC_MESSAGES/api.po Updated: locale/zh_cn/LC_MESSAGES/examples.po +3, -1 After change the document, sphinx-intl update differences 3. Sphinx i18n feature
  35. 35. Translate po files #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "" generated.po pot po sphinx-intl Translator #: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1 msgid "Serialize ``obj`` to a JSON formatted :class:`str`." msgstr "序列 ``obj`` 以JSON格式 :class:`str` 。" generated.po Translate by using Vim, Emacs, OmegaT, ... 35 3. Sphinx i18n feature
  36. 36. Input po files 36 3. Sphinx i18n feature reST html po Translated $ make html ... Build finished. The HTML pages are in _build/html.
  37. 37. Big picture 37 3. Sphinx i18n feature reST pot html po make gettext sphinx-intl Translator make html Doc Author Translation Catalog Translated catalog
  38. 38. Demo 38
  39. 39. 39
  40. 40. Entire process to translate sphinx docs 40 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog Translator Translator Translator Autor / Translator upload Translator clone Translator
  41. 41. To Be 41 4. Automated translation process with several services reST pot html po make gettext sphinx-intl make html Doc Author Translation Catalog Translated catalog upload Translators To Be Automated To Be Parallel clone
  42. 42. Translation tool types  Vim / Emacs (Editors)  Edit local files  Translation support plugins are available.  OmegaT (Translation Tools)  Edit local files.  Translation support features.  Transifex (Services)  Edit online  Translation support features. 42 4. Automated translation process with several services po Translators To Be Parallel
  43. 43. Be Parallel by using Transifex Transifex provides... As API: Upload pot Download po As Web console: Glossary Translation memory Recommendation Auto-translation 43 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download
  44. 44. Translation on Transifex web console 44 4. Automated translation process with several services Original Text Translated Text (you should keep reST syntax) Suggestions from Translation Memory (TM) Original (pot) Translation (po) Copy orig to translation Machine translation Save Review (if needed) Translators Parallel 1 2 4 3 5 6
  45. 45. To Be Automated 45 4. Automated translation process with several services po Translators Parallel pot Upload pot Auto Update sphinx-intl transifex-client po transifex-client Download reST html make gettext make html Doc Author upload To Be Automated clone
  46. 46. To Be Automated 46 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
  47. 47. The procedure for automation 1. clone repository 2. make gettext 3. Upload pot 4. Download po 5. make html 6. Upload html 47 4. Automated translation process with several services pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated
  48. 48. 1. pip install sphinx sphinx-intl transifex-client==0.8 2. git clone https://github.com/shimizukawa/deepthought.git 3. cd deepthought/doc 4. sphinx-intl create-transifexrc # create ~/.transifexrc 5. sphinx-intl create-txconfig # create .tx/config 6. make gettext 7. sphinx-intl -p _build/gettext update-txconfig-resources # update .tx/config 8. tx push -s # push pot files to transifex 9. tx pull -l zh_cn # pull po files from transifex 10. make html SPHINXOPTS="-D language=ma" Shell commands for automation run.sh $ export SPHINXINTL_TRANSIFEX_USERNAME=mice $ export SPHINXINTL_TRANSIFEX_PASSWORD=42 $ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7 $ export SPHINXINTL_POT_DIR=_build/gettext $ run.sh 48 4. Automated translation process with several services
  49. 49. The drone.io 49 WebHook Deploy Clone repository Run shell script The drone.io is a continuous integration service. 4. Automated translation process with several services
  50. 50. GitHub + drone.io + S3 50 GitHub Amazon S3 Transifex 1. Clone repository 2. make gettext. 3. Upload pot 4. Download po 5. make html 6. Upload html 4. Automated translation process with several services 2 1 3
  51. 51. Be Automated 51 pot Upload sphinx-intl transifex-client po transifex-client Download reST html make gettext make html upload clone 1 2 3 45 6 To Be Automated Upload pot Download po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
  52. 52. Automated by drone.io 52 Upload potDownload po upload WebHook clone 1 5 6 make html make gettext2 3 4 1 WebHook Automated 4. Automated translation process with several services
  53. 53. View from doc author  Doc Author doesn't require annoying procedure. Update Translation Source Doc Author Notify See Commit 53 4. Automated translation process with several services
  54. 54. View from doc translators  No git  No file  No conflict  Update Automatically  They can get a translated HTML output w/o hand-build. Translators Parallel See Translate Translated Pages 54 4. Automated translation process with several services
  55. 55. The entire automated process 55 Update Translation Source Doc Author Translators Parallel Notify See See Publish Translate Commit Download Translations Notify Automated 4. Automated translation process with several services
  56. 56. Summary Tools 1. sphinx - documentation generator 2. docutils - base of sphinx 3. sphinx-intl - support utility for sphinx i18n 4. transifex-client - file transfer tool for Transifex Services 1. Transifex - translation support service 2. Drone.io - continuous integration service 56 4. Automated translation process with several services
  57. 57. 57
  58. 58. TIP: Drone.io limits 15mins for a build  Drone.io limits 15mins for a build.  Install from wheel package may help you 58 5. Tips, tricks, traps 1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf - 2. export PIP_FIND_LINKS=./wheelhouse 3. pip install sphinx sphinx-intl transifex-client==0.8 run.sh ex. https://bitbucket.org/sphinxjp/docutils-translation/
  59. 59. TRAP: Version of transifex-client  transifex-client 0.11b3 is not stable (for me)  Especially for Windows users  If you have encountered trouble with using newer version, please try: 59 5. Tips, tricks, traps $ pip install "transifex-client=0.8"
  60. 60.  Drone.io only list-up repositories which you have admin permission.  Prepare a empty repository to create a drone.io project. 60 TRICK: Preparing Drone.io project 5. Tips, tricks, traps
  61. 61. Examples  Sphinx-1.3 doc for "ja" translation https://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin  Sphinx-1.4 doc for "ja" translation https://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin  Docutils doc for "ja" translation https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin 61 5. Tips, tricks, traps
  62. 62. Questions? @shimizukawa Grab me after the session :) 62
  63. 63. Thanks :) 63

×