Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Silme & compare-locales

4,987 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Silme & compare-locales

  1. 1. Localization testing with compare-locales and Silme by Adrian Kalla
  2. 2. Me ● Computer Science studies in Wuerzburg/Germany ● Member of the Polish Mozilla-L10n-Community since 2002/2003 ● Internship@MoCo: August-December 2008
  3. 3. Why do we need L10n-tests? ● Localizers: – to easily find possible problems ● Developers: – to make sure a localization does not break everything
  4. 4. Silme & compare-locales ● Silme is a new localization library written in Python ● Compare-locales is a tool for testing the completeness and correctness of localizations
  5. 5. The world before Silme ● Mozilla Translator ● Narro ● Translate Toolkit ● Litmus ● compare-locales ● and much more...
  6. 6. Mozilla Translator ● Probably the oldest tool for Mozilla L10n ● Still maintained ● Causes some problems with file layout
  7. 7. Narro ● Powerful localization tool ● Translation suggestions ● Language pack creation ● Web-only
  8. 8. Translate Toolkit ● Collection of many small tools ● Main goal: convert every L10n file to PO to work with every L10n-Tool that supports PO files ● Requires back and forth converting to and from PO
  9. 9. Litmus ● Huge collection of manual tests ● Good for testing problems which cannot be tested automatically ● Requires much time for testing
  10. 10. compare-locales - History ● Started as a small Perl script for testing localizations for missing and obsolete files and entities, only ● More was needed → rewrite as a bigger Python application – checking for other problems, like encoding or DTD parsing problems – statistics (e.g.: “unmodified: 500 strings”) ● Beginning with August 2008: second rewrite, based on the Silme library
  11. 11. Other Tools... ● Verbatim (mozootle) ● Probably every bigger localization team has made an own set of scripts and tools – And every time they start, they start with the question: ● “How do we access and parse the L10n-files...”
  12. 12. What are the common problems? ● focused on the front-end ● usually dependent on one data format ● not easy extensible ● continuous wheel reinvention: – missing layer between apps and data
  13. 13. Silme - Goals ● generic l10n operations simplified to maximum ● format independency ● platform independency ● easy extensible ● make localization over time easier
  14. 14. Silme - Target ● L10n tools developers ● Localizers with beginner programming knowledge ● Application developers ● Buildsystem admins
  15. 15. Silme – The Name ● Silme (seel-may) is a letter in the Tengwar alphabet (Tolkien)
  16. 16. Silme - Features ● strong diff support ● extensible input/output (file, zip, sql, cvs, hg, svn, ...) ● extensible format support (dtd, prop., po, l20n, ...) ● modular (core, diff, formats, io) ● multilocale
  17. 17. Silme - Structure ● silme.core ● silme.diff ● silme.io ● silme.format ● silme.fp
  18. 18. Silme - API - silme.core ● silme.core.Entity ● silme.core.EntityList ● silme.core.L10nObject ● silme.core.L10nPackage
  19. 19. Silme - API - silme.core.Entity # properties offlineApps.manageUsage=Show settings # dtd <!ENTITY netError.search.button quot;Szukajquot;> # gettext msgid quot;YaST installation sourcequot; msgstr quot;Źródło instalacji YaSTquot; id value entity -> offlineApps.manageUsage Show settings netError.search.button Źródło instalacji YaST YaST installation source Szukaj id value
  20. 20. Silme - API - silme.core.EntityList itemHistory.label Browsing History itemHistory.accesskey B itemPasswords.label Saved Passwords itemCookies.label Cookies entityList = EntityList() entityList.id = “sanitize.dtd” entity = Entity(‘itemHistory.label’) entity.setValue(‘Browsing History’) entityList.addEntity(entity)
  21. 21. Silme - API - silme.core.L10nObject entity ntemporary nn disabledn <!ENTITY itemCookies.label quot;Cookiesquot;> comment entity <!-- ntemporary temporary disabled n disabledn <!ENTITY itemDisabled “ Dis a b le d ” > entity --> <!ENTITY itemCookies.accesskey quot;Cquot;> <!ENTITY itemCache.label n quot;Cachequot;><!ENTITY itemCache.accesskey quot;aquot;> entity entity
  22. 22. Silme - API - EntityList, L10nObject ● Why both? – EntityLists: ● easy and fast ● no file layout preservation – L10nObject: ● preserves file layout for 1:1 reconstruction ● time consuming
  23. 23. Silme - API - silme.core.L10nPackage l 10nPackage = { id: ‘mozapps’, objects: {}, packages: { ‘d o w n l o a d s ’ : . . . ‘h e l p ’ : { id: ‘help’, packages: {}, objects:{ ‘h e l p. d t d ’ : L 1 0 n O b j e c t , ‘h e l p. p r o p e r t i e s ’ : L 1 0 n O b j e c t } } ‘p r o f l e ’ : . . . }
  24. 24. Silme - API - silme.diff ● silme.diff.(Entity|EntityList|L10nObject| L10nPackage)Diff ● silme.core.*.diff() ● silme.core.*.applyDiff() entityDiff = entity.diff(entity2) entityListDiff = entityList.diff(entityList2) entity2 = entity.applyDiff(entityDiff) entityListDiff2 = entityList.applyDiff(entityList2)
  25. 25. Silme - API - silme.io ● silme.io.manager ● silme.io.file|svn|cvs|hg|jar|mysql ● *.getL10nPackage() ● *.writeL10nPackage()
  26. 26. Silme - API - silme.format ● silme.format.*.Parser ● silme.format.*.Structure ● silme.format.*.Serializer
  27. 27. Silme - API - silme.fp ● silme.fp.object ● silme.fp.diff
  28. 28. Silme - Examples 1) Count the number of entities in a directory 2) Find all entities containing the word “karta”
  29. 29. Silme - Future TODO: 1)PEP08 2)Documentation 3)Stable release
  30. 30. compare-locales “2.0” ● Main ideas: – use all the powers of Silme in a tool that would provide at least the same functionality like the first Python version of compare-locales (for the beginning) – use it to ensure that Silme works
  31. 31. compare-locales - log ● First attempt: make the existing compare- locales use Silme as the I/O API – It worked (somehow), but with the use of many hacks ● Final attempt: write it from scratch
  32. 32. compare-locales - Requirements ● L10n.ini ● specific output style ● L10nPackage and L10nObject statistics ● filtering packages and objects ● collecting errors and warnings ● INI-Files ● adding new features
  33. 33. compare-locales – L10n.ini File ● Specifies where to look for L10n-files in Mozilla source
  34. 34. compare-locales – Output Style ● Tree-like with short paths ● As requested by a community member: – A second one with full paths
  35. 35. compare-locales – Statistics ● Provide statistics for entities, files and problems regarding them
  36. 36. compare-locales – Filtering ● Filter unneeded elements for testing
  37. 37. compare-locales – Errors ● There are many things, which should not be in a L10n-file. Find and report them: – Encoding problems: – Duplicated entities:
  38. 38. compare-locales – New features ● Some of the features on previous slides, like finding duplicate entities or checking for correct encoding, are already new features ● But there is more. For example: – Checking access keys for correctness – Comparing not only Mozilla source files, but extensions too – Spell checking of strings – Your_feature_could_be_here...
  39. 39. compare-locales – Access Keys ● Basic access keys checks can be done automatically: – access key present in the corresponding label – access key with only one character – are this policies being followed: https://developer.mozilla.org/en/XUL_Accesskey_FAQ_and_Policies
  40. 40. compare-locales – Extensions ● Compare extensions, too: ● Useful for extensions authors to check if missing entities will not break the whole application ● And for AMO: an extension could be checked while adding it
  41. 41. compare-locales – New features ● How complicated is it to add a feature? – Finding duplicate entities: ● 3 lines of code – Finding broken access keys (basic version): ● ~ 5 lines – Adding support for comparing extensions: ● Only a few hours of work
  42. 42. Limitations ● What is not possible with Silme and compare- locales: – Testing how the localization works when using it with the application it was written for ● MozMill can do automated UI testing – It will not make a coffee for you ;)
  43. 43. Links ● Silme: – http://diary.braniecki.net/tag/silme/ (Gandalfs blog posts about Silme) – http://hg.mozilla.org/users/zbraniecki_mozilla.com/silme/ (HG repository) ● Compare-locales: – http://pypi.python.org/simple/compare-locales/ (Python version) – http://hg.mozilla.org/users/akalla_aviary.pl/silme-patched/ (Silme version, HG repository) – http://mxr.mozilla.org/seamonkey/source/toolkit/locales/compare-locales.pl (Old Perl version)
  44. 44. Q&A
  45. 45. Thank you

×