Every year, VMware has hundreds of releases for its virtualization products that cover data centers, networking, storage, cloud management, and digital workspace. Testing many different products while adapting to their development lifecycle introduces some special challenges for the product globalization team. Vincent Truong describes the processes and testing techniques a large-size software company needs to correctly support the globalization of its products. Vincent examines the agile process of globalization—internationalization and localization—testing and how the test coverage has changed for cloud management products. He will cover each phase of the development lifecycle and testing concepts such as persona-based testing, UI mockups, code scanning, selection of a development framework, automation for UI/API/integration tests, manual internationalization and localization testing, the VMware Globalization Agile Scorecard, dashboard, and the defect heatmap. These testing techniques and processes are generally applicable to any type of product development.

1.
T12
Special
Topics
10/5/17
11:15
Globalization
Testing
for
Cloud
Products
Presented
by:
Vincent
Truong
VMWare
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Vincent
Truong
VMWare
Vincent
Truong
has
eighteen
years
of
hands-­‐on
experience
in
globalization
quality
engineering.
He
is
a
globalization
quality
engineering
manager
at
VMware
and
has
been
managing
the
QE
efforts
for
VMware
products
under
the
cloud
management
business
unit
(CMBU)
for
five
years.
He
serves
as
a
globalization
program
manager,
helping
drive
the
globalization
efforts
for
CMBU.
Previously,
Vincent
spent
thirteen
years
at
Macromedia
and
Adobe,
helping
drive
the
globalization
QE
efforts
for
Flash,
Dreamweaver,
Contribute,
and
Flex.
Vincent
has
considerable
experience
working
with
Scrum
as
a
ScrumMaster
and
Product
Owner.
His
interests
include
globalization
testing,
agile
localization
methodologies,
and
machine
translation.

3. © 2016 VMware Inc. All rights reserved.© 2016 VMware Inc. All rights reserved.
Globalization Testing for Cloud Products
Vincent Truong
Globalization QE Manager

4. What is Globalization?
Definition from Wikipedia:
In computing, internationalization and localization are means of adapting computer
software to different languages, regional differences and technical requirements of a
target market (locale)
Internationalization is the process of designing a software application so that it can
potentially be adapted to various languages and regions without engineering changes
Localization is the process of adapting internationalized software for a specific region
or language by adding locale-specific components and translating text. Localization
(which is potentially performed multiple times, for different locales) uses the
infrastructure or flexibility provided by internationalization (which is ideally performed
only once, or as an integral part of ongoing development)
2

5. What is Globalization? (continued)
Globalization (g11n) = Internationalization (i18n) + Localization (L10n)
– Internationalization (i18n): architecting and coding for language/locale independence;
allows localization for target audiences that vary in culture, region, or language
• i18n Development- Modifying software to run in non-English environments
• i18n QE- Verifying that the software behaves as expected on non-English environments
and can accept non-English input/output
– Localization (L10n): adapting to specific languages/locales by translating text and adding
locale-specific formatting to images and style sheets
• Translation- Translating the English content into non-English languages for both product
UI and documentations
• Linguistic Review- Verifying syntactical and technical accuracy of translated strings
3

6. Why Globalize?
• Make your product functionally ready for international users
• Enable world-wide collaboration on through your product
• Give your international users the choice to use their country's dates,
numbers, currencies, keyboard layouts, text sorting, and search
• Let your international users use your application in their native
language
4

7. Some example: Disneyland site in English
5

8. Some example: Disneyland site in French
6

9. Buying tickets through the French site
7

10. Examples of i18n Issues
• Build installation on non-English OS is failed.
• Non-ASCII data blocks product functionality
• Different system locale
• Text file encoding
• Non-ASCII hostname of test environment blocks functionality
• Date time format bug:
8

11. I18n Character Corruption
9

12. Other examples of i18n Issues
– Hard-coded Strings
– Truncation and UI Layout Issues
10

13. About VMware
• Company overview: VMware virtualizes computing, from the data
center to the cloud to mobile devices
11

14. VMware products
12

15. Cloud Management Platform
13

16. A long time ago…Waterfall Development
• I18n testing waterfall process
14
Test/validate
i18n bug fixes
for patch
releases
Maintenance
Install and
conduct i18n
testing on the
product
Conduct l10n
testing on the
product
Conduct l10n
testing on the
product’s
docs
Verification
Create i18n
test cases
and
automation
Implementation
Evaluate the
i18n test
scope
Design

17. I18n testing
• Multiple test passes
• Huge test matrix to support (thousands of test cases, multiple Operating Systems, multiple
browsers, multiple locales to test against)
15

18. New products…VMware Cloud Services
16

19. VMware Cost Insight
17

20. New challenges
• Many new agile requirements
• Continuous delivery
• Different cadence for different teams
18

21. New way to deliver: Continuous Delivery Pipeline (Generalized)
I18n architecture
• Use a
common
framework
• Make use
of i18n
libraries
• Use CLDR
Developer
• I18n unit
tests
Review
• I18n code
scanning
• Pre-checkin
tests
• Peer code
review
Integrate/Build
• Integration
tests
• Full
regression
test
Staging
• Manual
i18n testing
• End to End
i18n testing
• Manual
l10n testing
• I18n
automation
• I18n
truncation
detector
• I18n
Heatmap
for bug
trending
Production
• I18n
Scorecard
• User
Acceptance
Testing
• Monitoring
issues
19

22. I18n architecture
20
•Use a common framework if possible for all of your products
•Make use of internationalization libraries such as ICU (International
Components for Unicode)
•Use the CLDR (Common Locale Data Repository)

23. Developer phase
21
I18n unit tests
•Non-ascii characters I/O issues are a important problem
•We find i18n bugs due to character corruption or loss of data
•This is tested early using superstrings containing various non-ascii
characters

24. Review phase
22
•Code scanning analysis to find i18n issues such as
• Hard coded strings
• Functions or methods for currencies, number formats, date
formats, sorting
• Check file encoding
•Pre-Checkin test to verify if this change cause issues with
existing functionality
•Lightweight peer Code Review

25. Staging phase
23
•Manual i18n testing: test and validate new stories for i18n impact
•End to End i18n testing: test the entire application stack.
Focused on what the user (or persona) sees/experiences
•Manual l10n testing: localization testing by native speakers to
validate correctness of translation in context
•I18n automation: validate existing i18n functionality
•I18n truncation detector: detect potential layout issues in
localized user interface
•I18n Heatmap: used for bug trend analysis, test optimization

26. I18n automation coverage
24

27. 25
I18n Heatmap concept
•The heatmap is a visual representation of a product and the i18n and
l10n bugs we have for each component/functionality in our bug
database
•The concept is simple: each square represent a feature and its color
changes as more bugs are logged against that feature (ie the density
of g11n bug increases). Bugs are also weighted by priority & severity
•The color for each square is generated dynamically from the number
of bugs for that functionality
•Each square’s size varies proportionally depending on the amount of
test cases the team has for it

28. 26
I18n Heatmap design

29. Globalization Testing process for Cloud Products
27
Sprint 4Sprint 3Sprint 1 Sprint 2
G11N QEs:
• Test new features for i18n
• If l10n is supported, test new features with pseudo-
localization or machine translation
• Validate existing features via automation
Continuous release
I18n Automation I18n Automation I18n Automation
User Acceptance tests
New features testing,
l10n enablement
New features testing,
l10n enablement
New features testing,
l10n enablement
New features testing,
l10n enablement
I18n Automation
End to End testsEnd to End tests End to End tests