1. Automation Test Framework
Continue Improvement
Welcome Presentation
Presented by Christina Geng

2. AGENDA
❶ Case Introduction
❷ Testing Strategy Evolution
❸ Case ROI Analysis
❹ Success factor
❺ Revelation

3. CASE INTRODUCTION
Title： 7 Times Evolution of Test Automation Framework under JS Product
for Efficiency Improvement
Target：
❶ Build a robust test automation framework
❷ Manage test cases efficiently
❸ Reduce test case running time and maintenance cost
❹ Shared framework under scrum development

4. WHO AND WHAT?
Data Analysis Visualization
• reporting, online analytical processing
• analytics
• data mining
• process mining
• event processing
• business performance management
• benchmarking
• text mining
• predictive analytics

5. WHO AND WHAT?
Data Analysis Visualization
• reporting, online analytical processing Programming language:
Javascript/Java
• analytics
• data mining Provide service:
• process mining
The Visualization Service is a software module which provides various type of visualization objects that can be used in different
• event processing applications on different platforms (Web, Mobile and Desktop).
• business performance management
• benchmarking Visualization types:
• A basic visual component like chart (Bar, Column Pie, Line, etc ), Geo Map, Gauge, Table or Grid.
• text mining • An advanced visual component like (Tree map, Candlestick, Tagcloud)
• predictive analytics • Above visual component with interactions and animations
• Combination of multiple visual components, it could be composed by same type of visual components or different type of
component.

6. AGENDA
❶ Case Introduction
❷ Testing Strategy Evolution
❸ Case ROI Analysis
❹ Success factor
❺ Revelation

7. What’s more…
2012.12
Developer environment
2012.09 2012.10
Full Automation
Test Link
2011.12 2012.032012.06
Jenkins
Data Driven
Selenium
2011.6
We have it!

8. VERSION-1
Pie Bar Line Backgrounditle
T Legend Layout Tooltip Theme ValueAxis Animation
Pie y y y y y y
Bar y y y y y y y y y
Line y y y y y y y y y
Scheduler: ASTEC
TC Management: Excel
TC: test page
Test tool: SilkTest
Test
page
SilkTest

9. VERSION-1
+
1. First automation framework setup
- writing test case
2. Overview functional test case coverage
3. Reduce testing overlap for shared
component
HT
Test
Test ML
page
page
resource
SilkTest maintenance
the relation between TC and record

10. What’s more…
2012.12
Developer environment
2012.09 2012.10
Full Automation
Test Link
2011.12 2012.032012.06
Jenkins
Data Driven
Selenium
2011.6
We have it!

11. VERSION-2
Scheduler: ASTEC
TC Management: Excel/XML (test manager)
TC: test page
Test tool: Selenium Test Driver
Other tools
• Auto generate test plan from Excel to xml DIV DIV
• Test Manager Test
DIV
page
DIV
Selenium

12. VERSION-2
+
1. Reduce test pages
- test driver
2. Reduce resource running cost
3. Test steps turns to be keyword driven in xml
case unit
4. Easy case migration
5. Write test case in .xls, auto generate .xml DIV
DIV DIV
DIV
Test
Test
DIV
DIV
page
page
DIV
DIV
Selenium
test data redundancy
test case sync

13. What’s more…
2012.12
Developer environment
2012.09 2012.10
Full Automation
Test Link
2011.12 2012.03 2012.06
Jenkins
Data Driven
Selenium
2011.6
We have it!

14. VERSION-3
Unified Test Driver
Abstract Test Data
And How?
Wrapper API
Discard Test Page

15. VERSION-3
backgroud
color
visible
legend
position
verticalOffset
horizontalOffset testUtil.generateChart(viz/line; TestData.a1a1m1m1)
visible
labelFont
labelColor
formatString testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])
initChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
layout
padding
tooltip
enable updateChartsByProperties([['layout.padding',[50,50,50,50]],['legend.visible',false]]);
customization
text
mainValue
subValue
title
main sub
Selectability
mode
deSelectale
defaultSelectedItems
xAxisMember
yAxisMember
bar
barMarkerSize
measureMember
Wrapper API

16. VERSION-3
testUtil.generateChart(viz/line; TestData.a1a1m1m1; ['legend.visible',false])
public
mySQL
private
testUtil.generateChart(viz/line; TestData.a1a1m1m1; legendVisibleTrue)
Abstract Test Data

17. VERSION-3
titleVisibleTrue = [['title.visible',true]];
titleVisibleFalse = [['title.visible',false]];
titleNormal = [['title.visible',true],['title.text',"Set
customer title"]];
titleLong =
[['title.visible',true],['title.text',testData_Text.Lon
g]];
titleAlignmentCenter =
[['title.visible',true],['title.alignment',"center"]];
titleAlignmentLeft =
[['title.visible',true],['title.alignment',"left"]];
titleAlignmentRight =
[['title.visible',true],['title.alignment',"right"]];
legendVisibleTrue = [['legend.visible',true]];
legendVisibleFalse = [['legend.visible',false]];
legendHierarchicalTrue = [['legend.visible',true],
['legend.isHierarchical', true]];
legendPositionLeft = [['legend.visible',true],
['legend.position',"left"]];
legendPositionRight = [['legend.visible',true],
['legend.position',"right"]];
legendPositionTop = [['legend.visible',true],
['legend.position',"top"]];
legendPositionBottom = [['legend.visible',true],
['legend.position',"bottom"]];
<testsetcase id="test.workflow.layout.line.showTitle" enabledtype="enabled"/>
<testsetcase id="test.workflow.layout.line.hideLegend" enabledtype="enabled"/>
<testsetcase id="test.workflow.layout.line.hideAxis" enabledtype="enabled"/>
<testsetcase id="test.workflow.layout.line.showAxisTitle" enabledtype="enabled"/> Discard Test Page

18. VERSION-3
selenium1
selenium2
{StartBrowser}
IE, Chrome, Firefox (selenium2)
RemoteWebDriver webDriver = null;
webDriver = new FirefoxDriver();
ImageComparer
Robot()
Unified Test Driver

19. VERSION-3
selenium1
{StartBrowser}
Safari (selenium1)
selenium2 SeleniumServer seleniumServer = new SeleniumServer();
seleniumServer.boot();
selenium = new DefaultSelenium("localhost", 4444, "*safariproxy "+
safariLocation,urlRoot);
ImageComparer CommandExecutor executor = new SeleneseCommandExecutor(selenium);
DesiredCapabilities dc = new DesiredCapabilities();
Robot() webDriver = new RemoteWebDriver(executor, dc);
Unified Test Driver

20. VERSION-3
selenium1
selenium2 {TakeSnapshot}
Point p = this.getElementOffset(targetElement);
Dimension d = this.getElementSize(targetElement);
BufferedImage bi = new Robot().createScreenCapture(p.x, p.y, d.width, d.height);
ImageComparer
Robot()
Unified Test Driver

21. VERSION-3
selenium1
selenium2 {FileCompare}
cmd.exe ImageComparer.exe –file filepath1 filepath2 – threshhold 100
ImageComparer
Robot()
Unified Test Driver

22. What’s more…
2012.12
Developer environment
2012.09 2012.10
Headless
Test Link
2011.12 2012.03 2012.06
Full Automation
Data Driven
Selenium
2011.6
We have it!

23. VERSION-4
Table Text
Tooltip Selection Animation
Button Scroll CSS
Checkbox Hover Theme
Link Gesture
Gesture Upload
Dropdown Download

24. VERSION-4
Selection

25. 揭示研发管理白金定律，分享那些激动人心的创新与变革，使得团队获得过多源动力与更大的推动力！

26. VERSION-4
var eventObj = {
type : originalEvent.type,
timeStamp : originalEvent.timeStamp,
pageX : originalEvent.pageX,
pageY : originalEvent.pageY
};
eventRecord.push(eventObj);
var handler = function(event){
var clickEvt = document.createEvent('MouseEvents');
clickEvt.initMouseEvent(event.type, event.pageX, event.pageY…);
d3.select('.main .plot')[0][0].dispatchEvent(clickEvt)
};
testAPI_lasso_heatmap = [
{"type":"mouseover","timeStamp":1348739963751,"pageX":126,"pageY":94},
{"type":"mouseover","timeStamp":1348739963751,"pageX":485,"pageY":94},
{"type":"mousemove","timeStamp":1348739963766,"pageX":485,"pageY":208},
{"type":"mousemove","timeStamp":1348739963773,"pageX":126,"pageY":208}
]

27. VERSION-4
Record
Replay

28. VERSION-4
diff image location
diff image folder start to check
p4 folder
replace BM directly

29. What’s more…
2012.12
Developer environment
2012.09 2012.10
Headless
Test Link, Jenkins
2011.12 2012.03 2012.06
Full Automation
Data Driven
Selenium
2011.6
We have it!

30. VERSION-5
How to arrange test case?

31. VERSION-5
TC (.xls/.xml)
import
Test
Link
export
Executable TP
save Result log

32. VERSION-5
Continue Integration

33. VERSION-5
.xml TC mngmt
.xls
Test Case Pool
Manual Auto
3. Build info 5. result write back
4. get test plan
2.TP/Platform/build info Test Driver
CI Server Test Executor
5. result log Selenium2 Java/C++
1. Trigger
4. get assets
libs resources
Build Machine Web Server Code repository
Where we are.

34. VERSION-5
.xml TC mngmt
.xls
Test Link Test Case Pool
Manual Auto
3. Build info 5. result write back
4. get test plan
2.TP/Platform/build info Test Driver
CI Server
5. result log
Java Tool
Test Executor
Selenium2 Java/C++
Jenkins
1. Trigger
4. get assets
libs resources
Build Machine Web Server Code repository
Perforce
Where we are.

35. What’s more…
2012.12
Developer environment
2012.09 2012.10
Headless
Test Link, Jenkins
2011.12 2012.03 2012.06
Full Automation
Data Driven
Selenium
2011.6
We have it!

36. Engineering

37. VERSION-6
http://casperjs.org/

38. VERSION-6
Paint
Layout Display
PhantomJS (Headless Webkit) http://phantomjs.org

39. VERSION-6
config
Browser?
CasperJS IE/Safari/FF/Chrome
TP Read JS list Open browser
PhantomJS: RunJS (test step)
1. open webkit
2. runJS
….
3. Take snapshot
4. close Take Snapshot
Result validation
not complete not complete
Save result log
Reporting
Test driver inner logic

40. And that’s OK

41. What’s more…
2012.12
Developer environment
2012.09 2012.10
Headless
Test Link, Jenkins
2011.12 2012.03 2012.06
Full Automation
Data Driven
Selenium
2011.6
We have it!

42. VERSION-7
.java TC Management
Test Case Pool
local
CI Server Test Executor
Test Driver
Selenium2 Java/C++
per check-in
trigger
result log
sync up Casper/PhantomJS
Build Machine
get resources P4 server
Test Executor

43. VERSION-7
.java TC Management
Test Case Pool
Eclipse Plug-in
local
CI Server Test Executor
Test Driver
Selenium2 Java/C++
per check-in
trigger
result log
Jenkins
Build Machine
sync up Casper/PhantomJS
get resources P4 server
Test Executor
Perforce

44. CASE ROI ANALYSIS
Functional Test: BAT (nightly), FA
1) Test machine
2) Assets (util, BM…)
1. Set up environment 3) TP
4) Libs
1) Browser compatibility
2. Test case execution
4.5h
47min
2) Run in headless
1) Email sending
3. Test result reporting 2) Sonar integrated with CI
1) BM update
4. Maintenance 2) TC management
1000+ cases

45. SUCCESS FACTOR ❶ ❷❸❹
Focus

46. SUCCESS FACTOR ❶ ❷❸❹
Team work

47. SUCCESS FACTOR ❶ ❷❸❹
Continue Improvement

48. SUCCESS FACTOR ❶ ❷❸❹
Learn 360

49. OUR REVELATION
不被传统的测试方法束缚；从产品用到的技术找到突破；持续改进的思想；不断尝试的理念

50. CONTRIBUTOR
Xiaolu, Ye
Christina, Geng
Helen, Jiang
Chen, Zhou
Peipei, Jiang
Coco, Dai
Grace, Zhao
Minghao, Lu
Kent, Wang
Izzie, Wang
Phil, Xu
NO
THE
SUPPORTED FROM
Daoyu, Wei
END Li, Zhao
Jimmy, Yang
Alex, Feng
Alex, Su
Jiwei, Li
... …