From zero to one - How we evolved our test automation processes and mindset in Deep Security

From Zero to One
How we evolved our test automation processes and
mindset in Deep Security

Copyright 2018 Trend Micro Inc.2
Introduction

What is Deep Security?

Deep Security
• Deep Security is a collaborative effort between Taiwan and
Ottawa, with about 200 R&D people working on both sides

Deep Security
• Aside from the 12/13-hour communication gap, we also
have to work through the language and cultural barriers
• Since we are such a diverse team, we have to be able to
work together to release Deep Security

Automation in Deep Security
• I’ve been in the DS automation team since I joined Trend
about x years ago
• Over the years, automation has evolved from just a simple
test tool, to a team-wide integration of DevOps principles
• The revolution has been quite beautiful !

Deep Security Automation Framework

13000
test cases

release branch
feature branch

From Zero to One

Introducing the Rogue One team
• We are a 3-person team responsible for
– Install / upgrade automation
– Test case development
– Innovating automation practices
– Provide guidance for feature teams for anything related to automation
• In Ottawa, we have a counterpart called the Phoenix team

9.5
9.6
10.0
12.0
11.0
Established 2017

• Initial goal of our team is to revamp the existing install /
upgrade automation to make it more efficient, flexible and
user friendly
• So we made Lift.

Lift

Environment Creation
• The main parts of install / upgrade automation is to be able
to spin up a host on the fly for testing
• In the old system, this is done by having a pool of VMs (all
supported platforms has one VM), and keeping a CLEAN
snapshot of each VM
• Whenever a test is started, this VM is started up and
reverted to the CLEAN state
• Guess what kind of issues we had with this setup?

• Issues:
– Only one test can run at a time
– Long test times
– If something fails and needs attention, other tests in sequence
cannot run
– If something needs updating, all snapshots need to be updated

• Ansible is an open-source tool for environment
orchestration.
• Ansible can help you with config management, deployment
and task automation.

Ansible
• Why use Ansible?
– No Agent
• SSH / Python
• Fast deployment
– Use YAML
• Easy to understand
– Many available modules
– Open-source
– Small learning curve
– Ansible Tower

Test Case Frameworks

Test Cases Framework
• JUnit was the first test framework for DS
• It was initially made to create automation tests for the FW /
DPI feature
• Overtime, it grew to support all features of DS, including AM
• Without a proper PR process, it became too big, too dirty
(dependencies everywhere! !) and very hard to maintain
• So we made DSAF.

• One of the reasons why we moved from JUnit to DSAF, is
because of flaky tests
• A test could fail ONCE out of 10x
• It’s a big problem especially when you run Nightly tests on
multiple combinations of environments
• How do you classify that test?
– Do you fix it?
– Do you remove it?
– What do you do with it?

• Test Case Robustness

Promote Solution to QA / SDET

Promote Solution
• The most difficult point is not task, it’s
human.
– Team member change
– Knowledge
– I have my own tool, script. That’s enough.
– Data analysis
– Fix once, solve everywhere

Promote Solution - Seed team
TaiwanCanada
Feature
Team
Feature
Team
Feature
Team
Feature
Team
Feature
Team
Feature
Team

Promote Solution - Continue
• Pull
– New features
– Utility, Tool and Report
• Push
– End old service
– Manager

How – Code Review
• Code Review
– Collaborator
– GitHub

How – Code Review
• Change Size
• Review Time < 1 hour
• Coding Rule

How – Code Review

How – Code Review
• Comment
• Code as Document

How – Continue
• Wiki
– Guide
– Document
– Help
• Jira Cases

Resource / Performance /
Dashboard for Environment

What issues we encountered
• Guess and Try
• Evidence
• Performance

Why dashboard
• Take aim & Focus
• Ignore detail information
• 2 dimensions for human
• Learning curve
• Sporadic

Tools
• Grafana
• InfluxDB
• Prometheus
• Jenkins
• ElasticSearch

Targets
• Disk
• Memory
• CPU
• Job queue
• Garbage

Auto Alert to Auto Fix
• Disk -> remove some files
• Memory, CPU -> record, scale up
• Job queue -> record, scale up
• Garbage -> remove it!

@app.route('/alert', methods=['POST'])
def get_alert():
j = request.get_json(force=True)
print j
return "SUCCESS"

Dashboard for Test Result

Test Result on Testrail…..

Test Result on Jenkins……

Jenkinis API
Python

What about the volume of logs?

Elasticsearch is a real-time distributed and open
source full-text search and analytics engine. It is
accessible from RESTful web service interface and
uses schema less JSON (JavaScript Object Notation)
documents to store data.

Push test results and logs to ES

DSL Query (Domain Specific Language)
POST _search
{
"query": {
"bool": {
"should": [
{
"match": {
"job_name":
"Lift/Integration/DSM_Upgrade/win2012R2x64_sql12_10_0"
}
}
]
}
}
}

Visualize data from ES

Root Cause Analysis

Data
Collection
What’s the
question we
want to ask?
A query for
answer the
question
Data
Visualization
Dashboard for
each question
How we made them…

Our Works

• Zero-Click
• Mothra
• Rose

Zero-Click

Zero-Click
• Have you wondered if your branch is sane enough to be merged to the
main branch?
• Have you wondered if the branches being merged into main branch are
safe?
• Have you ever encountered a bug after PR caused by a merge from a
buggy or untested branch?
• How much do you trust your own code?

Zero-Click

Zero-Click
• Runs smoke tests before merging code to main branch
• Auto-triggered for every PR, every commit
• Takes about 40 mins to run
• Failed 0-click job means a BUG, and will BLOCK your PR
• A guarantee that your branch is shippable

SonarQube

Zero-Click
• Zero-Click was the first collaboration between the automation teams in Taiwan
and Ottawa
• We spent 2 weeks, working closely together in one room in the Ottawa office

Consolidated Test Result (Mothra)

Multi pipelines across Canada & Taiwan
• Inconsistent data on TestRail
• Overview on all test environments
• The most important factors
– Build number
– Platform
– Feature
– More?

Issues
• Detail and Log
• New database or Existed database?
• ElasticSearch or PostgreSQL?
• How to push test result?
– Command line on multi-platform
• Dashboard for everyone, by everyone

Solution

Platform name
• TestRail Config ID
• Mothra ID
• Mapping table on every environment
• Mapping table on Mothra
– Allow strings

Big Data

Relationship of Source Code and Test System (ROSE !)

What are the problems ROSE is
trying to solve?

1. Where can I find the automation pipeline and test suite to test my code
change?
2. Are the tests from Zero-Click enough to test my code change (PR)?
3. Is it possible to find the relationship between the source code and the
existing tests so that we can determine which tests should run for the code
changes?
4. What’s the code coverage rate of our automation testing?
5. What are the features we don’t have tests for?

Automation
Pipelines
Run Test
Collect Test
Coverage Data
Mapping
Coverage Data
with Tests
Developer
Create a Pull
Request
Analyze Code
Change and
Recommend
Covered Tests
Run
Recommended
Tests Before the
Pull Request
Pass the
Tests and
Finish the
Pull Request

How do we find the relationship
between the source code and testing
in DSM?

COCO
Listener
COCO
Web
Service
COCO
Portal
COCO
Reports

How do we learn from data
collected by COCO?

MATT (Machine Determined Testing)

Multi-label classification
DSM Installation Anti-Malware

COCO
Java
File
Java
File
Java
File
MATT
Java
File
Java
File
Java
File
Unknown
Java files
Predicted Result

Result

Result (Cont.)

Data-Driven Decision Making

Code Coverage
Data
Decision/Learning
Increase Code
Coverage
Testing

https://code.fb.com/developer-tools/predictive-test-selection/

Code
Coverage

https://arxiv.org/abs/1810.05286

Product
Automation Testing
System Automation
Data Collection
Data Visualization
Data Analysis
Machine Learning
Microservices

Q&A

From zero to one - How we evolved our test automation processes and mindset in Deep Security

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to From zero to one - How we evolved our test automation processes and mindset in Deep Security

Similar to From zero to one - How we evolved our test automation processes and mindset in Deep Security (20)

More from Jen-Chieh Ko

More from Jen-Chieh Ko (20)

Recently uploaded

Recently uploaded (20)

From zero to one - How we evolved our test automation processes and mindset in Deep Security