Gainsight's one stop application for analyzing product health. Test reporting, analysis, and notification application.

1. ©2016 Gainsight
10th Jan 2018
Gainsight CodeVigil

2. ©2016 Gainsight
Agenda
● Overview of Automation at Gainsight
● Need for CodeVigil
● What is CodeVigil
● Tech Stack
● Technical Architecture
● Demo and Walk-through
● Q & A

3. ©2016 Gainsight
Gainsight Automation By the Numbers
~20 Automation
Suites
5% Failure is
~800 Tests
~16K Tests
Every Day
~500K Tests
Run Every Month ~10% increase
month-on-month

4. ©2016 Gainsight
Why Code Vigil
• With the huge scale and growth of automation comes the challenges of monitoring and analyzing
our ever growing test suites
• With ~16k tests being run, we need a way to monitor test results real-time to reduce the turnaround
time
• Analyzing test failures and getting insights from the runs also becomes important as we attempt to
maintain this huge volume of tests
• This led to the need for an application to monitor and analyze our test automation suites

5. ©2016 Gainsight
Code Vigil
• The application provides real time execution status to reduce the turnaround time
• Provides Failure trends and Top failures across the application to assess application health
• Insights of failures from different dimensions such as defects, new failure,failure trend etc. which
will help in distinguishing between random failures and product bugs
• Theme and Team levels aggregation of tests,executions and progress
• Slack notifications to concerned POC’s everytime a suite status is at risk (be it sanity failures, high
failure count etc).

6. ©2016 Gainsight
CodeVigil: Real-time monitoring
• How frustrating is it when you do a test run only to realize that a setup environment issue caused
some or all of the tests to fail?
• Going through live execution logs may be possible but definitely not convenient or helpful
• Using a custom Maven plugin and Elastic Search to store and query results, CodeVigil provides
real-time execution results
• Allowing with real-time monitoring, CodeVigil also provides live analytics such as new failures,
slowest tests etc. while allowing to drill down to individual test cases and view screenshots and logs
specific to that test case
• No need to wait for the end of the run to analyze results!

7. ©2016 Gainsight
CodeVigil: Real-time monitoring

8. ©2016 Gainsight
CodeVigil: Analytics and Metrics
• CodeVigil provides insights on defects and metrics on automation trends and execution
• Insights are generated on different dimensions such as tests run over time, unstable tests, top
defects, test execution time etc.
• Insights are at a build/team/org level. Is the same defect or issue causing failures org wide? Let
CodeVigil identify it!
• At Gainsight it’s particularly useful with our BDD based suite where Gherkins steps may be used
across teams and suites and CodeVigil gives a high level look at its status org wide
• Having the data stored in Elasticsearch makes it inexpensive and efficient to store, query, and
aggregate

9. ©2016 Gainsight
CodeVigil: Slack Notifications
• With thousands of test cases, multiple jobs, multiple teams, multiple builds, how do
Directors/Managers/Leads and other stakeholders get a view on potential risks?
• While test run results are typically available to view as reports (as they are with CodeVigil as well), they
might not always be relevant and it might not always be feasible to do that on a regular basis.
• CodeVigil allows people to subscribe to a particular test job for Slack notifications
• Slack notifications are sent based on a daily schedule and are triggered if a job does not meet one of a
set of criteria
• The criteria includes having a failure rate > 5%, any sanity failures, high number of unstable tests etc.
• It takes into account that a single run could be invalid and instead takes an aggregate of the last few runs
for each test suite.

10. ©2016 Gainsight
CodeVigil: Slack Notifications

11. ©2016 Gainsight
CodeVigil: Tech Stack
• Elastic Search
• Spring boot
• Angular 4
• Material Design

12. ©2016 Gainsight
CodeVigil: Architecture

13. ©2016 Gainsight
Q&A