TSQA 2020: SAS ENTERPRISE: ACHIEVING WARP SPEED BY TRANSFORMING AUTOMATION INTO CONTINUOUS RELEASE

Company Confidential– For Internal Use Only
Copyright © SAS Institute Inc. All rights reserved.
SAS Enterprise
Transforming Automation into Continuous Release

Trivia Time!
@ken_dickinson https://www.linkedin.com/in/kennethdickinson/

What will we talk about today?
● The differences between testing in an
enterprise environment
● One particular way to adapt the testing of
monthly releases into the Continuous
Delivery of features
● The types of testing and why we identify
them
● How to change the tires on a moving car

Hi, Nice to Meet You
Ken Dickinson
Director, Research & Development
SAS Institute
● Tenures in both tiny startups and
multiple fortune 500 enterprises
● Led teams across testing,
development, operations, and product
disciplines.
● Advocate for humanity in automated
testing

What is SAS?
● Founded in 1976 by grad students at
NCSU
● One of the “godfathers” of modern
analytics
● Approaching 14,000 employees worldwide
● Totally not evil...
Welcome to the Institute

What is the Enterprise ?
● Not a startup
● What has made us successful before
will make us successful in the future
● Change “must” apply to all
● Hidden innovators waiting to be
discovered
● Firepower

What is DevOps?
You are.
● Prepare yourself to become the ideas
you believe in
● Your conviction and passion will be
observed and measured
● Know when you have the answers, but
also when you don’t
● Much of this is about advocacy. Never
sleep

Where do we begin?
● Customer Intelligence is an AWS
based platform for Marketing
Analytics
● SAS but also SaaS
● Our only cloud native offering (for
now)
● Used by hundreds of enterprise
customers to connect to their own
customers
You guessed it, Marketing Analytics.

In the Beginning
● Single monolithic platform deploying all at
once
● Monthly scheduled releases
● Planned manual testing phases
● No production monitoring or observability
outside of the AWS console

An Ambitious Mission
● Turn monthly releases into Continuous
Delivery
● Normalize automation first culture
● Unify and streamline automation development
● Shrink the barrier of entry to automation
contribution
● Test everything, everywhere
● Begin pushing the AI envelope

This is Software Engineering, not Testing
● We are not goalies or quality police
● Quality is everyone’s responsibility
● We implement quality into systems
● Sometimes we write tests, sometimes we
don’t
● Our mandate is to produce valuable changes
to our customers that are of the highest quality
possible

Ok Fine, But How?
Unit Tests
Contract Tests
API Tests
Acceptance Tests
Exploratory | Usability Tests
Functional Automation
Security and Vulnerability
Performance and Load Testing

Why Layers?
• Small, Purposeful, Libraries. Separating tests into distributed layers encourages the
development of maintainable libraries with clear purpose.
• Clear Expectations. When tests are defined on a given layer, they inherit clearly defined
expectations of what the do and don’t do.
• Reporting & Triage. Testing on specific layers of a stack create more meaningful
reporting and actionable logging

Layers of Testing
Unit Testing
Designed to validate behavior on a function or method level. Can and should use mocked
dependencies. Run as part of the build process for the service they represent. Measured in
Code Coverage, which will be held above a defined percentage.
public void TestPhoneValidator()
{
string goodPhone = "(123) 555-1212";
string badPhone = "555 12"
PhoneValidator validator = new PhoneValidator();
Assert.IsTrue(validator.IsValid(goodPhone));
Assert.IsFalse(validator.IsValid(badPhone));
}

Layers of Testing
Consumer Driven Contracts
Designed to validate schema of API contracts between dependencies and dependents
(some call this consumers and providers). Runs as part of the build process, should be
included for every service level dependency.
Given "there is no user called Mary"
When "creating a user with username Mary"
POST /users { "username": "mary", email: "...", ...
}
Then
Expected Response is 200 OK

Layers of Testing
Contract Testing Tooling
One straightforward way to manage consumer driven contracts is with an
open source framework called PACT. I also recommend a plug in called
PACT Broker to visualize it.

Layers of Testing
API Testing
An extension of contract testing, some will know this as integration testing. Designed to
validate the functionality of our private and public API’s. Targets request/response behavior
of each endpoint using a shared requests library across all teams.
When "creating a user with a username with 21 characters"
POST /users { "username": "thisisalooongusername", email: "...", ... }
Then
Expected Response is 400 Bad Request
Expected Response body is { "error": "username cannot be more than 20
characters" }

Layers of Testing
Acceptance Testing
Designed to validate the acceptance criteria and the general health of our user experience.
Written in Behavior Driven Development (BDD) language and only expressed thru customer
accessible interfaces. Gating deployments of released features.

What is the Value of BDD
Single source of truth. When requirements are built around user behavior, there can be a
single point of reference for stakeholder, developer, and tester.
Human readability. Scenarios can be well understood by both technical and non technical
audiences.
Built for collaboration. Because they are written in a language that all contributing parties
understand, cross discipline collaboration becomes far more natural.
Designed for authenticity. By driving tests from customer facing interfaces, BDD scenarios
allow us to test our applications the way our users do.

Layers of Testing
Functional Automation
Similar to Acceptance Testing, but of lesser impact and non gating. This automation will be
written to accommodate corner cases and negative approaches. Written in BDD language
and only expressed thru customer accessible interfaces. Executed with each deployment but
non gating.

Layers of Testing
Performance and Load Testing
Designed to discover and communicate inefficiencies in performance that can impact the
customer experience or stability of the system. These are not functional performance
standards, which are run inside of pipeline and considered part of functional testing.

Too Much Noise
A word about ephemeral environments and CI/CD
● Performance testing can be difficult in a constantly changing continuous environment
with unpredictable usage patterns
● To solve for this we leverage ephemeral environments for each deployable unit (usually
a service)
● Requests and responses are measured in
isolation, automatically creating defects
when exceeding a standard deviation
● Environments are created and destroyed
automatically as part of the deployment
process

Layers of Testing
Passive IAST
Passive or “Self Sufficient” Interactive Application Security Testing (IAST) will be run in every
environment and hardness the full potential of our preexisting functional automation to
detect vulnerabilities in our application security.

Layers of Testing
Exploratory Testing
Designed to expose defects based upon exploratory testing charters. Can be executed by
anyone. Will not be run inside of any deployment pipeline but is often a requirement for
production exposure of a Feature in its definition of done (DoD)

What goes into a charter?
● Start with with “Why”
● Guide them to the neighborhood, not the
house
● Identify personas
● Maximize creativity
● Focus on customer experience
More art than science

Test Executions
Conditional Executions
Most will know this as “Tagging”. Tests will be decorated for each independently deployable
component of the system, usually by service or client. Often more than one. Tags will also
be introduced to identify priority levels, which will dictate their behavior inside of our quality
gates.

Diversity in Executions
● Not in the business of managing selenium
grids
● Parallelization will allow you to cast a wider
net
● 3rd party executors will allow you to focus
on your own business logic
● One more reason to keep tests lean

Designing for CI/CD
What do I have to consider?
● Keep test scenarios agnostic of environment.
Each scenario should be reusable across all of
your environments
● Plan for instability. Continuously delivered
applications are often less stable that their
traditional counterparts. Embrace this and see
quality from a holistic point of view.
● Be mindful of your runtimes. Especially with
gated tests. CI/CD derives a significant portion of
its value from its short feedback loop, be careful
not to undermine that.
● Automate everything you can. The CI/CD delivery
model is designed for automation. Introducing
manual steps will mitigate its purpose.

Exit Criteria and Definition of Done
How do we know its ready?
We establish a Definition of Done (DoD) as a set of criteria necessary for the exposure of a
feature to customers in the Production environment. A feature is not considered finished
until it is exposed to all customers in the Production environment.
Example:
• All Acceptance Level tests have been written,
executed, and are passing in all environments
• Exploratory testing has been performed
• Performance metrics are within nominal margins
• Necessary Documentation and Training has been
completed

Decoupling Deployment from Release
Launching Darkly
● Every user story is associated to an Epic or
Feature which is initially flagged off for all
users outside of the development team.
Features will be built iteratively along with
the automated tests that validate their
health.
● When a feature has met its obligation to the
Definition of Done it becomes eligible for
exposure to customers in Production.

Dark Launches
What is the impact?
● Granular exposure. You will gain a new level of control over
your customer experience. You will decide who sees what
and when.
● Introduce a new level of readiness. When you gain the
ability to control the exposure of features, you can then
“release” them when your entire organization
is ready. From Support to PSO to Documentation to
Marketing, you can ensure an entire company level of
readiness.
● Remove development bottlenecks. Dark launches allow
for your development and testing organization to
continue work on new features while other parts of your
organization complete their contributions.

Dark Launches
But stay vigilant
● Management is key. Make sure your organization is prepared
to manage feature flags at scale. The absence of this is the
most common risk of process failure. Tools exist to assist
with this but responsibility will ultimately fall onto your
people.
● So is training. Have a plan in place that empowers your
workforce to learn these concepts. Dark launches will impact
parts of your organization beyond
development, so be sure to include them too.
● Start small and measure everything. There is no detail
involved in dark launches that requires an aburbt or total
switch. Start with a single feature and practice. Your
observations will become valuable lessons to your
implementation. Act upon them and follow your
instincts!

Key Takeaways
● Invest your time and energy into
Automation. These skills will be critical
in all things continuous.
● Design for portability. Your tests will
provide the most value when they can be
run in any environment, long running or
not.
● Become as collaborative as possible.
Contributing to shared goals and shared
approaches across teams and offices will
be critical to our success.

Key Takeaways
● Value innovation. Ensure that your
standardization is not limiting your
implementation. Design for growth and
experimentation.
● Find your balance and be patient. Each
application and each organization will
discover its own unique recipe
standardization and liberty.
● Encourage a culture of distributed
contribution. Be ready for code
contributions from a distributed base.
Remember, trading a little structure for a
lot of enthusiasm can be totally worth it!

sas.com
Interested in our work?
http://2.sas.com/TSQA
or
https://sas.com/careers

TSQA 2020: SAS ENTERPRISE: ACHIEVING WARP SPEED BY TRANSFORMING AUTOMATION INTO CONTINUOUS RELEASE

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Similar to TSQA 2020: SAS ENTERPRISE: ACHIEVING WARP SPEED BY TRANSFORMING AUTOMATION INTO CONTINUOUS RELEASE

Similar to TSQA 2020: SAS ENTERPRISE: ACHIEVING WARP SPEED BY TRANSFORMING AUTOMATION INTO CONTINUOUS RELEASE (20)

Recently uploaded

Recently uploaded (20)

TSQA 2020: SAS ENTERPRISE: ACHIEVING WARP SPEED BY TRANSFORMING AUTOMATION INTO CONTINUOUS RELEASE