SlideShare a Scribd company logo
INTEGRATION TESTING AS
VALIDATION AND MONITORING
Melissa Benua
Senior Backend Engineer
PlayFab, Inc
STARWEST 2015
The challenge: Monitoring SaaS products
Software as a service is exploding, and so is testing complexity:
1. Not enough just to run tests at build time, now you also need need
deploy-time integration tests and continuous network monitoring
2. Every layer of tests adds
complexity & maintenance costs
3. There are a limited amount
of engineer-hours in the day
4. Engineers want to use their time
with maximum efficiency
Time spent writing the same tests over again is time that could be spent doing more
interesting and important stuff!
EXISTING OPTIONS
Commercial products you can buy now!
3
Cloud Monitoring Services
Providers:
• Keynote
• Gomez
• Pingdom
Pros:
• Lightweight
• Integrated alerting
• Public vs. private status pages
Cons:
• Difficult to manage multiple contributors
• Can’t do complex checks easily (log in a user and verify inventory)
• Can get expensive or require enterprise contracts
Hosted Monitoring Services
Providers:
• Sensu
• System Center Operations
Manager (SCOM)
• Nagios
Pros:
• Extremely powerful
• Older technology
Cons:
• Complex to set up
• Single centralized server
• Overkill for many services hosted in the cloud
OUR APPROACH
Do it the PlayFab way!
October 5, 2015 PlayFab Confidential 6
Our Solution
1. Author one set of HTTP-level tests
• Same as how clients connect
• Self-contained and self-initializing
• Repeatable and reliable
2. Deploy tests both within the build environment
and within the monitoring cloud
3. Collect data from tests into one central location
4. Present data for use by both devops and customers
Pros:
• Efficient use of engineering resources
• VM hosting bill is very small
• Can run complex tests without
worrying about maintainability
Cons:
• Pipeline requires some maintenance
• Requires knowing how to use two
different clouds
• Must be able to do test setup from
within a different ecosystem
Our solution, cont’d
Goals:
• Minimize number of lines of code
duplicated per functional piece
• Reliable & trustworthy reporting
• Affordable cost
• Adequate geo-location
• Very low maintenance time cost
• Easy to access
• More free time for engineering!
Limitations:
• Smaller # of monitoring leaf nodes (~10 instead of ~100 or ~1000)
• Vulnerable to gaps in dev logic
• Not as straightforward to set up
• Monitoring is only as good as your testing!
TESTING SCENARIOS
One of these may look familiar!
October 5, 2015 PlayFab Confidential 9
Scenario A – RESTful API
Sample characteristics:
• Custom service in Java layered on Apache
• Private hosting
• Tests via Junit
• Authenticates using private login
• Connects to several different backend
services (mongodb, sql, analytics,
queueing, etc)
Scenario B – MVC Website
Sample characteristics:
• Built on .net MVC
• Hosted in Azure
• Testing via custom harness
• Authenticates using OAuth and Facebook
• Backends into locally-hosted SQL server
Scenario C - PlayFab
Characteristics:
• JSON API built on C# + management website
• https://api.playfab.com/documentation
• Hosted in Windows on AWS
• Tests via VSTest
• Many moving parts
• Game server hosting
• Client versus server authentication
• Third-party purchasing and auth providers
• Various backend data sources
IMPLEMENTING OUR SOLUTION
How to wire up the pipeline!
October 5, 2015 PlayFab Confidential 13
Architecture
14
Build Server
 Compiles code
 Runs tests
Production
Deploys
Web Server
 Collects Data
Web Site
 Displays Data
Developer
Writes
Tests
Europe
Microsoft Azure
US-West US-East Asia
Amazon Web Services
Submits Code
Utilized Tech
Test Framework
• VSTest or Junit or custom executor
• Must output a predictable, machine-readable format
(.TRX from VSTest comes with an XSD for easy parsing)
Execution + Communication Layer
• Consul or custom cross-DC chatter
• Consul API is in many languages, easy to secure and simple configure
• Regularly executes the test executable
• Shares test results as ‘service health checks’ across DCs
Custom Data Bridge
• Transform test framework output into Consul input
Picking Monitoring Tests
October 5, 2015 PlayFab Confidential 16
Full App
Integration Test
Suite
Internal Service A
Test Suite
Library Unit
Test Suite
Integration
Suite
Internal Service B
Test Suite
Integration
Suite
Picking Monitoring Tests, con’t
Must-haves:
• Happen at same layer clients access (HTTP,
generally)
• Cover key ‘P0’ functionality areas
• Cover areas with lots of ‘moving parts’
Nice-to-haves:
• All exposed APIs
• Third-party integrations
• Full success-testing run
Ideal world:
• Full integration test suite
Scenario Must-Have Test Cases
REST API
• Login/Authenticate
• Logout
• One test per downstream
service
• Stretch: one test per API
MVC Website
• One test per login method
(OAuth, Facebook)
• Key pages
• Basic SQL coverage
Deployment Pipeline
The fewer manual steps the better!
Sample flow:
Submit Code
to Repo
CI Runs
Build
CI Runs Tests
Deployment
Packages
Created
Tests
Deployed
into Monitor
Cloud
Storage
Cloud
Storage
Distributes
to VMs
Monitoring Cloud
Any cloud will do!
Number of regions is important
• Azure has https://azure.microsoft.com/en-us/regions/#services
• AWS has http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region
VMs can be teeny – no need for heavy compute or memory usage
Test Execution Frequency
How complex is it to run your tests?
• Run a simple executable?
• Have to download a lot of data?
• Long setup phase?
• How long does a full test pass take?
Periodic execution (every N seconds)
Faster is better! Pingdom ‘free’ tier is every 15 minutes per check
Ideal range is between 30 seconds and 5 minutes
Be careful not to drown your ‘real traffic’
• Test traffic hiding problems with real users is a legitimate issue!
• Try to stay under 10% of total traffic if possible
Collecting Results
Execute Tests
Put machine-readable test results into collator
• Consul accepts Datacenter, CaseName, Pass/Warn/Fail, Note (we store latency)
• Agents may be updated using SDK or direct to HTTP interface
• Example: http://localhost:8500/v1/agent/check/pass/mytestcase
• Full HTTP API: https://www.consul.io/docs/agent/http.html
Small adapter program reads test results and outputs to Consul Agent
(SDK or HTTP)
Output!
Alerting
Ideal to hear about outages as a push rather than a pull
Determine what ‘failure’ means to you
• Balance between false alarms and missing real alarms
Many options!
• Post alerts into VictorOps for paging
• Send email from monitoring website
• Send push notification through your cloud
Questions?
Melissa Benua
mbenua@gmail.com
https://www.linkedin.com/in/mbenua
http://www.slideshare.net/MelissaBenua
APPENDIX
Technical Details and Sample Config
October 5, 2015 PlayFab Confidential 26
Partial Consul Configuration
{
"datacenter": "prd-uswest1",
"retry_join_wan": [ “west.cloudapp.net",
“east.cloudapp.net" ],
"server": true,
"service": {
"name": "pfmonitor",
"checks": [
{
"script":
"C:WindowsSystem32WindowsPowerShellv1.0powershell
.exe -file c:runtests.ps1",
"interval": "120s"
}
]
}
}
Consul Commands
Full HTTP API: https://www.consul.io/docs/agent/http.html
Add a health check:
$body =
{
"ID": “mypath",
"Name": "Path Works",
"Notes": "Checking uptime and latency",
"HTTP": "http://my.service.com/path",
"TTL": "45s"
}
• Invoke-WebRequest http://localhost:8500/v1/agent/check/register -Body $body
List the health checks:
• Invoke-WebRequest http://localhost:8500/v1/health/checks/myservice
[
{
"Node": "somenode",
"CheckID": “mypath",
"Name": “Path Works",
"Status": "passing",
},
]
Consul Commands
Update a health check:
• Can add ?note=foo to pass details like latency
• Invoke-WebRequest
http://localhost:8500/v1/agent/check/pass/mypath
• Invoke-WebRequest
http://localhost:8500/v1/agent/check/warn/mypath
• Invoke-WebRequest
http://localhost:8500/v1/agent/check/fail/mypath

More Related Content

What's hot

Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable product
Julian Simpson
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
hugo lu
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
Dennis van der Stelt
 
Continuous delivery applied
Continuous delivery appliedContinuous delivery applied
Continuous delivery appliedMike McGarr
 
Continuous Integration and Continuous Deployment in Enterprise scenario
Continuous Integration and Continuous Deployment in Enterprise scenarioContinuous Integration and Continuous Deployment in Enterprise scenario
Continuous Integration and Continuous Deployment in Enterprise scenario
Davide Benvegnù
 
CI with TFS
CI with TFSCI with TFS
CI with TFS
Esteban Garcia
 
Continuous Integration and Builds
Continuous Integration and BuildsContinuous Integration and Builds
Continuous Integration and Builds
Bhavin Javia
 
Taking your version control to a next level with TFS and Git
Taking your version control to a next level with TFS and GitTaking your version control to a next level with TFS and Git
Taking your version control to a next level with TFS and Git
Alexander Vanwynsberghe
 
Continuous integration
Continuous integrationContinuous integration
Continuous integrationamscanne
 
Why NXTware Remote for Jenkins
Why NXTware Remote for JenkinsWhy NXTware Remote for Jenkins
Why NXTware Remote for Jenkins
ecubemarketing
 
Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration
Amazon Web Services
 
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
Sencha
 
Deploy applications with TFS Build
Deploy applications with TFS BuildDeploy applications with TFS Build
Deploy applications with TFS Build
Gian Maria Ricci
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014
David Funaro
 
Sencha Roadshow 2017: What's New in Sencha Test
Sencha Roadshow 2017: What's New in Sencha TestSencha Roadshow 2017: What's New in Sencha Test
Sencha Roadshow 2017: What's New in Sencha Test
Sencha
 
Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
David Funaro
 
Learn Key Insights from The State of Web Application Testing Research Report
Learn Key Insights from The State of Web Application Testing Research ReportLearn Key Insights from The State of Web Application Testing Research Report
Learn Key Insights from The State of Web Application Testing Research Report
Sencha
 
#speakgell - Continuous Integration in iconnect360
#speakgell - Continuous Integration in iconnect360#speakgell - Continuous Integration in iconnect360
#speakgell - Continuous Integration in iconnect360
Derek Chan
 
Standardizing Jenkins with CloudBees Jenkins Team
Standardizing Jenkins with CloudBees Jenkins TeamStandardizing Jenkins with CloudBees Jenkins Team
Standardizing Jenkins with CloudBees Jenkins Team
Deborah Schalm
 

What's hot (20)

Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable product
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Continuous delivery applied
Continuous delivery appliedContinuous delivery applied
Continuous delivery applied
 
Continuous Integration and Continuous Deployment in Enterprise scenario
Continuous Integration and Continuous Deployment in Enterprise scenarioContinuous Integration and Continuous Deployment in Enterprise scenario
Continuous Integration and Continuous Deployment in Enterprise scenario
 
Continuous Integration 101
Continuous Integration 101Continuous Integration 101
Continuous Integration 101
 
CI with TFS
CI with TFSCI with TFS
CI with TFS
 
Continuous Integration and Builds
Continuous Integration and BuildsContinuous Integration and Builds
Continuous Integration and Builds
 
Taking your version control to a next level with TFS and Git
Taking your version control to a next level with TFS and GitTaking your version control to a next level with TFS and Git
Taking your version control to a next level with TFS and Git
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Why NXTware Remote for Jenkins
Why NXTware Remote for JenkinsWhy NXTware Remote for Jenkins
Why NXTware Remote for Jenkins
 
Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration Continuous Delivery, Continuous Integration
Continuous Delivery, Continuous Integration
 
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
Sencha Roadshow 2017: Sencha Upgrades - The Good. The Bad. The Ugly - Eva Luc...
 
Deploy applications with TFS Build
Deploy applications with TFS BuildDeploy applications with TFS Build
Deploy applications with TFS Build
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014
 
Sencha Roadshow 2017: What's New in Sencha Test
Sencha Roadshow 2017: What's New in Sencha TestSencha Roadshow 2017: What's New in Sencha Test
Sencha Roadshow 2017: What's New in Sencha Test
 
Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
 
Learn Key Insights from The State of Web Application Testing Research Report
Learn Key Insights from The State of Web Application Testing Research ReportLearn Key Insights from The State of Web Application Testing Research Report
Learn Key Insights from The State of Web Application Testing Research Report
 
#speakgell - Continuous Integration in iconnect360
#speakgell - Continuous Integration in iconnect360#speakgell - Continuous Integration in iconnect360
#speakgell - Continuous Integration in iconnect360
 
Standardizing Jenkins with CloudBees Jenkins Team
Standardizing Jenkins with CloudBees Jenkins TeamStandardizing Jenkins with CloudBees Jenkins Team
Standardizing Jenkins with CloudBees Jenkins Team
 

Viewers also liked

Commander's Intent: Managing Through Uncertainty
Commander's Intent: Managing Through UncertaintyCommander's Intent: Managing Through Uncertainty
Commander's Intent: Managing Through Uncertainty
James Gwertzman
 
PlayFab Advanced Cloud Script
PlayFab Advanced Cloud ScriptPlayFab Advanced Cloud Script
PlayFab Advanced Cloud Script
Thomas Robbins
 
Say hello to the new PlayFab!
Say hello to the new PlayFab!Say hello to the new PlayFab!
Say hello to the new PlayFab!
Thomas Robbins
 
Owning Web Performance with PhantomJS 2 - Fluent 2016
Owning Web Performance with PhantomJS 2 - Fluent 2016Owning Web Performance with PhantomJS 2 - Fluent 2016
Owning Web Performance with PhantomJS 2 - Fluent 2016
Wesley Hales
 
No Free Lunch: Transactions in Online Games
No Free Lunch: Transactions in Online GamesNo Free Lunch: Transactions in Online Games
No Free Lunch: Transactions in Online Games
James Gwertzman
 
Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...
Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...
Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...
ProductNation/iSPIRT
 
Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
Behind the Scenes: Deploying a Low-Latency Multiplayer Game GloballyBehind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
James Gwertzman
 
Building the pipeline for FUN - Game Development
 Building the pipeline for FUN - Game Development Building the pipeline for FUN - Game Development
Building the pipeline for FUN - Game Development
FaunaFace, Inc
 
The Future is Operations: Why Mobile Games Need Backends
The Future is Operations: Why Mobile Games Need BackendsThe Future is Operations: Why Mobile Games Need Backends
The Future is Operations: Why Mobile Games Need Backends
James Gwertzman
 
Epic Fails in LiveOps
Epic Fails in LiveOpsEpic Fails in LiveOps
Epic Fails in LiveOps
James Gwertzman
 
Deploying a Low-Latency Multiplayer Game Globally: Loadout
Deploying a Low-Latency Multiplayer Game Globally: Loadout Deploying a Low-Latency Multiplayer Game Globally: Loadout
Deploying a Low-Latency Multiplayer Game Globally: Loadout
Amazon Web Services
 

Viewers also liked (11)

Commander's Intent: Managing Through Uncertainty
Commander's Intent: Managing Through UncertaintyCommander's Intent: Managing Through Uncertainty
Commander's Intent: Managing Through Uncertainty
 
PlayFab Advanced Cloud Script
PlayFab Advanced Cloud ScriptPlayFab Advanced Cloud Script
PlayFab Advanced Cloud Script
 
Say hello to the new PlayFab!
Say hello to the new PlayFab!Say hello to the new PlayFab!
Say hello to the new PlayFab!
 
Owning Web Performance with PhantomJS 2 - Fluent 2016
Owning Web Performance with PhantomJS 2 - Fluent 2016Owning Web Performance with PhantomJS 2 - Fluent 2016
Owning Web Performance with PhantomJS 2 - Fluent 2016
 
No Free Lunch: Transactions in Online Games
No Free Lunch: Transactions in Online GamesNo Free Lunch: Transactions in Online Games
No Free Lunch: Transactions in Online Games
 
Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...
Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...
Nuts and Bolts of Marketing & selling SaaS products to US customers from Indi...
 
Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
Behind the Scenes: Deploying a Low-Latency Multiplayer Game GloballyBehind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
Behind the Scenes: Deploying a Low-Latency Multiplayer Game Globally
 
Building the pipeline for FUN - Game Development
 Building the pipeline for FUN - Game Development Building the pipeline for FUN - Game Development
Building the pipeline for FUN - Game Development
 
The Future is Operations: Why Mobile Games Need Backends
The Future is Operations: Why Mobile Games Need BackendsThe Future is Operations: Why Mobile Games Need Backends
The Future is Operations: Why Mobile Games Need Backends
 
Epic Fails in LiveOps
Epic Fails in LiveOpsEpic Fails in LiveOps
Epic Fails in LiveOps
 
Deploying a Low-Latency Multiplayer Game Globally: Loadout
Deploying a Low-Latency Multiplayer Game Globally: Loadout Deploying a Low-Latency Multiplayer Game Globally: Loadout
Deploying a Low-Latency Multiplayer Game Globally: Loadout
 

Similar to Integration Testing as Validation and Monitoring

Modernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-ArchitectModernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-Architect
DevOps.com
 
.NET microservices with Azure Service Fabric
.NET microservices with Azure Service Fabric.NET microservices with Azure Service Fabric
.NET microservices with Azure Service Fabric
Davide Benvegnù
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
Amazon Web Services
 
AWS Summit Auckland - Application Delivery Patterns for Developers
AWS Summit Auckland - Application Delivery Patterns for DevelopersAWS Summit Auckland - Application Delivery Patterns for Developers
AWS Summit Auckland - Application Delivery Patterns for Developers
Amazon Web Services
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOps
Eklove Mohan
 
Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)
CIVEL Benoit
 
Cerberus_Presentation1
Cerberus_Presentation1Cerberus_Presentation1
Cerberus_Presentation1CIVEL Benoit
 
Next-gen Automation Framework
Next-gen Automation FrameworkNext-gen Automation Framework
Next-gen Automation Framework
Kumar Swamy Dontamsetti
 
Performance testing with 100,000 concurrent users in AWS
Performance testing with 100,000 concurrent users in AWSPerformance testing with 100,000 concurrent users in AWS
Performance testing with 100,000 concurrent users in AWS
Matthias Matook
 
Presentation on 3 Pillars of DevOps - Kovair DevOps
Presentation on 3 Pillars of DevOps - Kovair DevOpsPresentation on 3 Pillars of DevOps - Kovair DevOps
Presentation on 3 Pillars of DevOps - Kovair DevOps
Kovair
 
DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...
DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...
DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...
Amazon Web Services
 
Continuous Integration with Amazon ECS and Docker
Continuous Integration with Amazon ECS and DockerContinuous Integration with Amazon ECS and Docker
Continuous Integration with Amazon ECS and Docker
Amazon Web Services
 
Application Delivery Patterns for Developers - Technical 401
Application Delivery Patterns for Developers - Technical 401Application Delivery Patterns for Developers - Technical 401
Application Delivery Patterns for Developers - Technical 401
Amazon Web Services
 
Continuous Integration Testing for SAP
Continuous Integration Testing for SAPContinuous Integration Testing for SAP
Continuous Integration Testing for SAP
Worksoft
 
Cloud-based Test Microservices JavaOne 2014
Cloud-based Test Microservices JavaOne 2014Cloud-based Test Microservices JavaOne 2014
Cloud-based Test Microservices JavaOne 2014
Shelley Lambert
 
15-factor-apps.pdf
15-factor-apps.pdf15-factor-apps.pdf
15-factor-apps.pdf
Nilesh Gule
 
Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...
Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...
Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...
Startupfest
 
Anatomy of a Build Pipeline
Anatomy of a Build PipelineAnatomy of a Build Pipeline
Anatomy of a Build Pipeline
Samuel Brown
 
Patterns and practices for building enterprise-scale HTML5 apps
Patterns and practices for building enterprise-scale HTML5 appsPatterns and practices for building enterprise-scale HTML5 apps
Patterns and practices for building enterprise-scale HTML5 apps
Phil Leggetter
 
Continuous Integration for OpenVMS with Jenkins
Continuous Integration for OpenVMS with JenkinsContinuous Integration for OpenVMS with Jenkins
Continuous Integration for OpenVMS with Jenkins
ecubemarketing
 

Similar to Integration Testing as Validation and Monitoring (20)

Modernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-ArchitectModernizing Testing as Apps Re-Architect
Modernizing Testing as Apps Re-Architect
 
.NET microservices with Azure Service Fabric
.NET microservices with Azure Service Fabric.NET microservices with Azure Service Fabric
.NET microservices with Azure Service Fabric
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
 
AWS Summit Auckland - Application Delivery Patterns for Developers
AWS Summit Auckland - Application Delivery Patterns for DevelopersAWS Summit Auckland - Application Delivery Patterns for Developers
AWS Summit Auckland - Application Delivery Patterns for Developers
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOps
 
Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)Cerberus : Framework for Manual and Automated Testing (Web Application)
Cerberus : Framework for Manual and Automated Testing (Web Application)
 
Cerberus_Presentation1
Cerberus_Presentation1Cerberus_Presentation1
Cerberus_Presentation1
 
Next-gen Automation Framework
Next-gen Automation FrameworkNext-gen Automation Framework
Next-gen Automation Framework
 
Performance testing with 100,000 concurrent users in AWS
Performance testing with 100,000 concurrent users in AWSPerformance testing with 100,000 concurrent users in AWS
Performance testing with 100,000 concurrent users in AWS
 
Presentation on 3 Pillars of DevOps - Kovair DevOps
Presentation on 3 Pillars of DevOps - Kovair DevOpsPresentation on 3 Pillars of DevOps - Kovair DevOps
Presentation on 3 Pillars of DevOps - Kovair DevOps
 
DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...
DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...
DevOps on Windows: How to Deploy Complex Windows Workloads | AWS Public Secto...
 
Continuous Integration with Amazon ECS and Docker
Continuous Integration with Amazon ECS and DockerContinuous Integration with Amazon ECS and Docker
Continuous Integration with Amazon ECS and Docker
 
Application Delivery Patterns for Developers - Technical 401
Application Delivery Patterns for Developers - Technical 401Application Delivery Patterns for Developers - Technical 401
Application Delivery Patterns for Developers - Technical 401
 
Continuous Integration Testing for SAP
Continuous Integration Testing for SAPContinuous Integration Testing for SAP
Continuous Integration Testing for SAP
 
Cloud-based Test Microservices JavaOne 2014
Cloud-based Test Microservices JavaOne 2014Cloud-based Test Microservices JavaOne 2014
Cloud-based Test Microservices JavaOne 2014
 
15-factor-apps.pdf
15-factor-apps.pdf15-factor-apps.pdf
15-factor-apps.pdf
 
Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...
Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...
Jeremy Edberg (MinOps ) - How to build a solid infrastructure for a startup t...
 
Anatomy of a Build Pipeline
Anatomy of a Build PipelineAnatomy of a Build Pipeline
Anatomy of a Build Pipeline
 
Patterns and practices for building enterprise-scale HTML5 apps
Patterns and practices for building enterprise-scale HTML5 appsPatterns and practices for building enterprise-scale HTML5 apps
Patterns and practices for building enterprise-scale HTML5 apps
 
Continuous Integration for OpenVMS with Jenkins
Continuous Integration for OpenVMS with JenkinsContinuous Integration for OpenVMS with Jenkins
Continuous Integration for OpenVMS with Jenkins
 

Recently uploaded

Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
Bhaskar Mitra
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
Cheryl Hung
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Thierry Lestable
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
Fwdays
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 

Recently uploaded (20)

Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 

Integration Testing as Validation and Monitoring

  • 1. INTEGRATION TESTING AS VALIDATION AND MONITORING Melissa Benua Senior Backend Engineer PlayFab, Inc STARWEST 2015
  • 2. The challenge: Monitoring SaaS products Software as a service is exploding, and so is testing complexity: 1. Not enough just to run tests at build time, now you also need need deploy-time integration tests and continuous network monitoring 2. Every layer of tests adds complexity & maintenance costs 3. There are a limited amount of engineer-hours in the day 4. Engineers want to use their time with maximum efficiency Time spent writing the same tests over again is time that could be spent doing more interesting and important stuff!
  • 4. Cloud Monitoring Services Providers: • Keynote • Gomez • Pingdom Pros: • Lightweight • Integrated alerting • Public vs. private status pages Cons: • Difficult to manage multiple contributors • Can’t do complex checks easily (log in a user and verify inventory) • Can get expensive or require enterprise contracts
  • 5. Hosted Monitoring Services Providers: • Sensu • System Center Operations Manager (SCOM) • Nagios Pros: • Extremely powerful • Older technology Cons: • Complex to set up • Single centralized server • Overkill for many services hosted in the cloud
  • 6. OUR APPROACH Do it the PlayFab way! October 5, 2015 PlayFab Confidential 6
  • 7. Our Solution 1. Author one set of HTTP-level tests • Same as how clients connect • Self-contained and self-initializing • Repeatable and reliable 2. Deploy tests both within the build environment and within the monitoring cloud 3. Collect data from tests into one central location 4. Present data for use by both devops and customers Pros: • Efficient use of engineering resources • VM hosting bill is very small • Can run complex tests without worrying about maintainability Cons: • Pipeline requires some maintenance • Requires knowing how to use two different clouds • Must be able to do test setup from within a different ecosystem
  • 8. Our solution, cont’d Goals: • Minimize number of lines of code duplicated per functional piece • Reliable & trustworthy reporting • Affordable cost • Adequate geo-location • Very low maintenance time cost • Easy to access • More free time for engineering! Limitations: • Smaller # of monitoring leaf nodes (~10 instead of ~100 or ~1000) • Vulnerable to gaps in dev logic • Not as straightforward to set up • Monitoring is only as good as your testing!
  • 9. TESTING SCENARIOS One of these may look familiar! October 5, 2015 PlayFab Confidential 9
  • 10. Scenario A – RESTful API Sample characteristics: • Custom service in Java layered on Apache • Private hosting • Tests via Junit • Authenticates using private login • Connects to several different backend services (mongodb, sql, analytics, queueing, etc)
  • 11. Scenario B – MVC Website Sample characteristics: • Built on .net MVC • Hosted in Azure • Testing via custom harness • Authenticates using OAuth and Facebook • Backends into locally-hosted SQL server
  • 12. Scenario C - PlayFab Characteristics: • JSON API built on C# + management website • https://api.playfab.com/documentation • Hosted in Windows on AWS • Tests via VSTest • Many moving parts • Game server hosting • Client versus server authentication • Third-party purchasing and auth providers • Various backend data sources
  • 13. IMPLEMENTING OUR SOLUTION How to wire up the pipeline! October 5, 2015 PlayFab Confidential 13
  • 14. Architecture 14 Build Server  Compiles code  Runs tests Production Deploys Web Server  Collects Data Web Site  Displays Data Developer Writes Tests Europe Microsoft Azure US-West US-East Asia Amazon Web Services Submits Code
  • 15. Utilized Tech Test Framework • VSTest or Junit or custom executor • Must output a predictable, machine-readable format (.TRX from VSTest comes with an XSD for easy parsing) Execution + Communication Layer • Consul or custom cross-DC chatter • Consul API is in many languages, easy to secure and simple configure • Regularly executes the test executable • Shares test results as ‘service health checks’ across DCs Custom Data Bridge • Transform test framework output into Consul input
  • 16. Picking Monitoring Tests October 5, 2015 PlayFab Confidential 16 Full App Integration Test Suite Internal Service A Test Suite Library Unit Test Suite Integration Suite Internal Service B Test Suite Integration Suite
  • 17. Picking Monitoring Tests, con’t Must-haves: • Happen at same layer clients access (HTTP, generally) • Cover key ‘P0’ functionality areas • Cover areas with lots of ‘moving parts’ Nice-to-haves: • All exposed APIs • Third-party integrations • Full success-testing run Ideal world: • Full integration test suite
  • 18. Scenario Must-Have Test Cases REST API • Login/Authenticate • Logout • One test per downstream service • Stretch: one test per API MVC Website • One test per login method (OAuth, Facebook) • Key pages • Basic SQL coverage
  • 19. Deployment Pipeline The fewer manual steps the better! Sample flow: Submit Code to Repo CI Runs Build CI Runs Tests Deployment Packages Created Tests Deployed into Monitor Cloud Storage Cloud Storage Distributes to VMs
  • 20. Monitoring Cloud Any cloud will do! Number of regions is important • Azure has https://azure.microsoft.com/en-us/regions/#services • AWS has http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region VMs can be teeny – no need for heavy compute or memory usage
  • 21. Test Execution Frequency How complex is it to run your tests? • Run a simple executable? • Have to download a lot of data? • Long setup phase? • How long does a full test pass take? Periodic execution (every N seconds) Faster is better! Pingdom ‘free’ tier is every 15 minutes per check Ideal range is between 30 seconds and 5 minutes Be careful not to drown your ‘real traffic’ • Test traffic hiding problems with real users is a legitimate issue! • Try to stay under 10% of total traffic if possible
  • 22. Collecting Results Execute Tests Put machine-readable test results into collator • Consul accepts Datacenter, CaseName, Pass/Warn/Fail, Note (we store latency) • Agents may be updated using SDK or direct to HTTP interface • Example: http://localhost:8500/v1/agent/check/pass/mytestcase • Full HTTP API: https://www.consul.io/docs/agent/http.html Small adapter program reads test results and outputs to Consul Agent (SDK or HTTP)
  • 24. Alerting Ideal to hear about outages as a push rather than a pull Determine what ‘failure’ means to you • Balance between false alarms and missing real alarms Many options! • Post alerts into VictorOps for paging • Send email from monitoring website • Send push notification through your cloud
  • 26. APPENDIX Technical Details and Sample Config October 5, 2015 PlayFab Confidential 26
  • 27. Partial Consul Configuration { "datacenter": "prd-uswest1", "retry_join_wan": [ “west.cloudapp.net", “east.cloudapp.net" ], "server": true, "service": { "name": "pfmonitor", "checks": [ { "script": "C:WindowsSystem32WindowsPowerShellv1.0powershell .exe -file c:runtests.ps1", "interval": "120s" } ] } }
  • 28. Consul Commands Full HTTP API: https://www.consul.io/docs/agent/http.html Add a health check: $body = { "ID": “mypath", "Name": "Path Works", "Notes": "Checking uptime and latency", "HTTP": "http://my.service.com/path", "TTL": "45s" } • Invoke-WebRequest http://localhost:8500/v1/agent/check/register -Body $body List the health checks: • Invoke-WebRequest http://localhost:8500/v1/health/checks/myservice [ { "Node": "somenode", "CheckID": “mypath", "Name": “Path Works", "Status": "passing", }, ]
  • 29. Consul Commands Update a health check: • Can add ?note=foo to pass details like latency • Invoke-WebRequest http://localhost:8500/v1/agent/check/pass/mypath • Invoke-WebRequest http://localhost:8500/v1/agent/check/warn/mypath • Invoke-WebRequest http://localhost:8500/v1/agent/check/fail/mypath