This document discusses building an automated acceptance testing framework for financial applications. It describes a large hedge fund's need to reduce regression testing time and effort from 6-8 months to daily releases. SQS created the Safia framework to automate acceptance tests and treat them as executable documentation. This reduced test suites from over 600 to under 100 and man-days of effort from over 600 to under 20 within a year. The framework uses templates and archetypes to quickly create automated test cases for different financial instruments. It provides benefits like increased collaboration, accelerated learning and adoption, and reduced costs of ownership. The document emphasizes designing self-documenting, business-focused tests as specifications rather than scripts.
How to Say Yes to Self-Service in the Cloud and Become an IT HeroRightScale
We’ve all seen the trend: your internal customers, developers, and business users alike, want self-service access to IT resources they need. And increasingly they want those resources in the cloud via a single management console. Cloud computing offers them the agility they need to respond to market demands and the ability to scale up while keeping costs low.
IT can become a hero by providing a self-service portal where internal customers can provision their own cloud resources while IT can maintain the control and visibility an enterprise requires. The happy result: enterprise agility and enterprise control.
During this session, learn how to empower your internal customers to provision the necessary cloud resources when they need them but also ensure that they are well within IT approved guidelines.
To help illustrate the effectiveness of this approach, our presenters walk you through real-world examples and the overall impact on their organizations.
1. Create an IT vending machine with consistent and reproducible processes.
2. Manage multiple IT environments through a single-pane-of-glass.
3. Use cost planning and forecasting to fine-tune and understand cloud spend.
4. Discover reporting and auditing tools to ensure compliance.
Участник конкурса на "Лучшую муниципальную практику поддержки предпринимательства и улучшения инвестиционного климата" в рамках Премии "Бизнес-Успех" по Приволжскому федеральному округу
премиябизнесуспех.рф
How to Say Yes to Self-Service in the Cloud and Become an IT HeroRightScale
We’ve all seen the trend: your internal customers, developers, and business users alike, want self-service access to IT resources they need. And increasingly they want those resources in the cloud via a single management console. Cloud computing offers them the agility they need to respond to market demands and the ability to scale up while keeping costs low.
IT can become a hero by providing a self-service portal where internal customers can provision their own cloud resources while IT can maintain the control and visibility an enterprise requires. The happy result: enterprise agility and enterprise control.
During this session, learn how to empower your internal customers to provision the necessary cloud resources when they need them but also ensure that they are well within IT approved guidelines.
To help illustrate the effectiveness of this approach, our presenters walk you through real-world examples and the overall impact on their organizations.
1. Create an IT vending machine with consistent and reproducible processes.
2. Manage multiple IT environments through a single-pane-of-glass.
3. Use cost planning and forecasting to fine-tune and understand cloud spend.
4. Discover reporting and auditing tools to ensure compliance.
Участник конкурса на "Лучшую муниципальную практику поддержки предпринимательства и улучшения инвестиционного климата" в рамках Премии "Бизнес-Успех" по Приволжскому федеральному округу
премиябизнесуспех.рф
Marketing para prohibir la conservacióncongresombian
Participación de Iván Trujano, Biólogo Especialista en Manejo de Recursos Naturales en el Centro Regional para la Conservación Biológica, en el 1er congreso multidisciplinario de bienestar animal México 2015 Ciudad de México.
Manatíes en parques y acuarios de México y su acercamiento con la estrategia ...congresombian
Participación del Dr. Benjamín Morales Vela, Investigador Titular del Colegio de la Frontera, en el 1er congreso multidisciplinario de bienestar animal México 2015, Ciudad de México.
AUG NYC - May 24 talks.
1. Atlassian Test Case Management Options and Integrations - Blaine Pryce & Bob Ho, Column Technologies
Today’s Software Economy requires a high degree of automation to make any DevOps initiative successful. The sheer velocity of DevOps is driving the need for a more integrated approach to the QA and testing processes. Blaine & Bob will explore the Atlassian Test Case Management options and an integrated technology approach that can streamline the QA and testing processes for your organization. The featured integration use case will highlight integrating Test Automation/ Test Case Management/Test Data Management and Bug Tracking
2. How to Customize, Automate and Expand the Power of JIRA - Ethan Foulkes, cPrime
Everyone knows Jira is great for development and we are seeing it used more and more for building non-development related workflows. Come and learn how easy it is to go beyond the out of box capabilities and hear Ethan speak about how to bend Jira to support any business process.
“Machine Learning in Production + Case Studies” by Dmitrijs Lvovs from Epista...DevClub_lv
Epistatica is a data science spin-off from VIA SMS R&D SERVICES, searching its niche in European markets.
Dmitrijs is head of credit risk with VIA SMS R&D SERVICES, a fintech company, and member of the board at Epistatica, holds a PhD from RAS Institute for Information Transmission Problems and analyzed data for over 12 years.
Cloud computing business framework
Victor Chang, Leeds Beckett University
International conference on
“DATA, DIGITAL BUSINESS MODELS, CLOUD COMPUTING AND ORGANIZATIONAL DESIGN”
24-25 November 2014 ,
Université Paris –Sud
Accelerate Your Sap Testing with BquriousyadavSusheel
Enabling Business Readiness of SAP implementation with bqurious.
Due to the challenges imposed by conventional test automation approach, it is very critical to make well-considered choice for the optimum division of the available time and money to scale test automation process to higher maturity levels, ensuring high efficiency and in turn higher ROI from this investment.
Bqurious is platform agnostic Test automation & management platform accessible from any geography. Instead of weeks or months, bqurious can be setup in 5 minutes and ready to use.
Test Metrics in Agile: A Powerful Tool to Demonstrate ValueTechWell
Most understand that an agile development and testing approach improves quality and reduces risks in our projects. In some companies and culture however, there are skeptics. Is the move to agile—and therefore agile testing—really beneficial? Join Iuliia Zavertailo for a closer look at a Scandinavian insurance company that started with one manual tester and within three years moved toward opening a large test center in the Baltic. Behind this story were many small steps of demonstrating testing's value to the client through a well-defined set of agile metrics which quantitatively supported the importance and value of testing. Iullia gives examples of key performance indicators—test coverage, defect open and close rates, issues reported by customers, and regression test suite duration—and provides a roadmap for building a test metrics framework. She then discusses tools that support the agile test framework, provides guidance on how to analyze test statistics, and offers ways to present the facts that interest clients most.
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...Amazon Web Services
NAB has publicly stated its intent to transform technology and operations investing a further $1.5B over the next 3 years. A major element is a mass migration of applications to cloud to help unlock the cost, risk and agility benefits, and enable their teams to deliver better customer outcomes. During this session, you will hear how NAB formulated their strategy, why NAB partnered with AWS, lessons learned through its journey to the Cloud, and how this shift has brought value to both its employees and customers and will continue to yield gains to its business.
AWS Cloud Adoption Framework and WorkshopsTom Laszewski
The presentation covers the AWS Cloud Adoption Framework (CAF). AWS CAF helps organization accelerate their cloud adoption journey. The framework includes six perspectives - business, people, governance, security, operations, and platform. These six perspectives are used during CAF Envision, Alignment, and Cloud Capability Assessment workshops to enable the art of the possible, identify and mitigate organizational and technology impediments, and score the cloud capabilities of an organization.
Marketing para prohibir la conservacióncongresombian
Participación de Iván Trujano, Biólogo Especialista en Manejo de Recursos Naturales en el Centro Regional para la Conservación Biológica, en el 1er congreso multidisciplinario de bienestar animal México 2015 Ciudad de México.
Manatíes en parques y acuarios de México y su acercamiento con la estrategia ...congresombian
Participación del Dr. Benjamín Morales Vela, Investigador Titular del Colegio de la Frontera, en el 1er congreso multidisciplinario de bienestar animal México 2015, Ciudad de México.
AUG NYC - May 24 talks.
1. Atlassian Test Case Management Options and Integrations - Blaine Pryce & Bob Ho, Column Technologies
Today’s Software Economy requires a high degree of automation to make any DevOps initiative successful. The sheer velocity of DevOps is driving the need for a more integrated approach to the QA and testing processes. Blaine & Bob will explore the Atlassian Test Case Management options and an integrated technology approach that can streamline the QA and testing processes for your organization. The featured integration use case will highlight integrating Test Automation/ Test Case Management/Test Data Management and Bug Tracking
2. How to Customize, Automate and Expand the Power of JIRA - Ethan Foulkes, cPrime
Everyone knows Jira is great for development and we are seeing it used more and more for building non-development related workflows. Come and learn how easy it is to go beyond the out of box capabilities and hear Ethan speak about how to bend Jira to support any business process.
“Machine Learning in Production + Case Studies” by Dmitrijs Lvovs from Epista...DevClub_lv
Epistatica is a data science spin-off from VIA SMS R&D SERVICES, searching its niche in European markets.
Dmitrijs is head of credit risk with VIA SMS R&D SERVICES, a fintech company, and member of the board at Epistatica, holds a PhD from RAS Institute for Information Transmission Problems and analyzed data for over 12 years.
Cloud computing business framework
Victor Chang, Leeds Beckett University
International conference on
“DATA, DIGITAL BUSINESS MODELS, CLOUD COMPUTING AND ORGANIZATIONAL DESIGN”
24-25 November 2014 ,
Université Paris –Sud
Accelerate Your Sap Testing with BquriousyadavSusheel
Enabling Business Readiness of SAP implementation with bqurious.
Due to the challenges imposed by conventional test automation approach, it is very critical to make well-considered choice for the optimum division of the available time and money to scale test automation process to higher maturity levels, ensuring high efficiency and in turn higher ROI from this investment.
Bqurious is platform agnostic Test automation & management platform accessible from any geography. Instead of weeks or months, bqurious can be setup in 5 minutes and ready to use.
Test Metrics in Agile: A Powerful Tool to Demonstrate ValueTechWell
Most understand that an agile development and testing approach improves quality and reduces risks in our projects. In some companies and culture however, there are skeptics. Is the move to agile—and therefore agile testing—really beneficial? Join Iuliia Zavertailo for a closer look at a Scandinavian insurance company that started with one manual tester and within three years moved toward opening a large test center in the Baltic. Behind this story were many small steps of demonstrating testing's value to the client through a well-defined set of agile metrics which quantitatively supported the importance and value of testing. Iullia gives examples of key performance indicators—test coverage, defect open and close rates, issues reported by customers, and regression test suite duration—and provides a roadmap for building a test metrics framework. She then discusses tools that support the agile test framework, provides guidance on how to analyze test statistics, and offers ways to present the facts that interest clients most.
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...Amazon Web Services
NAB has publicly stated its intent to transform technology and operations investing a further $1.5B over the next 3 years. A major element is a mass migration of applications to cloud to help unlock the cost, risk and agility benefits, and enable their teams to deliver better customer outcomes. During this session, you will hear how NAB formulated their strategy, why NAB partnered with AWS, lessons learned through its journey to the Cloud, and how this shift has brought value to both its employees and customers and will continue to yield gains to its business.
AWS Cloud Adoption Framework and WorkshopsTom Laszewski
The presentation covers the AWS Cloud Adoption Framework (CAF). AWS CAF helps organization accelerate their cloud adoption journey. The framework includes six perspectives - business, people, governance, security, operations, and platform. These six perspectives are used during CAF Envision, Alignment, and Cloud Capability Assessment workshops to enable the art of the possible, identify and mitigate organizational and technology impediments, and score the cloud capabilities of an organization.
AWS re:Invent 2016: Fueling Migration: Shortcutting your Application Portfoli...Amazon Web Services
For many large enterprises, gathering data on your applications and developing a detailed business case for an all-in migration to AWS can be a time consuming activity. Instead of waiting on a full analysis to be completed, a global healthcare company decided to take a shortcut by reinventing the portfolio assessment process in a brilliantly simple manner. This new streamlined approach enabled them to immediately take advantage of the increased speed and innovative capabilities of AWS. In this session, IT directors and enterprise architects can learn these techniques to quickly identify which of their own applications fit cloud-friendly patterns; including, how to build a prioritized and multi-threaded migration queue that maximizes velocity and business value.
The Tester’s Role: Balancing Technical Acumen and User AdvocacyTechWell
Ten years ago, many of us started our careers in testing, generally moving from a different internal role. It was common for people who were product users to be hired to jump start their technical career. Now, we see the growth of tester positions that require coding experience or a computer science degree. Melissa Tondi discusses the changing landscape of the role of testers, the challenges when hiring developers with no previous testing experience, and a way to shift the pendulum back to balance technical acumen with a user advocacy role. Melissa leads a thoughtful discussion on what makes a good tester, how we can continue to promote our profession, and how to accentuate the value testers bring to organizations. She identifies factors that caused the test/QA role to become mainstream and how it shifted to become more technically focused. Melissa helps fill in the gaps with a test strategy that balances time for the test team to continue supporting the development efforts while equally emphasizing user advocacy tests. She presents recommendations you can take back to your team to achieve the right balance for your organization.
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld
VMworld 2013
Venkat Gopalakrishnan, VMware
Kurt Milne, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Automation Hub Best practices - using right KPIs to build a strong business c...Cristina Vidu
Wondering how to prioritize the most impactful ideas to increase your program’s ROI? Learn how to customize the assessments to:
Collect the right information from your business users
Compute the right KPIs relevant for the decision-making process
Accurately prioritize the automation opportunities pipeline
Confidently report to your stakeholders based on transparent and relevant evaluation
Target audience: CoE leads, Business Analysts, Business Process Owners
👩💻 Speaker: Teodora Hiritiu, Senior Product Manager - Automation Hub @UiPath
Watch Automation Hub Best Practices Session 2: Large scale roll-outs:
https://youtu.be/QH1XGH6aJOY
3. A story about building an executable documentation system
• Large hedge fund COTS
implementation
• Crippling regression test burden
Born out of
necessity
• 6-8 month cycle daily release
• 80 times defect reductionResults:
• and again
• Same patterns automation
platform
Then did it
again…
Safia– Archetypes and Templates, a voyage in automating acceptance
3
4. The problem quantified
Real data from the field
40
67.5
100
78
23
15
12
5 3
0
28
64
151
217
311
371
393
479
545
601
0 0
25
106
284
371
393
479
545
601
0
100
200
300
400
500
600
700
0
20
40
60
80
100
120
February April August November February April May June July August
#TestSuites
ManDaysEffort
Month
Time taken for Release Regression testing
Man Days Effort for Regression Test # Test Suites # Automated Suites
4
5. The confidence problem!
Where it came from
First Approach to Acceptance Testing:
UAT in a phase, after a period of development
5
8. Becoming acceptance driven…
Building Integrity in
8
Develop
Ready User
Story
Customer
Verification
Specify tests
/ examples
Stakeholder(s) & BA
Testers
Developers
Backlog
Demo /
Showcase
Automate
acceptance
tests
Session
Based
Testing
9. The initial effect of automated acceptance testing
Real data from the field
40
67.5
100
78
23
15
12
5 3
0
28
64
151
217
311
371
393
479
545
601
0 0
25
106
284
371
393
479
545
601
0
100
200
300
400
500
600
700
0
20
40
60
80
100
120
February April August November February April May June July August
#TestSuites
ManDaysEffort
Month
Time taken for Release Regression testing
Man Days Effort for Regression Test # Test Suites # Automated Suites
9
10. Knowledge and experience to design a framework
• binding tests to investment apps
A set of design patterns
• from business specialists
Inject knowledge
• easy to customise
Vanilla acceptance tests
• tests as specifications
Living documentation
What we gained from this
10
11. Illogical Architecture Diagram
How Safia Works
House
Adaptor
System
Adaptor
Trading system
Maps SAFIA terminology to
system terminology
e.g.
Calypso, Beauchamps, Summi
t
SAFIA automates at the
server level via the API
Safia fixtures
API
Fitnesse fixtures
Maps investment house
terminology to SAFIA
terminology e.g.
Trade Capture = Trade Entry
11
12. e.g.
FX, Bond, Equities
IRS, FRA, CDS
e.g.
Run a P&L report
Update a legal entity
Amend SDI
Amend fees
Creating automated test cases using Safia
How Safia Works
Test Archetype
“Test Function”
Create a trade
Instrument
Template
IRS template
Test
Create a valid IRS trade
Apply Template &
Archetype
12
14. It brings benefits on different levels
Why use a framework like this
14
Benefits
Increased
collaboration
Accelerating
implementation
& learning
Anyone can
interact with it
Teaching &
coaching aid
Reduced Cost
of Ownership
16. The path to faster release
Real data from the field
40
67.5
100
78
23
15
12
5 3
0
28
64
151
217
311
371
393
479
545
601
0 0
25
106
284
371
393
479
545
601
0
100
200
300
400
500
600
700
0
20
40
60
80
100
120
February April August November February April May June July August
#TestSuites
ManDaysEffort
Month
Time taken for Release Regression testing
Man Days Effort for Regression Test # Test Suites # Automated Suites
16
17. Tests as specs, the principles of test design
How we wrote the tests
17
What makes a
good acceptance
test?
Self-documenting
About business
intentions
Concise and
granularA specification
not a script
Anyone can
understand it
Based on testable
statements
18. ~One third is system understanding
Reducing the Cost of Ownership
18
19. Points to consider when automating
19
Theory ‘aint practice
Start early & iterate frequently
Distil and simplify
Tests or Documentation?
DON’T PANIC !!!
21. Questions
Thank you for listening
21
Please evaluate our
presentation by using the
evaluation booklets
which you can find in
your conference bag.
Thank you!
22. SQS Group Limited
Mike Scott & Tom Roden
mike.scott@sqs.com
tom.roden@sqs.com
SQS Group Limited
7-11 Moorgate, London EC2R 6AF
Internet: www.sqs.com
Editor's Notes
Borne out of necessity, not dreamed up in a lab
Story of building an automation solution – out of necessity in real world project, struggling with quality and agilityShow framework, but talk thru story of interesting points and learnings derived along the wayevolved thru rapid iteration as it was built, then gradually refined and re-used on similar systems, distilled and simplifying it to something useful, agnostic products and technologiesIt is centred on investment / trading platforms, but the principles and experiences are much more universal
Problem – R reached 100 man days– growing and would mean not running it allIntroduced automation, teaching ATDDThen agreed to do full at source automationRapid ROI – story goes on but first walkthrough the changes made
Problem faced by many agile teams – not quite solving acceptance testingFirst – separate business UAT team, defects slow all other work to a stand-stillTesting, particularly in this way is like shining a torch onto the product – narrow beam = focus depth in detail, wide beam = less depth on one thing but more wider coverageSo the business were sent into a room with their torch to find the product…..waiting…..
An amount of time, gore and screaming later, the product was in a roughly suitable state to go live (or rather the business were too battle weary by then)Not a production quality, shippable product until the very end (and then not really clear – wide beam dispersion on our testing torch)Long UAT phase, delayed release – worse still poor quality in live
The product come rolling down the hill into test (withdev’s and others running after it) – asked to push it back upAll you can do to stop the boulder from rolling down into the gutterTesting at the end always used to make me feel this way.
Main changes:Implementing Specification by Example c.f. GojkoSpecification workshopsAutomation at source (alongside dev) w/ ETDemo’s
Problem – R reached 100 man days – growing and would mean not running it allIntroduced automation, teaching ATDDThen agreed to do full at source automationRapid ROI – story goes on but first walkthrough the changes made
From better tests to templatesTemplatesArchetypesUsage examples
Agile acceptance testing = BDD automation (at source)Increased collab = building the right product (build integrity in), involving the whole team and BUSINESSTools cover a range of technologies, and are open sourceAdvantageous Versus traditional automation for numerous reasons (More robust, Quicker, Easier to maintain,Reliable source of system documentationGreat accelerator for automationUseful by-product ofSafia is as a teaching and coaching aid, showing people how to write fixtures and tests cleanly and to BDD principlesEasily accessible to anyone in an organisation, business or IT
The tests were completely robust right from the startNever duplicated anything, broke a load of tests nor had to refactor a bunch of stuffFirst design was perfect, never needed to revise itSomething rather than nothing – It can be daunting. Start with something, and then make it better. A simple incomplete test suite is far better than none at all.Iterate – First version will be imperfect and incomplete. Improve quality and completeness iterativelySooner rather than later – A simple test suite soon is more useful, to a more complex, more fully featured test suite 6 months or more from now.Lean start-up patterns – build-measure—refactor-learn-repeatChallengesTrusting the tests (when not reliable)Utilities (to manage dependencies and data, particularly for asynchronous systems)Example usage so not duplicatingWhat is isn’t:A silver bulletAbout toolsThe only way to do it
Was asked to get regression down to 0 people for 0 days i.e. full lights out automation.Was harder than imagined due to the challenges mentionedBuilding a continuous delivery pipeline means taking full control of dependencies including integration with other systemsIf you don’t design it into applications (as was the case here) it can be hard to make apps testable once they are built (this was a COTS implementation) – lesson work out what you need to measure in live and what you need to do to test an app as you build it, that will save time in the long run in testing, extension and maintenance
describe the behaviour being checked and test what it purports to Sometimes notbe about business rules and process, not implementation jargon, test namesbe focused on the specific thing being testedfail due to the assertion being testedNot cover too much at once to affect ease of reading but group similar tests P&L Test exampleBe a specification of business logic, not be a list of instructions login, go to screenof use to everyone who needs to use them– collective test ownershipClear, explicit expectation they are documentation
Metric to support our claim (rather than refute it – though we searched objectively).This illustrates the case (one of the best supporting evidence) for living documentation, writing tests as specs so all can use them – in the long term support of a product you need tests for various stakeholders that can eliminate a good chunk of this system understanding time and cost.