The document discusses automated functional testing for aerospace and defense systems using Eggplant software. It notes that A&D software is large, complex, mission-critical, and operates in stressful environments. It outlines challenges like controlling costs, ensuring compatibility with legacy and new technologies, and effectively testing dynamic user interfaces. The document proposes that Eggplant's automation intelligence suite can help maximize mission success by enabling various approaches: 1) Modeling user journeys and outcomes, 2) Anticipating real-world stresses, 3) Enabling third-party testing while protecting IP, 4) Ensuring end-to-end user experiences, 5) Predicting successful system launches, and 6) Tracking mission progress and recommending improvements.
2. Attributes of
A&D Software:
• It is large.
• It solves or models complex
mathematical and complex physical
problems.
• It is tailored to a single mission.
• It is embedded within a complex system
of many interacting hardware elements.
• It embodies significantly new functions
never before coded in software.
• It has a stressful operating
environment.
• It has complex interactions with highly
trained users.
• Its builders do not use it; its users do
not build it.
• Its operation risks human life and great
economic loss.
3. A&D
Challenges
• Defense and aerospace companies are under the same pressures as other
industries to cut software testing costs. How are your teams using test
automation to test faster, effectively, and efficiently to ensure quality and on-
time delivery?
Controlling costs
• Legacy infrastructures need to be compatible with modern systems. That’s not
something that typical test tools can test. Can your test teams handle the mix of
standards-based, proprietary, custom, popular — IBM Rational Doors, Git,
Jenkins, IBM RQM, JIRA — and new, emerging technology?
Technology compatibility
• GIS mapping, C2 systems, and air traffic control systems rely on dynamic GUIs to
provide critical information to users. Traditional testing tools really only work
with text boxes, labels, buttons, and tabs, which aren’t effective for testing a
plane on a radar screen. Can you test graphic displays quickly and effectively to
ensure they perform as expected?
Testing dynamic GUIs
• The majority of testers in aerospace and defense are invaluable product experts,
and in many cases, hard to replace. How will you empower your teams —
especially those with varying skill levels — to test productively and efficiently?
Resource challenges
5. What Has Changed in Testing…
Yes, I tested My system is meeting &
exceeding its desired
positive mission
outcomes
6. DoD Software Acquisition is Changing
Defense Science Board
To be considered minimally viable for a proposal, competing contractors should
have to demonstrate at least a pass-fail ability to construct a software factory
Require all programs entering Milestone B to implement these iterative processes
for Acquisition Category (ACAT) I, II, and III programs
The chief executive officers (CEOs) of DoD prime contractors should brief the
USD(A&S) at least annually to demonstrate progress on adapting modern software
practices, including their corporations’ proficiencies in establishing effective
software factories.
Defense Innovation Board
All software procurement programs should start small, be iterative, and build
on success ‒ or be terminated quickly.
Adopt a DevOps culture for software systems
Automate testing of software to enable critical updates to be deployed in
days to weeks, not months or years
7. Why?
DoD software is plagued by poor quality and slow delivery.
Measuring the health of software development activities within DoD programs using obsolete metrics is irrelevant
at best and, at worst, can be misleading.
Electronic Warfare, RADARS, Guidance Systems, and C2 systems get their functionality from Software. The ability
to rapidly develop and field new software in the face of threats and advances is critical to maintaining superiority.
Most developers don’t write most of their code: COTS, Libraries, OSS, GitHub... They need functional feedback
early in the process when cost to address is low.
Acquisition Instructions are increasingly calling for incremental software releases and shifting DT and OT left into
the development cycle.
8. What it Means to You
Held to a new (higher) standard
The Defense Innovation Board released a draft ‘Metrics for Software Development’ proposal in
July 2018:
• The time from program launch to deployment of the “simplest
useful functionality”
• Deploy high priority code updates much more quickly and in
much smaller increments than typical DoD “block” upgrades.
• How gracefully software fails, how many errors are caught and
resolved in testing, and how rapidly developers patch bugs are
excellent measures of software development prowess.
• The percent of bugs caught in testing versus those caught in
the field provide a measure of the both the quality of the code
and the thoroughness of the testing environment.
• a set of automated tests will be constructed that cover both
the specification for what the performance should achieve as
well as the code that is used to implement that function.
9.
10. Appendix E: Software Factory Source Selection Criteria SuggestionsExternal test frameworks (e.g., Jersey Test Framework, TestPlant/eggPlant)
11. Weaker
current
offering
Stronger
current
offering
Weaker Strategy Stronger Strategy
Challengers Contenders Strong Performers Leaders
Eggplant leverages AI and automation to
test UX and business outcomes.
In the past 18 months, Eggplant evolved
and improved in three important aspects: It
developed a scriptless AI model-based
language, extended functional testing to
include early steps of customer experience
testing, and addressed its API testing
weakness.
Customers using Eggplant are very positive
about the extended testing persona options
offered, the collaboration possible among
them, the level of reuse the tool provides,
and the levels of automation they achieve
(60% to 70%).
16. Managing and Scheduling Tests
Orchestrate complex, high-volume test execution
within a CI or DevOps environment.
Reduce effort by easily managing your test environment
and automating your test execution.
Reduce testing errors by having a centralized management tool to
control your environment and test configuration, allowing you to
get the most out of your automated test scripts.
Reduce execution time by automatically
executing tests in parallel.
Improve quality by always understanding your overall software quality
status and being able to quickly identify the areas that need focus, and by
setting up a continuous integration environment.
QA Managers
SUTs
Repository
Test Engineers
41. Managing and Scheduling Tests
Orchestrate complex, high-volume test execution
within a CI or DevOps environment.
Reduce effort by easily managing your test environment
and automating your test execution.
Reduce testing errors by having a centralized management tool to
control your environment and test configuration, allowing you to
get the most out of your automated test scripts.
Reduce execution time by automatically
executing tests in parallel.
Improve quality by always understanding your overall software quality
status and being able to quickly identify the areas that need focus, and by
setting up a continuous integration environment.
QA Managers
SUTs
Repository
Test Engineers
43. FastStart Services
Account
Manager
KICK-OFF
Continue
& grow
Deploy &
transition
Engage
Technical
Account Manager
Eggplant
Community Documentation
Accelerate your test automation project.
Training and best practices.
Creation of test automation framework and
first set of automation scripts.
Unlimited post-engagement email- and web-
based support.
Test environment setup and integration into
your existing CI/DevOps infrastructure.
Editor's Notes
(*) Overview of WHAT we do.
(*) Key differentiators (to be discussed on slides below):
Fusion engine (multi-layer)
UX testing
Validate UX, universal coverage, cross-platform, end-to-end, intuitive and productive, low maintenance
AI
UX. Increased coverage, focus execution, automate exploratory testing.
Productivity. Auto-generation, reduce scripting, easy maintenance and impact analysis.
User monitoring and analytics
Make the user the centre of testing
(*) Other differentiators to be mentioned on this slide:
Taking F->P on this slide.
Being able to use selenium assets (and in general asset re-use and integration)
NOTE: explain Eggplant Functional and then highlight these four key points.
Capabilities:
Set up and configure your systems and devices-under-test (SUTs) quickly and reliably; manage and co-ordinate multiple Eggplant Functional controllers for optimal parallel test execution.
Quickly define test runs across multiple suites, platforms, and devices.
Schedule tests for continuous integration, nightly builds, and to make best use of your environment.
Automatically reserve SUTs via your eggCloud device cloud when test runs are scheduled.
Collect and analyze test results at a summary level or drill-down to the necessary detail.
Understand your test status quickly with a configurable dashboard and test reporting.
Execute arbitrary CLI commands from Eggplant Manager. This makes it easy to control your full test execution environment from one place.
Outcomes:
Reduce effort by easily managing your test environment and automating your test execution.
Reduce errors by having a centralized management tool to control your environment and test configuration, allowing you to get the most out of your automated test scripts.
Reduce execution time by automatically executing tests in parallel.
Improve quality by always understanding your overall software quality status and being able to quickly identify the areas that need focus, and by setting up a continuous integration environment.
Set up in minutes with Eggplant Manager as part of eggBox, your test lab in a box.
NOTES: the two messages of this slide are:
We integrate well with other tools and so can be easily integrated into your existing environment.
We engage with domain specific 3rd party vendors to deliver complete solutions for specific use-cases.
(*) Overview of WHAT we do.
(*) Key differentiators (to be discussed on slides below):
Fusion engine (multi-layer)
UX testing
Validate UX, universal coverage, cross-platform, end-to-end, intuitive and productive, low maintenance
AI
UX. Increased coverage, focus execution, automate exploratory testing.
Productivity. Auto-generation, reduce scripting, easy maintenance and impact analysis.
User monitoring and analytics
Make the user the centre of testing
(*) Other differentiators to be mentioned on this slide:
Taking F->P on this slide.
Being able to use selenium assets (and in general asset re-use and integration)
NOTE: explain Eggplant Functional and then highlight these four key points.
Capabilities:
Set up and configure your systems and devices-under-test (SUTs) quickly and reliably; manage and co-ordinate multiple Eggplant Functional controllers for optimal parallel test execution.
Quickly define test runs across multiple suites, platforms, and devices.
Schedule tests for continuous integration, nightly builds, and to make best use of your environment.
Automatically reserve SUTs via your eggCloud device cloud when test runs are scheduled.
Collect and analyze test results at a summary level or drill-down to the necessary detail.
Understand your test status quickly with a configurable dashboard and test reporting.
Execute arbitrary CLI commands from Eggplant Manager. This makes it easy to control your full test execution environment from one place.
Outcomes:
Reduce effort by easily managing your test environment and automating your test execution.
Reduce errors by having a centralized management tool to control your environment and test configuration, allowing you to get the most out of your automated test scripts.
Reduce execution time by automatically executing tests in parallel.
Improve quality by always understanding your overall software quality status and being able to quickly identify the areas that need focus, and by setting up a continuous integration environment.
Set up in minutes with Eggplant Manager as part of eggBox, your test lab in a box.
NOTES: the two messages of this slide are:
We integrate well with other tools and so can be easily integrated into your existing environment.
We engage with domain specific 3rd party vendors to deliver complete solutions for specific use-cases.
