This presentation reports the progress of a project whose goal is to develop a model-driven, contract-based SOA testing environment in which grey-box testing strategies for large services architectures are driven by stochastic inference. In order to achieve this goal, a Bayesian network is built directly by mapping and model transformation from the services architecture design and test models. Because services architectures are complex and large systems, the generated BN and its associated probability tables may also be very large and the inference process may be excessively slow. Nevertheless, the structures of the services architecture design and test models let techniques of compilation improve the inference task, enabling the use of BN inference as a sustainable tool for driving failure discovering and troubleshooting strategies in large scale services architectures.

1. simplengineering
Service Oriented Architects
November 30, 2011
Steps towards model-based,
inference-driven SOA Testing
Ariele P. Maesano¹ ², Fabio De Rosa ²,
Libero Maesano ², Perre-Henri Wuillemin¹
{Ariele.Maesano, Pierre-Henri.Wuillemin}@lip6.fr
{ariele.maesano, fabio.de-rosa, libero.maesano}@simple-eng.com
¹ Laboratoire d’Informatique de Paris 6 (UPMC), Paris, France
² SIMPLE ENGINEERING, 31-35, rue St. Ambroise, 75011 Paris, France

2. simplengineering
Service Oriented Architects
Table of contents
1. SOA Functional Testing
2. Black-box and Grey-box Test
3. SOA Testing Environment
4. Model Driven Test Engineering
5. BN Inference Approach
6. Discussion & Future work
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 2

3. simplengineering
Service Oriented Architects
Motivation
In the next 20 years, tens, hundreds, thousands, …
applications, systems, devices will be connected and will
collaborate without human intermediation - that’s the Brian
Arthur’s second economy *
Distributed enterprise architectures involving companies,
government and for-no-profit organizations allow the
automation of business processes that support daily activities
Service Oriented Architecture (SOA) is a design and
implementation style that allows organizations to put in
practice dynamic collaborations of loosely coupled systems, in
order to achieve flexible, dependable and secure business
process automation
SOA is the paradigm able to support the second economy
* W. Brian Arthur, "The second economy." McKinsey Quarterly, October 2011
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 3

4. simplengineering
Service Oriented Architects
SOA conceptual framework
The collaboration among systems is carried out
through the exchange of services which are regulated
by contracts
Service contracts are formal models of the service
functions and of the provider and consumer interfaces
and external behaviors (including security and quality
of service aspects)
Service contracts do not include any information about
internals of the systems that comply with them -
system implementations are black boxes, hidden and
private
A services architecture is a network of participant
systems, playing roles bound by service contracts, that
collaborate in order to achieve business goals
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 4

5. simplengineering
Service Oriented Architects
SOA testing conceptual framework
Validation question: "Are you building the right services architecture?", is
translated in the question: "Are you formalizing the right service contracts and
participants’ roles (combinations of contract parts)?"
Verification question: "Are you implementing participant systems right, i.e. in
compliance with the validated service contracts and participants’ roles?“
A model-driven approach simplifies the validation and verification task:
 once the service contract model has been validated, the remainder of the
overall validation task is the verification of the correct application of the
model mapping and transformation, until the implementation
The last verification question (about compliance between implementation and
contract) “stays open”,
 SOA Architects and Integrators cannot employ program static check
methods or white-box testing
 The only available verification methods are black-box testing of individual
SUTs (System Under Test) and grey-box testing of interactions among
SUTs of the Services Architecture Under Test (SAUT), where they are
observable
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 5

6. simplengineering
Service Oriented Architects
Probabilistic Inference Approach
The ongoing research on SOA testing spans three main areas:
 the automated generation of test cases from contract models
(positive, negative, quality of service…)
 the automation of the test execution and evaluation environments
 inference-based methods and tools in order to help Testers to
efficiently manage test campaigns
There are not yet applications of probabilistic inference to the
management of SOA testing campaigns
 it appears to be adequate for discovery of failures and search and
identification of faulty participants in a services architecture, where
the only available information is the match/mismatch between the
actual external behavior and the expected one
Bayesian Networks for Services Architecture Testing (BN4SAT)
project(*) is centered on the development of a Bayesian network
(BN) inference approach and technology to services architecture
testing management
(*)The project is conducted in cooperation between the Laboratoire d'Informatique de Paris VI (LIP6 - Université Pierre et Marie Curie - Paris VI -
France), the Centre National de la Recherche Scientifique (CNRS - France) and Simple Engineering, a European group specialized in the design of
services architectures. The project is partially funded by the Association Nationale de la Recherche et de la Technologie (ANRT - France)
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 6

7. simplengineering
Service Oriented Architects
sd TestSuite_Wiring_003
Black-box Test usbg
:BankGateNode
usam
:AccountMngtNode
Black-box tests are run
against individual SUTs CheckBalanceResponder/inquiry(m-usbg-usam-001)
CheckBalanceRequester/inform(m-usam-usbg-002)
whose internals are hidden
Connections (channels) with
other participants are not
observed
Only the SUT responses to
WithdrawalResponder/invoke(m-usbg-usam-003)
WithdrawalRequester/report(m-usam-usbg-004)
stimuli are observed
CheckBalanceResponder/inquiry(m-usbg-usam-005)
CheckBalanceRequester/inform(m-usam-usbg-006)
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 7

8. simplengineering
Service Oriented Architects
Grey-box Test sd TestSuite_Wiring_003
usatm usbg usam
:ATM_Node :BankGateNode :AccountMngtNode
FindAccountResponder/inquiry(m-usatm-usbg-001)
Grey-box tests are FindAccountRequester/inform(m-usbg-usatm-002)
run against a SAUT CheckBalanceResponder/inquiry(m-usatm-usbg-003)
(network of SUTs) CheckBalanceResponder/inquiry(m-usbg-usam-001)
CheckBalanceRequester/inform(m-usam-usbg-002)
SUTs are involved CheckBalanceRequester/inform(m-usbg-usatm-004)
in an end-to-end CheckCredentialsResponder/invoke(m-usatm-usbg-005)
CheckCredentialsRequester/report(m-usbg-usatm-006)
interaction
Channels among
WithdrawalResponder/invoke(m-usatm-usbg-007)
WithdrawalResponder/invoke(m-usbg-usam-003)
SUTs are observed
WithdrawalRequester/report(m-usam-usbg-004)
WithdrawalRequester/report(m-usbg-usatm-008)
CheckBalanceResponder/inquiry(m-usatm-usbg-009)
CheckBalanceResponder/inquiry(m-usbg-usam-005)
CheckBalanceRequester/inform(m-usam-usbg-006)
CheckBalanceRequester/inform(m-usbg-usatm-010)
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 8

9. simplengineering
Service Oriented Architects
SOA testing environment
Testing and Test Control Notation (V.3)*
environment
Arbiter, Scheduler, Test components as TTCN-3
scripts
Engine as a plug-in (or a “service”)
* http://www.ttcn-3.org/
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 9

10. simplengineering
Service Oriented Architects
deployment BankNet DeploymentTokens_1
TTCN-3 «SUT»
usbg :BankGateNode
usbg4usatminterceptor : usbg4usaminterceptor :
BankGate4ATM BankGate4AccountMngt
usatminterceptor4usbg:
ATM4BankGate usaminterceptor4usbg:
AccountMngt4BankGate
«executionEnvironment»
:TTWorkbench
usbginterceptor4usatm: usam4usbginterceptor :
«SUT» BankGate4ATM AccountMngt4BankGate
«TestComponent» «TestComponent» «SUT»
usatm :ATM_Node
usbginterceptor : usaminterceptor : usam :AccountMngtNode
usatm4usbginterceptor BankGateInterceptorNode usbginterceptor4usam:
AccountMngtInterceptorNode
:ATM4BankGate BankGate4AccountMngt
ushwcstimulator:
arbiter :Arbiter
HW_Control4ATM
usatm4ushwcinterceptor : «TestComponent»
ATM4HW_Control
usatmstimulator :
ATM_StimulatorNode
scheduler :Scheduler
a4e : s4e :
Arbiter4Engine Scheduler4Engine
Automated test execution &
e4a : e4s :
Engine4Arbiter Engine4Scheduler
evaluation environment
:Engine
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 10

11. simplengineering
Service Oriented Architects
SOA Testing in WS* platform
SAUT representation = extended UDDI
V3 Data Structure + collection of
WSDLs
UML Interaction (XMI) + collection of
SOAP messages
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 11

12. simplengineering
Service Oriented Architects
BN Inference Approach
Bayesian network inference is used as a
 failure discovering driver, by getting the next test case to run in
order to reveal a failure. As
 troubleshooting driver, by getting the next test to run in order to
locate faulty participants
BN4SAT engine integrates failure discovering and
troubleshooting, by managing a well-adjusted dynamic test
sequence (the strategy) that, with a minimum number of test
case runs:
 reveals a maximum number of failures and
 locates a maximum number of faulty participants
The BN4SAT graph is built by the BN4SAT loader directly from:
 the SAUT deployment UDDI files
 the WSDL files for all the involved service contracts and
 the test suite files
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 12

13. simplengineering
Service Oriented Architects
BN4SAT Metamodel
The BN4SAT graph model has six Boolean stochastic variable
types:
Entity - business organization, {notFaulty, faulty}
Participant - participant system, {notFaulty, faulty}
Port - participant required interface, {notFaulty, faulty}
ActionType - type of communicative action (message, rpc,
rpc reply), {notFaulty, faulty}
Action (evidence): test case action {pass, fail}
Transaction: test case {pass, fail}
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 13

14. simplengineering
Service Oriented Architects
Discussion
A BN for SOA testing can exhibit a very large size, and can be very dense, therefore
with large conditional probability tables (inefficiency in time and space)
We have implemented and assessed different classical inference methods such as
value elimination, lazy propagation, Shafer-Shenoy and proved that they are not
appropriate to cope in a sustainable way with BN4SAT graphs
Inference by compilation: Bayesian network can be interpreted as a multi-linear
function(MLF) and therefore implemented by an arithmetic circuit (AC)
Darwiche “classical” method
Coarse Refined deterministic
SAUT + test suite Conjunctive Conjunctive Disjunctive Normal
Bayesian network Negational Form Arithmetic Circuit
specification Normal Form Normal Form
(Coarse CNF) (Refined CNF) (d-DNNF)
BN4SAT method (Model-driven inference by compilation)
- No intermediary generation of BN - Refined CNF in linear time (experimental results)
- No CNF optimization step - Push the boundaries with size limitation
Refined Conjunctive deterministic Disjunctive
SAUT + test suite Normal Negational Form
Normal Form (Refined Arithmetic Circuit
specification
CNF) (d-DNNF)
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 14

15. simplengineering
Service Oriented Architects
Thanks
Questions ?
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 15

16. simplengineering
Service Oriented Architects
SOA testing environment
Services Architecture Under Test - SAUT is a network of SUTs (System
Under Test) connected by channels conveying communicative actions
such as messages, remote procedure calls (rpc), rpc replies
Channels may be observable (grey-box stance). Conversely, SUTs
internals are never observable (black-box stance)
deployment BankNet Deployment
BankNet SAUT
atm4bg :ATM4BankGate
«SUT» «SUT»
usatm :ATM_Node usbg :BankGateNode
bg4atm :BankGate4ATM
usatm4ushwcinterceptor : bg4am :BankGate4AccountMngt
ATM4HW_Control
hwc4atm :HW_Control4ATM am4bg :AccountMngt4BankGate
«SUT» «SUT»
ushw c :HW_ControlNode usam :AccountMngtNode
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 16

17. simplengineering
Service Oriented Architects
SOA Test Running
Each SUT, in a well established SAUT, is initialized with a text context
(the SAUT state), that is coherent with the next selected test suite (i.e.,
a collection of test cases) is going to run
A test run takes place when Tester submits one or more stimuli to one
or more SUTs
Tester compares the SAUT actual response - the exchange of actions
between SUT's following the stimuli - with the expected response - the
expected action exchange in a specified SAUT state
A test matches (mismatches) if the actual response corresponds to
(differs from) the expected one
Test run has a verdict: pass - the test matches and there is no evidence
of failure; fail - there is evidence of a failure; inconclusive - a decision
cannot be reached; error - a run-time error has occurred in the test
environment
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 17

18. simplengineering
Service Oriented Architects
SOA Testing Environment
Test Components: specialized components allow reaping SUTs
actions, addressing actions to SUTs, comparing SUTs actions
with the expected ones, formulating local verdicts and
transmitting them to the Arbiter:
 Stimulators - send stimuli to one or more SUT, in order to trigger
test runs
 Interceptors - sit transparently between two connected
participants, intercept actions and perform different tasks
 Emulators - emulate the behavior of SUTs
Arbiter - the Arbiter collects the Test Components local test
verdicts and produces final test verdicts
Scheduler - the Scheduler drives the execution of the test runs
Engine, which is notified by the Arbiter with test verdicts,
manages the Scheduler by handling the test strategy on the
basis of probabilistic inference on the acquired test verdicts
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 18

19. simplengineering
Service Oriented Architects
Model-driven Approach
The starting point for testing a SAUT is in the Platform
Independent Model (PIM), specifically, the collection of
samples
A sample is a modeled, schema-compliant occurrence of
services interactions, that coordinates either the services
deliveries (positive sample), or the services refusals, when
the delivery conditions are not satisfied (negative sample)
A sample is composed of a:
 sample interaction, a UML Interaction together with its
MessageTypes instances, and a
 sample context, a collection of domain objects (instances)
representing the states of the participants' resources in
which the interactions take place
The sample collection is the starting point for the design
and the implementation of test cases
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 19

20. simplengineering
Service Oriented Architects
Test Engineering - Negative sample
class Sample oracle
sd TestSuite_Wiring_003
«MessageType»
m-usbg-usam-003 :Withdraw alInv okeMessage
usbg usam
acc# = 123456
:BankGateNode :AccountMngtNode
am = 1000,00
curr = USDollar
dateTime = 2011-06-15T11:28:38.100+01:00
WithdrawalResponder/invoke(m-usbg-usam-003)
«MessageType»
WithdrawalRequester/decline(m-usam-usbg-005) m-usam-usbg-005 :Withdraw alDeclineMessage
acc# = 123456
am = 1000,00
curr = USDollar
dateTime = 2011-06-15T11:28:14.109+01:00
code = NotEnoughBalance
Sample context (fact base)
CUSTOMER-NAME
CUSTOMER CUSTOMER-ACCOUNT ACCOUNT
customer# name
customer# customer# account# account# currency state balance
ABCDEFG Ariele
ABCDEFG ABCDEFG 123456 123456 EUR active 900.00
ABCDEFG Paolo
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 20

21. simplengineering
Service Oriented Architects
SOA Testing in WS* platform
The SAUT is translated into a set of WSDL files and an extended UDDI V3 Data
Structure, in order to capture use links (the observable channels in a specific
deployment scenario) - each participant system is represented as UDDI Business
Service
Sample collection is translated in a test suite (a collection of test cases), by the
mapping and mechanical transformation of the PIM into the WS* Interoperability PSM
A test case includes:
 a test case oracle - a XML infoset that represents the interchange of SOAP messages
between SUTs (resulting from the transformation of the sample UML Interaction), together
with the collection of the corresponding SOAP envelopes (resulting from the transformation
of the Message Type instances exchanged in the interaction)
 a test case fact base (test context) - a relational database resulting from the object/relation
transformation of the sample context
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 21

22. simplengineering
Service Oriented Architects
SAUT Description
UDDI v3 Business Service
deployment BankNet Deployment
«SUT» am4bg :AccountMngt4BankGate «SUT»
usbg :BankGateNode
bg4am :BankGate4AccountMngt usam :AccountMngtNode
bg4atm :BankGate4ATM
<?xml version="1.0" encoding="UTF-8"?>
<uddi:businessService businessKey="urn:My_US_BankNet" serviceKey="urn:My_US_BankNet:BankGate">
<uddi:name>BankGate</uddi:name>
<uddi:bindingTemplates>
<uddi:bindingTemplate bindingKey="urn:BankGate-AccountMngt:BankGate4AccountMngt:usbg4usam"
serviceKey="urn:My_US_BankNet:BankGate">
<uddi:description>urn:BankGateNode:usbg</uddi:description>
<uddi:accessPoint useType="endpoint">http://usbg.My_US_BankNet.com/BankGate4AccountMngt/usbg4usam</uddi:accessPoint>
<uddi:tModelInstanceDetails>
<uddi:tModelInstanceInfo tModelKey="urn:BankGate-AccountMngt">
<uddi:instanceDetails>
<uddi:overviewDoc>
<uddi:description>required</uddi:description> use links
<uddi:description>urn:IAccountMngt</uddi:description>
<uddi:description>urn:AccountMngtNode:usam:BankGate-AccountMngt:AccountMngt4BankGate:usam4usbg</uddi:description>
</uddi:overviewDoc>
</uddi:instanceDetails>
</uddi:tModelInstanceInfo>
</uddi:tModelInstanceDetails>
</uddi:bindingTemplate>
...
</uddi:businessService>
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 22

23. simplengineering
Service Oriented Architects
Test Case Oracle - Interaction
sd TestSuite_Wiring_002
usbg usam
:BankGateNode :AccountMngtNode
XML Infoset describing
WithdrawalResponder/invoke(m-usbg-usam-003) interaction among SUT
WithdrawalRequester/report(m-usam-usbg-004)
<?xml version="1.0" encoding="UTF-8"?>
<ss20tc:test-case xmlns:ss20tc="urn:simpleSOAD2.0:test-case" id="urn:InterExchangeNet:Scenario01:Wiring_001">
<ss20tc:sequence>
...
<ss20tc:message id="m-usbg-usam-003" sn="m033" from="urn:BankGateNode:usbg"
to="urn:AccountMngtNode:usam" mode="WithdrawalResponder/invoke"/>
<ss20tc:message id="m-usam-usbg-004" sn="m034" from="urn:AccountMngtNode:usam"
to="urn:BankGateNode:usbg" mode="WithdrawalRequester/report"/>
...
</ss20tc:sequence>
</ss20tc:test-case>
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 23

24. simplengineering
Service Oriented Architects
Test Case Oracle - Envelope
MessageType Instance
class Sample oracle
«MessageType»
m-usbg-usam-003 :Withdraw alInv okeMessage
acc# = 123456
am = 1000,00 SOAP Message
curr = USDollar
dateTime = 2011-06-15T11:28:38.100+01:00
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:ns="urn:ien:bank:services:withdrawal:wsdl:types"
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
<soapenv:Header>
<wsa:MessageID>m-usbg-usam-003</wsa:MessageID>
</soapenv:Header>
<soapenv:Body>
<ns:WithdrawalInvokeMsg xmlns:ns="urn:ien:bank:services:withdrawal:wsdl:types">
<ns:sessionId>
<ns:account>123456</ns:account>
<ns:dateTime>2011-06-15T11:28:38.100+01:00</ns:dateTime>
</ns:sessionId>
<ns:currency>USDollar</ns:currency>
<ns:amount>10000,00</ns:amount>
</ns:WithdrawalInvokeMsg>
</soapenv:Body>
</soapenv:Envelope>
ICSSEA 2011 - Steps towards model-based, inference-driven SOA Testing 30/11/2011 - 24