Practical Constraint Solving for Generating System Test Data

Practical Constraint Solving for
Generating System Test Data
Ghanem Soltana, Mehrdad Sabetzadeh, Lionel Briand
ESEC/FSE 2021
J1C2 track – ACM TOSEM 2020

Constraint Solving
• Process of finding solutions to a given set of constraints
(invariants, preconditions, transition guards in models,
etc.)
• Software engineering applications
• Model verification
• Model finding and slicing
• Automated test data generation
• etc.
2

Constraint Solving: State of the Art
• Exhaustive techniques:
• SAT
• SMT
• Alloy
• Constraint programming (UML2CSP)
• Non-exhaustive techniques:
• Metaheuristic-search
3

Problem Definition
• Valid test data generation from system specifications
• Recast as a constraint solving problem
• System constraints are complex
• Existing solvers are not adequate or not efficient
• We target the Object Constraint Language (OCL)
4

Instance models: Tax System
6
•Child care tax deduction example
•Invalid data helps reasoning about
robustness rather than reliability
Must be logically
and structurally
well-formed {
- receives_child_allowance = true
T1: ResidentTaxPayer
C1: Child
- receives_child_allowance = false
C1: Child
- receives_child_allowance = true
- receives_child_allowance = false
Synthetic test data

Object Constraint Language
• OCL is the standard constraint modeling language of UML
• OCL is a declarative language
• OCL is based on first order logic, set theory, and simple
mathematical notations
7
context TaxPayer inv:
self.children->forAll(c:Child| c.birthYear > self.birthYear + 16)
<latexit sha1_base64="OXbuCucRWZUHzAlg7E1YiXOqi18=">AAAKt3icjVbdb9s2EHe6r077SrfHvQgNCrQdp9jZ2nUFVLTIHlZgQDOgaTtYRkpRJ5kNRakkFcfl+BfudS/7b3aU5dhymsIEBNPHH4+/u+PdMa0F12Y4/G/n2kcff/LpZ9c/D7748quvv9m98e0LXTWKwTGrRKVepVSD4BKODTcCXtUKaJkKeJmeHvr1l2egNK/kczOvYVLSQvKcM2pQdLL7b5JCwaUV2vjTuCzcOFe0hFhWEohXO+OZmcaje6wkTf22qQzERjVAwhQPOvUIvRAgD860mQuIE2NyWnIxT9Jcg+KgJwGrpIFzEz6n50d0Dirk8uxhGCD3PGJTLjIF8sdHeaWeCHGbPTz0or9DFqVcmelfQFX4KGzBK8EP4ej+nQRkts4/CE5294bRsB3h5cmom+wNunF0cuP6P0lWsaYEaZigWo9Hw9pMLFWGMwEuSBoNNWWntIAxTiU6SE9s634X3kJJFiJx/KQJW2mwvsWeL5A9GS21npcpwV+/TftJSc2U6CbNedEoII0S/T0zRWtc6wtBNiU3UPalhiPFvqhzEEqDJIMcA9fSsuW8UADSWVWkzg7JMLpPhu4yhs6XkOge6b5LsJI2Z7DCPSBe34MDPNOPBOO0wAsqi8azenb4hwtsUFYKTmE+q1SmY1vVoNobiresyuYkFGAIXhhH8L5IzQ0/gzinQgNpd7Kq9LGLx2Ji9/fdhhBDtX/X2bvLBW0U+iEep5ObJHC9UI2fjybWxwMkcy1dDcYGYZjieqGqRmatpXGyMHg2Rdc7EoYe4vOBmlakUd+SYbgcS8xazoQbAyGM1t7yutJxShaijTRcA2eAnllzXBRFjvTUgUaFwNFpGcQ2ueUdccf1SGFWYg5BxqZU6U3SC8haTdikvACcAtSt0VcY1sGWYY8x7KST9UO/CvXF4logx53fU9GAm1isB87mCN0GCcxZ40i4jV7+zlnYikLFkMKz3K1b80H0U32Mzs6dxaBmbtPRH95OhXgqtaGSYY7pbeyGcyaaDN1pi23guaAGL7SV2+vengnWnwssFq0U+9LySq0kUMcPavOe+7OCXNFiLo7DVMVMD0PVCOhla7e0rlJPq1l3by9f/DVIWzLeC+wQhqZX57Q2UC/Ix6OrUqM7ojMOczLNk35VJctUW+TKArlhfd2oGjuWN78LRYfjRk+xEFyGYV2QBp8EF47t+woR3jT+DuK2ivdaT6MqbGL9/lhnObKfWGwY9ZSz834TqrmvrhutUGFvIWUjDFfVjNDsTaNNWp1vHicqWSAX5L1+ni+OGaGiqHy8M7I0e+FvFLTldvUPlWC1I9I3AjAMS7rv9H47N9PyADb6KAYCNdYtF6zCIM+QpPQntJvceDQZT006QT/ZBJ8wMPMEKL5JkgulLPcvBmefiCJyCAy7V9cFwI33Rl6Ds+1rZiXf7NUZVae9bh39RKKfsctGBwf3uiaLz5/R5mPn8uTFQTQaRqM/D/Ye/9Y9hK4Pvh/cHNwejAa/DB4Pfh8cDY4HbOfhzusdvvMm+DU4CfJguoBe2+n2fDfojeDt/yFnmd0=</latexit>

Challenges
• The complexity for solving OCL constraints emerges from
the expressiveness of the language:
• Four types of collections (bags, sets, ordered sets, and
sequences)
• Several predefined operations on collections and primitive types
• Arithmetic and logical operators
• Null/invalid variables
• User-defined methods
8

Existing Solvers
• UMLtoCSP, Alloy (UML2Alloy ) will serve as our baselines
• All make severe compromises in terms of supported
features or suffer from scalability issues
• No existing solution in our context
9

Basic idea: hybrid solver
• Search- and SMT-based solvers show different strengths.
- Meta-heuristic search work better than exhaustive
techniques for collection-related constraints
- SMT is effective for constraints involving quantifier-free
theories
• How about dividing constraint solving tasks among search
and SMT in an efficient way?
10

Delegation of solving tasks
11
p.disability
Rate
=
0
Exclusive to SMT
>=
1
size
p.addresses
Exclusive to search
<=
size
p.allowances
p.nbChildren
Size over collection
(for search)
Attribute with
primitive type
(for SMT)
Both search and SMT

Modify common instance model
<=
0
p.nbChildren
<=
size
p.allowances
3
<=
size
p.allowances
p.nbChildren
What search
gets as input
- nbChildren = 3
T1: TaxPayer
Instance model to fix
What SMT
gets as input

PLEDGE: Overview
13
Class diagram
OCL constraints (including
multiplicity constraints)
OCL
Initial instance
model
1. Pre-process
constraints
A single OCL
constraint in the
Negative Normal
Form (NNF)
Final instance
model
2. Apply
search/SMT
based solving
strategy
All constraints have to be
satisfied at the same time
NNF facilitates
dividing solving tasks and
interaction with SMT

PLEDGE: Step 1
14
• A single SAT-equivalent OCL constraint in NNF is produced
• The NNF constraint is what needs to be solved!
• NNF facilitates distribution of solving tasks across SMT and search

Step 1: Example
15
* context PhysicalPerson inv user_1:
let min:Integer = 1900 in self.birthYear >= min and
self.birthYear <= Constants.TAX_YEAR
* context PhysicalPerson inv user_1:
if (self.disabilityType = None) then
self.disabilityRate = 0 else
self.disabilityRate > 0 and self.disabilityRate <= 1 endif
* context PhysicalPerson inv mult_1:
self.addresses->size() >= 1
<latexit sha1_base64="xAK8q+nv89mdugi9PGlbdPY/c5M=">AAAL1XicjVZbb9xEFN6GW3G4tPDIy6hRpbYa3HVKoVS4alWQqIRoqJI21XoVje1j75Dx2MyMk2yt4QkBb/wwfgE/g3/AGa/34k1TdqRVnDPfnPnOfeJKcG2Gw38ubb319jvvvnf5fW/7gw8/+vjK1U+e67JWCRwkpSjVYcw0CC7hwHAj4LBSwIpYwIv4+LHbf3ECSvNS7ptpBeOC5ZJnPGEGRUdXL/0bxZBz2Qht3HVc5naUKVZAKEsJ1Ok95amZhMGdpKB19UtdGgiNqoGSGG86dgg9EyARnmgzFRBGxmSs4GIaxZkGxUGPvVskKaWBM0P2JlNEMrGHzEpJuDwhNaKOgvvEI0SAIQWX958gOAdFQhJ8PRwiiqCdmR9zZSYvgSnyIHQ4wmTqjq1tfhOSx6XUhkmj/f1Hh0cvv3v0zNuYBM/IjVZjyjWLueBmuo/+QzI/ol9uEjMBubh1iXnGjMMMCQgNF+0/wH1H+nV7SDsgIFOe/Q/Xoham49rqYWmqQGvQnz/Q/BXcuOncE3gR6loNrucdXdkZ+sN2kfMfQfexM+jW3tHVy39HaZnUBUiTCKb1KBhWZtwwZXgiwHoR+q1iyTHLYYSfErNHj5s2OS25jpKUZKXCnzSklXqrR5qzGbInY4XW0yKm+Ncd0+6jYGZCdR1nPK8V0FqJ/plTxSrc6wtB1gU3UPSlhiPFvqhzEEq9KIUMs7ql1RTTXAFI26g8ts2QDv0v6dCex7DpHOLfpd3vHKxg9Qkscfeo03dvF+90K8I4zfCCybx2rJ4+/sF6jVeUCo5helqqVIdNWYFq6xdLsEyn1FUMxaSw1EPva274CYQZwwyk7cmkLFzswpEYN7dv2zUhhur2Ldvcmm9oo9AP4SgeX6Oe7YVqtB+MGxcPkIlt6WowDWZgjPu5KmuZtpaG0czg0wm63lLikrRtFsy0Io365gzJfM0xKw2FrC2EJKxyllelDmM6E630qLvYo1bAKaBnVhzn+76lPXWgUSFwdFoKYRNdd464aXuksP6whiBNJkzpddIzyErDXKc8AxwDVK3RFxjWweZhDzHstJP1Q78M9WJzJZCjzu+xqMGOG+wLtskQugkSEtsYS8kmevkr28BGFMoEKTzN7Ko1b0Q/0Qfo7Mw2GNTUrjv6zceZEE/ajp9gjelN7IazRNQpurPJN4FnghlM6EZurntzJth/FlhsWjG2+nlKLSVQhfcq85r8WUIumL+L67BUsdIJUbWAXrV2W6sq9aQ87fL2fOKvQNqW8VpghzAsvrimtYFqRj4MLiqN7orOOKzJOIv6XZXOS21WKzPkmvVVrSqcWM78LhQdjhs9wUZwHoZ9QRp8MC0c2/cVIpxpOHDDtov3Rk+tShxi/flYpRmyHzc4MKoJT876Q6jirruujUKFs4W6cc9VeUpZ+nOtTVyerV8nSpkjF+S9ep9rjillIi9dvFM6N3vmbxS07Xb5HyrBbkelGwRgEmzpbtK749xMil1Ym6MYCNRYtVywC4M8QZLS3dAesqNgPJqYeIx+aiIFCHEE8PHTRAulSeZeDLZ5JHLfIpB0T9IFwI52AqfBNu1rZilfn9UpU8e9ae3fof4XOGX93d273ZDF50+w/tg5//F81w+GfvDT7s7Db7uH0OXBZ4NrgxuDYPDV4OHg+8He4GCQbB1u/br1x9af2y+27fZv27/PoFuXujOfDnpr+6//AN9t+II=</latexit>
PhysicalPerson.allInstances()->forAll(it5|
((it5.birthYear >= 1900 and it5.birthYear <= 2018) and
((it5.disabilityType = None and it5.disabilityRate = 0) or
(it5.disabilityRate > 0 and it5.disabilityRate <= 1 and not
(it5.disabilityType = None))) and it5.addresses->size() >= 1))
<latexit sha1_base64="SOzKYLZAIHRPXNsQF1M6qnODn4s=">AAALqnicjVZtb9s2EHayt07ZS7t93BehQYG4YBXJXbYMmIIW3YcVGNasaNoOtlFQ0snmQlEaSSVxOf6P/bX9lH3ajrLsWHZTmIBg+u65473w7phUnCkdhv/s7H7w4Ucff3LrU2/vs8+/+PL2na9eqrKWKZylJS/l64Qq4EzAmWaaw+tKAi0SDq+S8yeO/+oCpGKleKFnFYwLOhEsZynVSHpzZ+fvUQITJgxX2h3HxMQOc0kLiEUpgDi9lyzT0zh6GERpQerqz7rUEGtZA/ETPOvcYdScgKawVOkZh3ikdU4LxmejJFcgGaixdzqdIZ/yU7SoFAHl/KlQmooU1EH/wUleysecHzB99Jd34H6ChEk9/R2o9E9iP/ohDH0qMr/L+TH2B2F03G9YrVzGFE0YZ3r2Ap32Y/9XdGYpe819TrXjhn2/lCj7DuaJH94khwdHDU+UekN45dx+v79UQbNMglKgHpwo9hYO+o1j/b43ApGtJsF7c3s/DMJm+ZubqN3s99p1+ubOrX9HWZnWBQidcqrUMAorPTZUapZysN6oVlDR9JxOYIhbgUlWY9PcIevfQ0rmYwbwE9pvqJ63KmOu5tAOjRZKzYqE4K+TU25TUD0lqk5yNqklkFryrsylpBXyukQQdcE0FF2qZmhjl9QGCKneKIMc715jlilmEwkgrJGTxJqQhMF3JLSbGDpbQIIj0n4bsILWF3CNOyZO3/EAz3RrhHma4zkVk9pZ9ezJL9YzXlFKOIfZZSkzFZuyAtnUGRZKmc2Iz0ETnwlLPAy/YppdQJxTroA0kmlZuOTFQz42h4d2jYi5Orxvzf0FQ2mJcYiHyfgu8WwnVcMX0di4fIBIbWOuAm0830+QP5FlLbLG03g0d/hyiqG3xPcdxJU01Q1Job6Fhf5iLTArZe+vLYSktHKeV6WKEzInrfSSI+wkK+AMMDIrgQuCwJKOOlCoEBgGLYPYjO65QPRtxyi4QnczyNIplWrd6DlkpbGtmzwHnANUjdM3ONbCFmmPMe2kpXVTf53qJXMlkcM27gmvwY4Ntm9rcoRug4TUGm2Jv41e9tYa2MqEpvVaw7cClylCn+V21fX3op+qM8xMbg3egMyuZ+X94itDwhq1TZDgKuV1hrE3k23gOacab78R2+ve3hJsVkssdrgEB9/i/l1ToIqPK/2Oy3YNuWGkLo/Dusa24Puy5tAp7Za1qlJNy8v2km9WyQqk6S/vBLYITZObG4DSUM2Nj6Ob6qg9onUOCzjJR90WTBZ1OS+sOXLN+6qWFc43536bihbHtJpi19iEYRMRGl9By8B2Y4UI5xrO57hp+Z05VcsSJ153mlZZjtaPDU6XasrSq+7EqphrxWtzU+IgIkXNNZPlJaHZH7XSSXm1fhwvxQRtQbtXz3OdNCOUT0qX74ws3J7HGwlNb77+h0qwNRLhpgboFPu/exc4caanxQDWhi4mAjVWjS3YskFcoJHCndAI2WE0Hk51MsY4mZEEhDgD8IljRkulae7eF9Y85pPAItBv35lLgB3uR06DNc3T55q+PtgzKs87oz14SIJvcSQHg8FRO5HxsRStP402Ny8HQRQG0W+D/Uc/tc+mW71vend7B72o933vUe/n3mnvrJfu/Ld7bzfYPdwje8/3ft8bzqG7O63M173O2sv+ByXM6Co=</latexit>
SAT-equivalent NNF constraint
Input
constraints

PLEDGE: Step 2
16
Minimizes SMT
interventions
Uses a search-
based OCL
solver
Assigns attributes
with primitive
types

Search-based constraint solver
• Representation of solutions: instance model of class diagram
• Search space: valid and invalid instance models
• Solution: a non-empty instance model that satisfies the OCL
constraints
• Search algorithm: Alternating Variable Method (AVM)
• Creates new instance models by tweaking properties and
adding/removing instances and links to/from existing instance models
• Is guided by a distance-based fitness function that quantifies “how far”
an instance model is from satisfying the OCL constraints
17

Partial well-formedness check
18
• Replace by “true” clauses that are not exclusive to search
• Evaluate the obtained constraint over the instance model
forALL
PhysicalPerson
.allInstances() and
and
or
it5.birthYear
>= 1900
it5.birthYear
<= 2018
and
it5.disabilityType
= None
it5.disabilityRate
= 0
and
it5.addresses
->size() >= 1
and
it5.disabilityRate
> 0
it5.disabilityRate
<= 1
and
not
(it5.disabilityType
= None)
true true
true true
true true
true
remains as is
(because not
assigned to SMT)

Invoking the SMT solver
19
Fix the structure of the instance model and invoke SMT as follows:
• Transform the NNF constraint into a loop-free constraint that only
includes OCL constructs assigned to SMT
• Convert this loop-free constraint to the SMT-LIB language
• Run the SMT solver over the obtained SMT-LIB constraint

Loop-free constraint illustration
20
• Enables specific object attributes to be mapped to SMT variables
PhysicalPerson.allInstances()->forAll(p|
(p.disabilityType = None and p.disabilityRate = 0) or
(p.disabilityType <> None and (p.disabilityRate > 0 and p.disabilityRate <= 1)))
<latexit sha1_base64="B95IlyfiXiYodvfMfgw8EiXSMYs=">AAALDnicjVZfb9xEEL8r/4L518IjL1ajSkm1uHeB0D7UVavyQCVEA2raSnenaG2PfUvWa7O7TnJd9jsg8V2QeEBI8AIfgW/DrO3L2XdJyUqnOLO/mf3NzM7MRiVnSo9G/w6vvfHmW2+/s/Wu9977H3z40fUbHz9XRSVjOIwLXsiXEVXAmYBDzTSHl6UEmkccXkTHj93+ixOQihXimV6UMMtpJljKYqpRdHRjSKYRZEwYrrQ7jonMTlJJcwhFIYA4u6cs0fNwfDfOSVX+WBUaQi0rIH6EJx07hGoESITFSi84hFOtU5ozvphGqQLJQM28g/kC9yk/QD6FCCjnT4TSVMSgdnY/e5AW8hHnO+VPvrdTBglTNGKc6cUzpO2H/rdIx6ci8bt731Pt9ka7fiEvUrv/YKW3s6H4wB9dbPF+6I93d3e9KYikG5mj69ujYFQvf/Nj3H5sD9p1cHRj659pUsRVDkLHnCo1GY9KPTNUahZzsN60UlDS+JhmMMFPgYFXM1Pn1fq3UJL4GBf8Ce3XUq+rYs4aZE9Gc6UWeUTwr1NT7iOnek5UFaUsqySQSvK+zqmkJe71hSCqnGnI+1LNkGJf1IYHpd40gRQvRE3L5ItMAghrZBZZMyKj4EsyspsYulhCgn3S/jZgOa1OYIW7R5y9e3t4pltTzFKD51RklWP19PE31jNeXkg4hsVpIRMVmqIEWV99vL1FsiA+B018JizxMPqKaXYCYUq5AlJrxkXuchdO+MzcuWPXhJiqO7etub3cUFpiHMJJNLtJPNtL1eTZeGZcPkDEtqarQBvP9yPcz2RRiaT2NJw2Dp/OMfSW+L6DuDqjuhYptLdk6C/XEtOpRX9tISSmpfO8LFQYkUbUKe99LO8OOAGMTCdwQRBY0jMHCg0Cw6AlEJrpLReIXdsjBWfobgJJPKdSrZNuIJ1es065ARwDlLXTlzjWwpZpDzHtpJX1U79K9flmJ5GTNu4Rr8DODHZUa1KE/i+yblvW8Ktg0SrE1mhL/KtwYK+sgSvRLWKk8DS1Xc9fi36iDjExqTV4ARK7npTXq3catzXqKn7DWcyrBENvsiuFlFONl9+Iq9teMcGmFeGAWV6plQTK8F6pL7g/K8glo+v8QCxVrHTflxWHXrW2W12Tal6ctvd28+J3IHXLuBDYIjSNLq9ppaFsyIfjy0qjPaJ1DmsySqf9rkqWpdbUSoPc8B7D7ZxvU9GimFZzbAMtqKxkyRsYdgWh8aVxHtZ+pBDhHGOvIKx7eG/wVLLAEdafjmWSIveZwXFRzll81h9BJXO9dW0QSpwsJK+4ZrI4JTT5oVI6Ks7Wj+OFyJAL8u6e51pjQijPCpfthCzdbqKNgrrZrv5DI9jriHBjAHSMDd3NeafO9Dzfg7UpimlAi2XNBXswiBMkKdwJtZKdjGeTuY5mGCczlYAQRwBfK2Z6bjRO3XvBmkc8CywC/fYtdw6wk+2xs2BN/ZJZydcndULlcW9WB5+T4AucscHe3n47YvHxM15/6mx+PN8LxqNg/N3e9sOv2mfQ1uDTwc3BzmA8uDt4OPh6cDA4HMTDX4a/Df8c/uX97P3q/e790UCvDVudTwa95f39H+OfuMY=</latexit>
(t1.disabilityType = None and t1.disabilityRate = 0) or (t1.disabilityType <> None and
.disabilityRate > 0 and t1.disabilityRate <= 1))
and
(d1.disabilityType = None and d1.disabilityRate = 0) or (d1.disabilityType <> None and
.disabilityRate > 0 and d1.disabilityRate <= 1))
<latexit sha1_base64="K3vx7sYObTALon6gkpXwizFVvrs=">AAALdnicjVZtb9xEEL6Et+Lw0sJHJGQaVSTtyj0HAkGqq1blA5UQLahpK51P1doe+5as12Z3neS62n/Bn+In8E+Q+MKs7cud75qSlaL4Zp+ZfWZmZ2aTmjOlx+O/t7bfefe99z+49qG389HHn3x6/cZnz1XVyBSO04pX8mVCFXAm4FgzzeFlLYGWCYcXyckjt//iFKRilXim5zVMS1oIlrOUahS9urH1Z5xAwYThSrvjmCjsJJe0hEhUAoize8YyPYvCH9KSNPUfTaUh0rIB4id40olDqE6ARFiq9JxDFGud05LxeZzkCiQDNfX2dBhkTNGEcabnz5CNH/m/4Ck+FZk/2PyNarc53vcr6b9B7979peLepuZ9f3yJzXuRH+7ve7jp7WVvo5NdTmdTb0BnU3NBZ3OnpxODyFYz8Or67jgYt8vf/Aj7j91Rv56+unHtrzir0qYEoVNOlZqE41pPDZWapRysFzcKapqe0AIm+CkwwWpq2vtj/Vsoyfwcfcsrof1W6q2qmPMOOZDRUql5mRD879SU+yipnhHVJDkrGgmkkXyocyZpjXtDIYimZBrKoVQzpDgU9eFBqRdnkOPFa2mZcl5IAGGNLBJrxmQcfEfGdhND5wtIcEj6vw1YSZtTWOKOiLN3dIBnuhVjljo8p6JoHKsnj362nvHKSsIJzM8qmanIVDXItsSwSqpsTnwOmvhMWOJh9BXT7BSinHIFpNVMq9LlLprwqbl7164JMVV3b1tze7GhtMQ4RJNkepN4dpCqybNwalw+QKS2patAG8/3E9wvZNWIrPU0ijuHz2YYekt830FcPVPdihTaWzD0F2uBWal5f20hJKW187yuVJSQTrTSRg6xjayAM8DIrAQuCAJLBuZAoUFgGLQMIhPfcoHYtwNScI7uZpClMyrVOukOstLT1il3gBOAunX6Esd62CLtEaad9LJh6pepvthcSeSkj3vCG7BTg53bmhyh/4vE8nyIWH4VLFqF1BptiX8VDuy1NXAlulWKFJ7kdtXzt6Ifq2NMTG4NXoDMrifl7eqU88dCaSpSrEd1Fb/hPOVNhqE3xZVCyqnGy2/E1W0vmWDTSnCwLq7UUgJ1dFTrN9yfJeSSEXlxIJYqVrrvy4bDoFr7rVWTalad9fd28+KvQNqW8UZgj9A0ubymlYa6Ix+Fl5VGf0TvHNZkksfDrkoWpdbVSofc8B7D7ZzvU9GjmFYzbAM9qG5kzTsYdgWh8UVzEdZhpBDhHGOvIWp7+GDwNLLCETacjnWWI/epwXFRz1h6PhxBNXO9dW0QSpwspGy4ZrI6IzT7vVE6qc7Xj+OVKJAL8l49z7XGjFBeVC7bGVm43UUbBW2zXf5CI9jriHBjAHSKDd3NeafO9Kw8gLUpimlAi3XLBXswiFMkKdwJrZKdhNPJTCdTjJOJJSDEEcDXiokvjKa5ey9Y85AXgUWg378ZLwB2shs6C9a0L5mlfH1SZ1SeDGZ18A0JvsUZGxwcHPYjFh8/4fpTZ/Pj+UEQjoPw14PdBz/2z6Broy9GN0d7o3D0/ejB6KfR09HxKN36d/ur7dvbd7x/dr7cubXzdQfd3up1Ph8N1s74P8kV1oU=</latexit>
To loop-free constraint
- disabilityType = None
- disabilityRate = 0.32
T1: ResidentTaxPayer D1: Dependent
- disabilityType = A
- disabilityRate = 0
responsible
dependents

SMT-LIB constraint
21
Variable
mappings
(set-option:produce-models true)
; Function definitions [no function to define for our running example]
; Enumeration definitions
(declare-datatypes ()((country (LU)(FR)(GR)(BL)(Other))))
(declare-datatypes ()((disability (None)(Vision)(A)(B)(C)(D)(E))))
; Variable definitions
(declare-const X1 Bool)(declare-const X2 country)(declare-const X3 Bool)
(declare-const X4 Int)(declare-const X5 disability)(declare-const X6 Real)
; Constraint built from the quantifier-free constraint for the instance model in (b)
(assert (and (>= 1 1) (and true (and true (and (and (or (and true true) (and false
false)) true) (and (and (and (and true (or X1 (not (= X2 LU)))) (or (not X1) (or X3 (=
X2 LU)))) (>= 1 1)) (and (and (and (>= X4 1900) (<= X4 2018)) (or (and (= X5 None)
(= X6 0)) (and (and (> X6 0) (<= X6 1)) (not (= X5 None))))) (>= 1 1))))))))
(check-sat)(get-model)
Variables
{
{
Enumerations
{
Loop-free
constraint in
SMT-LIB
- birthYear = -1
- disabilityRate = 12.5
- disabilityType = None
- isResident = false
T1: TaxPayer
- country = LU
A1: Address
- isLocal = false
I1: Other
X1 X2
X3
X4
X6
X5
{

Limitations of PLEDGE
• PLEDGE is non-exhaustive
• Deriving loop-free constraints might lead to combinatorial
explosions in the size of the constraint
• Search is not equipped with fine-grained fitness function
calculation procedures for all its OCL constructs
22

Case study inputs
24
A B C
#Classes 64 57 55
#Enumerations 17 4 8
#Generalizations 43 35 20
#Attributes 344 492 195
#Associations 53 44 60
• Three industrial UML class diagrams, denoted A (taxation domain),
B (automotive domain), and C (satellite domain)
• Underlying NNF constraints
A B C
1 #Nodes in the AST 2004 1844 4014
2 #Universal quanti�ers 26 63 60
3 #Logical operators (and/or/not) 160 153 495
4
#Operations on objects and collec-
tions (e.g., oclIsTypeOf and size)
70 122 287
5
#Operations on primitive types (in-
cluding arithmetic operations)
211 161 357
6
#Sub-expressions including no quan-
ti�ers, and, and or
250 156 412

Research Questions
• RQ1. How does PLEDGE fare against the state-of-the-art
in terms of success rate and execution time?
• RQ2. Can PLEDGE generate large instance models in
practical time?
• RQ3. Does PLEDGE offer any scalability advantage over
the alternatives when PLEDGE is not the obvious best
choice according to the results of RQ1?
25

RQ1
• For the simplest case study (B), no difference between
PLEDGE and Alloy or UMLtoCSP
• For the other two, Alloy and UMLtoCSP were not
applicable, only pure search is
• Complexity of the constraints over primitive attributes
determine whether we fare better than search
• PLEDGE was much more successful fast than pure search
26

RQ2
• For our most complex case study, Case C (satellite),
PLEDGE generated valid data samples with over 10K
objects in less than 13 hours.
• One does not have to wait until testing time to initiate the
generation of data.
• One can generate large volumes of (logically valid) data
overnight, thus minimizing the wait time by test
engineers.
27

RQ3
• Only case B (automotive)
• Alloy and UMLtoCSP did not maintain the performance
edge they had for the simplest case study, for larger
instance models.
• The baseline solver was consistently outperformed by
both PLEDGE and Alloy.
• PLEDGE is the fastest among the alternatives considered
for generating large volumes of system testing data.
28

Summary of Results
• Three industrial case studies (automotive, government,
satellite)
• Large variability in complexity
• No difference between PLEDGE and alternatives for the
simplest data model (automotive) with small instance models
• Exhaustive techniques are not applicable for complex models
• PLEDGE is much more efficient than pure search
• PLEDGE can generate large volumes of (logically valid) data
overnight for all models, and is faster even for the simplest
model than alternatives
29

Paper Summary
• System test data generation needs to be automated
• Complex constraints result from system specifications
• Focus on Object Constraint Language (OCL)
• A new efficient, hybrid OCL constraint solver (search + SMT)
• Foundations underlying our solver can be exported to other
languages and contexts
• PLEDGE: https://sites.google.com/view/hybridoclsolver/
30

Practical Constraint Solving for Generating System Test Data

More Related Content

What's hot

Similar to Practical Constraint Solving for Generating System Test Data

More from Lionel Briand

Recently uploaded

Practical Constraint Solving for Generating System Test Data