Presentation @FSE2018 of a collaborative work between UBA and Medallia Corp to analyze end user programmed calculated fields.

1. Testing and Validating End User
Programmed Calculated Fields
Diego Garbervetsky* Víctor Braberman* Javier Godoy* Sebastian Uchitel*§
Guido de CasoΩ Ignacio PerezΩ Santiago PerezΩ
*ICC, UBA/CONICET, §Imperial College London, ΩMedallia Inc.

2. •Company specialized in customer
experience management
•Manages billons of surveys
• Automotive, Retail, Hotels, Tech, etc.
•Business intelligence

3. Persistent
storage
Feedback
collection
mechanisms
Feedback
processing
pipeline
Text analytics
Alert processing
In-memory engine
loader
In-memory query
engine
Survey
takers
Professional
Services
Administration
interface
Configures
calculated fields
Source of truth for
non-calculated fields
Materializes the
calculated fields in the
in-memory engine
Completes
surveys /
reviews
Listens for new
feedback and triggers
various stages
Offers configuration capabilities,
including the creation and
testing of new calculated fields
Performs
queries

4. End User Programmed Calculated fields
• Implemented by domain experts
• But not skilled developers
• Intricated conditions on database columns
• Error Prone
Professional Services
DSL
Library
(string + integer
manipulation + Medallia
specific functions)
Surveys DB
VM
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
Customer
DSL

5. Validating calculated fields programs
Random sampling from surveys database
Business Analyst validates program output for randomly selected registers
Coverage metrics for termination
id name points … CF_Point
s
123 John 200290 4
188 Alice 89000 2
…
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
Test
Execution
{output,
coverage}
Random row
sampling
Row# Row
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}

6. Inefficient: more cases to be validated
and less coverage
Can we do something more systematic
than random sampling?

7. Symbolic Exec Engine
SMT
Test Case
Execution
inputs {output,
coverage}
DSL
Potential approach: symbolic/concolic execution
• Exec with random input
• Collect path condition
• Generate new input using SMT-solver
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
Path
Generation
DSL
Library
PC : points= null
Model(!PC) = points=0
PC : points != null ∧ points <= 400000 ∧
points <= 200000 ∧ points <= 100000…
Model(!PC) = points=400001

8. Symbolic Exec Engine
SMT
Test Case
Execution
inputs {output,
coverage}
DSL
Potential approach: symbolic/concolic execution
• Exec with random input
• Collect path condition
• Generate new input using SMT-solver
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
Limitation: Inputs that are not real, may be difficult to interpret,
violate tacit invariants, etc.
id name points …
1 xxxx null
1 xxxx 0
…
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
Path
Generation
DSL
Library

9. Symbolic Execution + Real Data
Hypothesis: Large databases may have enough registers (inputs) to contain
one example for each path condition used in practice
SMT+Symbolic
Exec
Test Case
Generation
PC {output,
coverage}
PC2Query
Translator
queries input
DSL (JS)
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
id name points …
123 John 200290
DSL
Library
row.points < 400001
∧ row.points > 200000
SELECT * FROM row
WHERE row.points <
400001
AND 200000 < row.points
Test1 {
row.points = 200001;
CF_Points(row)
}

10. Problems in practice
• PC “polluted” with implementation details in library code
SMT+Symbolic
Exec
Test CasesPC {output,
coverage}
PC2Query
Translator
queries input
DSL (JS)
function Years(Date d1, Date d2) {
…
}
Type Date = <dateData: long, ..>
function CF_Age(row) {
var age = Years(lastSeen, signupDate);
if(age>3)
var hours = ParseInt(substr(travel_hours, 10));
}
DSL
Library
... ∧ (((4611686018427387903uL &
(13835058055282163712uL^row.lastSeen.dateData)) -
(4611686018427387903uL &
(13835058055282163712uL^row.signupDate.dateData)))
/864000000000L) / 365 <3
DSL
Library

11. Problems in practice
• Loops may generate unbounded PCs
• Call to native methods
SMT+Symbolic
Exec
Test CasesPC {output,
coverage}
PC2Query
Translator
queries input
DSL (JS)
function CF_Age(row) {
var age = Years(lastSeen, signupDate);
if(age>3)
var hours = ParseInt(substr(travel_hours, 10));
}
function ParserInt(Sring s) {
…
while(...) {
res += 10*ord(s[i])…
}
…
}
… travel_hours[0] = ord(…) … ∧ travel_hours[1] = xxx & yyy
∧ travel_hours[2]= … ∧ travel_hours[3] …
DSL
Library
DSL
Library

12. Approach
Observation:
• End user Programs are typically simple code with no loops
• DSL Library has loops, data type definitions,
calls on native methods
Key idea: Hybrid Symbolic + DB engine reasoning
• Keep SMT+Symbolic reasoning only at the end user DSL level.
• i.e., Do not interpret calls on library.
• Translate high level PC to DB queries
• i.e., Implement DSL library methods using DB Store procedures
Hypothesis
• All non-interpreted calls can be mapped onto stored procedures
• DB engine is fast enough to solve queries with stored procedures
DSL
DSL
Library
Surveys DB
VM
function CF_Points(row) {
var points = row.points;
if (points == null) return null;
if (points > 400000) return 5; // Platinum
if (points > 200000) return 4; // Gold
if (points > 100000) return 3; // Silver
if (points > 50000) return 2; // Frequent
return 1; // Basic
}
function CF_Age(row) {
var age = Years(lastSeen, signupDate);
if(age>3)
var hours = ParseInt(substr(travel_hours, 10));
…
}

13. function CF_Age(row) {
var age = Years(row.lastSeen, row.signupDate);
if(age>3) …
}
... ∧ (((4611686018427387903uL &
(13835058055282163712uL^row.lastSeen.dateData))
- (4611686018427387903uL &
(13835058055282163712uL^row.signupDate.dateData
))) /864000000000L) / 365 <3 …
function Years(Date d1, Date d2) {
…
}
Type Date = long
DSL
Library

14. function CF_Age(row) {
var age = Years(row.lastSeen, row.signupDate);
if(age>3) …
}
… row.lastSeen!=null ∧ row.signupDate!=null
... ∧ Years(row.lastSeen, row.signupDate) <3
function Years(Date d1, Date d2) {
…
}
Type Date = long
SELECT * FROM row WHERE …
AND row.signupDate NOTNULL
AND row.lastSeen NOTNULL
AND Years(lastSeen, signupDate) < 3 limit 1
Uninterpreted
Years(Date d1, Date d2)
DSL
Library
CREATE FUNCTION Years(d1 date, d2
date)
RETURNS integer AS $$
BEGIN
…
END;

15. function IsPlatinum(row){
return
IsPlatinum(row,CF_ComputePointsReqForCountry(row));
}
function IsPlatinum(row, pointsForPlatinum) {
var points = row.lastPoints.split(';’);
if (points.Length < row.numDays) return 0;
var isPlatinum = true;
for(i = 0; i < row.numDays; i++) {
var calcPoint = int.Parse(points[i]);
isPlatinum = isPlatinum && (calcPoint >
pointsForPlatinum);
}
if (isPlatinum) return 1;
return 0;
}
function CF_ComputePointsReqForCountry(row) {
if (row.countryCode==null) return null;
if (row.countryCode==1 || row.countryCode==44) return
150000;
if (row.countryCode==54 || row.countryCode==55) return
250000;
return null
}
…

16. function IsPlatinum(row){
return
IsPlatinum(row,CF_ComputePointsReqForCountry(row));
}
function IsPlatinum(row, pointsForPlatinum) {
var points = row.lastPoints.split(';’);
if (points.Length < row.numDays) return 0;
var isPlatinum = true;
for(i = 0; i < row.numDays; i++) {
var calcPoint = int.Parse(points[i]);
isPlatinum = isPlatinum && (calcPoint >
pointsForPlatinum);
}
if (isPlatinum) return 1;
return 0;
}
function CF_ComputePointsReqForCountry(row) {
if (row.countryCode==null) return null;
if (row.countryCode==1 || row.countryCode==44) return
150000;
if (row.countryCode==54 || row.countryCode==55) return
250000;
return null
}
…
... ∧ row.lastPoints !=null ∧ row.numDays !=
null ∧ 1 <row.numDays ∧ row.countryCode != null
∧ row.countryCode = 1 ∧ IsPlatinum(row, 150000)
= 1
SELECT * FROM row
WHERE row.countryCode = 1 …
AND IsPlatinum(row, 150000) == 1
Fast filtering using
DB indices
Uninterpreted IsPlatinum(row, points)
Applied over a
reduced set of
records

17. Protoype
• PEX for Symbolic Engine: C# + uninterpreted function for DSL Library
• SQL for BD: Store procedure implementation of DSL Library
PEX
Test Cases
{PCs}
{output,
coverage}
PC2Query
Translator
{Queries +
Stored
Procedures}
{input}
C# + annotations for
uninterpreted functions
JS-2-C#
(systematic)
DSL (JS) Synthetic inputs
DSL
Library
DSL-2-
StoreProcedure

18. Evaluation: subjects
Anonymized BD with ˜21K records for internal product database
9 user anonymized end user programs
• Inputs: integer, floats and strings.
• Outputs: enumerated types, integers and strings
• Invokes Medallia library and VM methods
• lookups on auxiliary tables, regular expression for searching over strings, etc.
• Many nested ifs, null-checks  intricated code

19. Findings
• ✔️ High coverage
• ✔️ Few inputs
(Random sampling requires >50% #rows)
• No row in DB to cover one PC in CF_4
• 😱 It was due to a bug and not due to
incomplete DB

20. Lessons learnt
• Symbolic execution is feasible under
certain scenarios (end user programs +
uninterpreted library calls)
• Databases can be a valid alternative to
SMT solving for meaningful test case
generation
• An optimized search over a database can
be better than solving a complex path
condition.

21. Next steps
•Automate steps in prototype
•Delegate completely next round
of evaluation
•Full implementation

22. Questions?