BSSML17 - Machine Learning Summer School in Curitiba: Association Discovery.

1. N O V E M B E R 2 9 , 2 0 1 7

2. BigML, Inc 2
Association Discovery
Finding Meaningful Correlations
Poul Petersen
CIO, BigML, Inc

3. BigML, Inc 3Association Discovery
Association Discovery
• An unsupervised learning technique
• No labels necessary
• Useful for data discovery
• Finds "significant" correlations/associations/relations
• Shopping cart: Coffee and sugar
• Medical: High plasma glucose and diabetes
• Expresses them as "if then rules"
• If "antecedent" then "consequent"
• Significance measures
• BigML: “Magnum Opus” from Geoff Webb

4. BigML, Inc 4Association Discovery
Clusters
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51

5. BigML, Inc 5Association Discovery
Clusters
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
similar

6. BigML, Inc 6Association Discovery
Anomaly Detection
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51

7. BigML, Inc 7Association Discovery
Anomaly Detection
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
anomaly

8. BigML, Inc 8Association Discovery
Association Discovery
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51

9. BigML, Inc 9Association Discovery
Association Discovery
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
{customer = Bob, account = 3421}

10. BigML, Inc 10Association Discovery
Association Discovery
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
zip = 46140{customer = Bob, account = 3421}

11. BigML, Inc 11Association Discovery
Association Discovery
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
zip = 46140{customer = Bob, account = 3421}
{class = gas}

12. BigML, Inc 12Association Discovery
Association Discovery
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
zip = 46140
amount < 100
{customer = Bob, account = 3421}
{class = gas}

13. BigML, Inc 13Association Discovery
Association Discovery
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
Thr Sally 6788 sign food 26339 51
zip = 46140
amount < 100
Rules:
Antecedent Consequent
{customer = Bob, account = 3421}
{class = gas}

14. BigML, Inc 14Association Discovery
Use Cases
• Market Basket Analysis: Items that go together
• Data Discovery: how do instances relate?
• Behaviors that occur together
• Web usage patterns
• Intrusion detection
• Fraud detection
• Bioinformatics
• gene expression associated with outcomes
• Medical risk factors

15. BigML, Inc 15Association Discovery
What is interesting?
• In-frequent patterns can be strong, but are they
interesting?
• Vodka and caviar
• Storms and high water sales
• Frequent patterns can be strong, but are they interesting?
• Coffee and milk
• High plasma glucose and diabetes
• “Frequency” isn’t the answer…
• Depends on the data and domain
• We need better metrics to define what is interesting
❌
✅
❌
✅

16. BigML, Inc 16Association Discovery
Association Metrics
Instances
A
C
Coverage
Percentage of instances
which match antecedent “A”

17. BigML, Inc 17Association Discovery
Association Metrics
Instances
A
C
Support
Percentage of instances
which match antecedent “A”
and Consequent “C”

18. BigML, Inc 18Association Discovery
Association Metrics
Instances
A
C
Confidence
Percentage of instances in
the antecedent which also
contain the consequent.
Coverage
Support

19. BigML, Inc 19Association Discovery
Association Metrics
C
Instances
A
C
A
Instances
C
Instances
A
Instances
A
C
Confidence
Instances
A
C
0%
A never
implies C
A sometimes
implies C
100%
A always
implies C

20. BigML, Inc 20Association Discovery
Association Metrics
Independent
A
C
C
Observed
A
Lift
Ratio of observed support
to support if A and C were
statistically independent.
Support == Confidence
p(A) * p(C) p(C)
Problem:
if p(C) is "small" then…
lift may be large.

21. BigML, Inc 21Association Discovery
Association Metrics
Observed
A
C
< 1
Negative
Correlation
Lift = 1
No Correlation
> 1
Positive
Correlation
Independent
A
C
Independent
A
C
Observed
A
C
C
Observed
A
Independent
A
C

22. BigML, Inc 22Association Discovery
Association Metrics
Independent
A
C
C
Observed
A
Leverage
Difference of observed
support and support if A
and C were statistically
independent.
Support - [ p(A) * p(C) ]

23. BigML, Inc 23Association Discovery
Association Metrics
C
Observed
A
Observed
A
C
< 0 Leverage = 0
No Correlation
> 0
Positive
Correlation
Observed
A
C
Negative
Correlation
-1…
Independent
A
C
—
Independent
A
C
—
Independent
A
C
—

24. BigML, Inc 24Association Discovery
Magnum Opus
• Select measure of interest: Levarage, Lift, etc
• System finds the top-k associations on that measure within
constraints
• Must be statistically significant interaction between
antecedent and consequent
• Every item in the antecedent must increase the strength
of association

25. BigML, Inc 25Association Discovery
Basic AD Configuration
1. Search Strategy: Support/Coverage/Confidence/Lift/Leverage
2. Max Number of Associations: 1 to 500 (default 100)
3. Max Items in Antecedent: 1 to 10 (default 4)
4. Complement Items: True / False
• False: Coffee and…
• True: Not Coffee and…
5. Missing Items: True / False
• False: Loan Description contains "Ferrari" and…
• True: Loan Description is missing and…

26. BigML, Inc 26Association Discovery
Data Types
numeric
1 2 3
1, 2.0, 3, -5.4 categoricaltrue, yes, red, mammal categoricalcategorical
A B C
date-time2013-09-25 10:02
DATE-TIME
YEAR
MONTH
DAY-OF-MONTH
YYYY-MM-DD
DAY-OF-WEEK
HOUR
MINUTE
YYYY-MM-DD
YYYY-MM-DD
M-T-W-T-F-S-D
HH:MM:SS
HH:MM:SS
2013
September
25
Wednesday
10
02
text
Be not afraid of greatness:
some are born great, some
achieve greatness, and
some have greatness
thrust upon 'em.
text
“great”
“afraid”
“born”
“some”
appears 2 times
appears 1 time
appears 1 time
appears 2 times

27. BigML, Inc 27Association Discovery
Items Type
itemscoffee, sugar, milk, honey,
dish soap, bread
items
• Canonical example: shopping cart contents
• Single feature describing a list of items
• Each item separated by a comma (default)

28. BigML, Inc 28Association Discovery
Use Cases
GOAL: Discover “interesting” rules about what store items
are typically purchased together.
• Dataset of 9,834 grocery cart transactions
• Each row is a list of all items in a cart at
checkout

29. BigML, Inc 29
Association Demo #1

30. BigML, Inc 30Association Discovery
Use Cases
GOAL: Find general rules that indicate diabetes.
• Dataset of diagnostic measurements of 768
patients.
• Each patient labelled True/False for diabetes.

31. BigML, Inc 31
Association Demo #2

32. BigML, Inc 32Association Discovery
Medical Risks
Decision Tree
If plasma glucose > 155
and bmi > 29.32
and diabetes pedigree > 0.32
and insulin <= 629
and age <= 44
then diabetes = TRUE
Association Rule
If plasma glucose > 146
then diabetes often TRUE

33. BigML, Inc 33Association Discovery
Advanced AD Config
1. Measures: Set a minimum criteria for AD measures
2. Minimum Significance: lower values reduce spurious rules
3. Consequent: Restrict rules to a specific consequent criteria
4. Discretization: How numeric values are handled
1. Pretty: rounds off discretized values: 20 instead of 20.234
2. Size: the number of ranges (default 5)
3. Type: equal population / width
4. Trim: Removes percentage of values from the tails

34. BigML, Inc 34
Association Demo #3

35. BigML, Inc 35Association Discovery
Summary
• Association Discover Purpose
• Unsupervised technique for discovering interesting
associations
• Outputs antecedent/consequent rules
• Metrics: Support / Coverage / Confidence / Lift / Leverage
• Items type:
• Configuration:
• Search strategy / Minimum Measures
• Complementary rules / Missing
• Consequent Filtering
• Discretization
• Additional Uses
• Understanding clusters and anomaly detectors