This presentation reports our experience on using the machine learning techniques in Apache Spark ecosystem to understand the user behavior in a number of applications. In this context, Spark makes the vast computing power of a large high-performance computing system available to the behavioral economists without requiring the application scientists to learn about parallel computing. To illustrate the effectiveness of this approach, we focus on a compute-intensive task of establishing baseline for studying the impact of policies on consumer behavior. The gold standard for this type of baseline is a randomized control group, however, this control group can only provide a group-level reference, not for individual consumers. In many cases, the self-selection bias along with other factors can make it extremely difficult to generate a unbiased control group. By harnessing the computing power of Spark, we are able to learn the behavior pattern for each individual user and therefore create a much more precise baseline for behavioral analysis. We will use two use cases to illustrate the approach: a residential electricity usage study and a traffic pattern prediction study.

1. Spark for Behavior
Analysis Research
Ling Jin, Sam Borgeson, Anna Spurlock, Annika Todd
Doris Lee, Alex Sim, John Wu
Lawrence Berkeley National Lab

2. www.lbl.gov

3. Adam Smith’s Most Famous Books?
Behavioral Analysis Research 3

4. One-minute Behavioral Economics
Simpler to analyze and predict Harder to predict -- Need
massive data and computers
Behavioral Analysis Research 4

5. How Are Blackjack and Electric
Power Grid Similar?
Bust when demand is larger
than supply Bust when over 21
Behavioral Analysis Research 5

6. • Residential electricity records: 100,000 households,
hourly electricity usage
• A region where electricity usage peaks in the
summer time
• Randomized controlled trial of new rate structures -
households are randomly placed in different groups
• Control: using existing fixed rate for electricity
• Active: households opted in to Time-of-Use Pricing
• Passive: households defaulted in to Time-of-Use Pricing
• Data collected hourly over three years, one pre-
rate, two after (labeled T-1, T, T+1)
6
Reducing Peak Demands Through
Pricing
Behavioral Analysis Research

7. Goal Method Policy Implication
Better baseline models of energy
use
Gradient tree boosting Better program evaluation
Define relevant household
characteristics:
Classify and segment
households using easily
accessible data
• Define representative load
shapes
Adaptive K-means clustering
• Estimate household-specific
cooling change points (AC set
point)
Piecewise linear regression,
bootstrapping
• Characterize customers into
“Lifestyle Groups”
Blend behavioral theory with
machine learning techniques
• Define relevant household
energy characteristics
Simple feature algorithms (e.g.,
mean, min, max, peak usage;
variance; entropy, etc.)
7
Research Examples
Behavioral Analysis Research

8. Baselines are Critical for Measuring
Changes
Behavioral Analysis Research 8
Current Standard: randomized control group

9. Predictions from new Baseline Technique
Active group Passive group
9
Group PT PT +1 MT −PT MT+1 −PT+1
Control 1.960 1.957 0.069 -0.020
Passive 1.849 1.897 -0.027 -0.080
Active 1.860 1.903 -0.164 -0.164
Observation:the Active group reduced their uses of electricity consistently over the
two yeas of study during the peak-demand hours
P: predicted
M: measured
Behavioral Analysis Research

10. Goal Method Policy Implication
Better baseline models of energy
use
Gradient tree boosting Better program evaluation
Define relevant household
characteristics:
Classify and segment
households using easily
accessible data
• Define representative load
shapes
Adaptive K-means clustering
• Estimate household-specific
cooling change points (AC set
point)
Piecewise linear regression,
bootstrapping
• Characterize customers into
“Lifestyle Groups”
Blend behavioral theory with
machine learning techniques
• Define relevant household
energy characteristics
Simple feature algorithms (e.g.,
mean, min, max, peak usage;
variance; entropy, etc.)
10
Research Examples
Behavioral Analysis Research

11. Daily Load Shape: Definition
Objective / Definition
• 24-hour electricity usage
pattern
• To capture hour variations:
don’t care about constant
usage
• Want to use load shape to
study mechanism that might
affect electricity usage,
therefore concentrate on
discretionary demand
Samples
• A load shape is the
pattern created by 24
hours of demand
data.
• Shapes are produced
by patterns of
occupancy,
equipment
ownership, and
behavior
5
Defining Load Shapes
Flat/unoccupied Solar Morn/work/eve
Out for lunch? AC dominated?
Active at night?Behavioral Analysis Research 11

12. Clustering Process
Behavioral Analysis Research 12
Load data
preprocessing
Load shape
clustering
Post clustering
processing
Smart meter customer-
day load shape data
(24 hourly observations
per customer-day)
Subsample of
100,000
customer-day
load shapes
De-min*
Normalize
Eliminate load shapes
with missing data or
extremely low power
demand (below 0.2kW)
Adaptive k-means
algorithm
(parameter θ chosen)
Hierarchical
merging
Iterative truncation
algorithm*
(parameter V chosen)
Definition of final
dictionary of load shapes
Assign full dataset of
customer daily load
shapes to the closest
dictionary load shape
* Innovations we introduce

13. Components of Clustering Process
• Data cleaning and normalization
• Remove households with very low demand (<0.2kW)
• Normalization: compute hourly usage to hourly contribution to daily usage
• Clustering
• Centroid-based methods: Kmeans, Adaptive Kmeans
• Hierarchical clustering: distance metric, linkage
• Density-based clustering: DBSCAN
• Model-based clustering: GMM
• Judging cluster quality
• Compactness: MIA, VRSE
• Distinctness: SMI
• Combined: DBI, Silhouette
13

14. Clustering Quality Index
14

15. Clustering Quality
15
More compact
More distinct
Puzzling??
Best choice?

16. Kmeans Produces More Balanced Clusters
16
Sizes of clusters as fractions of total number of samples
Behavioral Analysis Research

17. Summary
Ø Examined a good number of clustering methods for identifying
daily usage profiles
Ø Short observation:
– Centroid-based method (Kmeans) works the best on this set of data
Ø Longer observations:
– There are too many choices and no (really) clear winner
– Centroid-based methods produce more compact clusters
– Centroid-based methods produce more balanced cluster
– DBSCAN declare many (up to 90%) data points as background
– Our observation is different from previous published results (Chicco
2012)
Ø Future work
– Maybe a different clustering quality metric would work better?
– Should examine alternative profile generation methods, other than
clustering
Behavioral Analysis Research 17

18. Thank You.
John Wu
Email: John.Wu@nersc.gov