The document describes modeling uncertainty in middleware-based streaming applications for power grids. It presents a discrete-event model built in Ptolemy II to capture uncertainty from sources like middleware latency, network delays, and number of sensor streams. Monte Carlo simulations are run over this model by varying parameters like middleware concurrency and sensor streams. Regression analysis is then used to understand the relationship between these influential parameters and the end-to-end application run time.

1. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Modeling Uncertainty For Middleware-based
Streaming Power Grid Applications
Ilge Akkaya, Yan Liu, Edward A. Lee, Ian Gorton
8thMiddleware For Next Generation Internet Computing Workshop
Beijing, China
December 9, 2013
UC Berkeley Ilge Akkaya 1 / 24

2. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
1 Introduction
2 Uncertainty Models and Parameter-Space Exploration
3 Discrete-Event Modeling in Ptolemy II for Uncertainty
Analysis
4 Regression Analysis
5 Conclusion
UC Berkeley Introduction Ilge Akkaya 2 / 24

3. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Introduction
• Cyber-Physical systems rely on the interaction of cyber
and physical system components
• Contemporary "Cyber" systems are about behavioral
correctness and do not have temporal guarantees (e.g.: C
code running on an embedded system, middleware)
• Cyber systems used to regulate physical plants that may
have tight latency requirements ( e.g.: smart grid)
• Uncertainty is inevitable in complex system design:
network latency, execution time, queuing delays, black-box
middleware queues, etc.
• Model-based characterization of uncertainty is useful for
capturing possible worst-case scenarios
UC Berkeley Introduction Ilge Akkaya 3 / 24

4. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Introduction
• Increasing number of high-throughput
sensors (i.e. Phasor Measurement Units
(PMUs)) being integrated into the power
grid
• Wide-area management and control
applications need to satisfy
• Accuracy
• Responsiveness
• Scalability requirements
• Middleware provides coordination and
alignment, at the expense of becoming
the bottleneck
Data
Concentrator
Data
Concentrator
.........
Data
Concentrator
MIDDLEWARE
HPC
Cluster
UC Berkeley Introduction Ilge Akkaya 4 / 24

5. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Heterogeneous Modeling in Ptolemy II
[Edward A. Lee et al., 2010]
UC Berkeley Introduction Ilge Akkaya 5 / 24

6. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Overview: Application Model
• An executable discrete-event Ptolemy model for a
three-area distributed smart-grid application given above
• PMU: Phasor Measurement Unit
• PDC: Phasor Data Concentrator
• Area: Balancing Authority running on a High Performance
Computing (HPC) Cluster
UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 6 / 24

7. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Sources of Uncertainty in Communication
Architectures
• Middleware
• Architecture generally does not scale well with increasing
number of sensor nodes
• Variable data aggregation latency
• Distributed applications
• Distributed State Estimation example: Computationally
expensive, iterative algorithm
• Number of iterations is a function of data quality
• Network
• Link capacity, length, queuing behavior
UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 7 / 24

8. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Capturing Uncertainty
• Complexity and cost of real testbeds promote model-based
performance evaluation of middleware
• We perform uncertainty modeling and analysis of
end-to-end distributed smart grid applications
• Monte Carlo sampling over the parameter space to
encapsulate uncertainties in
• number of sensor streams
• middleware capacity
• application run time
UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 8 / 24

9. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Case Study: Distributed State Estimation
• Distributed State Estimation (DSE) is an algorithm used
for estimating power system state
• Weighted Least-Squares based algorithm
• Distributed version developed to meet tight timing
deadlines
• Typically run every minute ( deadline=60s), likely to run
more frequently on the future power grid
UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 9 / 24

10. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Design Workflow for Uncertainty Analysis
• Parametrization of
uncertainty sources in the
model
• Monte Carlo sampling of the
parameter space
• Execution of the
parameterized model
• Regression analysis
• Design refinement of
influential parameters
UC Berkeley Uncertainty Models and Parameter-Space Exploration Ilge Akkaya 10 / 24

11. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Modeling Middleware Architecture
• Middleware modeled
as a thread pool
• Thread pool size is an
uncertainty parameter
• Stochastic delay per
thread processing
[Rician]
• Model trained using
Apache ActiveMQ
TM
benchmark results
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 11 / 24

12. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Distribution Fitting
Ti ∼ Rice(ν(NPMUi ), σ(NPMUi ))
ν(NPMUi ) = 0.0302 log(NPMUi ) + 0.055
σ(NPMUi ) = 0.0007 ∗ NPMUi + 0.0414
where NPMUi is the number of PMU streams at i’th Area
real
estimate
0
2
4
6
8
10
12
14
16
18
20
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Middleware Per File Delay Rician Distribution Fitting
time(s)
frequency
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 12 / 24

13. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Monte Carlo Simulation in Ptolemy II
Top-level data flow model generates
random parameter values
Inner Discrete-Event model is exe-
cuted with these parameters for 100
complete iterations ( 6000 s)
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 13 / 24

14. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Sample Simulation Traces: Time
PMU-PDC
PDC-MW
MW-HPC
p2p_in
END
0.0
0.5
1.0
1.5
2.0
2.5
x102
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Area I
time(s)
events
PMU-PDC
PDC-MW
MW-HPC
p2p_in
END
0.0
0.5
1.0
1.5
2.0
2.5
x102
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Area II
time(s)
events
PMU-PDC
PDC-MW
MW-HPC
p2p_in
END
0.0
0.5
1.0
1.5
2.0
2.5
x10
2
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Area III
time(s)
events
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 14 / 24

15. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Sample Simulation Traces: Distribution
sensor-to-concentrator
concentrator-to-MW
iteration runtime
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
0.0 10.0 20.0 122.5
Sample Latency Distribution
time(ms)
EventID
per packet MW delay
0
2
4
6
8
10
12
14
16
18
20
22
24
-0.2 -0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3
Middleware Processing Delay
time(ms)
EventID
Partial Latency Distri-
butions for one sam-
ple run of the DE
model. Captures the un-
certainty in HPC com-
putation [green], sen-
sor network [red], mid-
dleware network [blue]
and middleware process-
ing [black]
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 15 / 24

16. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Monte Carlo Parameter Space
Table : Monte Carlo Variables and Respective Probability Mass
Functions (range format:initial:increment:final )
Variable Parameter Name PMF Range
PMU_Count_1 Uniform 10:10:500
PMU_Count_2 Uniform 10:10:500
PMU_Count_3 Uniform 10:10:500
x1 concurrencyLevel Uniform 2:2:20
x3 numberOfIterations Uniform 1:1:20
x2 max{PMU_Count_1,PMU_Count_2,PMU_Count_3}
y maximum end-to-end run time per parameter sample
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 16 / 24

17. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Exploratory Data Analysis: Middleware
Concurrency
End-to-end deadline: 60s
For maximum number of sensor streams > 280, deadline misses observed at some
runs with lower middleware concurrency
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 17 / 24

18. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Exploratory Data Analysis: Number of DSE
Iterations
100 150 200 250 300 350 400 450
Maximum Number of PMUs per area
Monte−Carlo Simulation Results For Mean DSE Runtime
NumIterations= 1
NumIterations= 2
NumIterations= 3
NumIterations= 4
NumIterations= 5
NumIterations= 6
NumIterations= 7
NumIterations= 8
NumIterations= 9
NumIterations= 10
NumIterations= 11
NumIterations= 12
NumIterations= 13
NumIterations= 14
NumIterations= 15
NumIterations= 16
NumIterations= 17
NumIterations= 18
NumIterations= 19
NumIterations= 20
No observable correlation between number of iterations and run time
UC Berkeley Discrete-Event Modeling in Ptolemy II for Uncertainty Analysis Ilge Akkaya 18 / 24

19. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Relation Between PMU Count / Concurrency and
Maximum Run Time
UC Berkeley Regression Analysis Ilge Akkaya 19 / 24

20. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Regression Analysis
• Goal: to discover the impact of the number of algorithm
iterations, middleware capacity and number of sensor
streams on the aggregate end-to-end run time
• Polynomial regression with independent variables x1, x2, x3
and dependent variable y
Max PMU Count
Concurrency Level
MaxRuntime(s)
Figure : Polynomial regression fit with 5% confidence intervals
UC Berkeley Regression Analysis Ilge Akkaya 20 / 24

21. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Regression Analysis: Results
• Number of iterations (x3) was found to have little effect on the
dependent variable
• Coefficient estimates for x3 in regression tests were found to be
unreliable, confidence intervals centered around zero
• Data visualization supports regression results: iteration run time
quite insignificant; middleware processing time dominates
0
100
200
300
400
500
0
5
10
15
20
0
10
20
30
40
50
60
Number of PMUs
Number of Iterations vs Total #PMUs vs Runtime
Number of Iterations
MaxEnd−to−endRuntime(s)
UC Berkeley Regression Analysis Ilge Akkaya 21 / 24

22. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Findings
• Concurrency level or the deployment architecture of the
middleware should be improved to scale
• Number of DSE iterations is a minor concern ( some bad
data is tolerable at the expense of more iterations)
• Model-based Parameter Space Exploration provides
pre-deployment results for typical middleware
requirements. Future directions include
• Replacing network fabric/ middleware and re-simulation for
architectural comparisons
• Considering more sources of uncertainty
• Correct-by-construction uncertainty in systems - no
surprises in the end product
UC Berkeley Conclusion Ilge Akkaya 22 / 24

23. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Conclusion
• Distributed power applications are data intensive and
algorithm run times/network latency variance is high
among different deployments
• Pre-deployment analysis on architecture requirements for a
variable number of sensor streams need to be tested under
varying middleware capacity and latency conditions
• Presented a model-based approach for quantifying
uncertainty in middleware based applications
• Monte Carlo sampled parameter space used in populating
an executable heterogeneous model
• Discrete-Event simulation carried out to collect end-to-end
run time measurements
• Regression analysis carried out to account for significant
parameters that have the largest influence on limiting
behavior
UC Berkeley Conclusion Ilge Akkaya 23 / 24

24. 40 60 80 100 120
40
60
80
mmModeling
Uncertainty
For
Middleware-
based
Streaming
Power Grid
Applications
Ilge Akkaya,
Yan Liu,
Edward A.
Lee, Ian
Gorton
Introduction
Modeling
Uncertainty
DE Modeling
Regression
Analysis
Conclusion
Thank You
Questions ?
UC Berkeley Conclusion Ilge Akkaya 24 / 24