Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Being Smarter than the Smart Meter
- Cloud Operational Grid Analytics
Jay Talreja
Senior Manager, Software Development
UGBU

2 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Agenda
 Introduction
 Smart Meters & Grid Analytics
 Data Model
 Filters, DataSets & Analytical Calc Engine
 Operations
 Conclusion
 Q & A

3 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
Startup that pioneered
Analytics within the
Smart Grid/Utilities Domain
2007
2010
2012

4 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
 Early adopters of
HBase
 Python with Thrift
2007
2010
2012

5 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
 Acquired by Oracle in
December 2012
2007
2010
2012

6 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 Smart Meter Characteristics:
 Sub daily energy reading (1 Min/5 Min/15 Min/30 Min/Hourly)
 Time Series - highly granular data
 Two Way Automated Communication
 Power Outage Notification
 Power Quality Monitoring
 Smart Meter Promises:
 Enable dynamic pricing
 Improved Outage Management
 Empower the end user with detailed energy usage

7 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 Smart Metering dawns a new age in Grid Analytics
 Real Time Accessibility
 Data explosion (~ 1000 fold increase in data collection)
 What makes the grid smart ?
 Not Big Data and the capacity to store it
 But the capability to identity patterns hidden within the terabytes of
data
 What is needed then is a Smart Grid platform that can:
 Help identify theft
 Help predict system failures before they occur
 Help utilities better manage their operations

8 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
How can I accurately
identify homes that
are consuming more
energy than their
neighbors ?
It’s going to be a hot
summer this year !!
Are my devices sized
correctly ? No
blackouts please !!
Revenue loss (due to
theft) is a growing
concern. How can I
identify theft patterns
now that I have smart
meters ?
Distribution
Planning
Revenue
Protection
Energy
Efficiency

9 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 The Smart Meter Analytics platform that we built is:
 Cloud based – results delivered via the Web
 Supports sub second (~ 100 ms) data retrieval that powers the
User Interface and enables data visualization
 Supports batch based analytics
 We have developed our own distributed framework in Python
 All interaction with Hbase is via the Thrift API
 Highly Configurable
 Allows analysts and other non technical groups to implement
generic algorithms
 Shared middle tier that serves both the User Interface and allows
exploratory analytics

10 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
With HBase and it’s distributed storage the platform can easily
scale to meet the analytics needs of the biggest utilities across
the globe
Smart Meter & Grid Analytics
 The biggest cluster (16 Nodes) (so far) has 2 years worth of
history for 4 Million Smart meters
 25 TB
 ~ 7 Billion Rows
 ~ 500 Billion Values
 Multiple clusters – shared as well as dedicated
 All new clusters to run on Oracle’s Big Data Appliance

11 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter Analytics Platform
HBase
(CDH4)
HDFS
(Oracle Hadoop
-CDH4)
E
T
L
Dataset
(Configurable
Query API)
Analytic
Engine
User
Interface
Smart
Grid
Data
Weather
3rd
Party
Data
Export
back to
Smart
Grid
Filter

12 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Data Model
Fact
Point
Time
Value
• Abstract data model
• Generic
• Extensible
• Storage in HBase mirrors
the Access Patterns
Fact Point Time Value
Hourly kWh Meter xyz June 13th 2013
13:00
0.0875
Hourly Temp. F KDCA June 13th 2013
13:00
75
Power Out
Event
Meter xyz June 13th 2013
13:00
NULL

13 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
NO SQL Data Sets
Dataset and Filters provide a configurable querying capability that
provides fast access to the terabytes of time series data stored in HBase

14 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Analytical CalcDB Routines
Filters, DataSets & Analytical Calc Engine
SELECT
WHERE
GROUP
BY
DATASET
FIELDS
FILTER
COMPONENTS
TIME WINDOWS
/METRICS

15 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• Filters are akin to the WHERE clause of a SQL query
• Data driven and configurable
• In Batch analytics
• To operate on a subset of the population that satisfy filtering criteria
• In the User Interface
• Visualize data for select points that satisfy the filtering criteria
Find all meters where the hourly
consumption for last week is between 0.5
and 1 kWh ? Also, only find meters that
are in my zip code

16 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• Operate on the point population as determined by the Filter
• Consist of fields (like columns in a SELECT query)
• Each dataset field can look at data for a different time period (time
windows) and aggregate it to a single value (Aggregate Functions)
• Datasets support aggregate metrics out of the box
• e.g. SUM/MIN/MAX/STD DEV./Nth Metrics etc
For the meters that I selected, give me
the average daily consumption over the
past week, past year and last summer

17 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• The Analytic calc engine can be compared to a DB routine
(function/stored procedure)
• A calc follows a graphical execution model and lets developers
implement custom logic
• Allow complex analytics to be run and let data be saved back to
Hbase
Compare the average consumption and
save only meters where last summer’s
consumption was greater

18 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Cluster Operations

19 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Conclusion
 The generic data model in conjunction with HBase’s schema less
storage has enabled us to build a Smart Meter Analytics platform that
can scale up to meet the Big Data needs in the Smart Grid/Utilities
Domain
 By mapping storage in HBase to the data access patterns we have
successfully used the platform to serve both real time and batch
analytics
 Filters ,DataSets offer a powerful, expressive, configuration based
querying capability and attempt to bridge the NoSQL gap
 From our experience, Hbase has proven to be a robust, resilient
distributed data storage with low latency random access easily
manageable by a few developers

20 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Questions ?