The document discusses a smart meter analytics platform that was acquired by Oracle in 2012. It uses HBase for distributed storage of smart meter data and provides real-time and batch analytics capabilities. Filters and datasets allow configurable querying of time series data stored in HBase to power a user interface and exploratory analytics. The platform can scale to support the analytics needs of large utilities with billions of data points from millions of smart meters stored across multiple HBase clusters.

1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Being Smarter than the Smart Meter
- Cloud Operational Grid Analytics
Jay Talreja
Senior Manager, Software Development
UGBU

2. 2 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Agenda
 Introduction
 Smart Meters & Grid Analytics
 Data Model
 Filters, DataSets & Analytical Calc Engine
 Operations
 Conclusion
 Q & A

3. 3 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
Startup that pioneered
Analytics within the
Smart Grid/Utilities Domain
2007
2010
2012

4. 4 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
 Early adopters of
HBase
 Python with Thrift
2007
2010
2012

5. 5 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
 Acquired by Oracle in
December 2012
2007
2010
2012

6. 6 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 Smart Meter Characteristics:
 Sub daily energy reading (1 Min/5 Min/15 Min/30 Min/Hourly)
 Time Series - highly granular data
 Two Way Automated Communication
 Power Outage Notification
 Power Quality Monitoring
 Smart Meter Promises:
 Enable dynamic pricing
 Improved Outage Management
 Empower the end user with detailed energy usage

7. 7 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 Smart Metering dawns a new age in Grid Analytics
 Real Time Accessibility
 Data explosion (~ 1000 fold increase in data collection)
 What makes the grid smart ?
 Not Big Data and the capacity to store it
 But the capability to identity patterns hidden within the terabytes of
data
 What is needed then is a Smart Grid platform that can:
 Help identify theft
 Help predict system failures before they occur
 Help utilities better manage their operations

8. 8 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
How can I accurately
identify homes that
are consuming more
energy than their
neighbors ?
It’s going to be a hot
summer this year !!
Are my devices sized
correctly ? No
blackouts please !!
Revenue loss (due to
theft) is a growing
concern. How can I
identify theft patterns
now that I have smart
meters ?
Distribution
Planning
Revenue
Protection
Energy
Efficiency

9. 9 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 The Smart Meter Analytics platform that we built is:
 Cloud based – results delivered via the Web
 Supports sub second (~ 100 ms) data retrieval that powers the
User Interface and enables data visualization
 Supports batch based analytics
 We have developed our own distributed framework in Python
 All interaction with Hbase is via the Thrift API
 Highly Configurable
 Allows analysts and other non technical groups to implement
generic algorithms
 Shared middle tier that serves both the User Interface and allows
exploratory analytics

10. 10 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
With HBase and it’s distributed storage the platform can easily
scale to meet the analytics needs of the biggest utilities across
the globe
Smart Meter & Grid Analytics
 The biggest cluster (16 Nodes) (so far) has 2 years worth of
history for 4 Million Smart meters
 25 TB
 ~ 7 Billion Rows
 ~ 500 Billion Values
 Multiple clusters – shared as well as dedicated
 All new clusters to run on Oracle’s Big Data Appliance

11. 11 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter Analytics Platform
HBase
(CDH4)
HDFS
(Oracle Hadoop
-CDH4)
E
T
L
Dataset
(Configurable
Query API)
Analytic
Engine
User
Interface
Smart
Grid
Data
Weather
3rd
Party
Data
Export
back to
Smart
Grid
Filter

12. 12 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Data Model
Fact
Point
Time
Value
• Abstract data model
• Generic
• Extensible
• Storage in HBase mirrors
the Access Patterns
Fact Point Time Value
Hourly kWh Meter xyz June 13th 2013
13:00
0.0875
Hourly Temp. F KDCA June 13th 2013
13:00
75
Power Out
Event
Meter xyz June 13th 2013
13:00
NULL

13. 13 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
NO SQL Data Sets
Dataset and Filters provide a configurable querying capability that
provides fast access to the terabytes of time series data stored in HBase

14. 14 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Analytical CalcDB Routines
Filters, DataSets & Analytical Calc Engine
SELECT
WHERE
GROUP
BY
DATASET
FIELDS
FILTER
COMPONENTS
TIME WINDOWS
/METRICS

15. 15 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• Filters are akin to the WHERE clause of a SQL query
• Data driven and configurable
• In Batch analytics
• To operate on a subset of the population that satisfy filtering criteria
• In the User Interface
• Visualize data for select points that satisfy the filtering criteria
Find all meters where the hourly
consumption for last week is between 0.5
and 1 kWh ? Also, only find meters that
are in my zip code

16. 16 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• Operate on the point population as determined by the Filter
• Consist of fields (like columns in a SELECT query)
• Each dataset field can look at data for a different time period (time
windows) and aggregate it to a single value (Aggregate Functions)
• Datasets support aggregate metrics out of the box
• e.g. SUM/MIN/MAX/STD DEV./Nth Metrics etc
For the meters that I selected, give me
the average daily consumption over the
past week, past year and last summer

17. 17 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• The Analytic calc engine can be compared to a DB routine
(function/stored procedure)
• A calc follows a graphical execution model and lets developers
implement custom logic
• Allow complex analytics to be run and let data be saved back to
Hbase
Compare the average consumption and
save only meters where last summer’s
consumption was greater

18. 18 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Cluster Operations

19. 19 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Conclusion
 The generic data model in conjunction with HBase’s schema less
storage has enabled us to build a Smart Meter Analytics platform that
can scale up to meet the Big Data needs in the Smart Grid/Utilities
Domain
 By mapping storage in HBase to the data access patterns we have
successfully used the platform to serve both real time and batch
analytics
 Filters ,DataSets offer a powerful, expressive, configuration based
querying capability and attempt to bridge the NoSQL gap
 From our experience, Hbase has proven to be a robust, resilient
distributed data storage with low latency random access easily
manageable by a few developers

20. 20 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Questions ?