Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Being Smarter than the Smart Meter
- Cloud Operation...
2 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Agenda
 Introduction
 Smart Meters & Grid Analytic...
3 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
Startup that pioneered
Analytics within...
4 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
 Early adopters of
HBase
 Python with...
5 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Introduction
 Acquired by Oracle in
December 2012
2...
6 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 Smart Meter Character...
7 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 Smart Metering dawns ...
8 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
How can I accurately
id...
9 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter & Grid Analytics
 The Smart Meter Analy...
10 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
With HBase and it’s distributed storage the platfor...
11 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Smart Meter Analytics Platform
HBase
(CDH4)
HDFS
(O...
12 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Data Model
Fact
Point
Time
Value
• Abstract data mo...
13 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
NO SQL D...
14 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Analytical CalcDB Routines
Filters, DataSets & Anal...
15 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• Filter...
16 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• Operat...
17 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Filters, DataSets & Analytical Calc Engine
• The An...
18 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Cluster Operations
19 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Conclusion
 The generic data model in conjunction ...
20 Copyright © 2013 Oracle and/or its affiliates. All rights reserved.
Questions ?
Upcoming SlideShare
Loading in...5
×

HBaseCon 2013: Being Smarter Than the Smart Meter

1,096

Published on

Presented by: Jay Talreja, Oracle

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,096
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Transcript of "HBaseCon 2013: Being Smarter Than the Smart Meter"

  1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1 Being Smarter than the Smart Meter - Cloud Operational Grid Analytics Jay Talreja Senior Manager, Software Development UGBU
  2. 2. 2 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Agenda  Introduction  Smart Meters & Grid Analytics  Data Model  Filters, DataSets & Analytical Calc Engine  Operations  Conclusion  Q & A
  3. 3. 3 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction Startup that pioneered Analytics within the Smart Grid/Utilities Domain 2007 2010 2012
  4. 4. 4 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction  Early adopters of HBase  Python with Thrift 2007 2010 2012
  5. 5. 5 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction  Acquired by Oracle in December 2012 2007 2010 2012
  6. 6. 6 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  Smart Meter Characteristics:  Sub daily energy reading (1 Min/5 Min/15 Min/30 Min/Hourly)  Time Series - highly granular data  Two Way Automated Communication  Power Outage Notification  Power Quality Monitoring  Smart Meter Promises:  Enable dynamic pricing  Improved Outage Management  Empower the end user with detailed energy usage
  7. 7. 7 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  Smart Metering dawns a new age in Grid Analytics  Real Time Accessibility  Data explosion (~ 1000 fold increase in data collection)  What makes the grid smart ?  Not Big Data and the capacity to store it  But the capability to identity patterns hidden within the terabytes of data  What is needed then is a Smart Grid platform that can:  Help identify theft  Help predict system failures before they occur  Help utilities better manage their operations
  8. 8. 8 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics How can I accurately identify homes that are consuming more energy than their neighbors ? It’s going to be a hot summer this year !! Are my devices sized correctly ? No blackouts please !! Revenue loss (due to theft) is a growing concern. How can I identify theft patterns now that I have smart meters ? Distribution Planning Revenue Protection Energy Efficiency
  9. 9. 9 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  The Smart Meter Analytics platform that we built is:  Cloud based – results delivered via the Web  Supports sub second (~ 100 ms) data retrieval that powers the User Interface and enables data visualization  Supports batch based analytics  We have developed our own distributed framework in Python  All interaction with Hbase is via the Thrift API  Highly Configurable  Allows analysts and other non technical groups to implement generic algorithms  Shared middle tier that serves both the User Interface and allows exploratory analytics
  10. 10. 10 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. With HBase and it’s distributed storage the platform can easily scale to meet the analytics needs of the biggest utilities across the globe Smart Meter & Grid Analytics  The biggest cluster (16 Nodes) (so far) has 2 years worth of history for 4 Million Smart meters  25 TB  ~ 7 Billion Rows  ~ 500 Billion Values  Multiple clusters – shared as well as dedicated  All new clusters to run on Oracle’s Big Data Appliance
  11. 11. 11 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter Analytics Platform HBase (CDH4) HDFS (Oracle Hadoop -CDH4) E T L Dataset (Configurable Query API) Analytic Engine User Interface Smart Grid Data Weather 3rd Party Data Export back to Smart Grid Filter
  12. 12. 12 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Data Model Fact Point Time Value • Abstract data model • Generic • Extensible • Storage in HBase mirrors the Access Patterns Fact Point Time Value Hourly kWh Meter xyz June 13th 2013 13:00 0.0875 Hourly Temp. F KDCA June 13th 2013 13:00 75 Power Out Event Meter xyz June 13th 2013 13:00 NULL
  13. 13. 13 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine NO SQL Data Sets Dataset and Filters provide a configurable querying capability that provides fast access to the terabytes of time series data stored in HBase
  14. 14. 14 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Analytical CalcDB Routines Filters, DataSets & Analytical Calc Engine SELECT WHERE GROUP BY DATASET FIELDS FILTER COMPONENTS TIME WINDOWS /METRICS
  15. 15. 15 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • Filters are akin to the WHERE clause of a SQL query • Data driven and configurable • In Batch analytics • To operate on a subset of the population that satisfy filtering criteria • In the User Interface • Visualize data for select points that satisfy the filtering criteria Find all meters where the hourly consumption for last week is between 0.5 and 1 kWh ? Also, only find meters that are in my zip code
  16. 16. 16 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • Operate on the point population as determined by the Filter • Consist of fields (like columns in a SELECT query) • Each dataset field can look at data for a different time period (time windows) and aggregate it to a single value (Aggregate Functions) • Datasets support aggregate metrics out of the box • e.g. SUM/MIN/MAX/STD DEV./Nth Metrics etc For the meters that I selected, give me the average daily consumption over the past week, past year and last summer
  17. 17. 17 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • The Analytic calc engine can be compared to a DB routine (function/stored procedure) • A calc follows a graphical execution model and lets developers implement custom logic • Allow complex analytics to be run and let data be saved back to Hbase Compare the average consumption and save only meters where last summer’s consumption was greater
  18. 18. 18 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Cluster Operations
  19. 19. 19 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Conclusion  The generic data model in conjunction with HBase’s schema less storage has enabled us to build a Smart Meter Analytics platform that can scale up to meet the Big Data needs in the Smart Grid/Utilities Domain  By mapping storage in HBase to the data access patterns we have successfully used the platform to serve both real time and batch analytics  Filters ,DataSets offer a powerful, expressive, configuration based querying capability and attempt to bridge the NoSQL gap  From our experience, Hbase has proven to be a robust, resilient distributed data storage with low latency random access easily manageable by a few developers
  20. 20. 20 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Questions ?

×