• Save
HBaseCon 2013: Being Smarter Than the Smart Meter

Like this? Share it with your network

Share

HBaseCon 2013: Being Smarter Than the Smart Meter

  • 1,894 views
Uploaded on

Presented by: Jay Talreja, Oracle

Presented by: Jay Talreja, Oracle

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,894
On Slideshare
1,894
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1 Being Smarter than the Smart Meter - Cloud Operational Grid Analytics Jay Talreja Senior Manager, Software Development UGBU
  • 2. 2 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Agenda  Introduction  Smart Meters & Grid Analytics  Data Model  Filters, DataSets & Analytical Calc Engine  Operations  Conclusion  Q & A
  • 3. 3 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction Startup that pioneered Analytics within the Smart Grid/Utilities Domain 2007 2010 2012
  • 4. 4 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction  Early adopters of HBase  Python with Thrift 2007 2010 2012
  • 5. 5 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction  Acquired by Oracle in December 2012 2007 2010 2012
  • 6. 6 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  Smart Meter Characteristics:  Sub daily energy reading (1 Min/5 Min/15 Min/30 Min/Hourly)  Time Series - highly granular data  Two Way Automated Communication  Power Outage Notification  Power Quality Monitoring  Smart Meter Promises:  Enable dynamic pricing  Improved Outage Management  Empower the end user with detailed energy usage
  • 7. 7 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  Smart Metering dawns a new age in Grid Analytics  Real Time Accessibility  Data explosion (~ 1000 fold increase in data collection)  What makes the grid smart ?  Not Big Data and the capacity to store it  But the capability to identity patterns hidden within the terabytes of data  What is needed then is a Smart Grid platform that can:  Help identify theft  Help predict system failures before they occur  Help utilities better manage their operations
  • 8. 8 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics How can I accurately identify homes that are consuming more energy than their neighbors ? It’s going to be a hot summer this year !! Are my devices sized correctly ? No blackouts please !! Revenue loss (due to theft) is a growing concern. How can I identify theft patterns now that I have smart meters ? Distribution Planning Revenue Protection Energy Efficiency
  • 9. 9 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  The Smart Meter Analytics platform that we built is:  Cloud based – results delivered via the Web  Supports sub second (~ 100 ms) data retrieval that powers the User Interface and enables data visualization  Supports batch based analytics  We have developed our own distributed framework in Python  All interaction with Hbase is via the Thrift API  Highly Configurable  Allows analysts and other non technical groups to implement generic algorithms  Shared middle tier that serves both the User Interface and allows exploratory analytics
  • 10. 10 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. With HBase and it’s distributed storage the platform can easily scale to meet the analytics needs of the biggest utilities across the globe Smart Meter & Grid Analytics  The biggest cluster (16 Nodes) (so far) has 2 years worth of history for 4 Million Smart meters  25 TB  ~ 7 Billion Rows  ~ 500 Billion Values  Multiple clusters – shared as well as dedicated  All new clusters to run on Oracle’s Big Data Appliance
  • 11. 11 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter Analytics Platform HBase (CDH4) HDFS (Oracle Hadoop -CDH4) E T L Dataset (Configurable Query API) Analytic Engine User Interface Smart Grid Data Weather 3rd Party Data Export back to Smart Grid Filter
  • 12. 12 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Data Model Fact Point Time Value • Abstract data model • Generic • Extensible • Storage in HBase mirrors the Access Patterns Fact Point Time Value Hourly kWh Meter xyz June 13th 2013 13:00 0.0875 Hourly Temp. F KDCA June 13th 2013 13:00 75 Power Out Event Meter xyz June 13th 2013 13:00 NULL
  • 13. 13 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine NO SQL Data Sets Dataset and Filters provide a configurable querying capability that provides fast access to the terabytes of time series data stored in HBase
  • 14. 14 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Analytical CalcDB Routines Filters, DataSets & Analytical Calc Engine SELECT WHERE GROUP BY DATASET FIELDS FILTER COMPONENTS TIME WINDOWS /METRICS
  • 15. 15 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • Filters are akin to the WHERE clause of a SQL query • Data driven and configurable • In Batch analytics • To operate on a subset of the population that satisfy filtering criteria • In the User Interface • Visualize data for select points that satisfy the filtering criteria Find all meters where the hourly consumption for last week is between 0.5 and 1 kWh ? Also, only find meters that are in my zip code
  • 16. 16 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • Operate on the point population as determined by the Filter • Consist of fields (like columns in a SELECT query) • Each dataset field can look at data for a different time period (time windows) and aggregate it to a single value (Aggregate Functions) • Datasets support aggregate metrics out of the box • e.g. SUM/MIN/MAX/STD DEV./Nth Metrics etc For the meters that I selected, give me the average daily consumption over the past week, past year and last summer
  • 17. 17 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • The Analytic calc engine can be compared to a DB routine (function/stored procedure) • A calc follows a graphical execution model and lets developers implement custom logic • Allow complex analytics to be run and let data be saved back to Hbase Compare the average consumption and save only meters where last summer’s consumption was greater
  • 18. 18 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Cluster Operations
  • 19. 19 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Conclusion  The generic data model in conjunction with HBase’s schema less storage has enabled us to build a Smart Meter Analytics platform that can scale up to meet the Big Data needs in the Smart Grid/Utilities Domain  By mapping storage in HBase to the data access patterns we have successfully used the platform to serve both real time and batch analytics  Filters ,DataSets offer a powerful, expressive, configuration based querying capability and attempt to bridge the NoSQL gap  From our experience, Hbase has proven to be a robust, resilient distributed data storage with low latency random access easily manageable by a few developers
  • 20. 20 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Questions ?