• Save
HBaseCon 2013: Being Smarter Than the Smart Meter
 

HBaseCon 2013: Being Smarter Than the Smart Meter

on

  • 1,777 views

Presented by: Jay Talreja, Oracle

Presented by: Jay Talreja, Oracle

Statistics

Views

Total Views
1,777
Views on SlideShare
1,777
Embed Views
0

Actions

Likes
3
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

HBaseCon 2013: Being Smarter Than the Smart Meter HBaseCon 2013: Being Smarter Than the Smart Meter Presentation Transcript

  • Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1 Being Smarter than the Smart Meter - Cloud Operational Grid Analytics Jay Talreja Senior Manager, Software Development UGBU
  • 2 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Agenda  Introduction  Smart Meters & Grid Analytics  Data Model  Filters, DataSets & Analytical Calc Engine  Operations  Conclusion  Q & A
  • 3 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction Startup that pioneered Analytics within the Smart Grid/Utilities Domain 2007 2010 2012
  • 4 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction  Early adopters of HBase  Python with Thrift 2007 2010 2012
  • 5 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Introduction  Acquired by Oracle in December 2012 2007 2010 2012
  • 6 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  Smart Meter Characteristics:  Sub daily energy reading (1 Min/5 Min/15 Min/30 Min/Hourly)  Time Series - highly granular data  Two Way Automated Communication  Power Outage Notification  Power Quality Monitoring  Smart Meter Promises:  Enable dynamic pricing  Improved Outage Management  Empower the end user with detailed energy usage
  • 7 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  Smart Metering dawns a new age in Grid Analytics  Real Time Accessibility  Data explosion (~ 1000 fold increase in data collection)  What makes the grid smart ?  Not Big Data and the capacity to store it  But the capability to identity patterns hidden within the terabytes of data  What is needed then is a Smart Grid platform that can:  Help identify theft  Help predict system failures before they occur  Help utilities better manage their operations
  • 8 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics How can I accurately identify homes that are consuming more energy than their neighbors ? It’s going to be a hot summer this year !! Are my devices sized correctly ? No blackouts please !! Revenue loss (due to theft) is a growing concern. How can I identify theft patterns now that I have smart meters ? Distribution Planning Revenue Protection Energy Efficiency
  • 9 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter & Grid Analytics  The Smart Meter Analytics platform that we built is:  Cloud based – results delivered via the Web  Supports sub second (~ 100 ms) data retrieval that powers the User Interface and enables data visualization  Supports batch based analytics  We have developed our own distributed framework in Python  All interaction with Hbase is via the Thrift API  Highly Configurable  Allows analysts and other non technical groups to implement generic algorithms  Shared middle tier that serves both the User Interface and allows exploratory analytics
  • 10 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. With HBase and it’s distributed storage the platform can easily scale to meet the analytics needs of the biggest utilities across the globe Smart Meter & Grid Analytics  The biggest cluster (16 Nodes) (so far) has 2 years worth of history for 4 Million Smart meters  25 TB  ~ 7 Billion Rows  ~ 500 Billion Values  Multiple clusters – shared as well as dedicated  All new clusters to run on Oracle’s Big Data Appliance
  • 11 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Smart Meter Analytics Platform HBase (CDH4) HDFS (Oracle Hadoop -CDH4) E T L Dataset (Configurable Query API) Analytic Engine User Interface Smart Grid Data Weather 3rd Party Data Export back to Smart Grid Filter
  • 12 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Data Model Fact Point Time Value • Abstract data model • Generic • Extensible • Storage in HBase mirrors the Access Patterns Fact Point Time Value Hourly kWh Meter xyz June 13th 2013 13:00 0.0875 Hourly Temp. F KDCA June 13th 2013 13:00 75 Power Out Event Meter xyz June 13th 2013 13:00 NULL
  • 13 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine NO SQL Data Sets Dataset and Filters provide a configurable querying capability that provides fast access to the terabytes of time series data stored in HBase
  • 14 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Analytical CalcDB Routines Filters, DataSets & Analytical Calc Engine SELECT WHERE GROUP BY DATASET FIELDS FILTER COMPONENTS TIME WINDOWS /METRICS
  • 15 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • Filters are akin to the WHERE clause of a SQL query • Data driven and configurable • In Batch analytics • To operate on a subset of the population that satisfy filtering criteria • In the User Interface • Visualize data for select points that satisfy the filtering criteria Find all meters where the hourly consumption for last week is between 0.5 and 1 kWh ? Also, only find meters that are in my zip code
  • 16 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • Operate on the point population as determined by the Filter • Consist of fields (like columns in a SELECT query) • Each dataset field can look at data for a different time period (time windows) and aggregate it to a single value (Aggregate Functions) • Datasets support aggregate metrics out of the box • e.g. SUM/MIN/MAX/STD DEV./Nth Metrics etc For the meters that I selected, give me the average daily consumption over the past week, past year and last summer
  • 17 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Filters, DataSets & Analytical Calc Engine • The Analytic calc engine can be compared to a DB routine (function/stored procedure) • A calc follows a graphical execution model and lets developers implement custom logic • Allow complex analytics to be run and let data be saved back to Hbase Compare the average consumption and save only meters where last summer’s consumption was greater
  • 18 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Cluster Operations
  • 19 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Conclusion  The generic data model in conjunction with HBase’s schema less storage has enabled us to build a Smart Meter Analytics platform that can scale up to meet the Big Data needs in the Smart Grid/Utilities Domain  By mapping storage in HBase to the data access patterns we have successfully used the platform to serve both real time and batch analytics  Filters ,DataSets offer a powerful, expressive, configuration based querying capability and attempt to bridge the NoSQL gap  From our experience, Hbase has proven to be a robust, resilient distributed data storage with low latency random access easily manageable by a few developers
  • 20 Copyright © 2013 Oracle and/or its affiliates. All rights reserved. Questions ?