Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Meter Data Management Presentation


Published on

Meter Data Management Presentation

  1. 1. The Hidden Challenge in Meter DataManagement: What You Dont Know Will Hurt You! An InformationWeek Webcast Sponsored by
  2. 2. Webcast Logistics
  3. 3. Today’s Presenters Jill Feblowitz, Vice President, Utilities and Oil and Gas, IDC Energy Insights Kevin Brown, Chief Architect, IBM Informix Database
  4. 4. Dealing withSmart Grid DataJill Feblowitz, Vice President,IDC Energy Insights
  5. 5. Agenda Smart Meter Outlook Management Challenges IT Challenges The Way Forward Recommendations© IDC Energy Insights Page 5
  6. 6. It’s All About More Interaction© IDC Energy Insights Page 6
  7. 7. Smart Meter Outlook A Decade of Global Growth in Smart Metering 2015 AP and LA will drive the global smart 2009 North meter marketplace America overtakes Europe in Smart 2005 North Meter shipments 2002 First AMI American utilities deployments begin in begin large-scale North America AMI deployments 2001 Enel begins AMI deployment© IDC Energy Insights Page 7
  8. 8. Smart Meter Outlook Data Deluge  85 million installed meters by 2015 according to the IDC Energy Insights Smart Meter Tracker  Each meter has the potential of producing a vast amount of interval data: 12 consumption events for a year vs. 8760 billing events with a 1 hour interval  Not to mention other on-going communications and transactions© IDC Energy Insights Page 8
  9. 9. Smart Grid Outlook IT Spending on Smart Grid, 2011 Hardware Services Software It is not just about smart meters at the home – it‟s about making the grid smarter, too.  Smart meters at the transformer  Intelligence electronic devices.© IDC Energy Insights Page 9
  10. 10. Management ChallengesChange in Business Process… Meterto Cash Before Smart Meters 60 to 90 days Data Collection Bill Calc Fulfillment Payment & Collections Customer Relationship Management (CRM) Customer Bill Print EBPP information System (CIS) Meter Lock Box (residential) Bill Mail Credit and Complex Billing Collections Electronic Bill Advanced Meter Payment and Kiosk, Local Office, (C&I) Meter data Presentment Mail management (EBPP) Meter Reading Accounting Accounting Accounting Customer Service© IDC Energy Insights Page 10
  11. 11. Management Challenges (and Opportunities) Change in Business Process… Meter to Cash Future Customer Sees The Data 7 to 90 days Data Collection Data Transport Bill Calc Fulfillment Payment & and Assessment Collections CRM Smart Phone Smart Phone Web Portal Web Portal In-home EBPP Smart Display Meter Advanced Metering CIS Lock Box Infrastructure EBPP Bill Print Credit and Bill Mail Collections Advanced Meter Kiosk, Local Office, (C&I) Mail MDM Network Operating Accounting Accounting Accounting Center - Data. Analysis Customer Service © IDC Energy Insights Page 11
  12. 12. Management Challenges The business case must be supported. Type of Benefit Source of Benefit Revenue Assurance • Remote connect/disconnect • Theft and tamper detection • Analysis of billing data to detect unbilled accounts • Pre-payment Operational • Reduced truck rolls for connect/disconnect, high bill Efficiency complaints • More efficient deployment of workforce in outage based on data Deferred Capital • Support for demand response in capacity constrained Investment areas • Prioritization of equipment replacement Increased Reliability • Predictive analytics applied to condition-based monitoring • Automated switching routines to minimize outage impact© IDC Energy Insights Page 12
  13. 13. Management Challenges There Much More Left to Do with the Meter Data n=26Source: IDC Energy Insights, Utility CIO Survey, 2010n=26 © IDC Energy Insights Page 13
  14. 14. Management Challenges Supporting New Pricing, Services  Expanded offerings: energy  Still to come… charge efficiency, demand response, rates? green energy rate  New Pricing: Time-based, critical peak pricing  New Relationships: “Prosumer” and net metering for PV  Tools to Build Relationship and Awareness: carbon footprint, energy efficiency and savings  Pre-pay and budget notification© IDC Energy Insights Page 14
  15. 15. IT Challenges Data explosion calls for data storage and handling and much more.Managing Meter Data Managing T&D Grid DataAchieving acceptable levels of production for Determining the balance between centralized orbilling and customer presentment distributed (device) computingOptimizing performance and utilization of storage Optimizing performance and utilization of storagespecific to the workload specific to the workloadMaking the right data available for production Making the right data available for operations(billing and customer presentment) and analytics and analyticsRetaining and archiving billing data to meet Securing the smart grid telecommunicationsregulatory requirements and satisfy business network from incursion by hackerscontinuityManaging data, alerting about data irregularities, Managing data, alerting about data irregularities,and resolving inconsistencies and resolving inconsistenciesProtecting privacy of customer data Integrating old and new communication infrastructure to support secure data communicationsEstablishing consistent data synchronization, data Establishing consistent data synchronization, datamodels, and protocols models, and protocolsSecuring the AMI network from incursion by hackers Minimizing network traffic with high device data production given new devices that produce high data volumes© IDC Energy Insights Page 15
  16. 16. IT Challenges Examples of Volumes in our Study Number of Intervals Frequency of Data Processed Rention in Active Meters Currently Data Collection on a Daily Basis Database Deployed (per day) (gigabytes) (years) (million) Utility A 1.32 15 minute 3 times 4.752 1.5 Utility B 1.40 Hourly 2 times 12 3.0 Utility C 0.70 Hourly 1 time 4 2.0 Utility D 2.00 15 minute 6 times 70 1.1 “Utilities are not accustomed to managing or processing this much data, I mean there‟s been no reason to…Probably the largest amount of data they work with has been in the billing world and maybe some GIS, but this volume is much larger, especially on a daily basis, so it really pushes on how well you architect something and you‟re using the right tools in your systems and your databases are properly architected."© IDC Energy Insights Page 16
  17. 17. IT Challenges Details, details, details…examples Time to process and speed of processing – Re-interrogating the meters to get missing reads and avoid duplication – “ If a system goes down or the network is down and you don’t get it resolved quickly, then when you start piling up two days’ worth of data, then it becomes a challenge because it takes longer.” – Service levels mandated by PUCs or service level agreements between IT and the business  8:00 AM presentment of previous day’s usage  Need to respond within four hours if system goes down  Target of first time read success and presentation at SLA of 99.5% – “Where the bigger challenge is going to be is managing flows on the system if everybody wants to do load control simultaneously, that’s going to be a little more challenging to us and it’s going to be interesting to see how those challenges work out.” – “We’ve had everything on the MDM from a bad index to a bad spot on the disk space to we needed more horsepower.” Ease of access – “If you wanted to see a month’s worth of data on the meter, you’d have to go through 30 different files. It’s not very user friendly.” – “We have to go through essentially 60 millions rows of data per day.”© IDC Energy Insights Page 17
  18. 18. Looking Forward Approaches  Operational data stores – To bring in other data – To off load data so there is no impact on “production” of the MDM  Changed business process  Developed retention and archiving protocols  “Changed how EDI moves data from one platform to another”  “Changed database schemas on mainframe”  Re-partitioning – “low hanging fruit”  Adding servers and storage. Cluster servers.  Considering high end compression techniques “[We have been working with meter data for about 3 years, and in that time, we have] “changed out systems two or three times.” “We‟ve been pretty aggressive in providing a quality system to this market, so we have already taken the steps to get what we think is the right level of hardware and the software products…We‟ve spent some money, unfortunately.”© IDC Energy Insights Page 18
  19. 19. Looking Forward Managing the Information: Wish List  Shorten processing time to meet regulatory and internal SLAs  Build analytics into the database for faster querying  Reduce hardware and software costs related to server, storage and RDBMS.  “Loss less” storage is useful in areas where time series data must keep its fidelity, such as predictive maintenance© IDC Energy Insights Page 19
  20. 20. Recommendations  Develop a data retention policy and investigate what needs to be kept online and for how long. Do not forget to include customer needs for presentment. Start early to evaluate how others in your organization (load research, capital planning, etc.) will access data.  Ask vendors for a “proof of concept”. Most vendors are willing to help by running test data sets using their technology.  Start by understand what current and future requirements for processing speed, storage and data access will be for your company and how these demands will ramp up over time.  Do your due diligence. Based on scenarios, investigate your options for processing and storage and the total cost of ownership associated with these. Do not assume that by adding more servers and storage is the most cost effective approach.© IDC Energy Insights Page 20
  21. 21. Powering Large Volumes of MeterData with InformixKevin Brown, IBM Informix Chief Information Management © 2011 IBM Corporation
  22. 22. Information ManagementChanging Storage Requirements Changing Workloads For 10 Million Smart Meters: Today – Each meter read once per month Very soon – Each meter read once every 15 minutes Regulations – Need to keep data on line for 3 years (PUC) and, perhaps, save for 7 years Smart Meter Interval Data 350.4B# of RecordsPer Year for 10M meters 3.65B 120M Frequency Monthly Meter Daily Meter 15 Minute Meter of reads Reads Reads Reads 22 © 2011 IBM Corporation
  23. 23. Information ManagementKeeping up with Smart Meter Data Large amounts of data causes problems in 2 areas: 1) Storage management • Large storage = Expensive • Cumbersome to maintain – Requires sophisticated partitioning schemes – Reorganization often required, which leads to downtime 2) Query performance • Compliance Reports must be completed before the end of each day • Customer portal queries must be handled in a timely manner • Customer billing must be completed each day23 © 2011 IBM Corporation
  24. 24. Information ManagementInformix TimeSeries: The Solution for Managing Meter Data  Time Series: – A logically connected set of records ordered by time  Performance – Extremely fast data access • Up to 60 times faster than competition – Handles operations hard or impossible to do in standard SQL  Space Savings – Saves at least 50% over standard relational layout  Toolkit approach – Develop algorithms that run directly in the database  Easier – Extremely low maintenance24 © 2011 IBM Corporation
  25. 25. Information ManagementTypical Relational Schema for Smart Meters Data Index Smart_Meters Table Meter_id Time KWH Voltage ColN 1 1-1-11 12:00 Value 1 Value 2 …….. Value N Table Grows 2 1-1-11 12:00 Value 1 Value 2 …….. Value N 3 1-1-11 12:00 Value 1 Value 2 …….. Value N … … … … …….. … 1 1-1-11 12:15 Value 1 Value 2 …….. Value N 2 1-1-11 12:15 Value 1 Value 2 …….. Value N 3 1-1-11 12:15 Value 1 Value 2 …….. Value N … … … … …….. … • Each row contains exactly one record = billions of rows in the table • Additional indexes are required for efficient lookups • Data is appended to the end of the table as it arrives • Meter ID’s stored in every record25 • No concept of a missing row © 2011 IBM Corporation
  26. 26. Information ManagementSame Table using an Informix TimeSeries Schema(logical view) Smart_Meters Table Meter_id Series 1 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 2 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 3 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] 4 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …] … … Table grows • Each row contains a growing set of records = one row per meter • Data append to end of a row rather than to the end of the table • Meter IDs stored once rather than with every record • Data is clustered by meter id and sorted by time on disk • Missing values take no disk space, missing interval reads take 2 bytes26 © 2011 IBM Corporation
  27. 27. Information ManagementVirtual Table Interface makes Time Series data appearRelational TimeSeries Table TimeSeries Virtual Table Smart_meter SM_vt mtr_id Series (int) timeseries(mtr_data) mtr_id date col_1 col_2 1 [(Mon, v1, ...)(Tue,v1…)] 1 Mon Value 1 Value 2 ... [(Mon, v1, ...)(Tue,v1…)] 1 Tue Value 1 Value 2 ... 2 1 Wed Value 1 Value 2 ... 3 [(Mon, v1, ...)(Tue,v1…)] ... ... ... ... ... 4 [(Mon, v1, ...)(Tue,v1…)] 3 Mon Value 1 Value 2 ... 5 [(Mon, v1, ...)(Tue,v1…)] 3 Tue Value 1 Value 2 ... 6 [(Mon, v1, ...)(Tue,v1…)] 3 Wed Value 1 Value 2 ... ... ... ... ... ... 7 [(Mon, v1, ...)(Tue,v1…)] [(Mon, v1, ...)(Tue,v1…)] Execute procedure tscreatevirtualtable 8 („SM_vt‟, „Smart_meter‟);27 © 2011 IBM Corporation
  28. 28. Information Management100 Million Smart Meter Benchmark: Goals 1. Measure processing times for data collected over a 31 day period for:  100 million meters at 30-minute intervals in an 8 hour day. 2. Demonstrate consistent processing times over the 31 day period. 3. Demonstrate linear storage growth of data stored over the 31 day period. 4. Complete one day’s billing cycle while simultaneously processing and loading meter data for 100M within an 8 hour time period. 5. Demonstrate all processing can be done using a low-cost combination of commercially available hardware, storage, and software.28 © 2011 IBM Corporation
  29. 29. Information Management100 Million Smart Meter Benchmark: Operations  Operations performed each day – Load interval and register data for 100M meters at 30 minute intervals (49 records/day/meter) • 49 records/day/meter * 100M = 4.9 Billion records/day – Perform VEE on the data (validation/estimation/editing) – Run a daily billing cycle on 6% of the meters – Gather results for 31 days Gather Results Processing Run daily billing cycle Perform Validation, Estimation & Editing Load & Register Data (100 million meters) Processed over 30 minute intervals over 31 Days29 © 2011 IBM Corporation
  30. 30. Information Management100 Million Smart Meter Benchmark: Components AMT-Sybex Affinity Meterflow Software Stack Informix v11.70.xC3 with TimeSeries version 5.0 Monitor & Admin AMT-Sybex Affinity Meterflow Meter AIX v7.1 Informix 11.70 AIX v7.1 Hardware Stack IBM System P Series & IBM Power P750 XIV Storage System Storage 32 cores (3.5 GHz) – 16 active 15 X 2TB storage 500 GB RAM 6 X 6 FC connections @ 4GB 1 GB LAN Fiber dual port adapter - 1 active 2 X 8GB FC dual port adapter - 4 active ports of storage Upstream Management Storage Area Network (SAN) Systems Knowledge ApplicationsMDM, DMS, Data 4 X 8GB NMS Management 6 X 4GB30 © 2011 IBM Corporation
  31. 31. Information Management100 Million Smart Meter Benchmark: Load Results An “end to end” run of 100 Million Meters at 30 minute intervals was performed for 31 days of data The result: all data was prepared, loaded, validated as well as a billing cycle run in less than 8 hours The average time to do these operations remained consistent over the 31 days – Performance remained constant even as storage increase The billing cycle completed in less than 5 hours and ran concurrently Process Avg. Elapsed Time Avg. Throughput Rate Preparation and Technical 2 hrs 10 min 628,205 records/sec Verification Data Load 3 hrs 14 min 420,962 records/sec Validation, Estimation, and Editing 2 hrs 11 min 623,409 records/sec (VEE) Total Time: 7 hours and 35 minutes!31 © 2011 IBM Corporation
  32. 32. Information ManagementLoad Performance: Storage Comparison over Time Load Time over 31 Days Total Process Time over 31 Days 100 Million Meters @ 30 minute intervals 100 Million Meters @ 30 minute intervals Total Time - Minutes Total Time - Minutes No. of Days Storage in TB over 31 Days 100 Million Meters @ 30 minute intervals Disk Space in TB No. of Days No. of Days32 © 2011 IBM Corporation
  33. 33. Information ManagementSummary: Informix is the Answer for Smart Meter Data 1) The enormous volume of smart meter data can be overwhelming 2) Most smart meter data is time series data 3) Not all database servers are equal, choose one that handles relational and time series queries equally well For more information on our 100M Smart Meter Benchmark: © 2011 IBM Corporation
  34. 34. Information Management (Kevin Brown – (Jill Feblowitz - © 2011 IBM Corporation
  35. 35. Information Management35 © 2011 IBM Corporation