Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environment - OPower

3,169 views

Published on

Opower is a fast moving energy management SaaS company that collects sensor data from nearly all of the major utilities in the United States–meaning from more than 45 million American households–along with major utilities in 5 countries throughout Europe and AsiaPac. Opower manages more than 100 billion meter reads, ranging from high frequency power data (AMI), smart thermostats data, and weather data. Currently all data at Opower is stored in HBase or Hadoop (and is notably not security sensitive). This discussion will discuss Opower’s HBase architecture, highlight potential and current uses of data in HBase, share the vision of Opower’s future projects and directions, and reveal how Opower’s big data management has allowed the company to help its utility clients save enough energy to power a city of nearly 200,000 people and save utility customers more than $70 million since only 2008!

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environment - OPower

  1. 1. HBase to Save the Planet Alex Newman posix4e@apache.org Architect, Drawn to Scale Strategic Advisor, Opower
  2. 2. My life with HBase Drawn toFactset Cloudera Opower Scale
  3. 3. About OpowerOpower is a customer engagement platform for the utility industry
  4. 4. About Opower Home energy reports Customized utility billsEnergy efficiency programs for utilities
  5. 5. About Opower Opower runs on analyticsAnalytics run on Hadoop + HBase
  6. 6. Opower analysis relies on datafrom a variety of sources » Electric Utility Usage » Thermostat » Weather » Gas Utility Usage Data data data Data Data Storage & 4 Shared Energy Processing Signature Repository 3 1 2 Disaggregation OPOWER Algorithms Platform
  7. 7. Opower’s first architecture couldnot support their analytic vision MySQL Scalability? Performance? Data integration?
  8. 8. Opower’s first architecture couldnot support their analytic vision Analytic workflow instead of analytic apps: SQL -> CSV -> R -> too little, too slow
  9. 9. Problem #1 Data Lake CostUsage AMI Regional AMI Sensor Data Data Lake
  10. 10. Problem #2 Slower and slower queries Smart-grid-scale dataLots of supporting data: weather, demographics, etc.
  11. 11. Problem #3It was taking lots of “magic” Intense analytics Strange schemas Segmented queries
  12. 12. Hadoop + HBase at OpowerOpower determined that they needed an entirely new data architecture
  13. 13. NexGen Architecture @ Opower
  14. 14. Hadoop + HBase at Opower Early success: HBase AMI
  15. 15. What rockedEndless, cheap scalability
  16. 16. What rockedThe analytics team loved it!
  17. 17. What suckedHard on the ops team – still trying to grok it
  18. 18. What sucked NoSchema p1. Creating Schema Managing MetaDataSchema <=> Performance
  19. 19. What sucked HA Failover Snapshots
  20. 20. What sucked No secondary indexAggregation is slow (Rollup/OLAP) Poor Client Performance
  21. 21. It would be better if only …Developers were not forced to knowhow the data is stored, indexed, etc.
  22. 22. It would be better if only … There were nicer APIs and better query languages (SQL?)
  23. 23. It would be better if only … Version migrations were easy Hierarchical Tables
  24. 24. It would be better if only … Real-time tuning
  25. 25. It would be better if only … Did I mention HA?
  26. 26. In summaryHBase has helped Opower achieve their analytic vision But they’ve still got a long way to go HBase still has a long way to go
  27. 27. Questions? Alex Newman posix4e@apache.orgArchitect, Drawn to ScaleStrategic Advisor, Opower

×