• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Unleashing the Power of Big Data in the Utilities Industry, by IDC Energy Insights

Unleashing the Power of Big Data in the Utilities Industry, by IDC Energy Insights






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Unleashing the Power of Big Data in the Utilities Industry, by IDC Energy Insights Unleashing the Power of Big Data in the Utilities Industry, by IDC Energy Insights Document Transcript

    • Unleashing the Power of Big Data and Analytics for the Utility Industry WHITE PAPER Sponsored by: HP J i l l F e bl o wi t z S ep tem b er 2 01 2www.idc-ei.com IDC OPINION The proliferation of smart meters and sensors along with new channels for customer interaction have provided North American utilities withF.508.988.7881 unprecedented access to data from the grid and from their customers. The challenge that utilities now face is how to take advantage of that data to create value for the business. Fortunately, advances in technology have made it possible to quickly gain intelligence from large volumes and a variety of data types. Areas where Big Data andP.508.935.4400 analytics can provide value for operations and customer relations are emerging, along with some best practices in assembling an analytics approach. Big Data and analytics technologies describe a new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data byGlobal Headquarters: 5 Speen Street Framingham, MA 01701 USA enabling high-velocity capture, discovery, and/or analysis. Key takeaways include: ● Utilities recognize the significance of having an analytics strategy and view investment in business intelligence and analytics as important. ● Utilities in North America are considering the use of Big Data and analytics primarily for analysis of operations data. ● The current conversation is about using all the attributes of Big Data and analytics to support grid and network reliability. In the future, the conversation may extend to customer engagement and management. ● Applications of Big Data and analytics in utilities that show promise are predictive asset management, capital investment planning, fraud and theft detection, sentiment analysis, and social media scans. To gain full advantage from existing data sources, utilities need to formulate a Big Data strategy that includes evaluation of decision makers requirements, decision processes, existing and new technology, and the availability and quality of data. September 2012, IDC Energy Insights #EI236948
    • SITUATION OVERVIEWThe Utilities EnvironmentUtilities in North America are faced with declining revenue. Forexample, the U.S. Department of Energys Energy InformationAdministration expects total electricity consumption in the UnitedStates to fall by 1.3% in 2012 and grow by 1.3% in 2013, whereas inthe past, consumption grew at a fairly steady rate of 2–3%. Sales areexpected to be down by 3.2% this year. The reduction in revenue is inpart due to the slow recovery from the recession as commercial andindustrial consumption is down. The reduction may also be attributedto the influence of low gas prices on revenue for gas and electricutilities. While these may be short-term situations, greater energyefficiency achieved with equipment standards, energy efficiencymeasures, and changes in behavior are expected to impactconsumption, thus revenue, going forward. It is no longer a case ofbalancing supply and demand but optimizing supply- and demand-sideresources to meet demand.Aging infrastructure, unusual weather patterns, and cybersecurity anddata privacy threats are challenging the utilitys ability to maintainreliability of service. The industry has known for the past decade thataging delivery infrastructure is in need of new investment. Forexample, according to the American Water Works Association(AWWA), much of the drinking water infrastructure in the UnitedStates, comprising more than 1 million miles of pipes, is nearing theend of its useful life and approaching the age at which it needs to bereplaced. This aging infrastructure is leaky, is prone to breaks, and isresulting in degraded water service and increased water servicedisruptions. Utilities need to understand the operational thresholds thatcan safely be achieved by older infrastructure to avoid equipmentfailure. North America has also experienced an uptick in flooding,fires, and storms that have left utility customers without access toenergy and water. Regulators are demanding that utilities be betterprepared to restore services quickly in the event of extreme weather orother natural disasters. Today, long-term or seasonal weather modelsare insufficient; unusual weather patterns require better near-termforecasting capability. A new breed of viruses attacking processcontrol systems has raised awareness of security at utilities. Utilitiesneed to continuously scan for potential attacks and stay on top of thenext generation of security risks.The past five years have seen the installation of new "smart"technologies by electric, gas, and water utilities. Smart gridinvestments are made to improve grid reliability, support distributedand renewable energy, reduce operations and maintenance costs,increase energy delivery efficiency, improve system security, andenable energy efficiency and demand response. A growing movementPage 2 #EI236948 ©2012 IDC Energy Insights
    • toward the deployment of smart grid technologies was bolstered bystimulus efforts such as the American Recovery and Reinvestment Act(ARRA). IDC Energy Insights estimates that the installation of smartmeters will reach a peak in 2012, and by 2015, there will be over 90million smart meters installed in North America. ARRA funding hasalso introduced new technologies such as synchrophasors to the grid.Now that the ARRA funding has been allocated and largely spent,utilities are building out IT capabilities as the focus shifts to using thedata to justify return on investment. With new data available andexisting data more accessible, utilities are asking, "Now that we haveall this data, what do we do with it?"Todays consumers expect a rich "customer experience," based on theirexperience with retailers, banks, and other service providers. Theywant to have their needs quickly understood and met and want tointeract through multiple devices and social media. Customers arealready coming to expect this same level of care and insight from theirutilities. For example, customers may want to be notified of an outageon their phone and have that notice followed up with an update onexpected restoration. Customers may use tools provided by the utilityon a customer portal to develop a budget plan for their usage and havenotices sent to their phone as consumption nears the target budget forthe month. They want to receive information only about thoseefficiency programs or demand response programs where they meeteligibility requirements. They may want utilities to respond toquestions posed to them in their tweets. The ability to get rateincreases approved by regulators depends on demonstrating customersatisfaction. Customer care requires continuous attention on the part ofthe utility, yet care must be cost effective.New and disruptive technologies are gaining ground. In response torenewable portfolio standards and emissions regulations, utilities arediversifying their portfolios and exploring energy storage. Residentialand commercial/industrial customers continue to invest in distributedenergy resources such as solar and wind, despite some setbacks in theindustry. The expectation is that 120,000 plug-in electric vehicles(EVs) will be sold in North America in 2012. Utilities are lookingtoward demand response technologies to shift consumption to lower-cost periods. Commercial and industrial customers are evaluating theuse of smart building technology to lower costs and reduce theircarbon footprint. As penetration of new technologies grows, there willbe an impact on operations, commercial relationships, and distributioninfrastructure. Resources like distributed wind and solar come on andoff the grid. EV charging will be done at home, at work, at the store,and at fueling stations. Utilities will need to prepare for a moredynamic distribution grid through informed planning based onsimulations. In the future, utilities will need to respond quickly tosupply and demand coming on and off the grid.©2012 IDC Energy Insights #EI236948 Page 3
    • The Attributes of Big Data and AnalyticsBig Data and analytics technologies describe a new generation oftechnologies and architectures designed to economically extract valuefrom very large volumes of a wide variety of data (structured andunstructured) by enabling high-velocity capture, discovery, and/oranalysis. Big Data and analytics include infrastructure (servers, storage,clustering software, networks, etc.), data organization andmanagement (software that cleanses, normalizes, tags, and integratesdata), analytics, business intelligence and discovery (software for adhoc discovery and deep analytics); real-time analysis; and automated,rules-based transactional decision making), and decision support(collaboration, scenario evaluation, risk management, and decisioncapture and retention). Software associated with Big Data andanalytics includes graph or network analysis, text mining or sentimentanalysis, streaming data monitoring and analysis, noSQL databases,relational databases, text mining, and graph or network analysis.Heres where volume, variety, velocity, and value (discussed in thesections that follow) line up with the conditions at utilities.VolumeSmart grid and smart network devices are generating more data thanever before. For example, one distribution company with 2 millionmeters estimates that it processes 35GB of data a day based on 15-minute interval data collected four times a day from each smart meter.Utilities are investing in powerline sensors, advanced remote terminalunits, and intelligent electronic devices at the edges of the network. Itis not just smart grid and network investments that are generating data.Any facility that has SCADA systems or data historians has had accessto control data. In aggregate, these devices and control systems throwoff an even greater amount of data than smart meters. If the utilitywere to make all this data available and accessible to process, volumeswould meet the Big Data volume thresholds.VarietyThe bulk of the data that utilities are generating — smart meter– andgrid-generated data — is structured data. It is time series data recordedat uniform time intervals ranging from seconds to hours. Applicationsat the utilities, such as customer care and billing, work and assetmanagement, energy trading and risk management, enterprise resourceplanning, project management, geospatial information systems, and soforth, are also sources of structured data that are valuable for analysis.External data services, such as weather, market prices, and futures, canbe helpful as well. Traditional sources of unstructured data at theutility include emails from utility customers, recorded contact centercalls, contracts with suppliers in PDF format, engineering drawings,specifications, Web traffic, and interactions via social media.Page 4 #EI236948 ©2012 IDC Energy Insights
    • VelocityToday, velocity can be associated with discovery/analysis onoperational data from control system and equipment or grid sensors innear real time. The idea is to allow the utility to monitor conditions,send fault or tamper alarms, or predict potential incidents. The intervalconsumption data collected by the smart meter is collected onlyperiodically. However, "last gasp" data from the meter, especially inoutage situations, is returned to the headend system in near real time.To support the dynamic distribution network of the future, high-velocity discovery and/or analysis based on interactions betweenutilities and smart homes and buildings will be required. Google andFacebook have been strong proponents of Big Data and analytics bymonitoring Web traffic and social media to push advertisements. Thesame techniques could be applied by utilities to customer data.ValueValue is the fourth attribute of Big Data and analytics and must bepresent to justify investment in the infrastructure, data organizationand management, discovery, and decision support. In general, BigData and analytics provide the ability to filter through large amountsof data, unlock insights available through new data sources, andleverage and optimize untapped data. There are potential problems andopportunities that need time and resources to address. For example,there may be a specialized transformer that is likely to fail, but thetransformer needed is not available off the shelf and requires monthsfrom order to delivery. If these kinds of problems can be identifiedquickly, then a company can avoid potential problems or get ahead ofits competitors in the marketplace.FUTURE OUTLOOKUtilities look for information technology investments to provide valuein improving operations and customer satisfaction. More specificallythis means:● Protecting revenue (customer)● Improving customer engagement (customer)● Maintaining reliability and securing the grid (operational)● Reducing the cost of operations (operational)● Complying with environmental regulations (operational and customer)©2012 IDC Energy Insights #EI236948 Page 5
    • Business Intelligence and Analyticsat UtilitiesUtilities have come to recognize the importance of having an analyticsstrategy and view investment in business intelligence and analytics asimportant. According to IDCs 2012 Global Technology and IndustryResearch Organization IT Survey, 30.2% of utility respondents have astated analytics strategy. Utilities see the importance of investment inthis area. Fifty-four percent of United States–based utility respondentsrate investment as important or very important; 87.4% have invested inbusiness intelligence and business analytics; and 36.9% plan to investin a new solution or make major enhancements to existing solutions inthe coming year. The top motivations for investment are cost controlor reduction, financial reporting and analysis, optimization ofoperations, and workforce optimization.Utilities are using business analytics to support:● Customer satisfaction: Target marketing for energy efficiency and demand response, evaluation of the effectiveness of demand response and energy efficiency programs, product development of time-based rates or demand response incentives, analytical tools provided to customers to help them reduce their energy bills, understanding how customers prefer to receive communications about outages● Improving operations: Pinpointing the location of outages for workforce deployment, scanning for potential security breaches, detecting fraud and theft and identifying unbilled accounts, performing analysis of faults or voltage irregularitiesProbably the most immediate savings are being realized with fraud andtheft detection. Utilities are also starting to realize savings throughanalytics used to prioritize capital investments in transformers andother equipment.Still, a business intelligence/analytics strategy is not the same asmaking a commitment to Big Data and analytics. Big Data andanalytics could be a part of the strategy but will involve investmentsdifferent from business intelligence/analytics in technology, process,data, and personnel. For example, companies will likely need to hiredata scientists or invest in streaming analytics. There is still someconfusion over what Big Data and analytics entail. Close to half ofutility respondents were not familiar with the term. Whether or notutilities will choose to implement Big Data and analytics will dependon the answers to questions such as:● What business objective (s) do we want to achieve?● What decisions do we want to make using the technology?Page 6 #EI236948 ©2012 IDC Energy Insights
    • ● How quickly do we need to make decisions?● Are decisions to be automated or involve human interaction?● How much data is required to arrive at an answer?● How varied is the data involved?● Can we modify existing applications and infrastructure to get the answers we want?● Does it make sense to build a common analytics infrastructure to handle all our analytics needs?Big Data Use Cases for UtilitiesToday, utilities in the United States are considering the use of Big Dataand analytics primarily for analysis of operations data (see Figure 1).FIGURE 1Drivers of Big Data in the Utility IndustryQ. What are your organizations drivers for using Big Data technologies and approaches? Analysis of operations-related data Analysis of online customer behavior–related data Analysis of transactional data f rom sales systems Service innovation Analysis of machine or device data Nonanalytic workloads 0 5 10 15 20 25 30 35 40 (% of respondents)n = 124 (utilities)Source: IDCs Global Technology and Industry Research Organization IT Survey, 2012The current conversation is about using all the attributes of Big Data andanalytics to support grid and network reliability by analyzing structuredtime series data from SCADA and control systems, sensors, and smartmeters, and that is where most investment is just now being made.©2012 IDC Energy Insights #EI236948 Page 7
    • Probably the most publicized use of Big Data and analytics is by theindependent system operators (ISOs) that must handle large quantities ofdata every day just to keep the power markets functioning andtransmission functioning. For example, PJM has contracted with a publicpower authority to process synchrophasor data to improve transmissiongrid management and will use Hadoop to manage the data.In fact, data management in the sense of being able to quickly accessdata is the first thing that utilities think about when presented with theconcept of Big Data and analytics. Twenty-five percent think of thisterm as referring to a very large amount of data, while 11.8% thoughtreal-time streaming data, 12.8% said data of multiple types, and 2.6%said high-performance computing. It is likely that the initial focus willbe on accessing data more quickly, but the value comes in executingon the results of the analytics. So business users should be able to runtheir own analytics and use the data for decision making or produce anautomated action.Applications of Big Data and analytics that are showing promise inutilities include:● Fraud and detection. One transmission and distribution company is comparing customer load profiles with data from smart meters and line transformer loads that are separately metered to identify leakage.● Sentiment analysis. Analyzing the voice patterns of the customer during a call to indicate the level of customer satisfaction is just starting to be used in other industries and may help CSRs reorient the approach to the call. Sentiment analysis may also be a rich source of data for marketing.● Predictive asset management. Predictive analytics are being used by a few utilities to forecast potential performance or equipment failures by studying the load on each asset along the distribution network and the rated capacities of devices (switch, reclosure, breaker, transformer, etc.). Detection of water leakages can also help predict potential water pipeline failures.● Capital investment planning. One ISO with the ultimate goal of informing capital investment strategy and improving grid reliability has recently invested in in-memory analytics to gain access to timely information about grid and asset status.● Social media scans. There may be value to utilities in analyzing unstructured data found in social media or emails. Some utilities are already doing scans of the social networks to understand how their customers are reacting to events such as unplanned outages.As the distribution network sees the addition of customer-owneddistributed resources, energy storage, electric vehicles, and automatedPage 8 #EI236948 ©2012 IDC Energy Insights
    • demand response, the attributes of Big Data and analytics — velocity,volume, variety, and value — will be called upon to support the virtualpower plant (VPP). Virtual power plants are a way to aggregatedistributed energy resources and demand response as if they were asingle power plant. The VPP can be used to balance the grid or enhancerevenue. Some utilities are looking at the potential for developing adistribution marketplace, similar to an ISO, to help manage the highpenetration of solar in some neighborhoods in order to take strain offfeeders when abundant solar power is available. Big Data and analyticswill be needed to support this type of market.Challenges for UtilitiesProbably the biggest barrier to the adoption of Big Data and analyticsat utilities is the lack of awareness. Other barriers are:● Concern for privacy. Smart meter installations have generated backlash from customers about safety (RF frequency) and concern about privacy of data. While utilities will likely invest in analytics to do social media scans to gauge sentiment, because of privacy concerns, they are not likely trawl for individual customer preferences on social media sites.● Lack of competition. U.S. utilities are still largely regulated, with the customer not having much supplier choice, compared with the Western European utilities market, which is deregulated and competitive. Therefore, utilities in the United States are less motivated to make investments in customer-related analytics.● Need to understand how Big Data works with real-time operational systems. Utilities have already made a considerable investment in real-time systems to run the transmission grid, and while there is some movement to apply IT to operational data, the business value is yet to be determined, especially around where speed to answer is essential.● Uncertainty about costs and requirements. Utilities are uncertain about the costs and requirements for building a Big Data and analytics infrastructure. There is also uncertainty about how Big Data and analytics will fit in with legacy systems.Overview of HP Big Data OfferingsHP offers a wide range of information management and analytics(IM&A) capabilities drawn from its enterprise services, softwareproducts, and cloud and security platforms to address Big Datachallenges and opportunities in the utility industry.©2012 IDC Energy Insights #EI236948 Page 9
    • HP IM&A services include information strategy and organization,information management and architecture, business analytics andinformation delivery, and social intelligence. HP provides theseconsulting services around its own Big Data software assets, Autonomyand Vertica, and third-party software assets, in particular SAP HANAand Microsoft SharePoint/BI platform. HP supports this with a variety ofsoftware and solution delivery models including on-premise installation,hosted, software as a service (SaaS), cloud computing, and multitenantSaaS (cloud deployment).HP focuses its capabilities to enable utilities to proactively manageinformation-related business risk, enhance customer experiences, andoptimize business performance to maintain reliability, reduce the costsof operation, and protect revenue.HPs Autonomy software asset helps utilities develop connectedintelligence from structured and unstructured data for actionabledecisions that improve business performance.HPs Vertica analytics database delivers scalable performance on BigData queries enabling real-time decision making to be embedded inutility processes in order to optimize business performance.Vertica and Autonomy deliver a powerful combination for real-timeanalytics and decision making using structured and unstructured dataacross the enterprise.Strengths and ChallengesHP offers a product and services portfolio that is consistent with whatIDC Energy Insights expects from a market-leading technologyprovider to the utility industries. In addition to the software andservices capabilities noted previously, HP provides the followinginfrastructure component to enable Big Data application deployments:● Cloud computing. HP can offer clients a variety of enterprise- grade cloud computing solutions, including public, hybrid, and private. The ability to offer private cloud to utilities is considered a strength for a number of reasons, not the least of which is a tighter security model.● Security. HP offers clients a security strategy through the HP Security Framework, designed to offer end-to-end information security plans and execution road maps. Because of the sensitive nature of customer data and the requirements and penalties imposed by the regulators, security is top of mind for industry IT executives.Page 10 #EI236948 ©2012 IDC Energy Insights
    • ● Mobility. HPs approach to enabling enterprise mobility is suited for organizations that wish to reach their constituents across multiple networks and devices by delivering applications, content, and services in a scalable, secure, and reliable way. This approach leverages HPs global applications services capabilities to provide the architecture, systems engineering, development, and support services. Combined, they help an organization simplify its applications and extend them where necessary as well as build mobile business-to- business, business-to-consumer, and business-to-employee applications. This approach also leverages HPs service-oriented architecture–based integration architecture and is enabled by development and security frameworks that help create componentized building blocks from monolithic legacy applications to develop and deploy mobile applications.HP faces several unique market challenges as well as many of thesame market challenges as other enterprise vendors servicing the BigData marketplace:● Demonstrating leadership in the transition from information management to Big Data. HP has made inroads into utilities with Exastream, a platform for creating, managing, and delivering communications to the utility customer. HP had a very compelling case for the use of OpenView in the management of smart meter data but was not able to gain significant traction in utilities. HP must more effectively demonstrate leading capabilities at the component level with high-profile Autonomy and Vertica wins as well as highlight its ability to deliver end-to-end Big Data software, services, and hardware with lighthouse and marquee clients.● Offering competitive, differentiated solutions portfolios. Utilities are not early adopters of Big Data and analytics, but they are now focused on analytics to make use of smart grid data. Many of the major professional service firms are offering to help with a business intelligence/analytics strategy. HPs competitors are also focused on expanding their solution portfolios on the Big Data trend in terms of the breadth and depth of product capabilities, IT infrastructure, and professional services. HP must leverage its industry position and demonstrate the value of a Big Data relationship with HP to gain Big Data market share.● Channel network management. HPs channel strength is also a weakness. Successful execution of HPs Big Data strategy to broaden its portfolio will require that HP reinforce its position in the enterprise space without alienating the channel that has been so beneficial to the organization.©2012 IDC Energy Insights #EI236948 Page 11
    • ● Focus on security. With the introduction of new viruses that can invade process control systems, and with the addition of IP- addressable devices on the grid, system vigilance has increased. Regulators now require utilities to comply with security and reliability standards for transmission networks and generation assets, and this requirement may soon extend to protecting distribution assets and networks as well. Concern with privacy of customer data, especially around the implementation of smart meters, requires more attention to the security area.PARTING THOUGHTSUtilities looking to improve their capabilities in Big Data and analyticsshould consider the following:● Recognize the value of untapped data assets in supporting fact- based decisions by generation, transmission, distribution and customer operations, marketing, energy efficiency, and demand response.● Recognize the implications of operating without critical information and build use cases to address the challenges.● Conduct a gap analysis to determine what new technology and staff investments are required.● Formulate a Big Data strategy that includes an evaluation of decision makers requirements, decision processes, existing and new technology, and availability and quality of data.● Consider cloud services and a shared services model with other utilities to reduce costs. This strategy will allow you to experiment prior to investment. Utilities with limited resources may want to consider this option in order to keep costs low.Copyright NoticeCopyright 2012 IDC Energy Insights. Reproduction without writtenpermission is completely forbidden. External Publication of IDCEnergy Insights Information and Data: Any IDC Energy Insightsinformation that is to be used in advertising, press releases, orpromotional materials requires prior written approval from theappropriate IDC Energy Insights Vice President. A draft of theproposed document should accompany any such request. IDC EnergyInsights reserves the right to deny approval of external usage for anyreason.This document was reprinted by HP with permission from IDC EnergyInsights.Page 12 #EI236948 ©2012 IDC Energy Insights