Energy Industry Trends:
Smart Grid, Big Data Management,
By Jonathan L. Tan
August 15, 2016
U.S. Energy Industry
• The energy industry is the totality of all of
the industries involved in the production and sale of energy,
including fuel extraction, manufacturing, refining and
• The energy industry is the third largest industry in the United
• U.S. energy companies produce oil, natural gas, coal,
nuclear power, renewable energy and fuels, as well as
electricity, smart grid, and demand response technologies.
• Growing consumer demand and world class innovation –
combined with a competitive workforce and supply chain
capable of building, installing, and servicing all energy
technologies – make the United States the world’s most
attractive market in the $6 trillion global energy market.
U.S. Energy Industry
Energy Industry Subsectors (Source: Select USA)
• Renewable Energy - Bloomberg New Energy Finance (BNEF) expects that by 2030 the
share of renewables in U.S. power generation mix to reach 27 percent. BNEF projects that just under
$700 billion will be invested in the U.S. renewable energy sector during the next two decades.
• Renewable Fuels - US’s Ethanol market is largest in world
• Oil & Gas -The United States is undergoing a revolution in oil and natural gas production from
• Coal The United States holds the world’s largest estimated recoverable reserves of coal and is a
net exporter of coal. Approximately 72 percent of coal production originated in five states: Wyoming,
West Virginia, Kentucky, Pennsylvania, and Texas.
• Nuclear Energy - The United States operates the most nuclear reactors, has the largest
installed nuclear power capacity, and generates the most nuclear power in the world. Nearly 20
percent of U.S. electricity is produced at 100 nuclear reactors in 31 states.
• Energy Efficiency - The market for achieving greater energy efficiency in the United States is
large and growing. Combined financing and investment in building, industrial, and supply side energy
efficiency doubled in 2012, exceeding $15 billion in funds due to an Executive Order from the Obama
Administration in 2012
• Smart Grid -The United States is an international leader in the development and deployment of
smart grid technologies and services. The smart grid subsector is defined by the electric grid
equipment and services required for the modernization of distribution and transmission systems, as
well as the Information and Communication Technologies (ICT) that support a fully networked grid and
enable two-way communications and electric flows. Overall grid modernization investments are
projected to achieve $130 billion in annual benefits for the U.S. economy by 2019, and the smart grid
deployment by utilities in the United States is expected to create $15 billion to $31 billion annually by
2014 in potential sales for the ICT industry.
1. The transition towards more renewable energy and diversified
supplies is creating opportunities and challenges for the security of
the global energy infrastructure.
• As renewables are now part of the energy portfolio and are rapidly
gaining market share, they bring along benefits such as energy
mix diversification, with distributed generation growing at a fast
pace worldwide and its installed capacity expected to more than
double in the next decade.
• At the same time, as the energy generation portfolio transitions
and diversifies further, new challenges are emerging, which
require changes to the electric utility business model and
regulatory policies to ensure secure and reliable supply.
2. Digital disruption is creating new opportunities – but also threats.
• On the one hand, technology is instrumental for realizing
intelligent grids and interconnected assets;
• On the other hand, it introduces new threats such as the possibility
of cyber attacks.
• The increasing interconnectivity and proximity of energy systems
means that conflicts can have ripple effects on energy markets
• New technologies, such as batteries and grid-embedded
generation, are making the cybersecurity of grid systems more
• Global inexperience in handling large-scale cyberattacks,
combined with the greater capabilities of state and non-state
actors, has increased the likelihood that future wars and attacks
will have a larger cyber component.
3. The rebalancing of energy supply and demand is leading to a new
global energy security order.
• Recent drops in oil prices have led to a significant shift in wealth
from net oil exporters to oil importers.
• At the same time, the development of unconventional sources of oil
and gas, as well as the recent economic slowdown in emerging
markets such as China and India, have contributed to price
readjustments against the backdrop of a general shift in energy
• Geopolitical shifts, the new distribution of powers and energy trade
flows will create challenges and opportunities for energy security in
the new energy architecture.
WHAT IS U.S. ELECTRICITY GENERATION BY
• In 2015, the United States generated about 4 trillion kilowatt hours of
electricity. About 67% of the electricity generated was from fossil fuels
(coal, natural gas, and petroleum).
• Major energy sources and percent share of total U.S. electricity
generation in 2015:
– Coal = 33%
– Natural Gas = 33%
– Nuclear = 20%
– Hydropower = 6%
– Other renewables = 7%
– Biomass = 1.6%
– Geothermal = 0.4%
– Solar = 0.6%
– Wind = 4.7%
– Petroleum = 1%
– Other gases = <1%
2016 will be the
first year that Coal
will overtaken by
Total US energy production
increases for the sixth year in a row
Source: U.S. Energy Information Administration
• Smart City Market - will grow to $3.3 Trillion by 2025 –with $100 billion
investment projected for Infrastructure technologies in the next ten years
with annual spend projected to $16 billion per year by 2020
• Internet of Things – 7 billion mobile devices in 2014, 25 billion connected
devices by 2015 and 50 Billion by 2020.
• IoT solutions will grow from $1.9 trillion in 2013 to $7.1 trillion in 2020
• Investment by in Analytics is accelerating:
For instance, IBM has spent $6 billion in analytics R&D and $14
billion in analytics acquisitions and now has 8000 consultants
dedicated to analytics and 200 mathematicians developing
RELATED MARKET TRENDS
• The global Energy Analytics and Utility Analytics
market is estimated to grow from $1.42 billion in 2013
to $4.74 billion in 2018. This represents a Compound
Annual Growth Rate (CAGR) of 27.3% from 2013 to
• In terms of regions, North America is expected to be
the biggest market.
Utility Companies that use analytics get $10.66
for every $1 they spend on Analytics.
RELATED MARKET TRENDS
• At nearly $1.3 trillion in estimated global revenue for 2014, the market for
advanced energy products and services is as large as apparel and fashion
and almost four times the size of the semiconductor industry worldwide,
according to a report commissioned by Advanced Energy Economy (AEE).
• In the United States, advanced energy market revenue grew 14% last year –
five times the rate of the U.S. economy overall – to just under $200 billion,
making it bigger than the airline industry, equal to pharmaceuticals, and
nearly equal to consumer electronics in this country.
• The study, conducted by Navigant Research, found that advanced energy in
the United States was an estimated $199.5 billion market in 2014, up 14%
from 2013 ($169 billion), and five times the rate of growth of the U.S.
• Areas of growth included:
• Solar energy (up 39%) and
• Natural gas generating equipment (48%), in long-anticipated response
to lower-priced natural gas supplies.
• Wind power, which suffered a severe setback in 2013 due to the on-
again, off-again federal production tax credit (PTC), rebounded in 2014
with four-fold growth, to $8.2 billion, and a pipeline of projects that could
result in revenue rivaling the $25 billion realized in 2012, its biggest year
ADVANCED ENERGY MARKET FACTS (Navigant)
• With global revenue up 12% over 2013, 2014 was the biggest growth
year for advanced energy worldwide since AEE began tracking these
markets in 2011.
• U.S. advanced energy revenue has grown 38% over the four years
from 2011 to 2014. U.S. advanced energy represents 15% of the
• With the addition of new revenue data on residential energy efficient
lighting, Building Efficiency is now the largest segment of the U.S.
advanced energy market, with revenue of $60.1 billion in 2014.
• Counting only those products for which AEE has four years of data,
revenue growth in Building Efficiency is 43% over four years.
• In 2013, the severe downturn in wind energy due to uncertainty over
the federal production tax credit (PTC), with revenue dropping to $2.1
billion from $25.5 billion in 2012, was enough to offset growth in other
segments of U.S. advanced energy to show an overall decline of 4%
year over year. But wind bounced back in 2014, to $8.2 billion, nearly
four times 2013 revenue.
ADVANCED ENERGY MARKET FACTS (Navigant)
• Including wind, the Electricity Generation segment of advanced
energy grew 47% overall in the U.S., to $45.8 billion in 2014.
• Solar photovoltaic (PV) revenue was up 39%, to $22.5 billion,
capping four-year growth of 173%.
• Natural gas generating equipment (combined cycle and simple cycle
gas turbines) saw an increase for the first time in four years, with
U.S. revenue up 48% to $6.4 billion – a sign that the natural gas
revolution is now translating into new orders, not just higher
utilization of existing natural gas power plants.
• The U.S. Fuel Production segment reached $49 billion in 2014, up
from a revised 2013 total of $48.4 billion, with $39.1 billion of that
total from sales of ethanol.
• In Transportation, U.S. revenue for hybrid vehicles was down 19%
but up 34% for plug-in electric vehicles, while natural gas-powered
vehicles jumped 26% in revenue.
• Revenue from electric vehicle charging stations was up 31% in
the U.S., to $201.5 million, up seven-fold from 2011.
ADVANCED ENERGY MARKET FACTS (Navigant)
Utility 2.0 Problems
Greatest earnings pressure in recent memory
Recession has led to unprecedented decline in load and
revenues (sales volume)
Margins down in deregulated markets due to supply/demand
Antiquated systems and tools
Aging work force
Data Deluge from AMI – 2.0
Lack of clarity in regulatory and legislative direction
Increasingly hostile rate environment—necessary upward
adjustments on hold
Large variance in state regulatory direction
Regulatory developments that reward energy efficiency and
GHG reduction yet to materialize
Federal energy policy mired in politics
Viability of status quo approach to business in question
“Load destruction” threatens 100 yr old cents/kwh revenue
Returns to current and future investments in low carbon
Pressure to move forward with Smart Grid and energy efficiency
remains high, but incentives/returns are contingent on new rate
Utility 2.0 Roadmap
Grow the traditional business
Stay on message: reliable and affordable
Continue to invest in renewables, clean coal, nuclear as an “act
Capital and O&M prioritization and “re-casing”
Data Analytics & Technology enablement
Legislative solutions—e.g., decoupling, save-a-watt, etc.
Smart Grid-enabled products and services—energy
management, distributed gen, PHEV, AMI 2.0
In core business—regulated and unregulated
Along value chain—upstream, downstream
In crucial IP—low-carbon, renewables, energy technology
Evolve customer relationships
Increased control of usage, load and spend
Redefined view of role relative to utility
Better understanding of segment-specific needs
Better Power Control, Load Management, Reliability, Cost,
ELECTRIC UTILITY 2.0 or UTILITY OF THE FUTURE
• Manage Carbon across the enterprise
• Pursue all cost-effective energy efficiency
• Integrate Cost Effective Renewable Energy Resources into the Power
• Incorporate Smart Grid Technologies for Consumer and Environmental benefit
• Conduct Robust and Transparent Resource Planning
Performance parameters are:
• Customer service
• Adoption of smart grid technologies and services and support for alternate
The utility of the future leverages core competencies and builds adjacent core
competencies, and develops these new models to scale, as the organization learns
to become customer focused, providing efficient, reliable and resilient clean
energy. This is achieved by data systems that allow for optimal life-cycle asset
management and operating intelligence. Distributed generation resources
integrated in the generation mix, allow for automated load management through
connected data with customers (demand) and power generation (supply) to be fully
aligned in real-time.
U.S. IOU CAPEX 2000 – 2010: T $69B D $160B
THE CLASSIC ELECTRIC GRID
1. Generation Sources
2. Transmission Lines
3. Substations (Ex: PG&E has 900)
4. Distribution Feeders (1,500+)
5. Power Poles (millions)
6. Fuses (50,000)
7. Tap Lines
8. Transformers (1,000,000)
9. Service Lines (4,500,000)
DISTRIBUTION SYSTEMS NEED TO GET SMART
• Large – 50% to 100% of a utility’s assets
• Old – Typically 40 to 60 years
• Invisible – Little monitoring below the substation
• Unreliable – Costs $3M per outage; regulators
− System Average Interruption Duration Index (SAIDI) 2 hrs/year
− Worst U.S. utilities have SAIDI of 6 to 7 hours
− Best are 1 hour or less
− Japan averages 3 minutes . . . Tokyo was 29 seconds!!! . . .
• Stressed – the “Edge”
− Demand Response
− Solar PV & other variable and distributed generation
− Plug-in Vehicles
Smart Grid Essentials
1. All power generation, distributed energy assets (smart grid), including
DER assets, data systems, are integrated and connected ( via ICT)
2. System of system (SoS) interoperability model -nexus of energy with
water/waste, transportation, industry, education, health, government,
and the public (citizens, visitors, vendors, etc.) Fully integrated data
systems (meter, IoT, weather, OEM, remote sensing, legacy, billing,
CIM, GIS, SCADA, etc.)
3. Leveraging IoT –sensors, oem, meters, smart phones/mobile devices,
4. Leveraging analytics – fully deployed (descriptive, predictive &
prescriptive) through IoT, legacy systems, real-time data, etc.
5. Big data management standards met
6. Sustainability and removal of carbon fuels from power generation
7. Intelligent real-time applications leveraging customer usage and DR
8. Distributed grid optimization (DGO) integration of renewables, micro-
grids, smart buildings, water/waste water systems, etc.
9. Adaptive planning applications and adaptive management systems
10. Distributed grid optimization (DGO) including renewables (carbon
SMART GRID CAPEX INVESTMENT TRENDS
World Energy Investment Outlook 2014 Factsheet Overview
Smart Grid Technology Investment: Forecasts for
• Smart meters
• Communications systems
• Distribution automation
• Substation automation
• IT systems
– Infrastructure management systems (including distribution
management, outage management, MDMS, demand response)
– Billing and customer care (and web sites)
– Storage, servers, hardware, etc.
– Other IT (including workforce management, SOA)
• Home area networking (HAN)
• Consulting and systems integration
• Other (including project management and customer communications)
SMART GRID INVESTMENT MIX
SA = Substation Automation
DA = Distribution Automation
HANs = Home area networks
SI = Systems Integration
Through 2030, the top spending countries will be: China (US$99 billion by 2030),
United States (US$60 billion by 2030), India, France, Germany, Brazil, Spain,
United Kingdom, Japan and Korea.
UTILITY DATA MANAGEMENT:
Why are utilities turning to Big Data solutions?
• Before smart grids, the data that a utility would collect from its customers
frequently amounted to little more than a monthly meter reading: one data
point a month per customer.
• The advent of AMI has increased the level of data collection dramatically.
The Columbia Water and Light Department, for example, has a pilot
where meters provide data on around half a dozen household circuits
every minute. That is more than 43,000 data points per customer per
• Even if, as is often the case, readings are taken at more infrequent
intervals, say every five, 10 or 15 minutes, the increase is very significant.
And this is just one element of what is currently termed ‘Big Data’, data
sets that are terabytes to exabytes in size and come from a range of
sources, which puts their management and analysis beyond the scope of
traditional IT tools
MARKET DRIVERS (BIG DATA AND ANALYTICS)
• The key driver for the adoption of data management strategies is clearly
the need to handle and analyze the large amounts of information
utilities are now faced with.
• Utilities are entering a new era of customer expectation and
technological innovation, and analytics is a core requirement for the
truly modern digital utility. Those with pervasive analytic adoption will be
able to internally comprehend and act upon inefficiencies and
operational pain points to not only maximize existing assets but provide
the best service possible to their customers.
• For retailers in unregulated markets, analytics can be the differentiating
factor for top-line growth through enhanced service and higher margins.
And for both regulated and unregulated markets, analytics also enables
a level of transparency that will enable each utility to respond to
changes and issues faster and delivery services with fewer incidents
OTHER PROCESSES THAT UTILITIES HOPE TO
ENHANCE THROUGH BETTER ANALYSIS OF DATA
-Business operation efficiency. Data analytics will allow utilities to see where
consumers may be stealing electricity and will allow for better asset management
as well as better system planning.
-Implementing efficiency measures. Data allows for time-of-use rates which in turn
allow customers to monitor their consumption and save money by shifting use away
from times when rates are higher, while also reducing the need for more expensive
forms of peak generation to be built.
-Developing new business models. By adding intelligence to the grid operators can
offer new services, such as energy management, engineering, high-speed Internet
services, cable TV and private network links.
-Improving grid resilience and load management. Distribution companies must
reduce power demand to prevent outages, for example installing energy load
management systems to help retailers and customers manage their energy use.
Better data will also allow utilities to identify and rectify problems with the grid more
quickly, in some cases even before the customer has a chance to report the issue
-Engaging customers. Having more information about customers and their usage
patterns is generally considered important in helping deliver a better, more
Big Data Infrastructure Essentials
Typical Big Data Management Infrastructure
BIG DATA INFRASTRUCTURE
DATA SYSTEM INTEGRATION
Point to Point systems architecture System of Systems via Hub and
BIG DATA INFRASTRUCTURE ESSENTIALS
Data Management in the cloud is essential for scalability, security,
technology investment CAPEX mitigation, economic adaptability, and ease of use
APPLICATION AREAS: SYSTEMS
UTILITY DATA MANAGEMENT
Data type Technology involved Notes
AMI Smart meters Increased sampling frequency leads to
1,000 to 10,000- fold increase in data
Distribution Grid equipment Real-time monitoring and control requires
Automation many more granular readings than those
taken by smart meters. The GridSim
simulation package described by
Anderson et al (2011), for example, uses
a default sample rate of 30 samples per
second, per sensor.
Third-party Off-grid data sets Utilities are increasingly also having to
integrate and handle highly granular data
from other sources, such as pricing
details for demand response or
forecasting information for renewable
Asset Mgmt Firmware for all Maintaining smart grid technology once it
smart devices and has been rolled out requires frequent
associated operating firmware upgrades and the like, which
systems equates to a considerable amount of
asset management data
Sources of Big Data in Utilities
BROADLY SPEAKING THERE ARE FIVE OPTIONS
FOR DEALING WITH UTILITY DATA ANALYTICS:
1. Develop the systems in house.
2. Buy systems from an established operational technology (OT) vendor.
3. Buy systems from an established Information Technology (IT) vendor.
4. Buy and integrate point products from specialist vendors.
5. Outsource data analytics to a third party.
Data analytics option Benefits Disadvantages
Develop systems in house Highly tailored to the Lack of development skills
utility’s requirements and resources
Rely on OT vendor system Close alignment with Analytics and IT integration
operational processes capabilities might not be
and requirements; good as good as for IT vendors
integration with OT
Rely on IT vendor system Good integration with IT May include unnecessary
systems; robust design and support features that
support increase cost: lack of
alignment with operational
Rely on point product or
pure-play vendor system Close alignment with Potential lack of
operational processes cross-product and IT system
and good integration integration; lifetime support
with OT concerns
Rely on third-party
service provider system Reduced cost, improved Evolving concept with
analytics limited track record; data
PROS AND CONS OF DIFFERENT DATA ANALYTICS
OPTIONS FOR UTILITIES
IDEAL UTILITY ANALYTICS VENDOR
• Data Analytics expertise
• OT/IT experience
• Deep subject matter expertise (SME)
LEADING UTILITY ANALYTICS VENDORS ARE
FOUND IN THESE CATEGORIES
1. Analytics and Applications (Real-Time Intelligence)
2. Data Management + Movement – Platforms
3. Data Infrastructure + Storage – Private/Public Clouds
• As in most nascent technology sectors, the smart grid data analytics
market can be characterized as a rather fragmented marketplace
with a mix of large, established players and smaller, specialized
firms that come from many different industries. While the majority of
the vendors, both large and small, come from the information
technology (IT) sector, others may come from the telecom or even
the automobile sector.
• Among the large group of IT players, there are the big, established
companies like Accenture, Capgemini, HP, IBM, Microsoft, Oracle,
SAIC, SAP, Siemens, and Teradata. Another large technology
group is represented by the Indian service companies such as
Infosys and the “pure plays” – mostly with meter data management
(MDM) expertise – like Aclara Software, Ecologic Analytics, eMeter,
Itron, Olameter, and NorthStar Utilities. Telvent is a relatively large
IT provider that offers a portfolio of MDM software, as well as
distribution management system (DMS) and outage management
system (OMS) solutions with data analytics capability.
Competitive Landscape (from Navigant Research)
• Interestingly, there is no pure niche smart grid data analytics vendor in
the marketplace. However, there is a small group of software
companies that have developed a special niche area that they have
found can be leveraged in the smart grid data analytics market.
OPOWER and OSIsoft represent this market segment.
• Although telecom companies play a fairly large role in the smart grid
market, they do not appear to be prominent players with respect to
smart grid data analytics – at least at this time. However, it would not
surprise Pike Research if they soon make inroads into this particular
sub-segment of the smart grid market since they can leverage a long
history of managing and analyzing reams of data. AT&T is a good
example of such a vendor.
Competitive Landscape (from Navigant Research)
• The auto companies, offering plug-in hybrid and electric vehicles,
could also become serious competitors in the smart grid data
analytics marketplace. For example, Toyota launched a home
energy system in 2012 to help individuals monitor and manage
(even remotely) their energy use.
• Pike Research believes that when the vendors begin to face the
issue of scale, this competitive landscape will shift. The increasing
need for scalability to handle ever-increasing amounts and
complexity of data will provide a major advantage to those vendors
that have the current software and other resource capabilities to
handle the explosion of data.
• Once scale and scalability become the key issue for utilities, many
smaller vendors could lose their competitive advantage. To compete
effectively in this marketplace, vendors must demonstrate that they
possess deep utility industry know-how and understanding of the
different technological and data analytics challenges that utilities
face when transitioning to a smart grid operation.
Competitive Landscape (from Navigant Research)
• A solid background in managing and interpreting data for other industry
clients, especially in the telecom, banking, or retail sectors – be it in the
area of business intelligence, information management, data
correlation and modeling techniques, data mining, database/data
warehousing, or predictive analytics and forecasting – will be
considered an advantage by potential utility clients as they try to
address their smart grid data analytics issues.
• Moreover, in this early adoption phase, vendors need to be sensitive to
the fact that many utilities are not ready and willing to handle too much
change at once, preferring instead a more cautious, incremental
approach. The ability to integrate data and the results of analytics into
business processes is another key competitive factor.
Competitive Landscape (from Navigant
By 2020, the deployment of smart
technologies in the electric grid, transport,
buildings, logistics, and industrial motors
will save 15% of global emissions and
almost a $1 trillion in savings per year in
energy savings to global industry.
Source: The Climate Group (Accenture)
• The energy industry is diverse, complex and growing
• Shale energy is driving oil and gas production and
providing energy independence from foreign sources.
• The digital revolution is also impacting the energy
industry with IoT, and data management and analytics.
• The Clean Revolution is providing new regulatory
mandates for clean energy and innovation in Renewables
energy resources (Solar, Wind, Energy Storage, etc.)
which are further driving states, utilities and industry from
• Global urbanization and smarter cities are also driving
demand in the future for more efficient, resilient and
reliable clean energy.
• Successful Investor Owned Utilities (IOU’s) and
some Muni’s (like SMUD) will transform into hybrid
companies providing more than just energy.
• They first need to decide whether they will build
centralized power plants or distributed generation
resources through solar, wind, energy storage,
• IOU’s will also provide network and data content
through their connected network in the form of energy
conservation services, allowing consumers control
over their own energy usage through demand
response, and services derived by distributed
generation optimization (DGO).
• Utilities that rightly leverage and integrate their big data,
technology, and analytics will be most likely to manage
the complexities of the future state where energy
markets and the regulatory environment continue to be
unpredictable, and consumers become more
sophisticated about their energy options and demand
more cost /energy efficient, reliable, and resilient grid.
• Fully interoperable Data Management Systems will
enable the optimization of the lifecycle management of
both data and physical infrastructure assets that will
enable adaptability in planning, management and real-
time business mobile operating intelligence.
• Optimized management of integrated enterprise data
will ultimately improve growth and profits and provide
enterprise value for investors.
• Tomorrow’s winners will be those who successfully
manage two key transitions:
– from a technology standpoint, their operators must be
able to benefit from digital innovation while at the
same time building up a solid defense to cyber
– From a legal framework standpoint, their structures
will have the necessary flexibility to adapt to an ever-
• Tomorrow’s challenge for the energy sector will be to
achieve effective collaboration between all stakeholders
involved, for the world to meet its future energy needs in
a sustainable way.
Thank you for your time
Jonathan L. Tan
gzz cleantech consulting