Smart grid talk_puspanjali (1)

ROLE OF BIG DATA
ANALYTICS IN SMARTGRID
Dr. Puspanjali Mohapatra
Department of CSE
IIIT Bhubaneswar
(puspanjali@iiit-bh.ac.in)
ROLE OF BIG DATA ANALYTICS IN SMART GRID

Big Data is watching you.
He who owns data, owns Future.

Outline
• Introduction to Smart Grid
• Introduction to Big data
• Industry and Utility Perspective
• Methodological stages and solution Approach
• Platforms and Architectures
• Application of Big Data in Smart Grid
• Conclusion
• References

Introduction to Smart Grid
• To a meter engineer- it is an advanced metering information.
• To a protection and control engineer- it is substation and distribution
automation.
• To a control room operator- it is distribution and operation
management.
• To a design and planning engineer- it is asset management.
• To an IT engineer- it is a challenge to bring all these together.

As per the definition provided by NIST, USA
• Smart grid is defined as a modernized grid that enables bidirectional flow
of energy and uses two ways communication and control capability that will
lead to an array of new functionalities and application.
• Finally we can conclude that Smart Grid is a multidisciplinary research area
with a lot of scope.

Sources of non-electrical and electrical big dataset in smart
grids

• The sources of data flow from
• smart meters
• PMUs, µPMUs
• ﬁeld measurement devices
• remote terminal units (RTUs)
• smart plugs
• programmable thermostats
• smart appliances
• sensors installed on grid-level equipment (e.g., transformers, network switches)
• asset inventory
• supervisory control and data acquisition (SCADA) system
• geographic information system (GIS)
• weather information, trafﬁc information, and social media .

Data Centers for Big Data

Big Data in Smart Grid
• Big data in smart grids are
• heterogeneous
• with varying resolution
• mostly asynchronous
• stored in different formats (raw or processed) at various locations.
• Smart meters collect data in every 15 minutes and these are stored in billing centers (1
million of smart meters may collect 3 TB of data every year).
• PMUs measure high-resolution voltage and current in the power grid and report at a 30-
60 times per second rate as time-synchronized phasors to phasor data concentrators
(PDCs) (40 TB of data every year).

5 V’s of Big Data

5 V’s of Big Data
• Volume
• in the order of thousands of terra bytes
• Variety
• structured/unstructured, synchronous/asynchronous
• Velocity
• real-time, second/minute/hour resolution
• Veracity
• inconsistencies, redundancies, missing data, malicious information
• Values
• technical, operational, economic

Big Data Management

Big Data Management
• Mega-corporations such as Google, Microsoft, Amazon have matured
data-mining and processing tools that allow for quick and easy processing
of large amounts of data.
• Big data analytics is more than just the data management; it is rather an
operational integration of big data analytics into power system decision-
making frameworks.
• With the development of proper business model for the key stakeholders
(e.g., electric utilities, system operators, consumers, aggregators).

Pattern of big data volume in electric utilities

Big Data from the perspective of electric utilities
• Electric utility is a very complex structure having close dependencies and
interactions among communications, IoT, and human factors.
• Recent concerns on increased security and reliability of critical
infrastructure are leading to the need of integrated energy system, which
integrates various critical infrastructure (electrical, gas, thermal, and
transportation).
• Future power grid management systems will be processing overwhelming
amounts of heterogeneous data.

Current utility status of Big Data implementation

Current identified barriers for big data

Challenges for the implementation of big data
analytics in electrical industries
• The Challenges are
• skill shortage
• data management issues
• lack of proper business models
• lack of management support
• Operational integration of big data to utility decision framework and its
value proposition to different stakeholders (e.g., utilities, system operators,
aggregators, consumers)and professional training are the key challenges to
be considered.
• Various industries like Siemens, GE, ABB, OSI-Soft, and so on are
developing big data platform and analytics for power grids.

Big data platforms developed by different companies
Company Name Name of the Platform
Simens EnergyIP Analytics(used by more than 50 utilities with a total of 28 millions installed smart
devices ).
GE • PREDIX (Industrial IOT platform which collects data from existing grid management systems,
smart meters, and grid sensors
• Native data collected from Grid IQ Insight which utilizes PREDIX platform.
ABB ABB Asset Health Centre (It monitors and establishes end-to-end asset management, business
processes for reducing costs, minimizing risks, improving reliability, and optimizing operations
across the electric utility) .
Smart Asset
Management Platform
OSI – Soft (for the purpose of real time monitoring asset health).

High-level overview of GE big data analytics platform

Key Challenges and Solutions to Apply Big Data to
Smart Grid

Development of business models for big data analytics
• Google, Facebook, Amazon disruptively transformed their business via big
data analytics, but electric utilities are still in the initial stage.
• The business models should be justiﬁed on the basis of market
opportunity/volume, required investment, and values to different
stakeholders.
• Recent research has estimated a cumulative $20 billion market value
between 2013 and 2020, growing to nearly $4 billion a year by 2020 .
This shows huge market potentials for big data analytics to electric
utilities.

Global utility analytics spending

Key stages of big data analytics

• Data Acquisition
• Collection of data from multiple heterogeneous sources with different formats and
features.
• Private information, consumer behaviours are to be protected.
• Data Confidentiality and security is maintained through data encryption and
decryption.
• Data Storage
• Data storage primarily belongs to data management (i.e. Data fusion, data integration,
and data transformation.
• Each data object has an associated key and each working node stores a group of keys
to make storage flexible.
• Data Analytics
• is designed to identify hidden and potentially useful information and patterns within
large dataset that can be transformed into knowledge.
• various algorithms (e.g., clustering, correlation, classification, categorization,
regression, feature extraction) are used to extract valuable information.

Different analytics techniques and their key applications

Application specific analytics applied to smart grid

Types of Data Analytics in Smart Grid
• Event analytics
• Detection of abnormal operating conditions including fault detection, system
outage detection, detection of malicious attacks, and theft of electricity are some
of the key application areas for event analytics.
• State analytics
• includes state estimation, system identification, and grid topology
identifications
• customer analytics
• includes customer classification/categorization, correlation between consumer
behavior and energy consumption patterns, and demand response
• operational analytics
• includes energy/load forecast, energy management and dispatch of resources .

GE Predix platform for big data analytics

• PREDIX is an IOT based data analytics platform which is the
foundation of Grid IQ Insight (a big data analytic architecture).
• This cloud based horizontal architecture consists of four layers.
• The bottom most layer is basically a physical layer consists of utility assets,
operational systems, and external data.
• the second layer is primarily a cloud based API and utility specific data layer
(e.g. analytics, dashboards).
• The third layer primarily includes grid applications.
• the fourth layer focuses on the visualization and operational integrations.

IBM based Lockheed Martin big data analytics
architecture for smart grid applications

IBM based Lockheed Martin big data analytics
architecture for smart grid applications
It consists of
• a four vertical layered reference architecture, where the left most
layer deals with data sources.
• the second layer consists of big data platforms and capabilities.
• the third layer deals with data analytics and customer insights.
• the last layer is designed to integrate data analytics results for various
operations.

SAP reference architecture for big data processing

SAP reference architecture for big data processing
This is a combination of
• horizontal and vertical reference architectures developed by SAP.
• vertical layers include data sources and data ingestion.
• horizontal layers include applications, real time data accelerated analytics, and data
management (e.g., storage, data processing and deep analytics).

Oracle big data analytics reference architecture

Oracle big data analytics reference architecture
It consists of vertical and horizontal layers.
• The vertical layers include data sources, data acquisition, data organization (to
ensure data quality for analytical operations), data analytics, decision making
(recommendation, alerts, dashboards), and data management (e.g., storage,
data security, governance).
• horizontal layers include technology platforms and integration layers for
operational integration to electric utility operational framework.

Big Data Platforms
• Hadoop
• Spark
• Storm
• Drill
• HPC

Hadoop
• is an open source framework for storing and processing large datasets using
Map Reduce programming model.
• it consists of storage part (known as hadoop distributed file systems (HDFS)
and processing part (known as Map Reduce programming model).
• splits files into large blocks and distributes them across nodes so as to process
data in parallel.
• HDFS not only ensures high availability, but also high fault tolerance
against hardware failures.
e.g. OSI-Soft is a hadoop based database and data analytics platform in electric
utility used in the PI system.

MapReduce
• MapReduce [OSDI’04] provides
• Automatic parallelization, distribution
• I/O scheduling
• Load balancing
• Network and data transfer optimization
• Fault tolerance
• Handling of machine failures
• Need more power: Scale out, not up!
• Large number of commodity servers as opposed to some high end
specialized servers
40
Apache Hadoop:
Open source implementation of
MapReduce

MapReduce workflow
42
Worker
Worker
Worker
Worker
Worker
read
local
write
remote
read,
sort
Output
File 0
Output
File 1
write
Split 0
Split 1
Split 2
Input Data Output Data
Map
extract something you care
about from each record
Reduce aggregate,
summarize, filter, or
transform

44
http://kickstarthadoop.blogspot.ca/2011/04/word-count-hadoop-map-reduce-example.html
Example: Word Count

MapReduce
47
Hadoop
Program
Master
fork fork fork
assign
map
assign
reduce
Worker
Worker
Worker
Worker
Worker
read
local
write
remote
read,
sort
Split 0
Split 1
Split 2
Input Data
Map Reduce
Output
File 0
Output
File 1
write
Output Data
Transfer
peta-scale
data
through
network

Google File System (GFS)
Hadoop Distributed File System (HDFS)
• Split data and store 3 replica on commodity servers
48

MapReduce
49
Masterassign
map
assign
reduce
Worker
Worker
Worker
Worker
Worker
local
write
remote
read,
sort
Output
File 0
Output
File 1
write
Split 0
Split 1
Split 2
Split 0
Split 1
Split 2
Input Data Output Data
Map Reduce
HDFS
NameNode
Read from
local disk
Where are the chunks of input
data?Location of the chunks of
input data

Spark
• is a fast, in-memory, open-source big data processing engine which is designed
to overcome the disk I/O limitations of Hadoop.
• perform in-memory computations and allow the data to be cached in memory.
• It eliminates Hadoop’s disk overhead limitation for iterative tasks.
• It is 100 times faster than Hadoop MapReduce when data can fit into the
memory and 10 times faster when data resides in the disk.

Storm
It is an open source distributed
• real-time computation system, that can reliably process unbounded
streams of data.
• It is scalable, fault-tolerant, and easy to set up and operate, thereby
having several use cases, including realtime.
• It uses analytics, online machine learning, and real-time computation.

Apache Drill
• It is an open source software framework that supports data-intensive distributed
applications for interactive analysis of large-scale datasets.
• It is able to scale 10,000+ servers and process peta bytes of data and trillions of
records within seconds.
• It can discover schemas on-the-fly, thereby delivering self-service data exploration
capabilities on data stored in multiple formats in files or databases.
• It can seamlessly integrate with several visualization tools, thereby making big-
data platform interactive.

HPC
• HPC is a vertical scale up platform for big data processing which
consists of a powerful machine with thousands of cores.
• Due to high quality hardware implementation it is highly fault tolerant
and hardware failures are extremely rare.
• It can process terabytes of data.

Comparison of various big data analytics platforms

Potential applications of big data analytics in smart
grids

Application of Big Data in Smart Grid
The big data has potential to
• improve reliability and resiliency of power grid.
• deliver optimum asset management and operations.
• improve decision making by sharing information/data.
• to support rapid analysis of extremely large data sets for performance
improvement.

Energy Management related Applications
• Energy management in real time.
• Energy price and load forecasting using machine learning, deep
learning algorithms.
• Extracting hidden usage patterns of consumers from big data.
• Dimensionality reduction of large scale of big data.
• Energy management of large public bindings.
• to use smart meter data and applied time-based Markov Model and
clustering algorithms to identify end users’ energy consumption
dynamics.

Improvement of smart grid reliability and stability
• Social media like Twitter data may enhance the system relaibility.
• GIS, GPS, and weather data is used in outage management.
• Application of SCADA big data for voltage instability detection.
• PMU big data could be used for stability margin prediction and real-
time asset health monitoring.
• big data can greatly benefit for generator performance monitoring for
improving market and operational efficiency.

Visualization
• single line diagram, 2D, and 3D charts/plots were used for grid
visualization.
• for the big data visualization in smart grid Scatter diagram, parallel
coordinate, and Andrew curve may be used.
• RTDMS providing several visualization options including dashboard
display for situational awareness, voltage angle contour plots, voltage
magnitude plot, frequency plot, oscillatory mode plot, etc. may be
used.

Parameter or State estimation
• Parameter and state estimations are essential for power system
planning, operation, and control.
• Estimations are used for several applications including operational
resource planning, real-time system monitoring, and resilient control
design against cyber- and/or physical-attacks.

Application to Cyber Physical Systems
• Due to close interdependencies between power and communication
infrastructure, the future grids subject to increased risk of malicious
attacks.
• Integration of big data analytics provides an excellent opportunity to
timely identify such malicious attacks and prevent the system from
huge damages.

Conclusion
• Here, utility and industry perspectives on current status of big data
implementation in power system is presented.
• Key technical, security, and regulatory challenges for deploying big data to
smart grid are identified.
• Value proposition of big data analytics to key stakeholders (e.g., consumers,
electric utilities, and system operators) is described with respect to
operational integration of big data to utility’s decision frameworks.

References
1. J.Q.Trelewicz,“Big data and big money : The role of data in the ﬁnancial sector,” IT Professional, vol. 19, no. 3,
pp. 8–10, 2017.
2. E. I. Lab, “Big data in banking for marketers how to derive value from big data,” White Paper.
3. U. Srinivasan and B. Arunasalam, “Leveraging big data analytics to reduce healthcare costs,” IT professional,
vol. 15, no. 6, pp. 21–28, 2013.
4. M.M.Islam,M.A.Razzaque,M.M.Hassan,W.N.Ismail,andB.Song,“Mobile cloud-based big healthcare data
processing in smart cities,” IEEE Access, vol. 5, pp. 11887–11899, 2017.
5.M. Marjani, F. Nasaruddin, A. Gani, A. Karim, I. A. T. Hashem, A. Siddiqa, and
I.Yaqoob,“BigIoTdataanalytics:Architecture,opportunities,andopenresearch challenges,” IEEE Access, vol. 5, pp.
5247–5261, 2017.
6. M. Satyanarayanan, P. Simoens, Y. Xiao, P. Pillai, Z. Chen, K. Ha, W. Hu, and B. Amos, “Edge analytics in the
internet of things,” IEEE Pervasive Computing, vol. 14, no. 2, pp. 24–31, 2015.
7. S.K.SharmaandX.Wang,“Live data analytics with collaborative edge and cloud processing in wireless iot
networks,” IEEE Access, vol. 5, pp. 4621–4635, 2017.
8. X. He, Q. Ai, R. C. Qiu, W. Huang, L. Piao, and H. Liu, “A big data architecture design for smart grids based on
random matrix theory,” IEEE Transactions on Smart Grid, vol. 8, no. 2, pp. 674–686, March 2017.
9. Y. Sun, H. Song, A. J. Jara, and R. Bie, “Internet of things and big data analytics for smart and connected
communities,” IEEE Access, vol. 4, pp. 766–773, 2016.
10. Bhattarai, B. P., Paudyal, S., Luo, Y., Mohanpurkar, M., Cheung, K., Tonkoski, R., ... & Manic, M. (2019). Big
data analytics in smart grids: State-of-the-art, challenges, opportunities, and future directions. IET Smart
Grid, 2(2), 141-154.

THANK YOU !!!!

????

Smart grid talk_puspanjali (1)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Smart grid talk_puspanjali (1)

Similar to Smart grid talk_puspanjali (1) (20)

Recently uploaded

Recently uploaded (20)

Smart grid talk_puspanjali (1)