SG data analytics

1. Smart Grid Data Analytics
Technologies, Trends and Challenges
Dr. Danish Mahmood
SZABIST- Islamabad
International Symposium on Emerging trends in Computing and Engineering-2022 (ISETCE-2022)

2. Questions and Questions to discuss…
• Why we want to induct smartness in power sector?
• Is it necessary or world can survive well without it?
• Does smart grid generate big data?
• How data is generated?
• How to Minimize Peak Power Demand at SH, SB and SC scale using Analytics and CI
• How Pakistan is dealing with forecasting Peak Power Demand
• Is the problem of capacity payments and circular debt political or technical?
• What possible architectures can be utilized for SG Batch and Streaming
Applications?
• What are the possible future directions for globe and especially for
Pakistan?
Challenges
and
future
directions

3. • Electricity is integral part of our lives
• Increasing power demand day by day require increase in power generation
accordingly
• The problem we realized about a decade earlier is
• Curtailing natural resources
• Serious ecological and environmental threats- green house gas emissions/ global
warming
• Solution lies in such a power system that
• Ensures clean and green energy- mitigate ecological concerns- can not be done
overnight
• Ensures two way communication between user and producer
• Work on some power demand response programs
• Rely on distributed energy sources – mitigate security concerns
Why we want to induct smartness in power sector?
Is it necessary or world can survive well without it?

4. According to Global Electricity Review 2022
Wind and solar, the fastest growing sources of electricity, reach a record ten
percent of global electricity in 2021
dataset comprises annual power generation and import data for 209 countries covering the period 2000 to 2020. For 2021,
added data for 75 countries which together represent 93% of global power demand.
Download full data from https://ember-climate.org/insights/research/global-electricity-review-2022/#supporting-material-downloads
https://ember-climate.org/insights/research/global-electricity-review-2022/
Set target of min 85% by 2050

5. What strategies to be adopted to create equilibrium
between demand and supply?
• Induction of Distributed Energy Resources
• Demand/Supply side management
• Classifying and identifying types of consumers
• Residential, industrial, government,
Hybrid
• Optimizing misc. uncertainty factors
• RE Gen, Demand and Distribution losses
etc.
• Offering such policies and regulations that
gives certain incentives to consumers
• Pricing mechanisms like RTP, IBR, Day
Ahead pricing and Direct load control etc.
• Ensuring Advanced metering infrastructure availability
• Two way communication amongst demand and supply side

6. Data Sources Technology Utilization and Applications
AMI Smart Meters Load/ power consumption scheduling and
forecasting power demands
Distribution
automation
Grip Equipment Sensors are deployed in Distribution system
which take periodic samples.
Real time monitoring, Theft control, OPF etc
Off Grid Data Third party data Climatic and demographic data.
Define policies, consumer behaviors
demand response programs and long term
predictions
Smart Grid Big Data
How data can be collected, utilized and applied for useful insights?
Distributed Data Gathering: What are the recommended tools for such scenarios?
Is it necessary to bring data towards computational engine or it can be computed at edges?
If smart meter data are collected every 15 sec, over I million devices results in 2920TB per year.

7. Per hour power
consumption and
generation
Is there rise or fall in
power demand as per
routine?
What are reasons
behind?
What is going to happen
if such rise or fall persists
To manage such demand/ supply
fluctuation, orchestrate an
automated system (DSM/ SSM)
Take actions to minimize
Smart Grid Analytics
Peak Power Demand
Peak demand refers to the times of day when electricity consumption is at its highest.

8. Min. Peak Power Demand
Generic System Model: HEMS
• Home management systems
• Minimizing electricity consumption
• Minimizing electricity bills
• Offering incentives to install small scale RE generation
plants
• Constraints
• User Comfort
• Policies and implementations of policies (Net metering)
• Solution
• Optimal home Appliance Load Scheduling
Danish Mahmood et al., "Realistic Scheduling Mechanism for Smart Homes." Energies 2016, 9(3), 202;
Major problem in engineering industries
Lab results or test environment results do not match with implemented results
Why?
Majorly- User Comfort Problem
Solution
Can this User comfort be quantized???

9. Min. Peak Power Demand
Danish Mahmood, et.al.,, "A review on optimization strategies integrating renewable energy sources focusing uncertainty factor – Paving path to eco-friendly smart cities",
Sustainable Computing: Informatics and Systems Volume 30, June 2021, 100559
Generic System Model: Smart City
• Problems
• Real time data gathering and
computing
• Predicting RE generation near
real time
• Uncertainty increases as RE
increases
• Ensuring unit commitment
(RE share takes load as
forecasted).

10. Pakistan Energy generation share- 2021
Clean energy- 41%
Clean energy without Nuclear- 33-34%
Non RE Share- 59%
NEPRA, “Indicative Generation Capacity Expansion Plan (IGCEP) 2021-30 Report,” NEPRA, 2021 [Online]. Available:
https://nepra.org.pk/Admission%20Notices/2021/06%20June/IGCEP%202021.pdf

11. Peak Power Demand Forecast in Pakistan
Year Actual Peak Demand (MW) Projected Peak Demand (MW)
2018-2019 25627 27261
2019-2020 26252 27128
2020-2021 27414 24106
NEPRA, “Indicative Generation Capacity Expansion Plan (IGCEP) 2021-30 Report,” NEPRA, 2021 [Online]. Available:
https://nepra.org.pk/Admission%20Notices/2021/06%20June/IGCEP%202021.pdf
Multiple linear regression model is utilized
This model cater for only three independent variables i.e. ,GDP, Population and Number of Consumers
The existing model use yearly peak load. However results can be more accurate if monthly peak is taken.

12. Precise peak power demand prediction
• We have all the data for this,
• Inspite of using only GDP, Population and Number of Consumers for peak
power demand forecasting on yearly basis
• All we need is to identify and expand target variables
• Climatic data as in coming years solar energy production is planned massively.
• Economic features as month wise GDP, industrial growth, inflation, etc.
• Demographic variables like population growth and per capita income
• Historic electrical and non electrical data based on occasions, holidays,
temperatures (Monsoon season) etc.

13. Smart Grid Distributed analytics
• Incubation of multiple RE sources and MGs in centralized power
generation plant.
• Two basic tasks
• How to store that huge data
• How to compute that huge data
• Solution lies in Distributed Computing

14. Climate Data.
Other Sensory Data
Communication data
User preferences Data
Smart Meter Data
Ingestion Name node
Data node 2
Data node 1
Data node 4
Data node 3
Data node 6
Data node 5
Gateway- name node
No change in data- Customers Data, or raw data
No
change
in
data-
only
addition,
coz
to
change
you
have
to
load
whole
data
in
ram
again
File size of 192 mb is to be stored ===one will ask Gateway to store it in HDFS, Gateway will communicate with Name node informing that
192mb is to be stored so give resources, HDFS name node will tell that it has max. data block size of 64mb, now gateway node will make three
parts of 192mb as B1, B2 and B3. now name node allocate resources (datanodes to store these three blocks)
B1
B2
B3
Default replication policy is 3
Secondary
name node
Hadoop 1– 64mb
Hadoop 2- 128mb
Hadoop 3- no replication- RAID disks
use only 30% of storage
A node that connects to the Hadoop cluster,
but does not run any of the daemons
Hadoop Storage

15. Possible Architecture: streaming operations of SG
Climate data
Other Sensory Data
Communication data
Smart Meter Data
User preference data
ELT
SQOOP/
Structured Data
HDFS
FLUME/ Point to Point
delivery tool for
unstructured data- no
storage just dumping
into hdfs
KAFKA- runs its own
cluster not hadoop
cluster
Can store the data- by
default it can store for
7 days,
Flume reads the data and sends in to
HDFS-we can not analyze data on the fly.
It’s a message queue system/ Based
on PS model. It is point to multipoint.
Spark Streaming/
flink/ storm
- Real time data
analytics
Pull architecture
Possible applications:
• Net metering
• AMI data manipulations
• Short term prediction of Peak Demand
• Demand and Supply Equilibrium

16. HDFS
Batch Processing
Default HADOOP behavior is sequential
Load whole table in RAM to do some thing
Map
Reduce
HIVE allows you to
write SQL on top of
hadoop
Hive FB created it in 2005
Allows you to create tables and store the data
Allows you to write SQL on hadoop
Converts the query into a map reduce program
SPARK In Memory
Processing- SPARK
SQL
For big queries like traverse the
whole data etc, hive or SQL is
great-Load whole table and do
the computation- SPARK SQL is
fastest due to in memory
processing.
HBase
ZOOKEEPER
Cluster
HBase depends on ZooKeeper
By default HBase manages the ZooKeeper
instance
E.g., starts and stops ZooKeeper
HMaster and HRegionServers register
themselves with ZooKeeper
Possible architecture: Batch processing operations of SG
Possible applications
• Peak power demand forecast
• Minimizing effect of RE uncertainty
• Clustering and maintaining OPF

17. HDFS
What if I intend to work on just one row of the data???
Here we need MPP engines, like IMPALA, HAWQ, LLAP,
DRILL, phinix
MPP engines
Impala Hawq/LLAP
DRILL
Phoenix
HBase
PRESTO
• Write an impala query and it is faster than hive
• Then why not to use hive--- reliability issues
• IMPALA queries are in RAM- once lost is lost…
• NoSQL DB of Hadoop near real-time db
• Install HBASE on top of hadoop-- Random reads and writes
are allowed
• It has its own language. Do not allow SQL queries drawback
• Phoenix has SQL –
• Combining Phoenix and HBASE is deadly- writing SQL on no
SQL
ZOOKEEPER Cluster
Hbase + Phoenex = OLTP-
Online Transactional processing
System
OLTP Systems
HBase depends on ZooKeeper
By default HBase manages the ZooKeeper instance
E.g., starts and stops ZooKeeper
HMaster and HRegionServers register themselves with ZooKeeper
What if you want to run a query on just one row. Why to load whole data into RAM
Tools w.r.t applications related with real time processing

18. Future directions
• Uncertainty analysis
• RE integration in Smart homes, buildings and cities
• Security cyber physical systems
• SG data ingestion architectures and tools
• SG data querying (stream and real time solutions)
architectures and frameworks
• Optimal ESS utilization (state of change, Sharing
economy using ESSs, Electric cars )