Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Sandy May & Richard Conway, Elastacloud
Azure Databricks
Monitoring Solar Farms
#SAISDD11
About Us
• Richard Conway Founder and Director of Elastacloud, a UK Cloud Data Analytics Consultancy, Azure
MVP/Microsoft ...
What we’ll cover in this presentation
• A little about solar and renewables data
acquisition
• A little about Azure and ho...
The Solar Farm
4
images from mnn.com
5
Courtesy Wikipedia – installed UK capacity ~
< 10GW
Solar Irradiance
6
Inverter
W/m2
String
String
String
DC to AC National
Grid
Modbus Collection of Data
7
Inverter
Portal
Consumer
Logging
Get Data
TCP Port 502
Things to note:
- Inverter can generall...
Cloud Gateway Patterns from Farm
8
Inverter Modbus
Collector
ServerPyranometer
Weather
station
Generally
terrible
router
C...
All sources of information
9
Farm
weatherbit.io
darksky
openweather
Met office
cloud cover
suntimes
Azure
Balancing Energy
10
production
time - intraday
shortfall
surplus
- Shortfalls lead to buy
back to fulfil contracts
- Surpl...
Our Azure Architecture
11
Data Bricks
Event Hubs
Scada API for
Actual Power &
Recorded
Irradiance
Weather API’s
(MetOffice...
Demos and Code!
12
Thanks and Questions?
13
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1

Share

Download to read offline

Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to Monitor 1,000+ Solar Farms in Real Time with Sandy May and Richard Conway

Download to read offline

Renewables AI is at the forefront of innovation in the solar energy market. As the name suggests, we use AI to make predictions on energy output from large portfolios of solar farms. This talk lays out the fundamental architecture, technology and approaches that make the platform work beginning with key features of the Azure Databricks cloud and how it works seamlessly with Azure Data Lake and Azure Event Hubs. There will be good coverage of ML and DL Pipelines and how they are used with image recognition and machine learning through Structured Streaming to make real-time decisions.

Key Takeaways: Prediction of next day irradiance and power ratios with real-time accuracies of 95% Structured streaming of IoT data from hundreds of thousands of inverters at 5 minute intervals Real-time joining of weather data and several other external datasets Use of Deep Learning Pipelines and advanced time series methods to predict 48 hours of future energy production Near-real time processing of image data at frequent intervals to predict cloud cover from onsite cameras and drones Analysis of data and preventative maintenance of fan failures in solar inverters

Using Azure Databricks, Structured Streaming, and Deep Learning Pipelines to Monitor 1,000+ Solar Farms in Real Time with Sandy May and Richard Conway

  1. 1. Sandy May & Richard Conway, Elastacloud Azure Databricks Monitoring Solar Farms #SAISDD11
  2. 2. About Us • Richard Conway Founder and Director of Elastacloud, a UK Cloud Data Analytics Consultancy, Azure MVP/Microsoft Regional Director + Sandy May Cloud Big Data Surgeon • Microsoft Azure Gold Partner, Cloud Platform and Data Analytics, OSS Partner of the year 2015, Microsoft Partner of the Year nominee 2018 • Co-founder of UK Azure User Group, IoT and Data Science Innovators UK, UK Cloud Infrastructure User Group • Author of data science degree academy.microsoft.com • Running AzureCraft in UK annually • Contributors to open source, several Apache projects including Storm, Spark, Libcloud and Parquet • 50+ people, offices in London, Nottingham and Spain 2#SAISDD11
  3. 3. What we’ll cover in this presentation • A little about solar and renewables data acquisition • A little about Azure and how we design things • Using Databricks in a fun way • Orchestrating the future 3
  4. 4. The Solar Farm 4 images from mnn.com
  5. 5. 5 Courtesy Wikipedia – installed UK capacity ~ < 10GW
  6. 6. Solar Irradiance 6 Inverter W/m2 String String String DC to AC National Grid
  7. 7. Modbus Collection of Data 7 Inverter Portal Consumer Logging Get Data TCP Port 502 Things to note: - Inverter can generally only accept a single Modbus TCP connection - As such usage of Hub and Spoke Models are needed to relay to cloud - Modbus def example: 29;1;INV_EFFICIENCY;3;2;29;;;0.1;0;%;2 Register No. Min/Max/Avg Units Msg Type Per minute Redundant Consumer TCP Port xxx
  8. 8. Cloud Gateway Patterns from Farm 8 Inverter Modbus Collector ServerPyranometer Weather station Generally terrible router Cloud Gateway Export Meter Event Hub Message Stream Low spec AWS Server Data Orchestration API with 24 hour lag Event Hub Message Stream Takes around 8-9 hours!
  9. 9. All sources of information 9 Farm weatherbit.io darksky openweather Met office cloud cover suntimes Azure
  10. 10. Balancing Energy 10 production time - intraday shortfall surplus - Shortfalls lead to buy back to fulfil contracts - Surpluses need selling - Grid needs to be balanced at all times - Need to understand intraday pricing market to understand exposure - Insure risk through PPAs for lower returns - National Grid has balancing cost - Internal transactions need balancing too - Need to understand interconnects
  11. 11. Our Azure Architecture 11 Data Bricks Event Hubs Scada API for Actual Power & Recorded Irradiance Weather API’s (MetOffice, DarkSky & WeatherBit) 5 minute weather data streamed for over 1,200 solar farms in Europe; data used to create Solar Irradiance predictions, and Power output for farms using service Actuals collected for model training and evaluation Predictions and model performance recorded in Azure SQL Data Warehouse and reported on using Power BI
  12. 12. Demos and Code! 12
  13. 13. Thanks and Questions? 13
  • DerekBevan1

    Nov. 30, 2019

Renewables AI is at the forefront of innovation in the solar energy market. As the name suggests, we use AI to make predictions on energy output from large portfolios of solar farms. This talk lays out the fundamental architecture, technology and approaches that make the platform work beginning with key features of the Azure Databricks cloud and how it works seamlessly with Azure Data Lake and Azure Event Hubs. There will be good coverage of ML and DL Pipelines and how they are used with image recognition and machine learning through Structured Streaming to make real-time decisions. Key Takeaways: Prediction of next day irradiance and power ratios with real-time accuracies of 95% Structured streaming of IoT data from hundreds of thousands of inverters at 5 minute intervals Real-time joining of weather data and several other external datasets Use of Deep Learning Pipelines and advanced time series methods to predict 48 hours of future energy production Near-real time processing of image data at frequent intervals to predict cloud cover from onsite cameras and drones Analysis of data and preventative maintenance of fan failures in solar inverters

Views

Total views

427

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

22

Shares

0

Comments

0

Likes

1

×