The document describes an algorithm for short-term temperature and hourly precipitation prediction using neural networks. It involves collecting data from various weather stations, preprocessing the data by interpolating missing values and normalizing it. The data is then segmented and fed into separate neural networks for temperature and precipitation prediction. Different combinations of input variables, network architectures and hyperparameters are tested. The best model is selected and used to make online predictions through a web interface. Future work may involve improving prediction accuracy through additional modeling techniques.
This master's thesis project aims to develop a magnetic component design environment to improve the design process. The environment will allow for optimal inductor and transformer design through automated optimization algorithms and precise loss and thermal models. As part of this project, a core loss measurement system was developed to build a loss map database for the design environment to utilize. Initial testing of the design environment showed good accuracy, with relative errors under 10% when comparing calculated and measured loss values.
This document summarizes research to control spectrum analyzers remotely using Python for an axion dark matter experiment. A Python program was developed to control Keysight spectrum analyzers in two modes - XSA and Noise Figure. A web-based monitoring system was also updated with real-time graphs and an automated logbook to track refrigerator temperatures. The system was tested with a small cavity experiment and successfully detected test signals. Future work includes testing the system in a real axion experiment.
Wearable Gait Classification Using STM SensortileShayan Mamaghani
- Successful and efficient classification of gait behavior.
- Automated real-time discrimination.
- Used STM Sensortile in a dual sensor data acquisition module and a Beaglebone for processing.
- Utilized the FANN neural network library to train and test the system.
Big Fast Data in High-Energy Particle PhysicsAndrew Lowe
Experiments at CERN (the European Organization for Nuclear Research) generate colossal amounts of data. Physicists must sift through about 30 petabytes of data produced annually in their search for new particles and interesting physics. The tidal wave of data produced by the Large Hadron Collider (LHC) at CERN places an unprecedented challenge for experiments' data acquisition systems, and it is the need to select rare physics processes with high efficiency while rejecting high-rate background processes that drives the architectural decisions and technology choices. Although filtering and managing large data sets is of course not exclusive to particle physics, the approach that has been taken is somewhat unique. In this talk, I describe the typical journey taken by data from the readout electronics of one experiment to the results of a physics analysis.
Combining out - of - band monitoring with AI and big data for datacenter aut...Ganesan Narayanasamy
Andrea Bartolini presented a method for combining out-of-band monitoring with artificial intelligence and big data analytics to enable datacenter automation. Their system, called D.A.V.I.D.E., uses fine-grained power and performance monitoring of nodes through an embedded system. Data is collected and analyzed using MQTT, Cassandra, and Apache Spark. An autoencoder was trained on historical monitoring data to learn normal behavior and is used to detect anomalies through reconstruction error at the edge in real-time. Future work includes extending this approach for security and expanding it to larger systems.
The document describes the development of a new motionlogger actigraph. It discusses actigraphy technology, uses in sleep and medical research, and key design considerations for motionlogger devices. Specifically, it outlines the importance of low noise, a sensitive accelerometer, precise filtering, avoiding data collection during high current operations, and using a stable power supply to achieve high accuracy compared to polysomnography.
Synchrophasor Applications Facilitating Interactions between Transmission and...Luigi Vanfretti
Distribution grid dynamics will become increasingly complex due to the transition from passive to active networks arising from the increase of renewable energy sources at medium and voltage level. A successful transition requires to increase the observability and awareness of the interactions between Transmission and Distribution (T&D) grids, particularly to guarantee adequate operational security.
This presentation explores how different technical means can facilitate interactions between TSOs and DSOs with the utilization of GPS-time-synchronized phasor measurements (aka Phasor Measurement Units (PMUs)) with millisecond resolution. If made available in actual T&D networks, such high-sampled data across operational boundaries allows an opportunity to extract information related to different time-scales.
As part of the work carried out in the EU-funded FP7 IDE4L project (http://ide4l.eu/), a specific use case, containing PMU-based monitoring functions, has been defined to support the architecture design of future distribution grid automation systems. As a result, the architecture can accommodate for key dynamic information extraction and exchange between DSO and TSO.
This presentation presents the use case and focuses on the technical aspects related to the development and implementation of the PMU-based monitoring functionalities that can provide means to facilitate technical co-operation between transmission and distribution operations.
Real Time Monitoring and Electro Magnetic Interference causing Data corruptionRekaNext Capital
On site, where there are motors and generators, the sensor reading are affected by Electro-Magnetic Interference. The presentation shares some Good Engineering Practices to minimize Data Corruption. Real Time Monitoring Data Quality is even more sensitive to Data Corruption as there is a huge amount of sensors per hour.
This master's thesis project aims to develop a magnetic component design environment to improve the design process. The environment will allow for optimal inductor and transformer design through automated optimization algorithms and precise loss and thermal models. As part of this project, a core loss measurement system was developed to build a loss map database for the design environment to utilize. Initial testing of the design environment showed good accuracy, with relative errors under 10% when comparing calculated and measured loss values.
This document summarizes research to control spectrum analyzers remotely using Python for an axion dark matter experiment. A Python program was developed to control Keysight spectrum analyzers in two modes - XSA and Noise Figure. A web-based monitoring system was also updated with real-time graphs and an automated logbook to track refrigerator temperatures. The system was tested with a small cavity experiment and successfully detected test signals. Future work includes testing the system in a real axion experiment.
Wearable Gait Classification Using STM SensortileShayan Mamaghani
- Successful and efficient classification of gait behavior.
- Automated real-time discrimination.
- Used STM Sensortile in a dual sensor data acquisition module and a Beaglebone for processing.
- Utilized the FANN neural network library to train and test the system.
Big Fast Data in High-Energy Particle PhysicsAndrew Lowe
Experiments at CERN (the European Organization for Nuclear Research) generate colossal amounts of data. Physicists must sift through about 30 petabytes of data produced annually in their search for new particles and interesting physics. The tidal wave of data produced by the Large Hadron Collider (LHC) at CERN places an unprecedented challenge for experiments' data acquisition systems, and it is the need to select rare physics processes with high efficiency while rejecting high-rate background processes that drives the architectural decisions and technology choices. Although filtering and managing large data sets is of course not exclusive to particle physics, the approach that has been taken is somewhat unique. In this talk, I describe the typical journey taken by data from the readout electronics of one experiment to the results of a physics analysis.
Combining out - of - band monitoring with AI and big data for datacenter aut...Ganesan Narayanasamy
Andrea Bartolini presented a method for combining out-of-band monitoring with artificial intelligence and big data analytics to enable datacenter automation. Their system, called D.A.V.I.D.E., uses fine-grained power and performance monitoring of nodes through an embedded system. Data is collected and analyzed using MQTT, Cassandra, and Apache Spark. An autoencoder was trained on historical monitoring data to learn normal behavior and is used to detect anomalies through reconstruction error at the edge in real-time. Future work includes extending this approach for security and expanding it to larger systems.
The document describes the development of a new motionlogger actigraph. It discusses actigraphy technology, uses in sleep and medical research, and key design considerations for motionlogger devices. Specifically, it outlines the importance of low noise, a sensitive accelerometer, precise filtering, avoiding data collection during high current operations, and using a stable power supply to achieve high accuracy compared to polysomnography.
Synchrophasor Applications Facilitating Interactions between Transmission and...Luigi Vanfretti
Distribution grid dynamics will become increasingly complex due to the transition from passive to active networks arising from the increase of renewable energy sources at medium and voltage level. A successful transition requires to increase the observability and awareness of the interactions between Transmission and Distribution (T&D) grids, particularly to guarantee adequate operational security.
This presentation explores how different technical means can facilitate interactions between TSOs and DSOs with the utilization of GPS-time-synchronized phasor measurements (aka Phasor Measurement Units (PMUs)) with millisecond resolution. If made available in actual T&D networks, such high-sampled data across operational boundaries allows an opportunity to extract information related to different time-scales.
As part of the work carried out in the EU-funded FP7 IDE4L project (http://ide4l.eu/), a specific use case, containing PMU-based monitoring functions, has been defined to support the architecture design of future distribution grid automation systems. As a result, the architecture can accommodate for key dynamic information extraction and exchange between DSO and TSO.
This presentation presents the use case and focuses on the technical aspects related to the development and implementation of the PMU-based monitoring functionalities that can provide means to facilitate technical co-operation between transmission and distribution operations.
Real Time Monitoring and Electro Magnetic Interference causing Data corruptionRekaNext Capital
On site, where there are motors and generators, the sensor reading are affected by Electro-Magnetic Interference. The presentation shares some Good Engineering Practices to minimize Data Corruption. Real Time Monitoring Data Quality is even more sensitive to Data Corruption as there is a huge amount of sensors per hour.
One-day ahead Power Forecasting is more and more required on the energy markets, and its accuracy is more and more crucial since it affects the net income of operators. 1. Weather Numerical Prediction, including a meso scale downscaling, provides a global prediction. A RANS CFD-tools is used for the micro-scale downscaling, providing a precise wind forecast at each wing generator hub. 2. To improve the reliability of this forecast, especially in the short term range, the use of "fresh" SCADA data is performed. Attention is focused on the Active Power, but other signals such as temperature and local wind characteristics can be taken into account. 3. In order to erase systematic errors and bias from the downscaled NWP based forecast (1.), as well as to mix it with the persistent model (2.), an Artificial Neural Network is trained using long term history. This paper explains first the method used and the choices made, especially concerning the Machine Learning parameters. A second part presents some results on some real cases, with different time horizons.
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Igor Sfiligoi
Presented at PEARC20.
This talk presents expanding the IceCube’s production HTCondor pool using cost-effective GPU instances in preemptible mode gathered from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and the Google Cloud Platform. Using this setup, we sustained for a whole workday about 15k GPUs, corresponding to around 170 PFLOP32s, integrating over one EFLOP32 hour worth of science output for a price tag of about $60k. In this paper, we provide the reasoning behind Cloud instance selection, a description of the setup and an analysis of the provisioned resources, as well as a short description of the actual science output of the exercise.
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Igor Sfiligoi
- IceCube is a neutrino observatory that detects high-energy neutrinos from astrophysical sources to study violent cosmic events. It uses over 5000 optical sensors buried in Antarctic ice to detect neutrinos.
- A cloud burst was performed using over 50,000 GPUs across multiple cloud providers worldwide to simulate photon propagation through ice for IceCube data analysis. This was the largest cloud simulation ever and demonstrated the ability to burst at exascale scales.
- The simulation helped improve IceCube's neutrino detection and pointing resolution to identify the first known source of high-energy neutrinos, a blazar, demonstrating IceCube's potential for multi-messenger astrophysics.
This document introduces SkyhookDM, a system that offloads computation from clients to storage nodes. It does this by embedding Apache Arrow data access libraries inside Ceph object storage devices (OSDs). This allows large Parquet files to be scanned and processed directly on the OSDs without needing to move all the data to clients. Experiments show SkyhookDM reduces latency, CPU usage, and network traffic compared to traditional approaches. It has also been integrated with the Coffea analysis framework. Ongoing work involves optimizing Arrow serialization for network transfers.
An Energy-Efficient and Delay-Aware Wireless Computing System for Industrial ...Arun Das
This document describes an energy-efficient and delay-aware wireless computing system for industrial wireless sensor networks. The system uses a fog-based architecture of interconnected servers that can communicate with each other to monitor and control industrial processes remotely. It aims to improve power efficiency by putting servers into sleep modes to reduce power consumption while still meeting delay requirements. The system dynamically adjusts the number of active and sleeping servers to maintain low power usage and acceptable communication delays. Evaluation shows the system is able to reduce power consumption over time while satisfying internal processing delays.
Timing-pulse measurement and detector calibration for the OsteoQuant®.Binu Enchakalody
The document describes calibration procedures for an OsteoQuant pQCT scanner. It discusses:
1) Measuring motor and detector timing pulses using a USB counter to synchronize data collection with position. Measurements were accurate to within 0.13%.
2) Correcting for detector dead time using a polynomial model to linearize photon counts versus tube current data. Corrections were stable to within 0.5% error.
3) Correcting for beam hardening effects using polynomial and bimodal energy models to linearize projection values with absorber thickness. A secondary correction further improved stability of different-date corrections to below 1% error.
Solar panel monitoring solution using IoT-Faststream TechnologiesSudipta Maity
Faststream Technologies offers an automated IOT based solar panel monitoring/troubleshooting system that allows for automated solar panel monitoring from anywhere over the internet. As part of our solution, we make use of several IoT gateways suitable for different needs, based on SoCs like STM32, ESP32, ublox, CC3200, SiliconLabs, to monitor the solar panel parameters, in turn, providing Solar Plant Insights.
Our system constantly monitors the solar panel and transmits various parameters to the Cloud over the IoT system. Here we make use of the IoT platform to transmit solar power parameters to Amazon/ Azure cloud /IOT server via the gateway (over WiFi and Ethernet). A powerful web interface allows viewing of data in meaningful formats, enabling users to make decisions.
Monitoring of Transmission and Distribution Grids using PMUsLuigi Vanfretti
My presentation on "Monitoring of Transmission and Distribution Grids using PMUs" for the Workshop on Energy Business Opportunities in NY State.
The Center for Integrated Electrical Energy Systems (CIEES) at Stony Brook University and the Center for Future Energy Systems (CFES) at Rensselaer Polytechnic Institute will be holding a one day Workshop on Energy Business Opportunities in NY State.
The document provides release notes for Logger Pro 3.2, describing:
1) New features in version 3.2 including video analysis, strip charts, improved rate functions, and LabPro operating system updates.
2) Installation notes for compatibility with TI Connect and removing older versions.
3) Known issues with the software such as limitations of supported monitors and potential problems connecting sensors.
SENSOR SELECTION SCHEME IN TEMPERATURE WIRELESS SENSOR NETWORKijwmn
In this paper, we propose a novel energy efficient environment monitoring scheme for wireless sensor
networks, based on data mining formulation. The proposed adapting routing scheme for sensors for
achieving energy efficiency from temperature wireless sensor network data set. The experimental
validation of the proposed approach using publicly available Intel Berkeley lab Wireless Sensor Network
dataset shows that it is possible to achieve energy efficient environment monitoring for wireless sensor
networks, with a trade-off between accuracy and life time extension factor of sensors, using the proposed
approach.
IRJET- Land Mine Data Collection System using Long Range WiFi and P2P Com...IRJET Journal
This system collects data from sensors in landmines including time of blast, weight, temperature, climate factors like rainfall, and direction of movement. This data helps determine the cause of mine blasts and informs soldiers of conditions. Sensors transmit data via long range WiFi to a base station where the information is displayed. Collecting this critical data enhances military investigations and situational awareness for troops.
"Building and running the cloud GPU vacuum cleaner"Frank Wuerthwein
This talk, describing the "Largest Cloud Simulation in History" (Jensen Huang at SC19), was given at the MAGIC meeting on Dec. 4th 2019. MAGIC stands for "Middleware and Grid Interagency Cooperation", and is a group within NITRD. Current federal agencies that are members of MAGIC include DOC, DOD, DOE, HHS, NASA, and NSF.
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...Igor Sfiligoi
NRP Engagement webinar: Description of the 380 PFLOP32S , 51k GPU multi-cloud burst using HTCondor to run IceCube photon propagation simulation.
Presented January 27th, 2020.
Burst data retrieval after 50k GPU Cloud runIgor Sfiligoi
We ran a 50k GPU multi-cloud simulation to support the IceCube science. This talk provided an overview of what happened to the associated data.
Presented at the Internet2 booth at SC19.
A Scalable and Distributed Electrical Power Monitoring System Utilizing Cloud...Ryousei Takano
This document describes a scalable and distributed electrical power monitoring system using cloud computing. Low-cost power measuring units collect data from current sensors and send it to data collecting units. These units then push the data to a data store hosted on Google App Engine. This cloud-based system allows visualization of power consumption across a large campus from any application accessing the data through a REST API. The system is scalable, low-cost, and easy to develop applications for power monitoring and planning energy savings.
This document provides an overview and agenda for a presentation on creating a "Hello World" program with Cisco's Data in Motion (DMo) software. The presentation introduces DMo and how it can manage and analyze data at the edge. It discusses how DMo represents a paradigm shift with edge intelligence and provides examples of railway and utilities use cases. The document explains DMo's programming model involving dynamic data definitions, patterns, conditions, and actions. It also demonstrates how to set up a DMo instance, create timer and event rules to read a light sensor and control an LED based on the sensor readings.
Application of machine learning and cognitive computing in intrusion detectio...Mahdi Hosseini Moghaddam
This document describes a proposed hardware-based machine learning intrusion detection system using cognitive processors. It discusses the need for new intrusion detection approaches due to limitations of signature-based methods. The proposed system collects network packet data using a Raspberry Pi and classifies it using a Cognimem CM1K cognitive processor chip, which implements restricted coulomb energy and k-nearest neighbor algorithms. The document outlines the system architecture, data collection and normalization methodology, and analysis of results from testing the CM1K chip on both custom and NSL-KDD network datasets, finding accuracy levels around 70-80% but slower processing times than a software simulation of the chip's algorithms. Future work areas include adding more packet features, using
Neural network modeling and control of data centers is presented. Data centers consume significant and increasing amounts of energy. A neural network model is developed and trained using steady state and transient data from a physical data center setup to map temperature outputs. The neural network accurately models temperatures with 95% accuracy. A neural network controller is then designed using the inverse model to stabilize temperatures according to reference values in response to varying workloads and power consumption. The controller successfully regulates temperatures in real-time simulation. Future work includes implementing the control on an actual system and expanding the control parameters.
The document provides an overview of SCADA and central control applications for power systems. It discusses:
1. SCADA architecture and components, including human-machine interfaces, application servers, communication servers, remote terminal units, and communication protocols.
2. Key SCADA functions like data acquisition, monitoring and event processing, control capabilities, and data storage.
3. Examples of centralized control applications for power system operation involving multiple actors like transmission system operators and generation companies.
ROLE OF DIGITAL SIMULATION IN CONFIGURING NETWORK PARAMETERSDeepak Shankar
Selecting the right Ethernet standard and configuring all the network devices in the embedded systems accurately is an extremely hard and rigorous job. The configuration depends on the topology, workloads of the connected devices, processing overhead at the switches, and the external interfaces. Network calculus, mathematical models and analytical techniques provide worst case execution time (WCET), but their probability of activity is extremely wide. This leads to overdesign which leads to higher costs, power consumption, weight, and size. Simulating the network is the best way to measure the throughput of the entire system. Digital system simulation provides better latency and throughput accuracy, but the accuracy is still limited because it does not consider the latency associated with the network OS, cybersecurity processing and scheduling. In many cases, these factors can reduce the throughput by 20-40%.
In this paper, we will present our research on modeling the entire Ethernet network, including the workloads, network flow control, scheduling, switch hardware, and software. To substantially increase the coverage and compare topologies, we have developed a set of benchmarks that provides coverage for different combination of deterministic, rate-constrained, and best effort traffic. During the presentation, we will cover the benchmarks, the list of attributes required to accurately model the traffic, nodes, switches, and the scheduler settings. We will also look at the statistics and reports required to make the configuration decision. In addition, we will discuss how the model must be constructed to study the impact of future requirements, failures, network intrusions, and security detection schemes.
Key Takeaways:
1. Learn how to efficiently use network simulation to design Ethernet systems
2. Develop a reusable benchmark and associated statistics to test different configurations
3. The role and impact of the CDT slots, guard band, send slope, idle slope, shuffle scheduling, flow control and virtual channels
One-day ahead Power Forecasting is more and more required on the energy markets, and its accuracy is more and more crucial since it affects the net income of operators. 1. Weather Numerical Prediction, including a meso scale downscaling, provides a global prediction. A RANS CFD-tools is used for the micro-scale downscaling, providing a precise wind forecast at each wing generator hub. 2. To improve the reliability of this forecast, especially in the short term range, the use of "fresh" SCADA data is performed. Attention is focused on the Active Power, but other signals such as temperature and local wind characteristics can be taken into account. 3. In order to erase systematic errors and bias from the downscaled NWP based forecast (1.), as well as to mix it with the persistent model (2.), an Artificial Neural Network is trained using long term history. This paper explains first the method used and the choices made, especially concerning the Machine Learning parameters. A second part presents some results on some real cases, with different time horizons.
Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...Igor Sfiligoi
Presented at PEARC20.
This talk presents expanding the IceCube’s production HTCondor pool using cost-effective GPU instances in preemptible mode gathered from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and the Google Cloud Platform. Using this setup, we sustained for a whole workday about 15k GPUs, corresponding to around 170 PFLOP32s, integrating over one EFLOP32 hour worth of science output for a price tag of about $60k. In this paper, we provide the reasoning behind Cloud instance selection, a description of the setup and an analysis of the provisioned resources, as well as a short description of the actual science output of the exercise.
Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...Igor Sfiligoi
- IceCube is a neutrino observatory that detects high-energy neutrinos from astrophysical sources to study violent cosmic events. It uses over 5000 optical sensors buried in Antarctic ice to detect neutrinos.
- A cloud burst was performed using over 50,000 GPUs across multiple cloud providers worldwide to simulate photon propagation through ice for IceCube data analysis. This was the largest cloud simulation ever and demonstrated the ability to burst at exascale scales.
- The simulation helped improve IceCube's neutrino detection and pointing resolution to identify the first known source of high-energy neutrinos, a blazar, demonstrating IceCube's potential for multi-messenger astrophysics.
This document introduces SkyhookDM, a system that offloads computation from clients to storage nodes. It does this by embedding Apache Arrow data access libraries inside Ceph object storage devices (OSDs). This allows large Parquet files to be scanned and processed directly on the OSDs without needing to move all the data to clients. Experiments show SkyhookDM reduces latency, CPU usage, and network traffic compared to traditional approaches. It has also been integrated with the Coffea analysis framework. Ongoing work involves optimizing Arrow serialization for network transfers.
An Energy-Efficient and Delay-Aware Wireless Computing System for Industrial ...Arun Das
This document describes an energy-efficient and delay-aware wireless computing system for industrial wireless sensor networks. The system uses a fog-based architecture of interconnected servers that can communicate with each other to monitor and control industrial processes remotely. It aims to improve power efficiency by putting servers into sleep modes to reduce power consumption while still meeting delay requirements. The system dynamically adjusts the number of active and sleeping servers to maintain low power usage and acceptable communication delays. Evaluation shows the system is able to reduce power consumption over time while satisfying internal processing delays.
Timing-pulse measurement and detector calibration for the OsteoQuant®.Binu Enchakalody
The document describes calibration procedures for an OsteoQuant pQCT scanner. It discusses:
1) Measuring motor and detector timing pulses using a USB counter to synchronize data collection with position. Measurements were accurate to within 0.13%.
2) Correcting for detector dead time using a polynomial model to linearize photon counts versus tube current data. Corrections were stable to within 0.5% error.
3) Correcting for beam hardening effects using polynomial and bimodal energy models to linearize projection values with absorber thickness. A secondary correction further improved stability of different-date corrections to below 1% error.
Solar panel monitoring solution using IoT-Faststream TechnologiesSudipta Maity
Faststream Technologies offers an automated IOT based solar panel monitoring/troubleshooting system that allows for automated solar panel monitoring from anywhere over the internet. As part of our solution, we make use of several IoT gateways suitable for different needs, based on SoCs like STM32, ESP32, ublox, CC3200, SiliconLabs, to monitor the solar panel parameters, in turn, providing Solar Plant Insights.
Our system constantly monitors the solar panel and transmits various parameters to the Cloud over the IoT system. Here we make use of the IoT platform to transmit solar power parameters to Amazon/ Azure cloud /IOT server via the gateway (over WiFi and Ethernet). A powerful web interface allows viewing of data in meaningful formats, enabling users to make decisions.
Monitoring of Transmission and Distribution Grids using PMUsLuigi Vanfretti
My presentation on "Monitoring of Transmission and Distribution Grids using PMUs" for the Workshop on Energy Business Opportunities in NY State.
The Center for Integrated Electrical Energy Systems (CIEES) at Stony Brook University and the Center for Future Energy Systems (CFES) at Rensselaer Polytechnic Institute will be holding a one day Workshop on Energy Business Opportunities in NY State.
The document provides release notes for Logger Pro 3.2, describing:
1) New features in version 3.2 including video analysis, strip charts, improved rate functions, and LabPro operating system updates.
2) Installation notes for compatibility with TI Connect and removing older versions.
3) Known issues with the software such as limitations of supported monitors and potential problems connecting sensors.
SENSOR SELECTION SCHEME IN TEMPERATURE WIRELESS SENSOR NETWORKijwmn
In this paper, we propose a novel energy efficient environment monitoring scheme for wireless sensor
networks, based on data mining formulation. The proposed adapting routing scheme for sensors for
achieving energy efficiency from temperature wireless sensor network data set. The experimental
validation of the proposed approach using publicly available Intel Berkeley lab Wireless Sensor Network
dataset shows that it is possible to achieve energy efficient environment monitoring for wireless sensor
networks, with a trade-off between accuracy and life time extension factor of sensors, using the proposed
approach.
IRJET- Land Mine Data Collection System using Long Range WiFi and P2P Com...IRJET Journal
This system collects data from sensors in landmines including time of blast, weight, temperature, climate factors like rainfall, and direction of movement. This data helps determine the cause of mine blasts and informs soldiers of conditions. Sensors transmit data via long range WiFi to a base station where the information is displayed. Collecting this critical data enhances military investigations and situational awareness for troops.
"Building and running the cloud GPU vacuum cleaner"Frank Wuerthwein
This talk, describing the "Largest Cloud Simulation in History" (Jensen Huang at SC19), was given at the MAGIC meeting on Dec. 4th 2019. MAGIC stands for "Middleware and Grid Interagency Cooperation", and is a group within NITRD. Current federal agencies that are members of MAGIC include DOC, DOD, DOE, HHS, NASA, and NSF.
NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...Igor Sfiligoi
NRP Engagement webinar: Description of the 380 PFLOP32S , 51k GPU multi-cloud burst using HTCondor to run IceCube photon propagation simulation.
Presented January 27th, 2020.
Burst data retrieval after 50k GPU Cloud runIgor Sfiligoi
We ran a 50k GPU multi-cloud simulation to support the IceCube science. This talk provided an overview of what happened to the associated data.
Presented at the Internet2 booth at SC19.
A Scalable and Distributed Electrical Power Monitoring System Utilizing Cloud...Ryousei Takano
This document describes a scalable and distributed electrical power monitoring system using cloud computing. Low-cost power measuring units collect data from current sensors and send it to data collecting units. These units then push the data to a data store hosted on Google App Engine. This cloud-based system allows visualization of power consumption across a large campus from any application accessing the data through a REST API. The system is scalable, low-cost, and easy to develop applications for power monitoring and planning energy savings.
This document provides an overview and agenda for a presentation on creating a "Hello World" program with Cisco's Data in Motion (DMo) software. The presentation introduces DMo and how it can manage and analyze data at the edge. It discusses how DMo represents a paradigm shift with edge intelligence and provides examples of railway and utilities use cases. The document explains DMo's programming model involving dynamic data definitions, patterns, conditions, and actions. It also demonstrates how to set up a DMo instance, create timer and event rules to read a light sensor and control an LED based on the sensor readings.
Application of machine learning and cognitive computing in intrusion detectio...Mahdi Hosseini Moghaddam
This document describes a proposed hardware-based machine learning intrusion detection system using cognitive processors. It discusses the need for new intrusion detection approaches due to limitations of signature-based methods. The proposed system collects network packet data using a Raspberry Pi and classifies it using a Cognimem CM1K cognitive processor chip, which implements restricted coulomb energy and k-nearest neighbor algorithms. The document outlines the system architecture, data collection and normalization methodology, and analysis of results from testing the CM1K chip on both custom and NSL-KDD network datasets, finding accuracy levels around 70-80% but slower processing times than a software simulation of the chip's algorithms. Future work areas include adding more packet features, using
Neural network modeling and control of data centers is presented. Data centers consume significant and increasing amounts of energy. A neural network model is developed and trained using steady state and transient data from a physical data center setup to map temperature outputs. The neural network accurately models temperatures with 95% accuracy. A neural network controller is then designed using the inverse model to stabilize temperatures according to reference values in response to varying workloads and power consumption. The controller successfully regulates temperatures in real-time simulation. Future work includes implementing the control on an actual system and expanding the control parameters.
The document provides an overview of SCADA and central control applications for power systems. It discusses:
1. SCADA architecture and components, including human-machine interfaces, application servers, communication servers, remote terminal units, and communication protocols.
2. Key SCADA functions like data acquisition, monitoring and event processing, control capabilities, and data storage.
3. Examples of centralized control applications for power system operation involving multiple actors like transmission system operators and generation companies.
ROLE OF DIGITAL SIMULATION IN CONFIGURING NETWORK PARAMETERSDeepak Shankar
Selecting the right Ethernet standard and configuring all the network devices in the embedded systems accurately is an extremely hard and rigorous job. The configuration depends on the topology, workloads of the connected devices, processing overhead at the switches, and the external interfaces. Network calculus, mathematical models and analytical techniques provide worst case execution time (WCET), but their probability of activity is extremely wide. This leads to overdesign which leads to higher costs, power consumption, weight, and size. Simulating the network is the best way to measure the throughput of the entire system. Digital system simulation provides better latency and throughput accuracy, but the accuracy is still limited because it does not consider the latency associated with the network OS, cybersecurity processing and scheduling. In many cases, these factors can reduce the throughput by 20-40%.
In this paper, we will present our research on modeling the entire Ethernet network, including the workloads, network flow control, scheduling, switch hardware, and software. To substantially increase the coverage and compare topologies, we have developed a set of benchmarks that provides coverage for different combination of deterministic, rate-constrained, and best effort traffic. During the presentation, we will cover the benchmarks, the list of attributes required to accurately model the traffic, nodes, switches, and the scheduler settings. We will also look at the statistics and reports required to make the configuration decision. In addition, we will discuss how the model must be constructed to study the impact of future requirements, failures, network intrusions, and security detection schemes.
Key Takeaways:
1. Learn how to efficiently use network simulation to design Ethernet systems
2. Develop a reusable benchmark and associated statistics to test different configurations
3. The role and impact of the CDT slots, guard band, send slope, idle slope, shuffle scheduling, flow control and virtual channels
Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...deawoo Kim
This document proposes Bird-MAC, a new energy-efficient MAC protocol for periodic structural health monitoring applications using wireless sensor networks. Bird-MAC optimizes the synchronization period between nodes to be equal to the data generation period, aligning these phases for minimum energy consumption. Unlike existing approaches, Bird-MAC does not require transmitters to wake up early. Instead, it uses a late-bird initiated scheme where the node that wakes up later initiates communication by transmitting a beacon signal. Simulation and experimental results show that Bird-MAC reduces energy consumption by up to 45% compared to existing partial synchronization protocols and achieves 100% delivery reliability.
How HPC and large-scale data analytics are transforming experimental scienceinside-BigData.com
In this deck from DataTech19, Debbie Bard from NERSC presents: Supercomputing and the scientist: How HPC and large-scale data analytics are transforming experimental science.
"Debbie Bard leads the Data Science Engagement Group NERSC. NERSC is the mission supercomputing center for the USA Department of Energy, and supports over 7000 scientists and 700 projects with supercomputing needs. A native of the UK, her career spans research in particle physics, cosmology and computing on both sides of the Atlantic. She obtained her PhD at Edinburgh University, and has worked at Imperial College London as well as the Stanford Linear Accelerator Center (SLAC) in the USA, before joining the Data Department at NERSC, where she focuses on data-intensive computing and research, including supercomputing for experimental science and machine learning at scale."
Watch the video: https://wp.me/p3RLHQ-kLV
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
This document provides an overview of a wheelchair system created by multiple student teams. It describes the motivation to create a research platform wheelchair that can map an indoor space. It outlines the hardware and software architecture developed so far, including odometry, motor control, power management, and Ethernet and CAN bus communication. Issues addressed this semester included odometry errors, improving motor control, and designing a new central computer. Future work includes implementing SLAM for navigation and adding feedback control and security to the system.
This document summarizes a study on improving energy efficiency in data centers through a cyber-physical approach combining hardware and software monitoring. The study developed an optimization framework that gathers data on environmental, server, and workload parameters in real-time to dynamically adapt and propose optimizations. It deployed a wireless sensor network in a supercomputer data center to monitor inlet/outlet temperatures and other environmental data at adjustable sampling rates, reducing the amount of collected data by up to 68% while still capturing useful information. The approach was tested in a real case study to holistically optimize energy use through integrated IT and cooling system management.
This document provides an overview of SCADA systems and their application in power system operation and control. It discusses:
- How SCADA systems enable centralized monitoring and control of dispersed power system assets under deregulated electricity markets.
- The typical components of a SCADA system including the human-machine interface, application servers, communication servers, remote terminal units, and communication infrastructure.
- The main functions of SCADA systems including data acquisition, event and alarm processing, control capabilities, data storage and analysis.
- Communication protocols commonly used in SCADA systems like DNP3, IEC 60870-5-101/104, and IEC 60870-6 for
Future Grid provides a data platform that can reduce data infrastructure costs by up to 90% and increase analytic performance by over 1000%. It eliminates complexity in data integration and management. The platform ingests data from sensors and devices, transforms and manages the data, and enables instant analysis upon data arrival. It has proven ability to manage billions of data points daily for utilities and provide real-time operational insights for tasks like voltage control and asset management. Customers can build real-time applications in weeks at a low cost using the platform's self-service capabilities.
The document describes Multilin's Intelligent Line Monitoring System, which provides an end-to-end solution for monitoring overhead utility networks using advanced analytics. The system uses sensors, communications equipment, and software applications to analyze data and provide utilities with actionable intelligence on faults, maintenance needs, and line capacity. It helps utilities improve reliability, efficiency, and asset utilization.
⭐⭐⭐⭐⭐ Learning-based Energy Consumption PredictionVictor Asanza
✅ Published in: https://doi.org/10.1016/j.procs.2022.07.035
As more people send information to the cloud-fog infrastructure, this brings many problems to the management of computer energy consumption. Therefore, energy consumption management of servers, fog devices and cloud computing platform should be investigated to comply with the Green IT requirement. In this paper, we propose an energy consumption prediction model consisting of several components such as hardware design, data pre-processing, characteristics extraction and selection. Our main goal is to develop a non-invasive meter based on a network of sensors that includes a microcontroller, the MQTT communication protocol and the energy measurement module. This meter measures voltage, current, power, frequency, energy and power factor while a dashboard is used to present the energy measurements in real-time. In particular, we perform measurements using a workstation that has similar characteristics to the servers of a Datacenter locate at the Information Technology Center in ESPOL,
which currently provide this type of services in Ecuador. For convenience, we evaluated different linear regression models to select the best one and to predict future energy consumption based on the several measurements from the workstation during several hours which enables the consumer to optimize and to reduce the maintenance costs of the IT equipment. The supervised machine learning algorithms presented in this work allow us to predict the energy consumption by hours and by days.
⭐ The matlab code used for data processing are available in: https://github.com/vasanza/Matlab_Code/tree/EnergyConsumptionPredictionDatacenter
⭐ The dataset used for data processing are available in:https://ieee-dataport.org/open-access/data-server-energy-consumption-dataset
✅ Read more related topics:
https://vasanza.blogspot.com/
A Low-cost and Scalable Visualization System for Electricity ConsumptionRyousei Takano
This document summarizes a low-cost and scalable system for visualizing electricity consumption. The system uses inexpensive current sensors connected to data collection units that transmit power usage data to a Google App Engine cloud server. This allows visualization apps to retrieve the data and monitor consumption across many sensors over time. The system was demonstrated at the SC11 conference with data from 82 sensors across Japan and the US being visualized.
"A session in the DevNet Zone at Cisco Live, Berlin. Analytics of network telemetry data (such as flow records, IPSLA measurements, and time series of MIB data) helps address many important operational problems. Traditional Big Data approaches run into limitations even as they push scale boundaries for processing data further. One reason for this is the fact that in many cases, the bottleneck for analytics is not analytics processing itself but the generation and export of the data on which analytics depends. Data does not come for free. The amount of data that can be reasonably collected from the network runs into inherent limitations due to bandwidth and processing constraints in the network itself. In addition, management tasks related to determining and configuring which data to generate lead to significant deployment challenges.
This presentation provides an overview of DNA (Distributed Network Analytics), a novel technology to analyze network telemetry data in distributed fashion at the network edge, allowing users to detect changes, predict trends, recognize anomalies, and identify hotspots in their network. Analytics processing occurs at the source of the data using an embedded DNA Agent App that dynamically configures data sources as needed and analyzes the data using an embedded analytics engine. This provides DNA with superior scaling characteristics while avoiding the significant operational and bandwidth overhead that is associated with centralized analytics solutions. An ODL-based SDN controller application orchestrates network analytics tasks across the network, providing a network analytics service that allows users to interact with the network as a whole instead of individual devices one at a time. DNA is enabled by the IOx App Hosting Framework and integrated with light-weight embedded analytics engines, CSA (Connected Service Analytics) and DMO (Data in Motion). "
1. Short Term Temperature and Hourly
Precipitation Prediction System
Nithyakumaran Gnanasekar
Under the Guidance of
Arthur Helmicki, Victor Hunt and Paul Talaga
3. Motivation
• During winter, ice accumulates onto the bridge stays,
which later drops down on the traffic when the
temperature increases.
• In Dec 19 2012 one such incident was occurred,
where a huge piece of ice to crashed on a vehicle’s
windshield
• Huge amount of multi dimensional data is collected
from the bridge and local weather station every few
minutes.
• UCII has designed and developed a warning system
providing timely warning and alert messages.
4. Background
• Prediction System can be classified into
• Univariate models
• Multivariate models
• The models can be further classified into
• Statistical method
• neural network based methods.
• Use of pre trained neural networks to provide online prediction is
exploited in this work.
• Once the network is trained the multivariate prediction can be
performed in real time.
5. Background
List of Inputs
Temperature
Humidity
Wind speed
Solar Radiation
Hourly Precipitation
Input
5*24= 120 Input Neurons
Output
8 Temperature neurons
Jain, Abhishek. Predicting air temperature for frost warning using artificial neural networks. Diss. uga, 2003
• Jain, Abhishek in his work used single layer neural
network to predict temperature for next eight
hours
• The Temperature Prediction was performed only
during early spring during sunrise.
• To warn the farmers about morning frost that
damage orchids
6. Suggested Algorithm
• Temperature Prediction System
• Hourly Precipitation Prediction
System
• Data Collection
• Data Segmentation
• Neural Network Design
• Train Network
• Decay and Weights
• Input/output variable selection
• Predict Data
• Algorithm
• Variable Selection
• Stations Selection
• Number of hours of input data
• Output representation
• Bagging
7. Algorithm- Temperature Prediction System
Data Collection Segment Data Train
Network
Predict SectionTrain Section
Predict Data Web Interface
8. Data Collection
Stations Type Time Interval
Local Weather Stations 5,10,12,15 min
Airports 60 – 30 min
On Bridge Sensors 10,15 min
Data Collection:
• Data is collected at irregular Time intervals
• Cubic Spline interpolation is used to fix
missing values
• The data is aggregated to be hourly
frequency.
Station 1
Station 2
Station 15
South Tower
Cubic Spline Imputation
Aggregate to Hourly Data
.
.
.
.
.
.
.
.
.DB
Stations, Airports, Sensors
9. Algorithm- Temperature Prediction System
Data Collection
Segment Data Train
Network
Predict Data Web Interface.
.
Training Section Cubic Spline Interpolation
Predict Section
10. Segment Data
Data
-7 to 0
0-15
15-28
28-34
Hourly Aggregated data Classifier
Normalized data with Min Max Information
Normalize
Normalize
Normalize
Normalize
Data
Data
Data
Data
• The Hourly Aggregated Data from Data
Collection section is used as input.
• The Data is first segmented into four buckets.
• This is accordance with neural network design.
• The Mean temperature for every 24 hours is
calculated and is used for classification
• Every variable is then normalized to read
between 1-0.
• The Min Max information required for de-
normalization is attached along with the data.
Temperature
Hourly
Precipitation Humidity
Wind
Speed
Solar
Radiation
Max 7.016667 3.704762 97 35.30526 643.75
Min -10.1696 0 35.1875 0 0
Min Max Information Stored along with Data
11. Expert System Design
-7 to 0
0-15
15-28
28-34
Classifier
Output
The Classifier sends the input based on
mean temperature to corresponding
network.
Every input is given to only one neural
network for training.
Only one output is computed at any
given time.
This system performs well and reduces
error due to large denormalization
* All four Blocks are classical back propagation network with one hidden layer
12. Algorithm- Temperature Prediction System
Data Collection
Segment Data Train
Network
Predict Data Web Interface
Training Section Cubic Spline Interpolation Segment Data
Predict Section Normalized Data with Min Max info
Segment Data
.
.
13. Train Network
• The train network can be tuned for
parameter
• Decay
• Number of weights
• The network splits the training set into two set
• 90% data for Training
• 10% for evaluating.
• The 10% of evaluation sample is a
random set picked from training set.
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
-0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
RMSE(C)
Decay Parameter
Decay Parameter
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
0 5 10 15 20 25 30 35
RMSE(C)
Number of Neurons
Number of Neurons
14. Train Network
• The decay and weights can be
provided as sequence.
• The network would iterate through
the combination
• Provide a report on such a data set. S.no neuron decay rmse sse maxerror meanerror
1 1 1.00E-01 2.290552 23588.85 8.499794 0.30368227
2 1 1.00E-05 2.572076 29743.62 10.92812 -0.09750547
3 1 1.00E-02 2.472744 27490.63 10.154952 -0.07979373
4 2 1.00E-01 2.283635 23446.58 8.616908 0.49384269
5 2 1.00E-05 3.73943 62869.07 14.439244 -0.97010542
6 2 1.00E-02 2.284121 23456.56 8.953333 -0.08007818
7 3 1.00E-01 2.157424 20926.54 7.961871 0.39193959
8 3 1.00E-05 6.009436 162365.5 20.995079 2.84870665
9 3 1.00E-02 2.322157 24244.28 11.334819 0.87290343
10 4 1.00E-01 2.143207 20651.63 7.892792 0.36539226
11 4 1.00E-05 3.81584 65464.6 12.887219 -0.53080024
12 4 1.00E-02 2.210153 21961.95 10.485526 0.57920437
13 5 1.00E-01 2.079384 19439.97 7.621765 0.32835461
14 5 1.00E-05 3.700756 61575.39 12.738093 0.48186848
15 5 1.00E-02 2.173325 21236.15 10.482595 0.32212886
16 6 1.00E-01 2.047381 18846.18 7.54331 0.28953601
17 6 1.00E-05 5.057563 115002.9 17.145608 0.18968488
18 6 1.00E-02 2.102648 19877.39 7.163767 -0.17827613
19 7 1.00E-01 2.064283 19158.65 7.525051 0.26114947
20 7 1.00E-05 4.861489 106258.8 12.917473 -1.98795079
$net
a 144-10-8 network with 1538 weights options were - decay=0.1
$minMax
row Temperature HourlyPrecipitation Humidity WindSpeed SolarRadiation diff
1 Max 7.016667 3.704762 97.0000 35.30526 643.75 2.413333
2 Min -10.169565 0.000000 35.1875 0.00000 0.00 -2.520000
$bestNet
"Best selected configuration has hidden neuron 10 and decay 0.1"
$rmse 2.290552
Network
Description
Min Max
Information Chosen
Network
Configuration
Detailed
Report for all
combination
15. Neural Network Training Section Cubic Spline Interpolation Normalized Data with Min Max info
Online Prediction and Web Section Trained Neural Network Segment Data
Algorithm- Temperature Prediction System
Data Collection
Train
Network
Predict Data Web Interface
Segment Data Train Network
Segment Data.
.
16. Output/Input Selection
• In the previous works listed in the
background
• All the variables collected were fed to
neural network as input.
• No were performed on the correlation
between the variables.
• Correlation analysis between the variables
led to using better set of input and in
improving prediction result.
19. Output/Input Selection
List of Inputs
Temperature
Hourly Precipitation
Solar Radiation
Pressure
Wind Speed
Humidity
Temperature Difference
Input
7*24= 168 Input Neurons
Output
8 Temperature Difference neurons
AfterCorrelationAnalysisandsimulation
http://10.39.8.247/predict/TemperatureDiff.php
0
5
10
15
20
25
30
Numberofneurons
Neuron
0.39
0.4
0.41
0.42
0.43
0.44
0.45
0.46
0.47
(C)
Root Mean Square Error for Variable Combination
20. Neural Network Training Section Cubic Spline Interpolation Normalized Data with Min Max info
Online Prediction and Web Section Trained Neural Network Segment Data
Algorithm- Temperature Prediction System
Data Collection
Predict Data Web Interface
Segment Data Train Network
Train
Network
Segment Data.
.
21. Predict Data
[1] [2]Data Classifier
[1] Data Collection Section [2] Data Segmentation Section Trained Neural Network
data w/o minMax Normalized Data with MinMax
Data
• Last 24 hours of data is picked. It is passed through data
collection section
• This data is then sent to classifier to identify the
appropriate Neural network
• This provides the minMax Information for normalization
• The data is then passed thorough segmentation section
• This is then provided to network to predict newer values
*Note Every Hour has 8 predicted values
22. Neural Network Training Section Cubic Spline Interpolation Normalize Data Multiple Sections
Online Prediction and Web Section Trained Neural Network Segment Data
Algorithm- Temperature Prediction System
Data Collection
Web Interface
Segment Data Train Network
Train
Network
Segment Data.
.
23. Web Interface
• Configuration
• PHP 5.4
• Zend Framework
• D3.js for plotting
• Uses MYSQL Database
• Complaint with UCII site.
24. Algorithm- Temperature Prediction System
Data Collection Segment Data Train Network Predict Data Web Interface
Train
Network
Segment Data.
.
Neural Network Training Section Cubic Spline Interpolation Normalize Data Multiple Sections
Online Prediction and Web Section Trained Neural Network Segment Data
25. Suggested Algorithm
• Temperature Prediction System
• Hourly Precipitation Prediction
System
• Data Collection
• Data Segmentation
• Neural Network Design
• Train Network
• Decay and Weights
• Input/output variable selection
• Predict Data
• Algorithm
• Variable Selection
• Stations Selection
• Number of hours of input data
• Output representation
• Bagging
26. Algorithm- Hourly Prediction System
Data Collection Train Network Predict Data Web Interface
Train
Network
Normalize Data.
.
Neural Network Training Section Cubic Spline Interpolation Multiple Sections
Online Prediction and Web Section Trained Neural Network Aggregate Data
27. Variable Selection-Correlation
• Output
• 4 hours of Probability data for
Hourly Precipitation
http://10.39.8.247/predict/correlation.php#div-1
All Variables
Temperature
Humidity
Dew point
Pressure
Wind Speed
Wind Speed Gust
Wind Direction
Wind Direction Degree
Daily Rain
Hourly Precipitation
Selected Variables
Hourly Precipitation
Humidity
Wind Direction
Daily Rain
Solar Radiation
Temperature
28. Cross Correlation Between Variables
HP Vs Humidity HP Vs Pressure HP Vs Solar Radiation
HP Vs Temperature HP Vs Wind Direction Degree HP Vs Wind Speed
*HP- Hourly Precipitation
29. Variable Selection-Cross Correlation
HP Vs Humidity HP Vs Pressure HP Vs Solar Radiation
HP Vs Temperature HP Vs Wind Direction Degree HP Vs Wind Speed
Selected Variables
Hourly Precipitation
Humidity
Wind Direction
Daily Rain
Solar Radiation
Temperature
All Variables
Temperature
Humidity
Dew point
Pressure
Wind Speed
Wind Speed Gust
Wind Direction
Wind Direction Degree
Daily Rain
Hourly Precipitation
Selected Variables
Hourly Precipitation
Humidity
Daily Rain
Wind Direction
Correlation
Cross Correlation
*Daily Rain is Picked for further analysis as it similar to Hourly Precipitation
37. Number of Hours of Input Data
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0 5 10 15 20 25 30
MAE
Number of hours
Number of Input hours of data required
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
0 5 10 15 20 25 30 35
MAE
Number of Neurons
Number of Neurons
• Number of Hours input data- 24
• Number of hidden Neurons=13
38. Input/Output Representation
• Input is 24 hour Normalized data from three stations and 3 variables from
the stations.
• Output is 4 variable. Each variable indicating probability of rain for every
hour ahead.
Date Time Hour1 Hour2 Hour3 Hour4
… 0 0 0 0
…. 0 1 0 0
… 1 0 0 0
…. 0 0 0 1
• Sample Actual Output Data Used for Training
• Rainfall>1(mm) ? Output=1 : Output =0
2^4= 16 different Combinations of possible Output
Distribution of Actual Output Classes
2011-2013
39. Bagging
• Bagging is way of
boosting the system
performance by
altering the class
distribution in the
training set.
• This ensures
prediction is not
always skewed
towards the largest
occurring class
45. Error Plot
Predicted
Error Computed for Random Sample data 2011-2013
*This data was not used for Training
Value 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Output Class 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111
Predicted
actual actual
counts percentage
47. Future Work
• Extending Hourly Precipitation algorithm to other stations and bridge
sensors
• Study of using predicted data as an input to another prediction to
creating a feed back network.
• Using radar data to predict hourly precipitation to increase accuracy.
56. Hourly Precipitation Vs Wind Direction
Back time lagged correlation
0
50
100
150
200
250
300
350
Degree
Direction Vs Degress
2013- 6 Hours Prior to Rainfall- Wind Direction Distribution
57. Temperature Prediction - Evening Drop
-3
-2
-1
0
1
2
3
4
5
6
12:00:00PM
2:24:00PM
4:48:00PM
7:12:00PM
9:36:00PM
12:00:00AM
2:24:00AM
4:48:00AM
7:12:00AM
9:36:00AM
(C)
Temperature Prediction 1-4-14
Temperature Prediction 1 Prediction2 Prediction3 Prediction 4 Prediction 5 Prediction 6 Prediction 7 Prediction 8
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
12:00:00 PM 2:24:00 PM 4:48:00 PM 7:12:00 PM 9:36:00 PM 12:00:00 AM 2:24:00 AM 4:48:00 AM 7:12:00 AM 9:36:00 AM
(C)
Mean Error