ORADIEX : A Big Data driven smart framework for real-time surveillance and analysis of individual exposure to radioactive pollution
•
0 likes•83 views
These slides were presented at the 2nd Big Data and Cyber Security Intelligence conference held in 2019 in France, Versailles.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to ORADIEX : A Big Data driven smart framework for real-time surveillance and analysis of individual exposure to radioactive pollution
Similar to ORADIEX : A Big Data driven smart framework for real-time surveillance and analysis of individual exposure to radioactive pollution (20)
More from Hadi Fadlallah
More from Hadi Fadlallah (20)
Recently uploaded
Recently uploaded (20)
ORADIEX : A Big Data driven smart framework for real-time surveillance and analysis of individual exposure to radioactive pollution
- 1. ORADIEX A Big Data driven smart framework for real-time surveillance and analysis of individual exposure to radioactive pollution Hadi Fadlallah, Yehia Taher, Rafiqul Haque, Ali Jaber
- 2. Plan • Introduction • Objective • Previous Work • Proposed system • Implementation • Experiments • Conclusion • Limitations • Future work 2/28
- 4. Rise of Internet of Things 4/28 Introduction
- 6. Objective • Scalable solution for engineering radiation data • Processing big data (huge volume, high speed) • Real-time monitoring 6/28 Objective
- 8. RaDEn Limitations Weak alert system Poor visualization Only raw data is stored Complex data retrieval 8
- 9. Proposed System • ORADIEX: Enhanced Radiation Data Engineering system • Scalability and fault-tolerance • Handles Big Data • Monitor radiation data in real-time and batch style • Send Email alert on radiation exposure • Allows historical data analysis 9/28
- 10. Proposed System 10/28 Proposed system Data Sources Processed Data Storage Data Ingestion Raw Data Storage Data Processing Data Visualization
- 12. Raw Data Storage 12/28 Proposed system Data nodes Ingested data Data retrieval
- 13. Data Processing 13/28 Proposed system Spark workers Processed data Ingested data
- 16. InfluxDB NOSQL database Time series database NOSQL JSON format 16/28 Implementation
- 17. Alarm System 17/28 Implementation Email notification configuration Alert configuration
- 18. Experiments • Dataset provided by the Lebanese Atomic Energy Commission • Confidentiality issues in accessing sensors, web server • Data: Beirut, from 2015-08-01 to 2016-08-01 • Radiation level, temperature, rain level, sensor battery power, data collection time and external battery power 18/28 Experiments
- 19. Experiments • Start required services • Sensor simulation, folder listener • Import to HDFS • Execute python script • Visualize data using Grafana 19/28 Experiments
- 24. Conclusion •Implemented radiation data engineering system •Improved version of our previous work RaDEn •Ensure scalability and fault-tolerance •Radiation monitoring (Real-time) •Data retrieval 24/28 Conclusion
- 25. Limitations • No sensors or web server access • Lack of documentation • Time limit 25/28 Limitations
- 26. Future work • Distributed search engines • Data enrichment/aggregation 26/28 Future work
- 27. Any Question?
Editor's Notes
- First, I will start my presentation with a brief introduction then I will illustrate this project objective and the most relevant previous work. Then, I will present our proposed system and how we implemented it and I will show the experiments we made. Finally, I will conclude and discuss our work.
- Radiation pollution is a critical concern due to high damage that it may cause to humans and environment. To minimize damages, controlling and monitoring is very important.
- In the past century, it was hard to have centralized radiation monitoring system due to the limitations of traditional networks. With the rise of internet of things, radiation measurement unit was integrated in wireless sensors, and used to transmit data via communication networks.
- As result, new challenges appeared. When sensors collect data in real-time it may result a massive amount of data, transferred in a high speed having a wide variety of formats. The traditional data technologies cannot handles any more this type of data. Also existing solutions are conventional and mostly handles data in batch style.
- In this experimental research, our objective is to build a scalable radiation data engineering platform that has: the ability to process and monitors huge amount of radiation data with high speed having different formats in real-time.
- Previously, we have proposed a radiation engineering system called RaDEn that relies on Big Data technologies that guarantee collecting hug amount of data in real-time, storing data in a scalable data lake, drawing real-time graph and raising alerts.
- But, this proposed system still has many limitations since it has a weak alert system which only show message boxes. A poor visualization layer since it uses a very basic tool. Data is stored only in raw format and data retrieval process is not user friendly and requires advanced programming level.
- In this research, we have proposed new system called ORADIEX which can be considered as an improved version of RadEn. ORADIEX allows sending email notifications when a radiation exposure occurs. It has a powerful visualization layer that allows to build monitoring dashboards. It stores raw and processed radiation data and allows users to perform data retrieval using a user friendly interface.
- The system architecture is composed of 6 layers: The data sources which consists of radiation sensors installed in different places, Flat files and Archive relational databases. The data ingestion layer, which is responsible of collecting data, sending it to the other layers The data processing layer, which is responsible of cleaning data and removing unwanted data. Then, it send it to the processed data storage layer The processed data storage layer is responsible of storing clean data in a scalable warehouse to be consumed by visualization layer The visualization layer is responsible of reading newly added data to the storage layer, drawing real-time graph, monitoring radiation level and sending email notifications when exposure occurs. The last layer, is the raw data storage layer which consists of a data lake that can be used in data retrieval or to reprocess data if an error was occurred in data processing. Next, we will describe briefly the data flow in ORADIEX
- First, the data ingestion layer. To read data with different formats from sensors and flat files we have used Apache Kafka, which is a distributed, scalable and fault-tolerant technology We have create two Kafka topics: one for real-time processing and one for batch style. Data are sent from the data sources to Kafka producers then are stored in kafka pipelines until they are consumed. At the same time, data is sent to the data storage layer via Apache flume agent (one for each kafka topic).
- The data storage layer has 2 components: The data repository: which consists of Hadoop distributed file system, which allow parallel computing and guarantee high scalability and fault-tolerance: the data comes from the ingestion layer to the Hadoop master node and then it is replicated over the slave nodes in a text file format. The metadata: which relies mainly on Apache Hive. it allows creating Tables on the top of HDFS directories, and let the user able to retrieve data from the repository using SQL-Like languages (Spark-SQL, HiveQL)
- The Data processing layer relies mainly on Apache Spark , which is a scalable, fault-tolerant, distributed data processing technology. The Apache spark master receive the data from the data ingestion layer and send the data to the spark workers to be cleaned then storing within a scalable data warehouse build using a NoSQL database called InfluxDB.
- When new data is stored within the scalable warehouse it is visualized in real-time by a service based application called Grafana which also monitor radiation level and send notification when an exposure occurs.
- TO implement this system, we have configured four (linux-based) virtual machines, one machine acts as master node, it contains Main installations such as Hadoop, Apache Kafka, Flume, Hive, Sqoop, Spark, InfluxDB and Grafana installations. Other machine act as Hadoop data nodes.
- Concerning the scalable warehouse, we have used InfluxDB which is a timeseries NoSQL database, where data is stored in JSON format as shown in the following image.
- As shown in this image, we can configure the alert system by defining the radiation level limit and setting up the email notification using Grafana user interface. Also, the alert value will be shown on the visualized graph
- We run the experiments with a dataset proceed by the LAEC. For confidentiality purposes data was given as flat files instead of giving us access to the sensors or the web server. The data is collected from one sensor located in Beirut 1 august two thousand fifty till 1 august two thousands sixty The dataset contains information such as ….
- First, we have to run the required services (Hadoop cluster, spark, kafka, flume agent and python script) To simulate reading data from sensor we have created a directory and a listener on the top of it: when any file is added to the folder, it will start sending it line by line to the kafka broker. We developed a python script to run an Apache Spark job to read data from kafka broker and send it to InfluxDb instance. Finally data is visualized using Grafana.
- This figure shows how data is stored and replicated within Hadoop cluster.
- The following figure shows a screehnshot of a realtime graph showing the radiation level, the rain level and the temperature values.
- This figure shows a data retrieval operation done using Grafana where we retrieved the mean radiation level in Beirut during the past hour and we visualized the result in a graph.
- The following figure shows the alert list where the result of a periodic radiation level check is saved.
- As a conclusion, we can say that we have designed and implemented a radiation data engineering system that: Is an improved version of our previous work RaDEn Ensure scalability and fault-tolerance Guarantee radiation monitoring Guarantee data retrieval operations on raw and processed data.
- This research has some limitations due to the following reasons: We didn’t get access to the sensors or web server Lack of big data technologies documentation The time limit constraint
- In the future, there are many improvements that can be made: We can use distributed search engines such as Solr and ElasticSearch We can enrich data by integrating it with online weather data and other measurement that may affect radiation level.