RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data
•
0 likes•132 views
These slides were presented at the 1st Big Data and Cyber Security Intelligence conference held in 2018 in Beirut, Lebanon.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data
Similar to RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data (20)
More from Hadi Fadlallah
More from Hadi Fadlallah (20)
Recently uploaded
Recently uploaded (20)
RaDEn : A Scalable and Efficient Platform for Engineering Radiation Data
- 1. RaDEn: A Scalable and Efficient Platform for Engineering Radiation Data Hadi Fadlallah, Yehia Taher, Ali Jaber
- 2. Plan • Introduction • Objective • Proposed system • Implementation • Experiments • Conclusion • Limitations • Future work 2/25
- 4. Rise of Internet of Things 4/25 Introduction 3 … 5
- 5. New Challenges 5/25 Introduction 3 … 5 Huge Volume High Speed Wide variety Traditional Solutions
- 6. Objective • Scalable solution for engineering radiation data • Processing big data (huge volume, high speed) • Real-time monitoring 6/25 Objective
- 7. Proposed system • RaDEn: Radiation Data Engineering system • Scalability and fault-tolerance • Handles big data • Monitor radiation data in real-time and batch style 7/25 Proposed system 7 … 12
- 8. Proposed system 8/25 Proposed system 7 … 12
- 9. Data ingestion 9/25 Proposed system 7 … 12
- 10. Data storage 10/25 Proposed system 7 … 12
- 11. Data processing 11/25 Proposed system 7 … 12
- 12. Data visualization • Acts with data processing engine • Real-time graph • Matplotlib python library 12/25 Proposed system 7 … 12
- 14. Alarm System 14/25 Implementation 13 … 14
- 15. Experiments • Dataset provided by the Lebanese Atomic Energy Commission • Confidentiality issues in accessing sensors, web server • Data: Beirut, from 2015-08-01 to 2016-08-01 • Radiation level, temperature, rain level, sensor battery power, data collection time and external battery power 15/25 Experiments 15 … 20
- 16. Experiments • Start required services • Sensor simulation, folder listener • Import to HDFS • Processing and visualization 16/25 Experiments 15 … 20
- 18. Experiments • Alert is raised in from of message boxes • Level -> title • Description -> body 18/25 Experiments 15 … 20
- 19. Experiments • Created Hive external table (table: radiation) • Ignore move messy data rows (view: vw_radiation) • Spark-SQL , HiveQL queries SELECT * FROM vw_radiation WHERE dose_rate > 50; 19/25 Experiments 15 … 20
- 21. Conclusion •Implemented radiation data engineering system •Relies on Apache Hadoop, Kafka, Sqoop, Flume and Spark •Ensure scalability and fault-tolerance •Radiation monitoring •Data retrieval 21/25 Conclusion
- 22. Limitations • Small data set • No sensors or web server access • Lack of documentation • Time limit 22/25 Limitations
- 23. Future work • Improve visualization (bokeh, Kibana) • Friendly user interface • Use ORC (Optimized Row Columnar) format • Distributed search engines 23/25 Future work
- 24. Thank you
Editor's Notes
- Radiation pollution is a critical concern due to high damage that it may cause to humans and environment. To minimize damages, controlling and monitoring is very important.
- In the past century, it was hard to have centralized radiation monitoring system due to the limitations of traditional networks. With the rise of internet of things, radiation measurement unit was integrated in wireless sensors, and used to transmit data via communication networks.
- As result, new challenges appeared: 1- when sensors collect data in real-time it may result a massive amount of data, which is transferred in a high speed. 2- the utilization of different types of sensors implies that we have different data formats. The traditional data technologies cannot handles any more this type of data. Also existing solutions are conventional and mostly handles data in batch style.
- In this experimental research, our objective is to build a scalable radiation data engineering platform that has: the ability to process and monitors huge amount of radiation data with high speed having different formats in real-time.
- Our proposed system is called RaDEn an abbreviation of radiation data engineering system It guarantees high scalability and fault-tolerance, handles big data And has the ability to monitor data in real-time and batch-style
- The system architecture is composed of 6 layers: The data sources which consists of radiation sensors installed in different places, Flat files and Archive relational databases The data ingestion layer, which is responsible of collecting data and send it to the data processing engine and data storage layer The data storage layer which allows storing huge volume of data, and allow end-user to search among the stored data The data processing engine it allows processing radiation data in real-time and raise alerts when high radiation level is detected The visualization layer, it allows showing real-time graphs The coordination layer: it guarantee the communication between the different technologies used in different layers. This task is done by Apache zookeeper which is required by data technologies. Next, we will describe the technologies that we have used in each layer
- First, the data ingestion layer. To read data with different formats from sensors and flat files we have used Apache Kafka, which is a distributed, scalable and fault-tolerant technology We have create two Kafka topics: one fro real-time processing and one for batch style. Data are sent from the data sources to Kafka producers then are sent distributed into kafka pipelines in parallel then until they are consumed. Data are sent to the data storage layer via Apache flume agent (one for each kafka topic) and at the same time it is sent to the processing engine. Also the system is able to import archival data from relational databases using apache sqoop import where we only have to specify the connection string of the relational database and the location into the hdfs
- The data storage layer has 2 components: The data repository: which consists of Hadoop distributed file system, which allow parallel computing and guarantee high scalability and fault-tolerance: the data comes from the ingestion layer to the Hadoop master node and then it is replicated over the slave nodes in a text file format. The metadata: which relies mainly on Apache Hive. it allows creating Tables on the top of HDFS directories, and let the user able to retrieve data from the repository using SQL-Like languages (Spark-SQL, HiveQL)
- The Data processing layer relies mainly on Apache Spark , which is a scalable, fault-tolerant, distributed data processing technology. The Apache spark master receive the data from the data ingestion layer and send the data to the spark workers to be processed then visualized in the data visualization layer. Beside of Spark, we have used pandas python library which contains many function to manipulate data.
- The data visualization layer relies mainly on a python library called Matplotlib, it a very simple library that allows user the draw real-time graphs.
- TO implement this system, we have configured three (linux-based) virtual machines, one machine acts as hadoop master node, and it contains apache kafka, flume, hive, sqoop and spark installations. Other machine act as Hadoop data nodes. We have used only one Kafka node and one Spark node due to the small dataset that we have received, but we can add more nodes when required
- We have written a python script that implement the following alarm system (based on the LAEC requirements) The alarm system work as the following: ….
- We run the experiments with a dataset proceed by the LAEC. For confidentiality purposes we they give us the data in form of flat files instead of giving access to the sensors or the web server. The data is collected from one sensor located in Beirut 1 august two thousand fifty till 1 august two thousands sixty The dataset contains information such as ….
- First, we have to run the required services (Hadoop cluster, spark, kafka, flume agent and python script) To simulate reading data from sensor we have created a directory and a listener on the top of it: when any file is added to the folder, it will start sending it line by line to the kafka broker. Each row is processed and visualized using the python script.
- The following figure shows some sequential screenshots of the real-time graph, we can se the evolution of the radiation level in function of date and time
- When there is an alert, it is raised in form of a message box like shown in the figure, the alarm level is written in the title and the description in the body
- On the top of the HDFS directory we have created a Hive external table, and we created a view that read from this table to ignore messy data rows and convert data types. Then we can retrieve data using SQL-Like languages such as spark-SQL and HiveQL.
- The figure show a screenshots of the results of the previous query.
- As a conclusion, we can say that we have designed and implemented a radiation data engineering system that: - can handles massive amount of data in real-time and at rest. - relies on scalable, fault-tolerant and distributed technologies such as Hadoop. - Allow users to retrieve stored data using SQL-Like languages Also, we have implemented an alarm system to monitor the radiation data and raise alert when high radiation level is detected.
- This research has some limitations due to the following reasons: It is not evaluated using big data due to the small dataset that we have received We didn’t get access to the sensors or web server Lack of big data technologies documentation The time limit constraint
- In the future, there are many improvements that can be made: Improving the visualization layer, using more powerful tools such as bokeh python library and Kibana which is a part of elastic search framework Design and implement user friendly interfaces Creating a data warehousing job that run every day and convert the newely stored files into ORC format which guarantee higher performance We can use distributed search engines such as Solr and ElasticSearch