This document reviews big data analytics (BDA) architecture trends and analysis. It discusses the evolution of data analytics from ancient times to modern technologies like Hadoop and Spark. It describes key features of BDA like flexibility, scalability, and fault tolerance. Common BDA architectures like lambda and kappa architectures are summarized. The lambda architecture uses batch, speed, and serving layers to handle both real-time and batch processing. The kappa architecture simplifies this by removing the batch layer and handling all processing through streaming. Overall, the document provides a high-level overview of BDA architectures and technologies.
IMMERSIVE TECHNOLOGIES IN 5G-ENABLED APPLICATIONS: SOME TECHNICAL CHALLENGES ...ijcsit
5G next-generation networking paradigm with its envisioned capacity, coverage, and data transfer rates
provide a developmental field for novel applications scenarios. Virtual, Mixed, and Augmented Reality will
play a key role as visualization, interaction, and information delivery platforms. The recent hardware and
software developments in immersive technologies including AR, VR and MR in terms of the commercial
availability of advanced headsets equipped with XR-accelerated processing units and Software
Development Kits (SDKs) are significantly increasing the penetration of such devices for entertainment,
corporate and industrial use. This trend creates next-generation usage models which rise serious technical
challenges within all networking and software architecture levels to support the immersive digital
transformation. The focus of this paper is to detect, discuss and propose system development approaches
and architectures for successful integration of the immersive technologies in the future information and
communication concepts like Tactile Internet and Internet of Skills.
Named Data Networking (NDN) is a recently designed Internet architecture that benefits data names
instead of locations and creates essential changes in the abstraction of network services from "delivering
packets to specific destinations” to "retrieving data with special names" makes. This fundamental change
creates new opportunities and intellectual challenges in all areas, especially network routing and
communication, communication security, and privacy. The focus of this dissertation is on the forwarding
aircraft feature introduced by NDN. Communication in NDN is done by exchanging interest and data
packets
IMMERSIVE TECHNOLOGIES IN 5G-ENABLED APPLICATIONS: SOME TECHNICAL CHALLENGES ...ijcsit
5G next-generation networking paradigm with its envisioned capacity, coverage, and data transfer rates
provide a developmental field for novel applications scenarios. Virtual, Mixed, and Augmented Reality will
play a key role as visualization, interaction, and information delivery platforms. The recent hardware and
software developments in immersive technologies including AR, VR and MR in terms of the commercial
availability of advanced headsets equipped with XR-accelerated processing units and Software
Development Kits (SDKs) are significantly increasing the penetration of such devices for entertainment,
corporate and industrial use. This trend creates next-generation usage models which rise serious technical
challenges within all networking and software architecture levels to support the immersive digital
transformation. The focus of this paper is to detect, discuss and propose system development approaches
and architectures for successful integration of the immersive technologies in the future information and
communication concepts like Tactile Internet and Internet of Skills.
Named Data Networking (NDN) is a recently designed Internet architecture that benefits data names
instead of locations and creates essential changes in the abstraction of network services from "delivering
packets to specific destinations” to "retrieving data with special names" makes. This fundamental change
creates new opportunities and intellectual challenges in all areas, especially network routing and
communication, communication security, and privacy. The focus of this dissertation is on the forwarding
aircraft feature introduced by NDN. Communication in NDN is done by exchanging interest and data
packets
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Architectural design of IoT-cloud computing integration platformTELKOMNIKA JOURNAL
An integration between the Internet of Things (IoT) and cloud computing can potentially leverage the
utilization of both sides. As the IoT based system is mostly composed by the interconnection of pervasive
and constrained devices, it can take a benefit of virtually unlimited resources of cloud entity i.e storage
and computation services to store and process its sensed data. On the other hand, the cloud computing
system may get benefit from IoT by broadening its reach to real world environment applications. In order
to incarnate this idea, a cloud software platform is needed to provide an integration layer between the IoT
and cloud computing taking into account the heterogenity of network communication protocols as well as the
security and data management issues. In this study, an architectural design of IoT-cloud platform for IoT and
cloud computing integration is presented. The proposed software platform can be decomposed into five main
components namely cloud-to-device interface, authentication, data management, and cloud-to-user interface
component. In general, the cloud-to-device interface acts as a data transmission endpoint between the whole
cloud platform system and its IoT devices counterpart. Before a session of data transmission established,
the communication interface contact the authentication component to make sure that the corresponding IoT
device is legitimate before it allowed for sending the sensor data to cloud environment. Notice that a valid IoT
device can be registered to the cloud system through web console component. The received sensor data
are then collected in data storage component. Any stored data can be further analyzed by data processing
component. User or any developed applications can then retrieve collected data, either raw or processed
data, through API data access and web console.
FAST PACKETS DELIVERY TECHNIQUES FOR URGENT PACKETS IN EMERGENCY APPLICATIONS...IJCNCJournal
Internet of Things (IoT) has been receiving a lot of interest around the world in academia, industry and telecommunication organizations. In IoT, many constrained devices can communicate with each other which generate a huge number of transferred packets. These packets have different priorities based on the applications which are supported by IoT technology. Emergency applications such as calling an ambulance in a car accident scenario need fast and reliable packets delivery in order to receive an immediate response from a service provider. When a client sends his request with specific requirements, fast and reliable return contents (packets) should be fulfilled, otherwise, the network resources may be wasted and undesirable circumstances may be counted. Content-Centric Networking (CCN) has become a promising network paradigm that satisfies the requirements of fast packets delivery for emergency applications of IoT. In this paper, we propose fast packets delivery techniques based on CCN for IoT environment, these techniques are suitable for urgent packets in emergency applications that need fast delivery. The simulation results show how the proposed techniques can achieve high throughput, a large number of request messages, fast response time and a low number of lost packets in comparison with the normal CCN.
A Comparative Study: Taxonomy of High Performance Computing (HPC) IJECEIAES
The computer technologies have rapidly developed in both software and hardware field. The complexity of software is increasing as per the market demand because the manual systems are going to become automation as well as the cost of hardware is decreasing. High Performance Computing (HPC) is very demanding technology and an attractive area of computing due to huge data processing in many applications of computing. The paper focus upon different applications of HPC and the types of HPC such as Cluster Computing, Grid Computing and Cloud Computing. It also studies, different classifications and applications of above types of HPC. All these types of HPC are demanding area of computer science. This paper also done comparative study of grid, cloud and cluster computing based on benefits, drawbacks, key areas of research, characterstics, issues and challenges.
final Year Projects, Final Year Projects in Chennai, Software Projects, Embedded Projects, Microcontrollers Projects, DSP Projects, VLSI Projects, Matlab Projects, Java Projects, .NET Projects, IEEE Projects, IEEE 2009 Projects, IEEE 2009 Projects, Software, IEEE 2009 Projects, Embedded, Software IEEE 2009 Projects, Embedded IEEE 2009 Projects, Final Year Project Titles, Final Year Project Reports, Final Year Project Review, Robotics Projects, Mechanical Projects, Electrical Projects, Power Electronics Projects, Power System Projects, Model Projects, Java Projects, J2EE Projects, Engineering Projects, Student Projects, Engineering College Projects, MCA Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, Wireless Networks Projects, Network Security Projects, Networking Projects, final year projects, ieee projects, student projects, college projects, ieee projects in chennai, java projects, software ieee projects, embedded ieee projects, "ieee2009projects", "final year projects", "ieee projects", "Engineering Projects", "Final Year Projects in Chennai", "Final year Projects at Chennai", Java Projects, ASP.NET Projects, VB.NET Projects, C# Projects, Visual C++ Projects, Matlab Projects, NS2 Projects, C Projects, Microcontroller Projects, ATMEL Projects, PIC Projects, ARM Projects, DSP Projects, VLSI Projects, FPGA Projects, CPLD Projects, Power Electronics Projects, Electrical Projects, Robotics Projects, Solor Projects, MEMS Projects, J2EE Projects, J2ME Projects, AJAX Projects, Structs Projects, EJB Projects, Real Time Projects, Live Projects, Student Projects, Engineering Projects, MCA Projects, MBA Projects, College Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, M.Sc Projects, Final Year Java Projects, Final Year ASP.NET Projects, Final Year VB.NET Projects, Final Year C# Projects, Final Year Visual C++ Projects, Final Year Matlab Projects, Final Year NS2 Projects, Final Year C Projects, Final Year Microcontroller Projects, Final Year ATMEL Projects, Final Year PIC Projects, Final Year ARM Projects, Final Year DSP Projects, Final Year VLSI Projects, Final Year FPGA Projects, Final Year CPLD Projects, Final Year Power Electronics Projects, Final Year Electrical Projects, Final Year Robotics Projects, Final Year Solor Projects, Final Year MEMS Projects, Final Year J2EE Projects, Final Year J2ME Projects, Final Year AJAX Projects, Final Year Structs Projects, Final Year EJB Projects, Final Year Real Time Projects, Final Year Live Projects, Final Year Student Projects, Final Year Engineering Projects, Final Year MCA Projects, Final Year MBA Projects, Final Year College Projects, Final Year BE Projects, Final Year BTech Projects, Final Year ME Projects, Final Year MTech Projects, Final Year M.Sc Projects, IEEE Java Projects, ASP.NET Projects, VB.NET Projects, C# Projects, Visual C++ Projects, Matlab Projects, NS2 Projects, C Projects, Microcontroller Projects, ATMEL Projects, PIC Projects, ARM Projects, DSP Projects, VLSI Projects, FPGA Projects, CPLD Projects, Power Electronics Projects, Electrical Projects, Robotics Projects, Solor Projects, MEMS Projects, J2EE Projects, J2ME Projects, AJAX Projects, Structs Projects, EJB Projects, Real Time Projects, Live Projects, Student Projects, Engineering Projects, MCA Projects, MBA Projects, College Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, M.Sc Projects, IEEE 2009 Java Projects, IEEE 2009 ASP.NET Projects, IEEE 2009 VB.NET Projects, IEEE 2009 C# Projects, IEEE 2009 Visual C++ Projects, IEEE 2009 Matlab Projects, IEEE 2009 NS2 Projects, IEEE 2009 C Projects, IEEE 2009 Microcontroller Projects, IEEE 2009 ATMEL Projects, IEEE 2009 PIC Projects, IEEE 2009 ARM Projects, IEEE 2009 DSP Projects, IEEE 2009 VLSI Projects, IEEE 2009 FPGA Projects, IEEE 2009 CPLD Projects, IEEE 2009 Power Electronics Projects, IEEE 2009 Electrical Projects, IEEE 2009 Robotics Projects, IEEE 2009 Solor Projects, IEEE 2009 MEMS Projects, IEEE 2009 J2EE P
Performance Analysis of Internet of Things Protocols Based Fog/Cloud over Hig...Istabraq M. Al-Joboury
The Internet of Things (IoT) becomes the future of a global data field in which the embedded devices communicate with each other, exchange data and making decisions through the Internet. IoT could improves the qualityoflife in smart cities, but a massive amount of data from different smart devices could slow down or crash database systems. In addition, IoT data transfer to Cloud for monitoring information and generating feedback thus will lead to highdelay in infrastructure level. Fog Computing can help by offering services closer to edge devices. In this paper, we propose an efficient system architecture to mitigate the problem of delay. We provide performance analysis like responsetime, throughput and packet loss for MQTT (Message Queue Telemetry Transport) and HTTP (Hyper Text Transfer Protocol) protocols based on Cloud or Fog serverswith large volume of data form emulated traffic generator working alongsidewith one real sensor. We implement both protocols in the same architecture, with low cost embedded devices to local and Cloud servers with different platforms. The results show that HTTP response time is 12.1 and 4.76 times higher than MQTT Fog and cloud based located in the same geographical area of the sensors respectively. The worst case in performance is observed when the Cloud is public and outside the country region. The results obtained for throughput shows that MQTT has the capability to carry the data with available bandwidth and lowest percentage of packet loss. We also prove that the proposed Fog architecture is an efficient way to reduce latency and enhance performance in Cloud based IoT.
Open Source Platforms Integration for the Development of an Architecture of C...Eswar Publications
The goal of the Internet of Things (IoT) is to achieve the interconnection and interaction of all kind of everyday
objects. IoT architecture can be implemented in various ways. This paper presents a way to mount an IoT architecture using open source hardware and software platforms and shows that this is a viable option to collect information through various sensors and present it through a web page.
Cooperative hierarchical based edge-computing approach for resources allocati...IJECEIAES
Using mobile and Internet of Things (IoT) applications is becoming very popular and obtained researchers’ interest and commercial investment, in order to fulfill future vision and the requirements for smart cities. These applications have common demands such as fast response, distributed nature, and awareness of service location. However, these requirements’ nature cannot be satisfied by central systems services that reside in the clouds. Therefore, edge computing paradigm has emerged to satisfy such demands, by providing an extension for cloud resources at the network edge, and consequently, they become closer to end-user devices. In this paper, exploiting edge resources is studied; therefore, a cooperative-hierarchical approach for executing the pre-partitioned applications’ modules between edges resources is proposed, in order to reduce traffic between the network core and the cloud, where this proposed approach has a polynomial-time complexity. Furthermore, edge computing increases the efficiency of providing services, and improves end-user experience. To validate our proposed cooperative-hierarchical approach for modules placement between edge nodes’ resources, iFogSim toolkit is used. The obtained simulation results show that the proposed approach reduces network’s load and the total delay compared to a baseline approach for modules’ placement, moreover, it increases the network’s overall throughput.
CIRCUIT BREAK CONNECT MONITORING TO 5G MOBILE APPLICATIONijcsit
Along by a continuous improvement to composite electronic devices, a safety to technicians takes
additionally become the matter to good concern, as a result to technicians' lives is in jeopardy while their
work through shutting down circuit breakers, even that even once the breaker takes been switched off,
someone will inadvertently flip to while a technician remains working. That should be a system to
guarantee safety that technicians. Also, individuals do not love switching all the time toward turn on / off
appliances like fans/lighting/air conditioners. It ends in wasted energy thanks to unnecessarily placing the
instrument. To address these issues, we tend to come up through the system through mobile app-controlled
circuit breakers that degrade wireless management to home appliances to hunt down a golem app. That
replaces a traditional breaker through the mobile app-controlled system in the on / off system, where no
one will activate the breaker, while not the word. The remote of home appliances helps a user to save
electricity. That enhances a quality of life and luxury. Additionally, a system includes the home security
mechanism against drone intrusion using the mobile app-controlled door lock system besides the
mechanism that sleuthing dangerous gas leaks. A formation of the system subtracts the degree of victim
associate ESP 32 microcontroller, the Bluetooth module, matrix 4x4 keyboards, and the paraffin gas
detector associate with a golem mobile application. The entire system is usually compact systems
Prediction Based Efficient Resource Provisioning and Its Impact on QoS Parame...IJECEIAES
The purpose of this paper is to provision the on demand resources to the end users as per their need using prediction method in cloud computing environment. The provisioning of virtualized resources to cloud consumers according to their need is a crucial step in the deployment of applications on the cloud. However, the dynamical management of resources for variable workloads remains a challenging problem for cloud providers. This problem can be solved by using a prediction based adaptive resource provisioning mechanism, which can estimate the upcoming resource demands of applications. The present research introduces a prediction based resource provisioning model for the allocation of resources in advance. The proposed approach facilitates the release of unused resources in the pool with quality of service (QoS), which is defined based on prediction model to perform the allocation of resources in advance. In this work, the model is used to determine the future workload prediction for user requests on web servers, and its impact toward achieving efficient resource provisioning in terms of resource exploitation and QoS. The main contribution of this paper is to develop the prediction model for efficient and dynamic resource provisioning to meet the requirements of end users.
An Event-based Middleware for Syntactical Interoperability in Internet of Th...IJECEIAES
Internet of Things (IoT) connecting sensors or devices that record physical observations of the environment and a variety of applications or other Internet services. Along with the increasing number and diversity of devices connected, there arises a problem called interoperability. One type of interoperability is syntactical interoperability, where the IoT should be able to connect all devices through various data protocols. Based on this problem, we proposed a middleware that capable of supporting interoperability by providing a multi-protocol gateway between COAP, MQTT, and WebSocket. This middleware is developed using event-based architecture by implementing publish-subscribe pattern. We also developed a system to test the performance of middleware in terms of success rate and delay delivery of data. The system consists of temperature and humidity sensors using COAP and MQTT as a publisher and web application using WebSocket as a subscriber. The results for data transmission, either from sensors or MQTT COAP has a success rate above 90%, the average delay delivery of data from sensors COAP and MQTT below 1 second, for packet loss rate varied between 0% - 25%. The interoperability testing has been done using Interoperability assessment methodology and found out that ours is qualified.
Moving Toward Big Data: Challenges, Trends and PerspectivesIJRESJOURNAL
Abstract: Big data refers to the organizational data asset that exceeds the volume, velocity, and variety of data typically stored using traditional structured database technologies. This type of data has become the important resource from which organizations can get valuable insightand make business decision by applying predictive analysis. This paper provides a comprehensive view of current status of big data development,starting from the definition and the description of Hadoop and MapReduce – the framework that standardizes the use of cluster of commodity machines to analyze big data. For the organizations that are ready to embrace big data technology, significant adjustments on infrastructure andthe roles played byIT professionals and BI practitioners must be anticipated which is discussed in the challenges of big data section. The landscape of big data development change rapidly which is directly related to the trend of big data. Clearly, a major part of the trend is the result ofthe attempt to deal with the challenges discussed earlier. Lastly the paper includes the most recent job prospective related to big data. The description of several job titles that comprise the workforce in the area of big data are also included.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Architectural design of IoT-cloud computing integration platformTELKOMNIKA JOURNAL
An integration between the Internet of Things (IoT) and cloud computing can potentially leverage the
utilization of both sides. As the IoT based system is mostly composed by the interconnection of pervasive
and constrained devices, it can take a benefit of virtually unlimited resources of cloud entity i.e storage
and computation services to store and process its sensed data. On the other hand, the cloud computing
system may get benefit from IoT by broadening its reach to real world environment applications. In order
to incarnate this idea, a cloud software platform is needed to provide an integration layer between the IoT
and cloud computing taking into account the heterogenity of network communication protocols as well as the
security and data management issues. In this study, an architectural design of IoT-cloud platform for IoT and
cloud computing integration is presented. The proposed software platform can be decomposed into five main
components namely cloud-to-device interface, authentication, data management, and cloud-to-user interface
component. In general, the cloud-to-device interface acts as a data transmission endpoint between the whole
cloud platform system and its IoT devices counterpart. Before a session of data transmission established,
the communication interface contact the authentication component to make sure that the corresponding IoT
device is legitimate before it allowed for sending the sensor data to cloud environment. Notice that a valid IoT
device can be registered to the cloud system through web console component. The received sensor data
are then collected in data storage component. Any stored data can be further analyzed by data processing
component. User or any developed applications can then retrieve collected data, either raw or processed
data, through API data access and web console.
FAST PACKETS DELIVERY TECHNIQUES FOR URGENT PACKETS IN EMERGENCY APPLICATIONS...IJCNCJournal
Internet of Things (IoT) has been receiving a lot of interest around the world in academia, industry and telecommunication organizations. In IoT, many constrained devices can communicate with each other which generate a huge number of transferred packets. These packets have different priorities based on the applications which are supported by IoT technology. Emergency applications such as calling an ambulance in a car accident scenario need fast and reliable packets delivery in order to receive an immediate response from a service provider. When a client sends his request with specific requirements, fast and reliable return contents (packets) should be fulfilled, otherwise, the network resources may be wasted and undesirable circumstances may be counted. Content-Centric Networking (CCN) has become a promising network paradigm that satisfies the requirements of fast packets delivery for emergency applications of IoT. In this paper, we propose fast packets delivery techniques based on CCN for IoT environment, these techniques are suitable for urgent packets in emergency applications that need fast delivery. The simulation results show how the proposed techniques can achieve high throughput, a large number of request messages, fast response time and a low number of lost packets in comparison with the normal CCN.
A Comparative Study: Taxonomy of High Performance Computing (HPC) IJECEIAES
The computer technologies have rapidly developed in both software and hardware field. The complexity of software is increasing as per the market demand because the manual systems are going to become automation as well as the cost of hardware is decreasing. High Performance Computing (HPC) is very demanding technology and an attractive area of computing due to huge data processing in many applications of computing. The paper focus upon different applications of HPC and the types of HPC such as Cluster Computing, Grid Computing and Cloud Computing. It also studies, different classifications and applications of above types of HPC. All these types of HPC are demanding area of computer science. This paper also done comparative study of grid, cloud and cluster computing based on benefits, drawbacks, key areas of research, characterstics, issues and challenges.
final Year Projects, Final Year Projects in Chennai, Software Projects, Embedded Projects, Microcontrollers Projects, DSP Projects, VLSI Projects, Matlab Projects, Java Projects, .NET Projects, IEEE Projects, IEEE 2009 Projects, IEEE 2009 Projects, Software, IEEE 2009 Projects, Embedded, Software IEEE 2009 Projects, Embedded IEEE 2009 Projects, Final Year Project Titles, Final Year Project Reports, Final Year Project Review, Robotics Projects, Mechanical Projects, Electrical Projects, Power Electronics Projects, Power System Projects, Model Projects, Java Projects, J2EE Projects, Engineering Projects, Student Projects, Engineering College Projects, MCA Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, Wireless Networks Projects, Network Security Projects, Networking Projects, final year projects, ieee projects, student projects, college projects, ieee projects in chennai, java projects, software ieee projects, embedded ieee projects, "ieee2009projects", "final year projects", "ieee projects", "Engineering Projects", "Final Year Projects in Chennai", "Final year Projects at Chennai", Java Projects, ASP.NET Projects, VB.NET Projects, C# Projects, Visual C++ Projects, Matlab Projects, NS2 Projects, C Projects, Microcontroller Projects, ATMEL Projects, PIC Projects, ARM Projects, DSP Projects, VLSI Projects, FPGA Projects, CPLD Projects, Power Electronics Projects, Electrical Projects, Robotics Projects, Solor Projects, MEMS Projects, J2EE Projects, J2ME Projects, AJAX Projects, Structs Projects, EJB Projects, Real Time Projects, Live Projects, Student Projects, Engineering Projects, MCA Projects, MBA Projects, College Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, M.Sc Projects, Final Year Java Projects, Final Year ASP.NET Projects, Final Year VB.NET Projects, Final Year C# Projects, Final Year Visual C++ Projects, Final Year Matlab Projects, Final Year NS2 Projects, Final Year C Projects, Final Year Microcontroller Projects, Final Year ATMEL Projects, Final Year PIC Projects, Final Year ARM Projects, Final Year DSP Projects, Final Year VLSI Projects, Final Year FPGA Projects, Final Year CPLD Projects, Final Year Power Electronics Projects, Final Year Electrical Projects, Final Year Robotics Projects, Final Year Solor Projects, Final Year MEMS Projects, Final Year J2EE Projects, Final Year J2ME Projects, Final Year AJAX Projects, Final Year Structs Projects, Final Year EJB Projects, Final Year Real Time Projects, Final Year Live Projects, Final Year Student Projects, Final Year Engineering Projects, Final Year MCA Projects, Final Year MBA Projects, Final Year College Projects, Final Year BE Projects, Final Year BTech Projects, Final Year ME Projects, Final Year MTech Projects, Final Year M.Sc Projects, IEEE Java Projects, ASP.NET Projects, VB.NET Projects, C# Projects, Visual C++ Projects, Matlab Projects, NS2 Projects, C Projects, Microcontroller Projects, ATMEL Projects, PIC Projects, ARM Projects, DSP Projects, VLSI Projects, FPGA Projects, CPLD Projects, Power Electronics Projects, Electrical Projects, Robotics Projects, Solor Projects, MEMS Projects, J2EE Projects, J2ME Projects, AJAX Projects, Structs Projects, EJB Projects, Real Time Projects, Live Projects, Student Projects, Engineering Projects, MCA Projects, MBA Projects, College Projects, BE Projects, BTech Projects, ME Projects, MTech Projects, M.Sc Projects, IEEE 2009 Java Projects, IEEE 2009 ASP.NET Projects, IEEE 2009 VB.NET Projects, IEEE 2009 C# Projects, IEEE 2009 Visual C++ Projects, IEEE 2009 Matlab Projects, IEEE 2009 NS2 Projects, IEEE 2009 C Projects, IEEE 2009 Microcontroller Projects, IEEE 2009 ATMEL Projects, IEEE 2009 PIC Projects, IEEE 2009 ARM Projects, IEEE 2009 DSP Projects, IEEE 2009 VLSI Projects, IEEE 2009 FPGA Projects, IEEE 2009 CPLD Projects, IEEE 2009 Power Electronics Projects, IEEE 2009 Electrical Projects, IEEE 2009 Robotics Projects, IEEE 2009 Solor Projects, IEEE 2009 MEMS Projects, IEEE 2009 J2EE P
Performance Analysis of Internet of Things Protocols Based Fog/Cloud over Hig...Istabraq M. Al-Joboury
The Internet of Things (IoT) becomes the future of a global data field in which the embedded devices communicate with each other, exchange data and making decisions through the Internet. IoT could improves the qualityoflife in smart cities, but a massive amount of data from different smart devices could slow down or crash database systems. In addition, IoT data transfer to Cloud for monitoring information and generating feedback thus will lead to highdelay in infrastructure level. Fog Computing can help by offering services closer to edge devices. In this paper, we propose an efficient system architecture to mitigate the problem of delay. We provide performance analysis like responsetime, throughput and packet loss for MQTT (Message Queue Telemetry Transport) and HTTP (Hyper Text Transfer Protocol) protocols based on Cloud or Fog serverswith large volume of data form emulated traffic generator working alongsidewith one real sensor. We implement both protocols in the same architecture, with low cost embedded devices to local and Cloud servers with different platforms. The results show that HTTP response time is 12.1 and 4.76 times higher than MQTT Fog and cloud based located in the same geographical area of the sensors respectively. The worst case in performance is observed when the Cloud is public and outside the country region. The results obtained for throughput shows that MQTT has the capability to carry the data with available bandwidth and lowest percentage of packet loss. We also prove that the proposed Fog architecture is an efficient way to reduce latency and enhance performance in Cloud based IoT.
Open Source Platforms Integration for the Development of an Architecture of C...Eswar Publications
The goal of the Internet of Things (IoT) is to achieve the interconnection and interaction of all kind of everyday
objects. IoT architecture can be implemented in various ways. This paper presents a way to mount an IoT architecture using open source hardware and software platforms and shows that this is a viable option to collect information through various sensors and present it through a web page.
Cooperative hierarchical based edge-computing approach for resources allocati...IJECEIAES
Using mobile and Internet of Things (IoT) applications is becoming very popular and obtained researchers’ interest and commercial investment, in order to fulfill future vision and the requirements for smart cities. These applications have common demands such as fast response, distributed nature, and awareness of service location. However, these requirements’ nature cannot be satisfied by central systems services that reside in the clouds. Therefore, edge computing paradigm has emerged to satisfy such demands, by providing an extension for cloud resources at the network edge, and consequently, they become closer to end-user devices. In this paper, exploiting edge resources is studied; therefore, a cooperative-hierarchical approach for executing the pre-partitioned applications’ modules between edges resources is proposed, in order to reduce traffic between the network core and the cloud, where this proposed approach has a polynomial-time complexity. Furthermore, edge computing increases the efficiency of providing services, and improves end-user experience. To validate our proposed cooperative-hierarchical approach for modules placement between edge nodes’ resources, iFogSim toolkit is used. The obtained simulation results show that the proposed approach reduces network’s load and the total delay compared to a baseline approach for modules’ placement, moreover, it increases the network’s overall throughput.
CIRCUIT BREAK CONNECT MONITORING TO 5G MOBILE APPLICATIONijcsit
Along by a continuous improvement to composite electronic devices, a safety to technicians takes
additionally become the matter to good concern, as a result to technicians' lives is in jeopardy while their
work through shutting down circuit breakers, even that even once the breaker takes been switched off,
someone will inadvertently flip to while a technician remains working. That should be a system to
guarantee safety that technicians. Also, individuals do not love switching all the time toward turn on / off
appliances like fans/lighting/air conditioners. It ends in wasted energy thanks to unnecessarily placing the
instrument. To address these issues, we tend to come up through the system through mobile app-controlled
circuit breakers that degrade wireless management to home appliances to hunt down a golem app. That
replaces a traditional breaker through the mobile app-controlled system in the on / off system, where no
one will activate the breaker, while not the word. The remote of home appliances helps a user to save
electricity. That enhances a quality of life and luxury. Additionally, a system includes the home security
mechanism against drone intrusion using the mobile app-controlled door lock system besides the
mechanism that sleuthing dangerous gas leaks. A formation of the system subtracts the degree of victim
associate ESP 32 microcontroller, the Bluetooth module, matrix 4x4 keyboards, and the paraffin gas
detector associate with a golem mobile application. The entire system is usually compact systems
Prediction Based Efficient Resource Provisioning and Its Impact on QoS Parame...IJECEIAES
The purpose of this paper is to provision the on demand resources to the end users as per their need using prediction method in cloud computing environment. The provisioning of virtualized resources to cloud consumers according to their need is a crucial step in the deployment of applications on the cloud. However, the dynamical management of resources for variable workloads remains a challenging problem for cloud providers. This problem can be solved by using a prediction based adaptive resource provisioning mechanism, which can estimate the upcoming resource demands of applications. The present research introduces a prediction based resource provisioning model for the allocation of resources in advance. The proposed approach facilitates the release of unused resources in the pool with quality of service (QoS), which is defined based on prediction model to perform the allocation of resources in advance. In this work, the model is used to determine the future workload prediction for user requests on web servers, and its impact toward achieving efficient resource provisioning in terms of resource exploitation and QoS. The main contribution of this paper is to develop the prediction model for efficient and dynamic resource provisioning to meet the requirements of end users.
An Event-based Middleware for Syntactical Interoperability in Internet of Th...IJECEIAES
Internet of Things (IoT) connecting sensors or devices that record physical observations of the environment and a variety of applications or other Internet services. Along with the increasing number and diversity of devices connected, there arises a problem called interoperability. One type of interoperability is syntactical interoperability, where the IoT should be able to connect all devices through various data protocols. Based on this problem, we proposed a middleware that capable of supporting interoperability by providing a multi-protocol gateway between COAP, MQTT, and WebSocket. This middleware is developed using event-based architecture by implementing publish-subscribe pattern. We also developed a system to test the performance of middleware in terms of success rate and delay delivery of data. The system consists of temperature and humidity sensors using COAP and MQTT as a publisher and web application using WebSocket as a subscriber. The results for data transmission, either from sensors or MQTT COAP has a success rate above 90%, the average delay delivery of data from sensors COAP and MQTT below 1 second, for packet loss rate varied between 0% - 25%. The interoperability testing has been done using Interoperability assessment methodology and found out that ours is qualified.
Moving Toward Big Data: Challenges, Trends and PerspectivesIJRESJOURNAL
Abstract: Big data refers to the organizational data asset that exceeds the volume, velocity, and variety of data typically stored using traditional structured database technologies. This type of data has become the important resource from which organizations can get valuable insightand make business decision by applying predictive analysis. This paper provides a comprehensive view of current status of big data development,starting from the definition and the description of Hadoop and MapReduce – the framework that standardizes the use of cluster of commodity machines to analyze big data. For the organizations that are ready to embrace big data technology, significant adjustments on infrastructure andthe roles played byIT professionals and BI practitioners must be anticipated which is discussed in the challenges of big data section. The landscape of big data development change rapidly which is directly related to the trend of big data. Clearly, a major part of the trend is the result ofthe attempt to deal with the challenges discussed earlier. Lastly the paper includes the most recent job prospective related to big data. The description of several job titles that comprise the workforce in the area of big data are also included.
International Journal of Engineering Research and Development (IJERD)IJERD Editor
journal publishing, how to publish research paper, Call For research paper, international journal, publishing a paper, IJERD, journal of science and technology, how to get a research paper published, publishing a paper, publishing of journal, publishing of research paper, reserach and review articles, IJERD Journal, How to publish your research paper, publish research paper, open access engineering journal, Engineering journal, Mathemetics journal, Physics journal, Chemistry journal, Computer Engineering, Computer Science journal, how to submit your paper, peer reviw journal, indexed journal, reserach and review articles, engineering journal, www.ijerd.com, research journals,
yahoo journals, bing journals, International Journal of Engineering Research and Development, google journals, hard copy of journal
An Comprehensive Study of Big Data Environment and its Challenges.ijceronline
Big Data is a data analysis methodology enabled by recent advances in technologies and Architecture. Big data is a massive volume of both structured and unstructured data, which is so large that it's difficult to process with traditional database and software techniques. This paper provides insight to Big data and discusses its nature, definition that include such features as Volume, Velocity, and Variety .This paper also provides insight to source of big data generation, tools available for processing large volume of variety of data, applications of big data and challenges involved in handling big data
Abstract: Knowledge has played a significant role on human activities since his development. Data mining is the process of
knowledge discovery where knowledge is gained by analyzing the data store in very large repositories, which are analyzed
from various perspectives and the result is summarized it into useful information. Due to the importance of extracting
knowledge/information from the large data repositories, data mining has become a very important and guaranteed branch of
engineering affecting human life in various spheres directly or indirectly. The purpose of this paper is to survey many of the
future trends in the field of data mining, with a focus on those which are thought to have the most promise and applicability
to future data mining applications.
Keywords: Current and Future of Data Mining, Data Mining, Data Mining Trends, Data mining Applications.
This article useful for anyone who want to introduce with Big Data and how oracle architecture Big Data solution using Oracle Big Data Cloud solutions .
Big data Mining Using Very-Large-Scale Data Processing PlatformsIJERA Editor
Big Data consists of large-volume, complex, growing data sets with multiple, heterogenous sources. With the
tremendous development of networking, data storage, and the data collection capacity, Big Data are now rapidly
expanding in all science and engineering domains, including physical, biological and biomedical sciences. The
MapReduce programming mode which has parallel processing ability to analyze the large-scale network.
MapReduce is a programming model that allows easy development of scalable parallel applications to process
big data on large clusters of commodity machines. Google’s MapReduce or its open-source equivalent Hadoop
is a powerful tool for building such applications.
BIG DATA SECURITY AND PRIVACY ISSUES IN THE CLOUD IJNSA Journal
Many organizations demand efficient solutions to store and analyze huge amount of information. Cloud computing as an enabler provides scalable resources and significant economic benefits in the form of reduced operational costs. This paradigm raises a broad range of security and privacy issues that must be taken into consideration. Multi-tenancy, loss of control, and trust are key challenges in cloud computing environments. This paper reviews the existing technologies and a wide array of both earlier and state-ofthe-art projects on cloud security and privacy. We categorize the existing research according to the cloud reference architecture orchestration, resource control, physical resource, and cloud service management layers, in addition to reviewing the recent developments for enhancing the Apache Hadoop security as one of the most deployed big data infrastructures. We also outline the frontier research on privacy-preserving data-intensive applications in cloud computing such as privacy threat modeling and privacy enhancing solutions.
Big data security and privacy issues in theIJNSA Journal
Many organizations demand efficient solutions to store and analyze huge amount of information. Cloud computing as an enabler provides scalable resources and significant economic benefits in the form of reduced operational costs. This paradigm raises a broad range of security and privacy issues that must be taken into consideration. Multi-tenancy, loss of control, and trust are key challenges in cloud computing environments. This paper reviews the existing technologies and a wide array of both earlier and state-ofthe-art projects on cloud security and privacy. We categorize the existing research according to the cloud reference architecture orchestration, resource control, physical resource, and cloud service management layers, in addition to reviewing the recent developments for enhancing the Apache Hadoop security as one of the most deployed big data infrastructures. We also outline the frontier research on privacy-preserving data-intensive applications in cloud computing such as privacy threat modeling and privacy enhancing solutions.
Big Data Summarization : Framework, Challenges and Possible Solutionsaciijournal
In this paper, we first briefly review the concept of big data, including its definition, features, and value. We then present background technology for big data summarization brings to us. The objective of this paper is to discuss the big data summarization framework, challenges and possible solutions as well as methods of evaluation for big data summarization. Finally, we conclude the paper with a discussion of open problems and future directions..
Similar to Review of big data analytics (bda) architecture trends and analysis (20)
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
2. k,((( 34
Authorized licensed use limited to: University of Waterloo. Downloaded on May 30,2020 at 04:01:40 UTC from IEEE Xplore. Restrictions apply.
3. data say they are doing it for the consumer’s benefit. But data
have a way of being used for purposes other originally
intended” [7].
Early of 1990s, the birth of the interconnected web of data
and accessible to anyone from anywhere, known as Internet.
The digital storage become more cost effective than manual
printing documents. Michael [8] describes that including the
sounds and images there are thousands of petabytes
information, the existence of 12,000 petabytes is not an
unreasonable guest. The web is increasing in size of 10-fold
each year, however, data will never be discovered values and
yield no insight. During the mid 1990s, the internet is
extremely popular, but structure relational databases cannot
cope with the variety of data types from different non-
relational databases. Thus, NoSQL system is created to
handle different languages and formats in a great flexible way.
Larry Page and Sergey Brin implement Google’s search
engine that can respond in a few seconds by returning desired
results, which processing and analyzing Big Data in
distributed method [9]. Richard comments that the purpose of
computing is insight, and not just numbers. In 1999, Kevin
introduces the term of “Internet of Things” to describe the
growing number of devices online to automated the
communication each other without a human interference;
Also, it utilizes the Internet to empower computers to sense
the world for themselves [10].
In the advent of Industry Revolution 4.0, which
developing in Germany 2013; it has been rapidly spread in
Europe and the world as a while. BDA is one of the key
adoptions and pillar for IoT initiative to improve decision
making [11]. It requires to process a large amount of data on
the fly and storing the data in various scalable storage
technologies. This lighting fast analytics implementation
allows the industries to gain rapid insights, provide prediction
for machinery, and share information. Intrinsically, it requires
a unified architecture to cater common operation for enabling
innovative applications.
B. Big Data General ‘Vs’ Concept
For understanding the Big Data concept, it always
considers the simple building block of data model which is
effectively communicating each and others. In 2001, Gartner
analyst, Doug introduces the 3Vs concept in the dimension of
data management, it consists of controlling data volume,
variety and velocity [12]. It characterizes the creation of data,
storage, retrieval and analysis. After a decade, IBM has been
coined two more worthy of Vs, which are Veracity and Value.
The following shows the brief description of 5Vs:
Volume: It implies to the enormous quantity of data is
generated.
Velocity: It refers to the speed at the data is created and
processed at staggering rate.
Variety: It defines as type of content of data analysis.
Veracity: It focuses on the quality and trust-worthiness
of the variability in the captured data.
Value: It raises to the significance of the data, which
delivering the insights and creating useful
model that answers sophisticated queries.
Inspired by the comprehensive discussion and relevant
comments on IBM website of Big Data Analytics hub, it
clusters the 5Vs into three groups [13]:
Volume
Velocity:
These translate into requirements of
hardware and software to deal with data.
Large scale distributed data processing
framework is required such as Hadoop.
Veracity
Velocity:
These translate into urgency of real-time
processing. The detection of possible data
corruption or manipulation is crucial with
high speed processing ability.
Value: This translates into the necessity of
interdisciplinary cooperation. This raise
the most difficult challenge for industrial
use of big data.
C. “Data at Rest” vs “Data in Motion”
There is no small task in gaining the insights of big data.
Firstly, “Data at Rest” refers the collected historical data from
various sources. It performs the analytics after the event
occurs. Thus, it is commonly used to discover behaviors and
patterns from the past records. Also, it refers to “batch
processing” method. To automate these tasks, there is a
scheduler application in place for executing the tasks
automatically. Secondly, “Data in Motion” refers to
processing and analyzing data in real-time as the event
happens. The latency is a key consideration, as a lag of
processing can be resulted the loss of opportunities.
Furthermore, hybrid of “Data at Rest” and “Data in Motion”
are common in the industries.
III. BIG COMPUTE FEATURES
For data intensive computing [14], the system should
encapsulate the sophisticated design technologies in storing,
managing and processing big data. There are two focus of key
areas, which are application and frameworks. These consists
the concept of data parallelism and task/application
parallelism. Data parallelism is distributed among servers,
and therefore can be processed in parallel. It has been claimed
that it opposes to task parallelism, furthermore, it is often the
simpler method to craft a parallel application [15].
The followings describe the generic features for Big
Compute:
• Being efficient in pre-processing raw data and
combining relevant data from multiple sources,
commonly known as ETL (Extract, Transform and
Load)
• Being flexible to apply various aggregation functions
and perform ad-hoc queries to compute large amount
of sources in discovering the high-level insights of
data
• Being cost effective to extends functionalities with
minimum costs and minimize maintenance cost for
keeping the system running smoothly
• Being low latency in harnessing real-time data for
analytics by optimizing the high volume operation
with minimal delay
• Being highly scalable to enlarge the growth of the
compute resources and storages with support easily
plug-in
• Being robustness and fault tolerance to have ability to
cope with erroneous input and without down any
failures
,(((RQIHUHQFHRQ2SHQ6VWHPV,26
4. 35
Authorized licensed use limited to: University of Waterloo. Downloaded on May 30,2020 at 04:01:40 UTC from IEEE Xplore. Restrictions apply.
5. • Being systematically governance to ensure data
availability, usability, integrity and security in used
Identifying the required features for a specific domain can
be difficult. In general, different application domains might
need different type of system. It is hard to meet all stockholder
needs with a singular design. As such, Cigdem [16] attempts
to use feature modelling technique [17]. It performs drill
down by distinguish domain scoping, which determining the
domain interest, the stockholders and their goals; and domain
modelling, which aiming to derive using a commodity
analysis. Figure 1 shows the feature model diagram. This
work provides insight in the overall feature space of BDA
system. It further assists for deriving the BDA architecture.
Figure 1: Feature Model
IV. REVIEW OF BIG DATA ARCHITECTURE FRAMEWORK
A reference architecture helps to build a blueprint of the
ultimate BDA system. It is based on a collection of
characteristics and features from common for a given set of
problems. The design of the architecture has to emerge the
fluent orchestration workflow to execute either in a
synchronous or asynchronous manner between the application
and its data. In many cases, it includes the support for the
hybrid mode of batch and real-time processing. The following
reviews of architecture frameworks broaden the perspective
and enabling problem solving with the right tools.
A. Lambda ‘λ’ Architecture
In 2011, one of the popular reference BDA architecture
design has been posted by Marz [18]. It is named as “Lambda
λ Architecture”. It is designed to combine of batch and real-
time processing paradigm in a parallel form. This method is
capable to solve many BDA use cases. In addition, it has the
robustness with fault tolerant strategy for serving wide range
of workloads. Technically, it is now feasible to run ad-hoc
queries against Big Data, but querying a petabyte dataset
every time you want to compute. Figure 2 shows the λ
architecture with three major layers.
Figure 2: λ Architecture
The batch layer pre-computes the master dataset, and
processes into batch views so that queries can be resolved with
low latency. This requires striking a balance of job between
pre-computation and execution time to complete the query.
By doing a little bit of computation on the fly to complete
queries, there save the process from needing to pre-compute
large batch views. In addition, it is not expected to update the
views frequently. The batch views may be a set of flat files
and it depends on chosen technologies. The key is to
precompute just enough information so that the query can be
completed quickly.
The serving layer indexes the views and provides
interfaces, thus, the pre-computed data can be speedily
queried. Both of the batch and speed layers are executed the
same processing logic, and then reconciles the results in
serving layer. It designates to be distributed among many
servers for scalability. There is a long-standing problem
where data is too normalized, there is a need to store some
information redundantly to improve response times. However,
denormalized the data may create huge complexity of keeping
it consistent. Thus, it need to be carefully construct this view
[19].
The speed layer is similar to batch layers. The objective is
to construct views that can be efficiently queried. It mainly
uses an incremental approach and handling real-time views.
These views are updated directly when new data arrives. It
compensates for the high latency of the batch layer to enable
up-to-date results for queries. However, incremental
computation has various new challenges and significant more
complex than batch computation. Especially, resource-
efficient manner with millisecond-level of latencies. Data
must be indexed in order to using of random-read/random-
write databases.
,(((RQIHUHQFHRQ2SHQ6VWHPV,26
6. 36
Authorized licensed use limited to: University of Waterloo. Downloaded on May 30,2020 at 04:01:40 UTC from IEEE Xplore. Restrictions apply.
7. B. Kappa ‘κ’ Architecture
Jay [20] describes that the alternatives are worth exploring
a part of λ architecture. He addresses the issue of maintaining
the codes in two complex distributed systems. There is
exactly painful development, as in operational burden.
Especially, the distributed components like Storm and
Hadoop. κ architecture has been introduced. In this approach,
re-processing will execute, whereby the processing code has
changed. Therefore, there is actually need to recompute the
result sets. The job doing the re-computation is just an
improved version of the same code, running on the same
framework, taking the same input data. Basically, it is
simplification of the λ architecture, where there have simply
removed the entire Batch Layer. Hence, it remains Speed
layer and Serve layer. Figure 3 shows the diagram of κ
Architecture.
The workflow can handle real-time data processing and
continuous data re-processing in a single stream computation
model. Streaming job reads the data and process them. When
re-processing is required, a second instance of the streaming
job is executed that starts processing the data from the
beginning of the retained data and redirects the output to a
separate table. When the second job that was executed has
caught up with the entire dataset, simply switch the
application to read from the new data view, stop the first job,
and delete the data view of the first job [21]. The entire multi
streams can spin up multiple consumers in parallel consuming
individual part of the data.
Figure 3: κ Architecture
Another pillar of κ architecture is the immutable data log.
This is similar in concept to the immutable Master Dataset in
Lambda architecture, but instead of using technologies such
as Hadoop/HDFS, κ architecture's immutable data log is
(usually) Kafka2
. It retains the full log of the data that it needs
to re-process. Data in Kafka is persisted to disk and replicated
for fault tolerance. Furthermore, growing of data in Kafka, it
doesn’t make the system slower, as it supports cluster
implementation by distributed across servers with over a
petabyte of storage.
C. Microservices Architecture
Fully built and deployed BDA solutions often include
many components of mix vendor software and open source
software as well. It uses physical servers, virtual machines
and docker containers. Nevertheless, application
programming interface (API) is a common method for
integrating the functions and also stitched together into
working pipeline for each data source. A container is similar
a very lightweight virtual machine, however, microservices
2
Apache Kafka is developed by LinkedIn and being contributed open source
community, as in Apache Software Foundation.
3
Apache Druid detail refers to “https://druid.apache.org/”
are even lighter. Based on the trends in BDA, most analytics
pipelines are easily deployed as an immutable microservices.
These microservices executes on its own process/container
and communicate in a self-regulate way without having to
depend on other services or application as a whole.
Microservices is commonly adopted Spark, Cassandra and
Kafka open source technology [22]. Figure 4 shows the
generic Microservices Architecture diagram as referring in
[23]. It can build on demand as needed in batch, speed and
serve layer.
Figure 4: Microservices Architecture
D. IOT Architecture
With the raise of Industry Revolution 4.0, the combination
of IOT and BDA with Artificial Intelligence are being driving
to optimize and automate production for industry. IOT is in
data-driven paradigm that uses real-time pervasive connected
sensors, simulations and event logs to deliver analytics
intelligent manufacturing through Internet/Intranet for every
area of the factory [24]. These IOT devices have been
deployed in daily operations to deliver operation efficiencies,
process innovation and environmental benefits. It also
presents the challenges in term of large-scale data
management, processing and analysis [25]. It consists of four
major bases; Time Series Store/Database (TSDB), Streaming
Message Queue (SMQ), Workflow Orchestration Engine
(WOE) and Distributed File System (DFS).
Time Series Store/Database (TSDB): It is an optimized
data management system for time-stamped or time-series data.
For processing the query of time series data, the time series
segment needs to be located. Then, there is a process of
retrieval based on a combination of one or more values of the
metadata, which commonly store in a relational database, such
as SQLite, PostgreSQL, MySQL or others. This mechanism
enables TSDB to have the low latency access for tracking,
monitoring, down sampling, and aggregating over time.
Typically, it has auto-shading and horizontal scaling with a
store-specific API or through a specific build connector. There
various open source TSDB, such as Apache Druid 3
,
InfluxDB4
, OpenTSDB5
and others.
Streaming Message Queue (SMQ): Machine-to-machine
uses message protocol for establishing communication with
publish-subscribe-based messaging to the servers; such as
MQTT (Message Queue Telemetry Transport), XMPP
(Extensible Messaging and Presence Protocol), DDS (Data
Distribution Service and others. It handles certain filters,
extraction and simple/complex calculation for process during
the streaming processes.
Workflow Orchestration Engine (WOE): It designs to
orchestrate enterprise level data processing operation, flow-
based controller, scheduler, data provenance with secure and
4
InfluxDB detail refers to “https://www.influxdata.com/”
5
OpenTSDB detail refers to “http://opentsdb.net/”
,(((RQIHUHQFHRQ2SHQ6VWHPV,26
8. 37
Authorized licensed use limited to: University of Waterloo. Downloaded on May 30,2020 at 04:01:40 UTC from IEEE Xplore. Restrictions apply.
9. durable for IOT and data analytics tasks. Furthermore, the
orchestration framework supports in distributed cluster and
extensibility with plug-in. Also, it has a diagrammatic of
views and modifiable behavior from web browser. There are
two popular open source orchestration workflow systems;
Apache Nifi/MiniFi6
is written in Java and Node-Red7
is in
JavaScript on top of Node.js platform.
Figure 5: IOT Architecture
Distributed File System (DFS): It is designed to store very
large data sets reliably, and to stream those data sets at high
bandwidth to application. It has highly fault-tolerant
capability, which has file system replicates, or copies, each
piece of data multiple times and distributes the copies to
individual nodes, placing at least one copy on a different
server rack than the others. As a result, the data on nodes that
crash can be found elsewhere within a cluster. This ensures
that processing can continue while data is recovered. There
choices of technologies for DFS, which is depending of the
“brotherhood” of applications, as the most famous open
source big data eco-system is Apache Hadoop [26]. Also,
there are Ceph8
, Alluxio9
, OpenIO10
and others.
E. NIST Big Data Reference Architecture (NBD-RA)
The National Institute of Standards and Technology
(NIST) has taken United State Federal Government for the
Big Data Research and Development Initiative responsibility.
It develops open standards and BDA architecture to accelerate
the adoption of the most secure and effective Big Data
techniques with technologies. White House announces this
initiative on March 28, 2012 [27]. It starts with fix federal
departments and agencies, which more than 80 projects
involve in this development.
NBD-RA is an elastic BDA architecture design. The
conceptual model design can be vendor-neutral, technology-
neutral, and infrastructure agnostic. The system consists of
five logical functional components; System Orchestrator,
Data Provider, Big Data Application Provider, Big data
Framework Provider and Data Consumer. Then, there are two
“Management” dimension and “Security Privacy”, which
overlaying those five components. Also, these two dimensions
provide services and functionality for BDA specific tasks.
Figure 6 shows the NBD-RA architecture, which is
referencing in [28].
6
Nifi/MiNifi detail refers to “https://nifi.apache.org”
7
Node-RED detail refers to “https://nodered.org”
8
Ceph detail refers to “https://ceph.io”.
9
Alluxio detail refers to “https://github.com/Alluxio/alluxio”.
Figure 6: NBD-RA Architecture
V. TRENDS AND ANALYSIS
The discussed architectures provide a structure with filling
a set of generic tools. However, the choice of technologies to
be used and integrated, which has much complexity. Firstly,
the consideration of BDA system is either on-premise, cloud
or hybrid. Secondly, the choice of data processing, analytics,
security with governance application technologies to be
developed; open source, commercial and hybrid. Finally, the
return of investment (ROI) by having the big data system, it is
driven by valuable AI use cases such as descriptive, predictive
and prescriptive analytics.
With on-premise BDA system, it provides high bandwidth
of transfer rate with more flexibility for accessing the system.
Nevertheless, it requires big capital outlay of investment with
high maintenance cost. Alternatively, big data in cloud
computing or hybrid cloud may be an alternative approach for
offering high availability that ranging from 99.9% to
99.99999%. Also, the promising support of expandability of
storage from gigabytes to petabytes [29]. However, there
are some native Hadoop options available in public clouds like
AWS, Google, Oracle, AliCloud and others. There may not
be the best suit for certain solutions for many applications, due
to the virtualization Hadoop performs slower workload for the
intensive application [30] [31]. Generally, all these
consideration needs a comprehensive requirements analysis
and budgeting cost.
Hadoop is one of BDA eco system, but it is not the only
the choice. Elasticsearch is the alternative BDA solution,
named ‘Elastic’. It is specialized for web search, network
traffics and log analysis. It based on Apache Lucene for low-
level indexing and analysis [32] [33]. NoSQL document-
oriented data stores is popular and on-demand nowadays,
MongoDB is one of widely used to provide durability with it
write-ahead logging techniques [34] [35]. Apache
Cassandra is one of the popular wide column-oriented enables
continuous availability, tremendous scale and data
distribution across multiple data centers and cloud availability
zones [36]. It has been deployed at certain technology giants,
such as Facebook, Netflix, Twitter, eBay and others.
Nevertheless, there are variety of choices for cloud computing
10
OpenIO detail refers to “https://www.openio.io”.
,(((RQIHUHQFHRQ2SHQ6VWHPV,26
10. 38
Authorized licensed use limited to: University of Waterloo. Downloaded on May 30,2020 at 04:01:40 UTC from IEEE Xplore. Restrictions apply.
11. technologies; Google BigTable, Amazon S3 Object storage,
Azure Cosmos DB, AlibabaCloud, ApsaraDB and others.
AI Analytics is important to every aspect of the
organization because it can help ROI at every level. Those
implemented analytics use cases need to be built around the
issues that are really clear, and the problems that businesses
are having today, to improve efficiency, effectiveness, and
specific issues such as customer satisfaction [37]. PWC
reports that 59% of executives say big data at their company
would be improved through the uses of AI [38]. By
developing best practices for quick ROI and momentum of
scale, it is critical for developing AI models, reusable building
blocks of data sets and working across organizational
boundaries to drive more valuable AI use cases [39].
VI. CONCLUSION
Nowadays, data is the fuel of an organization’s vehicle to
drive the business transformation. We are also witnessing the
growth and important of the hidden value of data. Therefore,
this paper contributes to various important aspect for
exploring BDA concepts with “V”s, features model, and key
component architectural components with trade-offs. BDA is
now being one of the main pillars of industry revolution 4.0,
as data analytics with AI are playing the crucial algorithmic
roles in producing accurate results.
REFERENCES
[1] H. Asaadi, D. Khaldi, and B. Chapman, “A Comparative Survey of
the HPC and Big Data Paradigms: Analysis and Experiments,” in
2016 IEEE International Conference on Cluster Computing
(CLUSTER), 2016, pp. 423–432.
[2] J. Yang, “From Google File System to Omega: A Decade of
Advancement in Big Data Management at Google,” in 2015 IEEE
First International Conference on Big Data Computing Service and
Applications, 2015, pp. 249–255.
[3] B. Marr, Big Data in Practice. John Wiley Sons, Inc., 2016.
[4] F. W. Kistermann, “The Invention and Development of the Hollerith
Punched Card: In Commemoration of the 130th Anniversary of the
Birth of Herman Hollerith and for the 100th Anniversary of Large
Scale Data Processing,” Ann. Hist. Comput., vol. 13, no. 3, pp. 245–
259, Jul. 1991.
[5] E. F. Codd, “A Relational Model of Data for Large Shared Data
Banks,” Commun. ACM, vol. 13, no. 6, pp. 377–387, Jun. 1970.
[6] J. Peeters, “Early MRP Systems at Royal Philips Electronics in the
1960s and 1970s,” IEEE Ann. Hist. Comput., vol. 31, no. 2, pp. 56–
69, Apr. 2009.
[7] R. Brueckner, “Where Did Big Data Come From?,” insidebigdata,
2013. [Online]. Available:
https://insidebigdata.com/2013/02/03/where-did-big-data-come-
from/. [Accessed: 12-Aug-2019].
[8] M. Lesk, “How Much Information Is There In the World?”,” 1997.
[Online]. Available: http://www.lesk.com/mlesk/ksg97/ksg.html.
[Accessed: 12-Aug-2019].
[9] B. Stone, “The Education of Google’s Larry Page,” Bloomberg
Businessweek, Apr-2012.
[10] K. Ashton, “That Internet of Things,” RFID J., 2009.
[11] A. Petrillo, “Fourth Industrial Revolution: Current Practices,
Challenges, and Opportunities,” in Digital Transformation in Smart
Manufacturing, R. Cioffi and F. De Felice, Eds. Intechopen, 2018.
[12] D. Laney, “3D Data Management: Controlling Data Volume, Velocity
and Variety,” 2001.
[13] S. Yin and O. Kaynak, “Big Data for Modern Industry: Challenges
and Trends [Point of View],” Proc. IEEE, vol. 103, no. 2, pp. 143–
146, Feb. 2015.
[14] S. Jha, J. Qiu, A. Luckow, P. K. Mantha, and G. C. Fox, “A Tale of
Two Data-Intensive Paradigms: Applications, Abstractions, and
Architectures,” CoRR, vol. abs/1403.1, 2014.
[15] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad:
distributed data-parallel programs from sequential building blocks,”
ACM SIGOPS Oper. …, pp. 59–72, 2007.
[16] C. A. Salma, B. Tekinerdogan, and I. N. Athanasiadis, “Feature
Driven Survey of Big Data Systems,” in Proceedings of the
International Conference on Internet of Things and Big Data, 2016,
pp. 348–355.
[17] K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterso,
“Feature-Oriented Domain Analysis (FODA) Feasibility Study,”
Pittsburgh, 1990.
[18] N. Marz, “How to beat the CAP theorem,” 2011. [Online]. Available:
http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html.
[Accessed: 13-Aug-2019].
[19] N. Marz and J. Warren, Big Data: Principles and best practices of
scalable realtime data systems. Manning Publications, 2015.
[20] J. Kreps, “Questioning the Lambda Architecture,” O’Reilly Media,
2014. [Online]. Available:
https://www.oreilly.com/ideas/questioning-the-lambda-architecture.
[Accessed: 13-Aug-2019].
[21] A. Kumar, Architecting Data-Intensive Applications. Packt
Publishing, 2018.
[22] G. Vetticaden, “Building Secure and Governed Microservices with
Kafka Streams,” Cloudera, 2018. [Online]. Available:
https://blog.cloudera.com/building-secure-and-governed-
microservices-with-kafka-streams/. [Accessed: 12-Aug-2019].
[23] J. Garrett, Data Analytics for IT Networks: Developing Innovative Use
Cases. Cisco Press, 2018.
[24] J. Davis, T. Edgar, J. Porter, J. Bernaden, and M. Sarli, “Smart
manufacturing, manufacturing intelligence and demand-dynamic
performance,” Comput. Chem. Eng., vol. 47, pp. 145–156, 2012.
[25] M. Mohammadi, A. Al-Fuqaha, S. Sorour, and M. Guizani, “Deep
Learning for IoT Big Data and Streaming Analytics: A Survey,” IEEE
Commun. Surv. Tutorials, vol. 20, no. 4, pp. 2923–2960, 2018.
[26] Z. Li and H. Shen, “Measuring Scale-Up and Scale-Out Hadoop with
Remote and Local File Systems and Selecting the Best Platform,”
IEEE Trans. Parallel Distrib. Syst., vol. 28, no. 11, pp. 3201–3214,
Nov. 2017.
[27] T. Kalil, “The White House Office of Science and Technology Policy:
Big Data is a Big Deal,” Office of Science and Technology Policy
(OSTP) Blog, 2012. [Online]. Available:
https://obamawhitehouse.archives.gov/blog/2012/03/29/big-data-big-
deal. [Accessed: 27-Aug-2019].
[28] “NIST Big Data Interoperability Framework: volume 8, reference
architecture interfaces,” Gaithersburg, MD, Jun. 2018.
[29] A. Zarrabi, E. K. Karuppiah, C. H. Ngo, K. K. Yong, and S. See,
“Gravitational Search Algorithm using CUDA,” in IEEE Parallel and
Distributed Computing, Applications and Technologies, PDCAT 2014,
2014, pp. 193–198.
[30] D. Nuñez, I. Agudo, and J. Lopez, “Delegated Access for Hadoop
Clusters in the Cloud,” in 2014 IEEE 6th International Conference on
Cloud Computing Technology and Science, 2014, pp. 374–379.
[31] M. E. Wendt, “Cloud-based Hadoop Deployments: Benefits and
Considerations,” 2014.
[32] J. Rosenberg, J. B. Coronel, J. Meiring, S. Gray, and T. Brown,
“Leveraging Elasticsearch to Improve Data Discoverability in Science
Gateways,” in Proceedings of the Practice and Experience in
Advanced Research Computing on Rise of the Machines (Learning),
2019, pp. 19:1--19:5.
[33] B. Dageville et al., “The Snowflake Elastic Data Warehouse,” in
Proceedings of the 2016 International Conference on Management of
Data, 2016, pp. 215–226.
[34] R. R. Shetty, A. M. Dissanayaka, S. Mengel, L. Gittner, R. Vadapalli,
and H. Khan, “Secure NoSQL Based Medical Data Processing and
Retrieval: The Exposome Project,” in Companion Proceedings of
the10th International Conference on Utility and Cloud Computing,
2017, pp. 99–105.
[35] B. Sendir, M. Govindaraju, R. Odaira, and P. Hofstee, “Low Latency
and High Throughput Write-Ahead Logging Using CAPI-Flash,”
IEEE Trans. Cloud Comput., p. 1, 2019.
[36] A. Lakshman and P. Malik, “Cassandra: A Decentralized Structured
Storage System,” SIGOPS Oper. Syst. Rev., vol. 44, no. 2, pp. 35–40,
Apr. 2010.
[37] S. Earley, “Executive Roundtable Series: Driving Higher ROI and
Organizational Change,” IT Prof., vol. 17, no. 6, pp. 60–64, Nov. 2015.
[38] “2018 AI predictions: 8 insights to shape business strategy,” PwC AS,
2018.
[39] “2019 AI Predictions: Six AI priorities you can’t afford to ignore,”
PwC AS, 2019.
,(((RQIHUHQFHRQ2SHQ6VWHPV,26
12. 39
Authorized licensed use limited to: University of Waterloo. Downloaded on May 30,2020 at 04:01:40 UTC from IEEE Xplore. Restrictions apply.