Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.
The impact of emerging IoT Technology and BigData. This is the slide presentation I did at the http://globalbigdatabootcamp.com/speakers/sanjay-sabnis/
Here's how big data and the Internet of Things work together: a vast network of sensors (IoT) collect a boatload of information (big data) that is then used to improve services and products in various industries, which in turn generate revenue.
Big data Analytics is a process to extract meaningful insight from big such as hidden patterns, unknown correlations, market trends and customer preferences
The impact of emerging IoT Technology and BigData. This is the slide presentation I did at the http://globalbigdatabootcamp.com/speakers/sanjay-sabnis/
Here's how big data and the Internet of Things work together: a vast network of sensors (IoT) collect a boatload of information (big data) that is then used to improve services and products in various industries, which in turn generate revenue.
Big data Analytics is a process to extract meaningful insight from big such as hidden patterns, unknown correlations, market trends and customer preferences
The slides defines IoT and show the differnce between M2M and IoT vision. It then describes the different layers that depicts the functional architecture of IoT, standard organizations and bodies and other IoT technology alliances, low power IoT protocols, IoT Platform components, and finally gives a short description to one of IoT low power application protocols (MQTT).
A Seminar Presentation on Big Data for Students.
Big data refers to a process that is used when traditional data mining and handling techniques cannot uncover the insights and meaning of the underlying data. Data that is unstructured or time sensitive or simply very large cannot be processed by relational database engines. This type of data requires a different processing approach called big data, which uses massive parallelism on readily-available hardware.
Big data is a term that describes the large volume of data may be both structured and unstructured.
That inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters.
Internet of Things (IoT) will enable dramatic society transformation. This seminar presents an introduction to the IoT and explains why IoT Security is important.
Then it presents security issues in wireless sensor networks that constitute a main ingredient of IoT.
Seminar given at Centre Tecnològic de Telecomunicacions de Catalunya (CTTC) on 28 January 2015.
A high level overview of common Cassandra use cases, adoption reasons, BigData trends, DataStax Enterprise and the future of BigData given at the 7th Advanced Computing Conference in Seoul, South Korea
The internet of things (IoT) is the internetworking of physical devices, vehicles, buildings and other items—embedded with electronics, software, sensors, actuators, and network connectivity that enable these objects to collect and exchange data.
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analyzed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.
IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...Guido Schmutz
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.
The slides defines IoT and show the differnce between M2M and IoT vision. It then describes the different layers that depicts the functional architecture of IoT, standard organizations and bodies and other IoT technology alliances, low power IoT protocols, IoT Platform components, and finally gives a short description to one of IoT low power application protocols (MQTT).
A Seminar Presentation on Big Data for Students.
Big data refers to a process that is used when traditional data mining and handling techniques cannot uncover the insights and meaning of the underlying data. Data that is unstructured or time sensitive or simply very large cannot be processed by relational database engines. This type of data requires a different processing approach called big data, which uses massive parallelism on readily-available hardware.
Big data is a term that describes the large volume of data may be both structured and unstructured.
That inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters.
Internet of Things (IoT) will enable dramatic society transformation. This seminar presents an introduction to the IoT and explains why IoT Security is important.
Then it presents security issues in wireless sensor networks that constitute a main ingredient of IoT.
Seminar given at Centre Tecnològic de Telecomunicacions de Catalunya (CTTC) on 28 January 2015.
A high level overview of common Cassandra use cases, adoption reasons, BigData trends, DataStax Enterprise and the future of BigData given at the 7th Advanced Computing Conference in Seoul, South Korea
The internet of things (IoT) is the internetworking of physical devices, vehicles, buildings and other items—embedded with electronics, software, sensors, actuators, and network connectivity that enable these objects to collect and exchange data.
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analyzed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.
IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...Guido Schmutz
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.
Internet of Things (IoT) - in the cloud or rather on-premises?Guido Schmutz
You want to implement a Big Data or Internet of Things (IoT) solution and like to know if it should be implemented in the cloud or on-premises. You are interested in the cloud offerings of vendors and what benefits they provide and if a similar solution would not be possible on-premises.
This presentation deals with this and other questions. Starting from a vendor-independent reference architecture and corresponding design patterns, different cloud solutions from various vendors are compared and rated. Additionally, it will be shown how such solution could be implemented on-premises and how a hybrid IoT solution could look like.
IoT Architecture - are traditional architectures good enough?Guido Schmutz
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.
Building a reliable and scalable IoT platform with MongoDB and HiveMQDominik Obermaier
Today’s Internet of Things (IoT) is enabling companies to blend together the physical and digital worlds, creating new business models and generating insights that increase productivity at once unimaginable levels. However, managing the ever growing volume of heterogeneous IoT data from disparate devices, systems and applications both on premise and in the cloud can be a challenging endeavour without a scalable and reliable IoT platform.
In this webinar, we will explore why and how companies are leveraging HiveMQ and MongoDB to build exactly that: a scalable and reliable IoT platform. Based upon a sample fleet management scenario, we will explain how telematics data can be routed via MQTT and efficiently stored to provide analytics and insights into the data.
Key Learnings
- Common challenges and pitfalls of IoT projects
- Required components for effectively handling data with an IoT platform
- HiveMQ for MQTT to enable bi-directional device communication over unstable networks
- MongoDB as the flexible and scalable modern data platform combining data from different sources and powering your applications
- Why MongoDB and HiveMQ is such a great combination
Pragmatic approach to Microservice Architecture: Role of MiddlewareAsanka Abeysinghe
Microservice Architecture (MSA) is emerging as a popular architecture pattern in today’s agile enterprise. Its iterative architecture and development methodologies are particularly attracting the interest of architects who need continuous delivery to fulfil business needs.
But, is every characteristics of MSA new or even pragmatic? Asanka Abeysinghe, vice president of solutions architecture at WSO2, will provide insights into MSA requirements from real-world examples and provide details of an architect friendly pragmatic approach for this architecture pattern.
Middleware plays a key role in successful MSA-based implementations. Using the correct middleware capabilities enable enterprises to fully leverage advantages provided by MSA, and ensures ease of implementation and faster time to market.
Asanka will explain essential middleware capabilities required to resolve the MSA puzzle, where you can also utilize supportive technologies such as Continuous Integration (CI), containerization and Container as a Service (CaaS).
Internet of Things (IoT) - in the cloud or rather on-premises?Guido Schmutz
You want to implement an Internet of Things (IoT) solution and would like to know if it should be implemented in the cloud or on-premises. You are interested in the cloud offerings of vendors and what benefits they provide and if a similar solution would not be possible on-premises.
This presentation deals with this and other questions. Starting from an vendor-independent reference architecture and corresponding design patterns, different cloud solutions from various vendors are compared and rated. Additionally it will be shown how such solution could be implemented on-premises and how a hybrid IoT solution could look like.
Increased Scalability: IoT devices need a lot of storage to share information for valuable purposes. Iot in cloud , like the StoneFly Cloud Connect to Microsoft Azure can provide customers with greater space which can increase as per the users demand. Helping to resolve the storage needs of customers.
Phoenix Data Conference - Big Data Analytics for IoT 11/4/17Mark Goldstein
“Big Data for IoT: Analytics from Descriptive to Predictive to Prescriptive” was presented to the Phoenix Data Conference on 11/4/17 at Grand Canyon University.
As the Internet of Things (IoT) floods data lakes and fills data oceans with sensor and real-world data, analytic tools and real-time responsiveness will require improved platforms and applications to deal with the data flow and move from descriptive to predictive to prescriptive analysis and outcomes.
Data Ingestion in Big Data and IoT platformsGuido Schmutz
Many of the Big Data and IoT use cases are based on combining data from multiple data sources and to make them available on a Big Data platform for analysis. The data sources are often very heterogeneous, from simple files, databases to high-volume event streams from sensors (IoT devices). It’s important to retrieve this data in a secure and reliable manner and integrate it with the Big Data platform so that it is available for analysis in real-time (stream processing) as well as in batch (typical big data processing). In past some new tools have emerged, which are especially capable of handling the process of integrating data from outside, often called Data Ingestion. From an outside perspective, they are very similar to a traditional Enterprise Service Bus infrastructures, which in larger organization are often in use to handle message-driven and service-oriented systems. But there are also important differences, they are typically easier to scale in a horizontal fashion, offer a more distributed setup, are capable of handling high-volumes of data/messages, provide a very detailed monitoring on message level and integrate very well with the Hadoop ecosystem. This session will present and compare Apache NiFi, StreamSets and the Kafka Ecosystem and show how they handle the data ingestion in a Big Data solution architecture.
Internet of Things - Are traditional architectures good enough?Guido Schmutz
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape?
Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks.
Similar to Internet of Things (IoT) and Big Data (20)
30 Minutes to the Analytics Platform with Infrastructure as CodeGuido Schmutz
Analytical platforms for PoCs and evaluation can be built in the cloud in an hour - with ready-made setup scripts. But if you put the services together freely, it gets more difficult. The open-source platform-in-a-box "Platys" (https://github.com/TrivadisPF/platys) shows that it is easier for test and PoC environments. In addition to possible uses and examples, we explain services and "just briefly" set up a data lake with a database, event broker, stream processing, blob store, SQL access and data science notebook.
Event Broker (Kafka) in a Modern Data ArchitectureGuido Schmutz
Today's modern data architectures and the their implementations contain an Event Broker. What are the benefits of placing an Event Broker in a Modern Data (Analytics) Architecture? What exactly is an Event Broker and what capabilities should it provide? Why is Apache Kafka the most popular realisation of an Event Broker?
These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Broker.
Then the session will highlight the different architecture styles which can be supported using an Event Broker (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Broker the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.
Big Data, Data Lake, Fast Data - Dataserialiation-FormatsGuido Schmutz
The concept of "Data Lake" is in everyone's mind today. The idea of storing all the data that accumulates in a company in a central location and making it available sounds very interesting at first. But Data Lake can quickly turn from a clear, beautiful mountain lake into a huge pond, especially if it is inexpertly entrusted with all the source data formats that are common in today's enterprises, such as XML, JSON, CSV or unstructured text data. Who, after some time, still has an overview of which data, which format and how they have developed over different versions? Anyone who wants to help themselves from the Data Lake must ask themselves the same questions over and over again: what information is provided, what data types do they have and how has the content changed over time?
Data serialization frameworks such as Apache Avro and Google Protocol Buffer (Protobuf), which enable platform-independent data modeling and data storage, can help. This talk will discuss the possibilities of Avro and Protobuf and show how they can be used in the context of a data lake and what advantages can be achieved. The support on Avro and Protobuf by Big Data and Fast Data platforms is also a topic.
ksqlDB is a stream processing SQL engine, which allows stream processing on top of Apache Kafka. ksqlDB is based on Kafka Stream and provides capabilities for consuming messages from Kafka, analysing these messages in near-realtime with a SQL like language and produce results again to a Kafka topic. By that, no single line of Java code has to be written and you can reuse your SQL knowhow. This lowers the bar for starting with stream processing significantly.
ksqlDB offers powerful capabilities of stream processing, such as joins, aggregations, time windows and support for event time. In this talk I will present how KSQL integrates with the Kafka ecosystem and demonstrate how easy it is to implement a solution using ksqlDB for most part. This will be done in a live demo on a fictitious IoT sample.
Kafka as your Data Lake - is it Feasible?Guido Schmutz
For a long time we discuss how much data we can keep in Kafka. Can we store data forever or do we remove data after a while and maybe having the history in a data lake on Object Storage or HDFS? With the advent of Tiered Storage in Confluent Enterprise Platform, storing data much longer in Kafka is much very feasible. So can we replace a traditional data lake with just Kafka? Maybe at least for the raw data? But what about accessing the data, for example using SQL?
KSQL allows for processing data in a streaming fashion using an SQL like dialect. But what about reading all data of a topic? You can reset the offset and still use KSQL. But there is another family of products, so-called query engines for Big Data. They originate from the idea of reading Big Data sources such as HDFS, object storage or HBase, using the SQL language. Presto, Apache Drill and Dremio are the most popular solutions in that space. Lately these query engines also added support for Kafka topics as a source of data. With that you can read a topic as a table and join it with information available in other data sources. The idea of course is not real-time streaming analytics but batch analytics directly on the Kafka topic, without having to store it in a big data storage.
This talk answers, how well these tools support Kafka as a data source. What serialization formats do they support? Is there some form of predicate push-down supported or do we have to always read the complete topic? How performant is a query against a topic, compared to a query against the same data sitting in HDFS or an object store? And finally, will this allow us to replace our data lake or at least part of it by Apache Kafka?
Event Hub (i.e. Kafka) in Modern Data ArchitectureGuido Schmutz
Today's modern data architectures and the their implementations contain an Event Hub. What are the benefits of placing an Event Hub in a Modern Data (Analytics) Architecture? What exactly is an Event Hub and what capabilities should it provide? Why is Apache Kafka the most popular realization of an Event Hub?
These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Hub.
Then the session will highlight the different architecture styles which can be supported using an Event Hub (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Hub the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaGuido Schmutz
Apache Kafka is a popular distributed streaming data platform and more and more is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. A lot of data necessary in stream processing is stored in traditional systems backed by relational databases. This session will present different approaches for integrating relational databases with Kafka, such as Kafka Connect, Oracle GoldenGate, ORDS APIs and bridging Kafka with Oracle AQ.
Event Hub (i.e. Kafka) in Modern Data (Analytics) ArchitectureGuido Schmutz
Today's modern data architectures and the their implementations contain an Event Hub. What are the benefits of placing an Event Hub in a Modern Data (Analytics) Architecture? What exactly is an Event Hub and what capabilities should it provide? Why is Apache Kafka the most popular realization of an Event Hub? These and many other questions will be answered in this session. The talk will start with a vendor-neutral definition of the capabilities of an Event Hub. Then the session will highlight the different architecture styles which can be supported using an Event Hub (Kafka), such as Streaming Data Integration, Stream Analytics and Decoupled Event-Driven Applications and how can these be combined into a unified architecture, making the Event Hub the central nervous system of an enterprise architecture. We will end with an overview of the Kafka ecosystem and a placement of the various components onto the Modern Data (Analytics) Architecture.
Building Event Driven (Micro)services with Apache KafkaGuido Schmutz
What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so.
Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Location Analytics - Real-Time Geofencing using Apache KafkaGuido Schmutz
An important underlying concept behind location-based applications is called geofencing. Geofencing is a process that allows acting on users and/or devices who enter/exit a specific geographical area, known as a geo-fence. A geo-fence can be dynamically generated—as in a radius around a point location, or a geo-fence can be a predefined set of boundaries (such as secured areas, buildings, boarders of counties, states or countries).
Geofencing lays the foundation for realizing use cases around fleet monitoring, asset tracking, phone tracking across cell sites, connected manufacturing, ride-sharing solutions and many others.
GPS tracking tells constantly and in real time where a device is located and forms the stream of events which needs to be analyzed against the much more static set of geo-fences. Many of the use cases mentioned above require low-latency actions taken place, if either a device enters or leaves a geo-fence or when it is approaching such a geo-fence. That’s where streaming data ingestion and streaming analytics and therefore the Kafka ecosystem comes into play.
This session will present how location analytics applications can be implemented using Kafka and KSQL & Kafka Streams. It highlights the exiting features available out-of-the-box and then shows how easy it is to extend it by custom defined functions (UDFs). The design of such solution so that it can scale with both an increasing amount of position events as well as geo-fences will be discussed as well.
Solutions for bi-directional integration between Oracle RDBMS and Apache KafkaGuido Schmutz
Apache Kafka is a popular distributed streaming data platform. A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. Data sources flowing into Kafka are often native data streams such as social media streams, telemetry data, financial transactions and many others. But these data stream only contain part of the information. A lot of data necessary in stream processing is stored in traditional systems backed by relational databases. To implement new and modern, real-time solutions, an up-to-date view of that information is needed. So how do we make sure that information can flow between the RDBMS and Kafka, so that changes are available in Kafka as soon as possible in near-real-time? This session will present different approaches for integrating relational databases with Kafka, such as Kafka Connect, Oracle GoldenGate and bridging Kafka with Oracle Advanced Queuing (AQ).
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaGuido Schmutz
Apache Kafka is a popular distributed streaming data platform. A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. Data sources flowing into Kafka are often native data streams such as social media streams, telemetry data, financial transactions and many others. But these data stream only contain part of the information. A lot of data necessary in stream processing is stored in traditional systems backed by relational databases. To implement new and modern, real-time solutions, an up-to-date view of that information is needed. So how do we make sure that information can flow between the RDBMS and Kafka, so that changes are available in Kafka as soon as possible in near-real-time? This session will present different approaches for integrating relational databases with Kafka, such as Kafka Connect, Oracle GoldenGate and bridging Kafka with Oracle Advanced Queuing (AQ).
Location Analytics Real-Time Geofencing using KafkaGuido Schmutz
An important underlying concept behind location-based applications is called geofencing. Geofencing is a process that allows acting on users and/or devices who enter/exit a specific geographical area, known as a geo-fence. A geo-fence can be dynamically generated—as in a radius around a point location, or a geo-fence can be a predefined set of boundaries (such as secured areas, buildings, boarders of counties, states or countries).
Geofencing lays the foundation for realizing use cases around fleet monitoring, asset tracking, phone tracking across cell sites, connected manufacturing, ride-sharing solutions and many others.
GPS tracking tells constantly and in real time where a device is located and forms the stream of events which needs to be analyzed against the much more static set of geo-fences. Many of the use cases mentioned above require low-latency actions taken place, if either a device enters or leaves a geo-fence or when it is approaching such a geo-fence. That’s where streaming data ingestion and streaming analytics and therefore the Kafka ecosystem comes into play.
This session will present how location analytics applications can be implemented using Kafka and KSQL & Kafka Streams. It highlights the exiting features available out-of-the-box and then shows how easy it is to extend it by custom defined functions (UDFs). The design of such solution so that it can scale with both an increasing amount of position events as well as geo-fences will be discussed as well.
Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. One option is to first persist the data into a data store and then use a traditional data visualisation solution to present the data. If latency is not an issue, such a solution might be good enough. An other question is which data store solution is necessary to keep up with the high load on write and read. If it is not an RDBMS but an NoSQL database, then not all traditional visualisation tools might already integrate with the specific data store. An other option is to use a Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solutions and then we show how the different blueprints can be implemented by mapping products onto the blueprints.
Kafka as an event store - is it good enough?Guido Schmutz
Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement?
This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?
Solutions for bi-directional Integration between Oracle RDMBS & Apache KafkaGuido Schmutz
A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. Today’s enterprises have their core systems often implemented on top of relational databases, such as the Oracle RDBMS. Implementing a new solution supporting the digital strategy using Kafka and the ecosystem can not always be done completely separate from the traditional legacy solutions. Often streaming data has to be enriched with state data which is held in an RDBMS of a legacy application. It’s important to cache this data in the stream processing solution, so that It can be efficiently joined to the data stream. But how do we make sure that the cache is kept up-to-date, if the source data changes? We can either poll for changes from Kafka using Kafka Connect or let the RDBMS push the data changes to Kafka. But what about writing data back to the legacy application, i.e. an anomaly is detected inside the stream processing solution which should trigger an action inside the legacy application. Using Kafka Connect we can write to a database table or view, which could trigger the action. But this not always the best option. If you have an Oracle RDBMS, there are many other ways to integrate the database with Kafka, such as Advanced Queueing (message broker in the database), CDC through Golden Gate or Debezium, Oracle REST Database Service (ORDS) and more. In this session, we present various blueprints for integrating an Oracle RDBMS with Apache Kafka in both directions and discuss how these blueprints can be implemented using the products mentioned before.
Fundamentals Big Data and AI ArchitectureGuido Schmutz
The right architecture is key for any IT project. This is especially the case for big data projects, where there are no standard architectures which have proven their suitability over years. This session discusses the different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Streaming Analytics architecture as well as Lambda and Kappa architecture and presents the mapping of components from both Open Source as well as the Oracle stack onto these architectures.
The right architecture is key for any IT project. This is valid in the case for big data projects as well, but on the other hand there are not yet many standard architectures which have proven their suitability over years.
This session discusses different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Event Driven architecture as well as Lambda and Kappa architecture.
Each architecture is presented in a vendor- and technology-independent way using a standard architecture blueprint. In a second step, these architecture blueprints are used to show how a given architecture can support certain use cases and which popular open source technologies can help to implement a solution based on a given architecture.
Location Analytics - Real-Time Geofencing using Kafka Guido Schmutz
An important underlying concept behind location-based applications is called geofencing. Geofencing is a process that allows acting on users and/or devices who enter/exit a specific geographical area, known as a geo-fence. A geo-fence can be dynamically generated—as in a radius around a point location, or a geo-fence can be a predefined set of boundaries (such as secured areas, buildings, boarders of counties, states or countries). Geofencing lays the foundation for realising use cases around fleet monitoring, asset tracking, phone tracking across cell sites, connected manufacturing, ride-sharing solutions and many others. Many of the use cases mentioned above require low-latency actions taken place, if either a device enters or leaves a geo-fence or when it is approaching such a geo-fence. That’s where streaming data ingestion and streaming analytics and therefore the Kafka ecosystem comes into play. This session will present how location analytics applications can be implemented using Kafka and KSQL & Kafka Streams. It highlights the exiting features available out-of-the-box and then shows how easy it is to extend it by custom defined functions (UDFs).
Most data visualization solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualization capabilities. One option is to first persist the data into a data store and then use a traditional data visualization solution to present the data. If latency is not an issue, such a solution might be good enough. An other question is which data store solution is necessary to keep up with the high load on write and read. If it is not an RDBMS but an NoSQL database, then not all traditional visualization tools might already integrate with the specific data store. An other option is to use a Streaming Visualization solution. This talk presents different architecture blueprints for integrating data visualization into a fast data solutions.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Knowledge engineering: from people to machines and back
Internet of Things (IoT) and Big Data
1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Internet of Things (IoT) and Big Data
30.9.2016 – DOAG 2016 Big Data Days
Guido Schmutz
2. Guido Schmutz
Working for Trivadis for more than 19 years
Oracle ACE Director for Fusion Middleware and SOA
Co-Author of different books
Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data
Member of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 25 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Internet of Things (IoT) and Big Data2
3. Agenda
1. Introduction
2. Towards an IoT Architecture
3. IoT Refererence Architecture
4. Summary
Internet of Things (IoT) and Big Data3
5. Internet of Things (IoT) Wave
Internet of Things (IoT): Enabling
communication between devices,
people & processes to exchange
useful information & knowledge
that create value for humans
Term was first proposed by Kevin
Ashton in 1999
Source: The Economist
Source: Ericsson, June 2016
Internet of Things (IoT) and Big Data5
6. Reasons why IoT opportunity is occurring now ?
Affordable hardware
• Costs of actuators & sensors have been
cut in half over last 10 years
Smaller, more powerful hardware
• Form factors of hardware have shrunk to
millimeter or even nanometer levels
Ubiquitous & cheap mobility
• Cost for mobile devices, bandwidth and
data processing has declined over last
10 years
Availability of supporting tools
• Big data tools & cloud based infrastructure
have become widely available
Mass market awareness
• IoT has surpassed a critical tipping point
• Vision of a connected world has reached
such a followership that companies have
initiated IoT developments
• Commitment is irreversible
Internet of Things (IoT) and Big Data6
8. Towards an IoT Architecture
Internet of Things (IoT) and Big Data15
9. Key Challenges for building an IoT application
1. Connect: How to collect data from intelligent devices?
• Abstract complexity associated with device connectivity
• Standardize integration of devices with enterprise
2. Analyze: How to analyze IoT data?
• Reduce noise and detect business event at real-time
• Enable historical big-data analysis
3. Integrate: How to integrate IoT data & events with enterprise infrastructure?
• Make enterprise processes IoT friendly
• Allow enterprise & mobile applications to control devices
Internet of Things (IoT) and Big Data18
10. Today) Existing Service-/API Architecture as a base
19
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
API Gateway
Enterprise Service Bus (ESB) / Data Integration
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST / SOAP
REST / SOAP
REST / SOAP
SOAP
Various
SQL
SOAP
REST
Service BusOracle Data Integrator
API Gateway
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data19 = one way = request/response
11. REST / SOAP
REST / SOAP
IoT 1a) Reuse exiting Service-/API-based Architecture
IoT Smart
Devices
20
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
Enterprise Service Bus (ESB) / Data Integration
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST / SOAP
REST
REST
JMS / REST
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
Service BusOracle Data Integrator
API Gateway
API Gateway
JMS
JMS
WeblogicJMS
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data20 = one way = request/response
12. IoT 1a) Challenges
Internet of Things (IoT) and Big Data21
• Do IoT devices contain enough resources to communicate directly over the internet
(HTTP or JMS) ?
• Should the device only collect data (sense) or is there also the way back necessary
(actuator) ?
• Can JMS be used from external devices (firewalls allow traffic over JMS) ?
• How many IoT devices are planned short and long term? How often do they send to
the backend? Are the JMS server as well as the ESB capable to deal with the
resulting message volume ?
• What are the operations on IoT messages / events, only simple transformations, filter
and routing operations?
13. REST / SOAP
REST / SOAP
IoT 1b) Reuse existing Service-/API-based Architecture
IoT Smart
Devices
22
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
API Gateway
Enterprise Service Bus (ESB) / Data Integration
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST
HTTP
REST
REST HTTP
JMS API Gateway
Service BusOracle Data Integrator
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
JMS
JMS
REST / SOAP
WeblogicJMS
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data22 = one way = request/response
14. REST / SOAP
REST / SOAP
IoT 2) Adding Event Hub and optional IoT Gateway
23
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
ESB / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST
Kafka / MQTT / REST
REST
Kafka / MQTT / REST
MQTT
Kura
Event Hub
Kafka
Service Bus
Oracle Data Integrator
REST
REST
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMS
API Gateway
API Gateway
REST
Kafka
REST / SOAP
REST
REST
SOA Suite
BPM Suite
Business Activity
Monitoring
MQTT
Internet of Things (IoT) and Big Data23
MQTT
= one way = request/response
15. How to implement an Event Hub?
Apache Kafka to the rescue
Distributed publish-subscribe messaging system
Designed for processing of high-volume, real
time activity stream data (logs, metrics
collections, social media streams, …)
Topic Semantic
does not implement JMS standard
Initially developed at LinkedIn, now part of
Apache
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Internet of Things (IoT) and Big Data24
16. Oracle’s Service Bus as a consumer of Kafka
Service Bus 12c
Cloud
Apps
Business
Service
Cloud
Proxy
Service
Kafka
Cloud
API
Mobile
Apps Pipeline
Routing
Kafka
Sensor / IoT
Web Apps
Business
Service
REST
Business
Service
WSDL
Backend
Apps
REST
Backend
Apps
WSDL
Proxy
Service
Kafka
Pipeline
Routing
Database
DB CDC
Stream
Processing
Internet of Things (IoT) and Big Data25
17. IoT 2) Solutions & Challenges
Internet of Things (IoT) and Big Data26
Solutions
• Event Hub solves the potential scalability issue of JMS
• IoT Gateway makes sure that lightweight sensors can connect to the internet / send their
data
Challenges
• Where to do complex analytics on the events? Is it scalable?
• Can we really send all data down to backend? Network bandwidth?
18. REST / SOAP
REST / SOAP
IoT 3) Adding Stream Processing / Analytics in Backend
27
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Stream Processing
ESP / CEP
DB
DB
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Stream Analytics
Service Bus
Oracle Data IntegratorKafka
MQTT
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMSAPI GatewayAPI Gateway
REST
MQTT
REST
REST
Kafka
Kafka / MQTT / REST
Kafka /
MQTT /
REST
REST
Kafka
REST / SOAP
REST
REST
Kura
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data27 = one way = request/response
KafkaMQTT
19. IoT 3) Solutions & Challenges
Internet of Things (IoT) and Big Data28
Solutions
• Stream Processing handles complex analytics on events in a scalable manner before
sending events to ESB / backend systems
Challenges
• Can we really send all data down to backend? Network bandwidth?
20. Oracle’s Stream Analytics as consumer of Kafka/MQTT
Oracle Stream Analytics
Stream Analytics
Kafka
Kafka
Mobile Apps
Kafka
Sensor / IoT
Web Apps
Machine Data
DB CDC
Kafka
MQTT
Internet of Things (IoT) and Big Data29
21. REST / SOAP
REST / SOAP
IoT 4) Adding Industry 4.0 Data Sources (machine data)
30
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Stream Processing
ESP / CEP
DB
DB
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Stream Analytics
Service Bus
Oracle Data IntegratorKafka
MQTT
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMSAPI GatewayAPI Gateway
REST
MQTT
REST
REST
Kafka
Kafka / MQTT / REST
Kafka /
MQTT /
REST
REST
Kafka
REST / SOAP
REST
REST
Kura
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data30 = one way = request/response
I 4.0
Machine
DB CDC GoldenGate
Kafka / MQTT / REST
KafkaMQTT
22. GoldenGate Gateway
Oracle’s GoldenGate for Change Data Capture of
existing database
Internet of Things (IoT) and Big Data31
Machine Data
DB
Kafka
Oracle GoldenGate
Delivery
Capture
Pump
JMS
Machine Data
DB Oracle GoldenGate
Delivery
Capture
Pump HBase
HDFS
23. REST / SOAP
REST / SOAP
IoT 5) Adding Stream Processing / Analytics at Edge
32
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Stream Processing
ESP / CEP
DB
DB
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Stream Analytics
Service Bus
Oracle Data IntegratorKafka
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMSAPI GatewayAPI Gateway
Kafka /
MQTT /
REST
REST REST
Kafka
Kafka
REST
RESTREST
REST / SOAP
REST
MQTT
SOA Suite
BPM Suite
Business Activity
Monitoring
ESP/CEP
Edge Analytics
MQTT
Internet of Things (IoT) and Big Data32
Kafka / MQTT / REST
MQTT
= one way = request/response
I 4.0
Machine
DB CDC GoldenGate
Kafka / MQTT / REST
Kafka
24. IoT 5) Solutions & Challenges
Internet of Things (IoT) and Big Data33
Solutions
• Stream Processing at the edge / gateway allows to reduce the amount of messages send
to the backend (cloud / on premises) if necessary
Challenges
• What if we need the raw events persisted? Where do to that?
25. REST / SOAP
REST / SOAP
IoT 6) Adding Raw Data Storage and Batch Analytics
34
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Service Bus
Oracle Data IntegratorKafka
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
JMSAPI GatewayAPI Gateway
REST REST
Kafka
Kafka
SQL
REST
REST
REST / SOAP
Stream Processing
ESP/CEP
DB
DB
Big Data Processing
HDFS
Batch
Processing
DB
Kafka
Kafka
HDFS
ESP/CEP
Edge Analytics
MQTT
MQTT
Stream Analytics
Hadoop / Spark
Oracle Big Data Appliance
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data34
Kafka / MQTT / REST
Kafka /
MQTT /
REST
= one way = request/response
I 4.0
Machine
DB CDC GoldenGate
MQTT
Kafka / MQTT / REST
Kafka
26. IoT 6) Solutions & Challenges
Internet of Things (IoT) and Big Data35
Solutions
• Adding Big Data platform allows to store all raw data in the distributed file system in a
scalable and reliable manner
Challenges
• How can we leverage the Big Data platform for more than just storing raw data? How does
it combine with the stream processing?
27. Continuous Ingestion / Fan-In from the Edge
DB Source
Big Data
Log
Stream
Processing
IoT Sensor
Event Hub
Topic
Topic
REST
Topic
IoT GW
CDC GW
Connect
CDC
DB Source
Log CDC
Native
IoT Sensor
IoT Sensor
37
Dataflow GW
Topic
Topic
Queue
MQTT GW
Topic
Dataflow GW
Dataflow
TopicREST
37
File Source
Log
Log
Log
Social
Native
Internet of Things (IoT) and Big Data37
Topic
Topic
28. Challenges for Ingesting Sensor Data
Internet of Things (IoT) and Big Data
• Multitude of sensors
• Multiple Firmware
versions
• Bad Data from
damaged sensors
• Data Quality
38
29. REST / SOAP
REST / SOAP
IoT 6a) Adding Data Mining / Machine Learning and
Model execution
40
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Service Bus
Oracle Data IntegratorKafka
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
JMSAPI GatewayAPI Gateway
REST REST
Kafka
Kafka
SQL
REST
REST
REST / SOAP
Stream Processing
ESP/CEP
DB
DB
Big Data Processing
HDFS
Batch
Processing
DB
Kafka
Kafka
HDFS
ESP/CEP
Edge Analytics
MQTT
MQTT
Stream Analytics
Hadoop / Spark
Oracle Big Data Appliance
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data40
Kafka / MQTT / REST
Kafka /
MQTT /
REST
= one way = request/response
I 4.0
Machine
DB CDC GoldenGate
MQTT
Kafka / MQTT / REST
Kafka
31. IoT Services
IoT Logical Reference Architecture
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Internet of Things (IoT) and Big Data43
32. IoT Services
IoT Logical Reference Architecture – Oracle
on premises
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Edge Analytics
Business Activity
Monitoring
SOA Suite
BPM Suite
Service Bus
Oracle Data Integrator
Stream Analytics
Big Data
Appliance
Stream Analytics
Service Bus
API Gateway
Internet of Things (IoT) and Big Data44
Oracle IoT CS
Gateway
Oracle IoT CS
Client Library
33. IoT Services
IoT Logical Reference Architecture – Oracle
Cloud Services
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Edge Analytics
Oracle BI CS
Oracle Big Data CS
Oracle SOA CS
Oracle Integration CS
Oracle IoT CS
Oracle Streaming
Analytics CS
Oracle Messaging CS
Oracle Big Data
Discovery CS
Oracle Mobile CS
Internet of Things (IoT) and Big Data45
Oracle IoT CS
Gateway
Oracle IoT CS
Client Library
Oracle Process CS
Oracle DataFlow ML CS
Big Data Preparation CS
Application Container CS
Container CS
34. IoT Services
IoT Logical Reference Architecture – Microsoft
Azure
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Azure IoTHub
Azure Streaming
Analytics
Azure Service Bus
Azure Power BI
HD Insight
Azure Storage
Azure IoT Gateway
SDK
Azure Event Hub
Azure Storage
Azure Machine
Learning
Internet of Things (IoT) and Big Data46
37. Summary
Treat events as events! Infrastructures for handling lots of events are available!
IoT tends to make Big Data / Fast Data infrastructures necessary
Know your use case/requirements to choose the right architecture!
• Can my existing backend landscape handle the new IoT load?
• Do I have to handle huge amount of events in “real-time”?
• Do I need to filter/aggregate data before invoking existing backend systems?
• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?
• What is the network bandwidth between device/gateway and cloud/backend?
• Centralized or Decentralized IoT solution?
Internet of Things (IoT) and Big Data49