Stream based data / event / message processing becomes preferred way of achieving interoperability and real-time communication in distributed SOA / microservice / database architectures.
Beside lambdas, Java 8 introduced two new APIs explicitly dealing with stream data processing:
- Stream - which is PULL-based and easily parallelizable;
- CompletableFuture / CompletionStage - which allow composition of PUSH-based, non-blocking, asynchronous data processing pipelines.
Java 9 will provide further support for stream-based data-processing by extending the CompletableFuture with additional functionality – support for delays and timeouts, better support for subclassing, and new utility methods.
More, Java 9 provides new java.util.concurrent.Flow API implementing Reactive Streams specification that enables reactive programming and interoperability with libraries like Reactor, RxJava, RabbitMQ, Vert.x, Ratpack, and Akka.
The presentation will discuss the novelties in Java 8 and Java 9 supporting stream data processing, describing the APIs, models and practical details of asynchronous pipeline implementation, error handling, multithreaded execution, asyncronous REST service implementation, interoperability with existing libraries.
There are provided demo examples (code on GitHub) using Completable Future and Flow with:
- JAX-RS 2.1 AsyncResponse, and more importantly unit-testing the async REST service method implementations;
- CDI 2.0 asynchronous observers (fireAsync / @ObservesAsync);
Microservices with Spring 5 Webflux - jProfessionalsTrayan Iliev
Spring 5 introduces new functional and reactive programming model for building web applications and (micro-)services.
The session @jProfessionals dev conference demonstrates how to build REST microservices using Spring WebFlux and Spring Boot using code examples on GitHub. It includes:
- Introduction to reactive programming, Reactive Streams specification, and project Reactor (as WebFlux infrastructure);
- Comparison between annotation-based and functional reactive ;programming approaches for building REST services with WebFlux;
- Router, handler and filter functions;
- Using reactive repositories and reactive database access with Spring Data;
- Building end-to-end non-blocking reactive web services using Netty-based web runtime;
- Reactive WebClients and integration testing;
- Realtime event streaming to WebClients using JSON Streams, and to JS client using SSE.
Spring 5 Webflux - Advances in Java 2018Trayan Iliev
Brief introduction to distributed stream processing, reactive programming, and novelties in Spring 5, Spring Boot 2, and reactive Spring Data + programming examples in GitHub. More information will be provided during upcoming Spring 5 course: http://iproduct.org/en/courses/spring-mvc-rest/
Sensor data is streamed in realtime from Arduino + accelerometeres, gyroscopes & compass 3D, ultrasound distance sensor, etc. using UDP protocol. The data processing is done with reactive Java alterantive implementations: callbacks, CompletableFutures and using Spring 5 Reactor library. The web 3D visualization with Three.js is streamed using Server Sent Events (SSE).
A video for the IoT demo is available @YouTube: https://www.youtube.com/watch?v=AB3AWAfcy9U
All source code of the demo is freely available @GitHub: https://github.com/iproduct/reactive-demos-iot
There are more reactive Java demos in the same repository - callbacks, CompletableFuture, realtime event streaming. Soon I'll add a description how to build the device and upload Arduino sketch, as well as describe CompletableFuture and Reactor demos and 3D web visualization part with Three.js. Please stay tuned :)
Reactive Microservices with Spring 5: WebFlux Trayan Iliev
On November 27 Trayan Iliev from IPT presented “Reactive microservices with Spring 5: WebFlux” @Dev.bg in Betahaus Sofia. IPT – Intellectual Products & Technologies has been organizing Java & JavaScript trainings since 2003.
Spring 5 introduces a new model for end-to-end functional and reactive web service programming with Spring 5 WebFlow, Spring Data & Spring Boot. The main topics include:
– Introduction to reactive programming, Reactive Streams specification, and project Reactor (as WebFlux infrastructure)
– REST services with WebFlux – comparison between annotation-based and functional reactive programming approaches for building.
– Router, handler and filter functions
– Using reactive repositories and reactive database access with Spring Data. Building end-to-end non-blocking reactive web services using Netty-based web runtime
– Reactive WebClients and integration testing. Reactive WebSocket support
– Realtime event streaming to WebClients using JSON Streams, and to JS client using SSE.
Presentation from BGOUG conference Nov 17, 2017.
Since September 2017, Java 9 is generally available. It offers many enhancements:
• Modularity – provides clear separation between public and private APIs, stronger encapsulation & dependency management.
• JShell – using and customizing Java 9 interactive shell by example
• Process API updates – feature-rich, async OS process management and statistics
• Reactive Streams, CompletableFuture and Stream API updates
• Building asynchronous HTTP/2 and WebSocket pipelines using HTTP/2 Client and CompletableFuture composition
• Collection API updates
• Stack walking, and other language enhancements (Project Coin)
Discussed topics are accompanied by live demos available for further review @ github.com/iproduct.
Making Machine Learning Easy with H2O and WebFluxTrayan Iliev
Machine learning is becoming a must for many business domains and applications. H2O is a best-of-breed, open source, distributed machine learning library written in Java. The presentation shows how to create and train machine learning models easily using H2O Flow web interface, including Deep Learning Neural Networks (DNNs). The session provides a tutorial how to develop and deploy fullstack-reactive face recognition demo using React + RxJS WebSocket front-end, OpenCV, Caffe CNN for image segmentation, OpenFace CNN for feature extraction, H20 Flow for face recognition interactive model training and export as POJO. The trained POJO model is incorporated in a real-time streaming web service implemented using Spring 5 Web Flux and Spring Boot. All demo is 100% Java!
Presentation from Angular Sofia Meetup event focuses on integration between state-of-the-art Angular, component libraries and supporting technologies, necessary to build a scalable and performant single-page apps. Topics include:
- Composing NGRX Reducers, Selectors and Middleware;
- Computing derived data using Reselect-style memoization with RxJS;
- NGRX Router integration;
- Normalization/denormalization and keeping data locally in IndexedDB;
- Processing Observable (hot) streams of async actions, and isolating the side effects using @Effect decorator with NGRX/RxJS reactive transforms;
- Integration of Material Design with third party component libraries like PrimeNG;
- more: lazy loading, AOT...
Microservices with Spring 5 Webflux - jProfessionalsTrayan Iliev
Spring 5 introduces new functional and reactive programming model for building web applications and (micro-)services.
The session @jProfessionals dev conference demonstrates how to build REST microservices using Spring WebFlux and Spring Boot using code examples on GitHub. It includes:
- Introduction to reactive programming, Reactive Streams specification, and project Reactor (as WebFlux infrastructure);
- Comparison between annotation-based and functional reactive ;programming approaches for building REST services with WebFlux;
- Router, handler and filter functions;
- Using reactive repositories and reactive database access with Spring Data;
- Building end-to-end non-blocking reactive web services using Netty-based web runtime;
- Reactive WebClients and integration testing;
- Realtime event streaming to WebClients using JSON Streams, and to JS client using SSE.
Spring 5 Webflux - Advances in Java 2018Trayan Iliev
Brief introduction to distributed stream processing, reactive programming, and novelties in Spring 5, Spring Boot 2, and reactive Spring Data + programming examples in GitHub. More information will be provided during upcoming Spring 5 course: http://iproduct.org/en/courses/spring-mvc-rest/
Sensor data is streamed in realtime from Arduino + accelerometeres, gyroscopes & compass 3D, ultrasound distance sensor, etc. using UDP protocol. The data processing is done with reactive Java alterantive implementations: callbacks, CompletableFutures and using Spring 5 Reactor library. The web 3D visualization with Three.js is streamed using Server Sent Events (SSE).
A video for the IoT demo is available @YouTube: https://www.youtube.com/watch?v=AB3AWAfcy9U
All source code of the demo is freely available @GitHub: https://github.com/iproduct/reactive-demos-iot
There are more reactive Java demos in the same repository - callbacks, CompletableFuture, realtime event streaming. Soon I'll add a description how to build the device and upload Arduino sketch, as well as describe CompletableFuture and Reactor demos and 3D web visualization part with Three.js. Please stay tuned :)
Reactive Microservices with Spring 5: WebFlux Trayan Iliev
On November 27 Trayan Iliev from IPT presented “Reactive microservices with Spring 5: WebFlux” @Dev.bg in Betahaus Sofia. IPT – Intellectual Products & Technologies has been organizing Java & JavaScript trainings since 2003.
Spring 5 introduces a new model for end-to-end functional and reactive web service programming with Spring 5 WebFlow, Spring Data & Spring Boot. The main topics include:
– Introduction to reactive programming, Reactive Streams specification, and project Reactor (as WebFlux infrastructure)
– REST services with WebFlux – comparison between annotation-based and functional reactive programming approaches for building.
– Router, handler and filter functions
– Using reactive repositories and reactive database access with Spring Data. Building end-to-end non-blocking reactive web services using Netty-based web runtime
– Reactive WebClients and integration testing. Reactive WebSocket support
– Realtime event streaming to WebClients using JSON Streams, and to JS client using SSE.
Presentation from BGOUG conference Nov 17, 2017.
Since September 2017, Java 9 is generally available. It offers many enhancements:
• Modularity – provides clear separation between public and private APIs, stronger encapsulation & dependency management.
• JShell – using and customizing Java 9 interactive shell by example
• Process API updates – feature-rich, async OS process management and statistics
• Reactive Streams, CompletableFuture and Stream API updates
• Building asynchronous HTTP/2 and WebSocket pipelines using HTTP/2 Client and CompletableFuture composition
• Collection API updates
• Stack walking, and other language enhancements (Project Coin)
Discussed topics are accompanied by live demos available for further review @ github.com/iproduct.
Making Machine Learning Easy with H2O and WebFluxTrayan Iliev
Machine learning is becoming a must for many business domains and applications. H2O is a best-of-breed, open source, distributed machine learning library written in Java. The presentation shows how to create and train machine learning models easily using H2O Flow web interface, including Deep Learning Neural Networks (DNNs). The session provides a tutorial how to develop and deploy fullstack-reactive face recognition demo using React + RxJS WebSocket front-end, OpenCV, Caffe CNN for image segmentation, OpenFace CNN for feature extraction, H20 Flow for face recognition interactive model training and export as POJO. The trained POJO model is incorporated in a real-time streaming web service implemented using Spring 5 Web Flux and Spring Boot. All demo is 100% Java!
Presentation from Angular Sofia Meetup event focuses on integration between state-of-the-art Angular, component libraries and supporting technologies, necessary to build a scalable and performant single-page apps. Topics include:
- Composing NGRX Reducers, Selectors and Middleware;
- Computing derived data using Reselect-style memoization with RxJS;
- NGRX Router integration;
- Normalization/denormalization and keeping data locally in IndexedDB;
- Processing Observable (hot) streams of async actions, and isolating the side effects using @Effect decorator with NGRX/RxJS reactive transforms;
- Integration of Material Design with third party component libraries like PrimeNG;
- more: lazy loading, AOT...
Rapid Web API development with Kotlin and KtorTrayan Iliev
Introduction to Kotlin and Ktor with flow, async and channel examples. Ktor is an async web framework with minimal ceremony that leverages the advantages of Kotlin like coroutines and extensible functional DSLs..
My jPrime 2016 presentation shows example of Domain-Driven Design (DDD), Event Sourcing (ES) and Functional Reactive Programming (FRP) using Reactor and Redux in a showcase of Java robotics - two small robots IPTPI (Raspberry Pi 2 + Ardiuno) and LeJaRo (LeJOS).
L'expérience du développement de CRESON, support pour des objets distants fortement cohérents dans Infinispan, par Etienne Riviere (UCLouvain).
Cet exposé présentera des résultats obtenus dans le cadre du projet européen LEADS que j'ai coordonné et où l'entreprise Red Hat était partenaire. Le code produit a été intégré dans le “staging" de la base de données NoSQL Infinispan, et évalué avec un équivalent open source de Dropbox développé par CloudSpaces, un autre projet européen.
Overview of Indiana University's Advanced Science Gateway support activities for drug discovery, computational chemistry, and other Web portals. For a broader overview of the OGCE project, see http://www.collab-ogce.org/ogce/index.php
Project Hydrogen: State-of-the-Art Deep Learning on Apache SparkDatabricks
Big data and AI are joined at the hip: the best AI applications require massive amounts of constantly updated training data to build state-of-the-art models AI has always been on of the most exciting applications of big data and Apache Spark. Increasingly Spark users want to integrate Spark with distributed deep learning and machine learning frameworks built for state-of-the-art training. On the other side, increasingly DL/AI users want to handle large and complex data scenarios needed for their production pipelines.
This talk introduces a new project that substantially improves the performance and fault-recovery of distributed deep learning and machine learning frameworks on Spark. We will introduce the major directions and provide progress updates, including 1) barrier execution mode for distributed DL training, 2) fast data exchange between Spark and DL frameworks, and 3) accelerator-awareness scheduling.
Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A...Databricks
"Project Hydrogen is a major Apache Spark initiative to bring state-of-the-art AI and Big Data solutions together. It contains three major projects: 1) barrier execution mode 2) optimized data exchange and 3) accelerator-aware scheduling. A basic implementation of barrier execution mode was merged into Apache Spark 2.4.0, and the community is working on the latter two. In this talk, we will present progress updates to Project Hydrogen and discuss the next steps.
First, we will review the barrier execution mode implementation from Spark 2.4.0. It enables developers to embed distributed training jobs properly on a Spark cluster. We will demonstrate distributed AI integrations built on top it, e.g., Horovod and Distributed TensorFlow. We will also discuss the technical challenges to implement those integrations and future work. Second, we will outline on-going work for optimized data exchange. Its target scenario is distributed model inference. We will present how we do performance testing/profiling, where the bottlenecks are, and how to improve the overall throughput on Spark. If time allows, we might also give updates on accelerator-aware scheduling.
"
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...Databricks
Yelp’s ad platform handles millions of ad requests everyday. To generate ad metrics and analytics in real-time, they built they ad event tracking and analyzing pipeline on top of Spark Streaming. It allows Yelp to manage large number of active ad campaigns and greatly reduce over-delivery. It also enables them to share ad metrics with advertisers in a more timely fashion.
This session will start with an overview of the entire pipeline and then focus on two specific challenges in the event consolidation part of the pipeline that Yelp had to solve. The first challenge will be about joining multiple data sources together to generate a single stream of ad events that feeds into various downstream systems. That involves solving several problems that are unique to real-time applications, such as windowed processing and handling of event delays. The second challenge covered is with regards to state management across code deployments and application restarts. Throughout the session, the speakers will share best practices for the design and development of large-scale Spark Streaming pipelines for production environments.
Get ready to lock and load through this quick overview of some of the newest most innovative, tools around. Source: http://www.takipiblog.com/7-new-tools-java-developers-should-know/
On-Prem Solution for the Selection of Wind Energy ModelsDatabricks
The renewable energy industry has only recently started to rely on data-driven models on applications that have traditionally required complex physical solutions. In this talk, we would like to show how we leverage Spark, Keras and (in our case, on-prem) high performance computing (HPC) infrastructure to potentially tackle common and interesting problems in the wind-related industry (saving hours of CPU-consuming simulations).
We use:
Apache Spark and Hive for data preparation and a combination of different data sources (some of them in the range of the petabytes scale).
Keras for model training/generation.
HPC for coordination and node-wide training of hyperparameters.
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Rapid Web API development with Kotlin and KtorTrayan Iliev
Introduction to Kotlin and Ktor with flow, async and channel examples. Ktor is an async web framework with minimal ceremony that leverages the advantages of Kotlin like coroutines and extensible functional DSLs..
My jPrime 2016 presentation shows example of Domain-Driven Design (DDD), Event Sourcing (ES) and Functional Reactive Programming (FRP) using Reactor and Redux in a showcase of Java robotics - two small robots IPTPI (Raspberry Pi 2 + Ardiuno) and LeJaRo (LeJOS).
L'expérience du développement de CRESON, support pour des objets distants fortement cohérents dans Infinispan, par Etienne Riviere (UCLouvain).
Cet exposé présentera des résultats obtenus dans le cadre du projet européen LEADS que j'ai coordonné et où l'entreprise Red Hat était partenaire. Le code produit a été intégré dans le “staging" de la base de données NoSQL Infinispan, et évalué avec un équivalent open source de Dropbox développé par CloudSpaces, un autre projet européen.
Overview of Indiana University's Advanced Science Gateway support activities for drug discovery, computational chemistry, and other Web portals. For a broader overview of the OGCE project, see http://www.collab-ogce.org/ogce/index.php
Project Hydrogen: State-of-the-Art Deep Learning on Apache SparkDatabricks
Big data and AI are joined at the hip: the best AI applications require massive amounts of constantly updated training data to build state-of-the-art models AI has always been on of the most exciting applications of big data and Apache Spark. Increasingly Spark users want to integrate Spark with distributed deep learning and machine learning frameworks built for state-of-the-art training. On the other side, increasingly DL/AI users want to handle large and complex data scenarios needed for their production pipelines.
This talk introduces a new project that substantially improves the performance and fault-recovery of distributed deep learning and machine learning frameworks on Spark. We will introduce the major directions and provide progress updates, including 1) barrier execution mode for distributed DL training, 2) fast data exchange between Spark and DL frameworks, and 3) accelerator-awareness scheduling.
Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A...Databricks
"Project Hydrogen is a major Apache Spark initiative to bring state-of-the-art AI and Big Data solutions together. It contains three major projects: 1) barrier execution mode 2) optimized data exchange and 3) accelerator-aware scheduling. A basic implementation of barrier execution mode was merged into Apache Spark 2.4.0, and the community is working on the latter two. In this talk, we will present progress updates to Project Hydrogen and discuss the next steps.
First, we will review the barrier execution mode implementation from Spark 2.4.0. It enables developers to embed distributed training jobs properly on a Spark cluster. We will demonstrate distributed AI integrations built on top it, e.g., Horovod and Distributed TensorFlow. We will also discuss the technical challenges to implement those integrations and future work. Second, we will outline on-going work for optimized data exchange. Its target scenario is distributed model inference. We will present how we do performance testing/profiling, where the bottlenecks are, and how to improve the overall throughput on Spark. If time allows, we might also give updates on accelerator-aware scheduling.
"
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...Databricks
Yelp’s ad platform handles millions of ad requests everyday. To generate ad metrics and analytics in real-time, they built they ad event tracking and analyzing pipeline on top of Spark Streaming. It allows Yelp to manage large number of active ad campaigns and greatly reduce over-delivery. It also enables them to share ad metrics with advertisers in a more timely fashion.
This session will start with an overview of the entire pipeline and then focus on two specific challenges in the event consolidation part of the pipeline that Yelp had to solve. The first challenge will be about joining multiple data sources together to generate a single stream of ad events that feeds into various downstream systems. That involves solving several problems that are unique to real-time applications, such as windowed processing and handling of event delays. The second challenge covered is with regards to state management across code deployments and application restarts. Throughout the session, the speakers will share best practices for the design and development of large-scale Spark Streaming pipelines for production environments.
Get ready to lock and load through this quick overview of some of the newest most innovative, tools around. Source: http://www.takipiblog.com/7-new-tools-java-developers-should-know/
On-Prem Solution for the Selection of Wind Energy ModelsDatabricks
The renewable energy industry has only recently started to rely on data-driven models on applications that have traditionally required complex physical solutions. In this talk, we would like to show how we leverage Spark, Keras and (in our case, on-prem) high performance computing (HPC) infrastructure to potentially tackle common and interesting problems in the wind-related industry (saving hours of CPU-consuming simulations).
We use:
Apache Spark and Hive for data preparation and a combination of different data sources (some of them in the range of the petabytes scale).
Keras for model training/generation.
HPC for coordination and node-wide training of hyperparameters.
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Reactive Java Robotics & IoT with Spring ReactorTrayan Iliev
On April 4-th, 2017 in cosmos coworking camp – Sofia, Trayan Iliev will talk about “Reactive Java Robotics and IoT with Spring Reactor” (http://robolearn.org/reactive-java-robotics-iot-spring-reactor/).
The event is organized by DEV.BG and it is part from the user group Internet of Things.
Program:
1. Robotics, IoT & Complexity. Domain-Driven Design (DDD). Reactive programming. Reactive Streams (java.util.concurrent.Flow);
2. High performance non-blocking asynchronous programming on JVM using Reactor project (using Disruptor/RingBuffer);
3. Implementig reactive hot event streams processing with Reactor: Flux & Mono, Processors;
4. End-to-end reactive web applications and services: Reactor IO (REST, WebSocket) + RxJS + Angular 2;
5. IPTPI robot demo – reactive hot event streams processing on Raspberry Pi 2 + Arduino with embedded and mobile interfaces: http://robolearn.org/
For the lecturer: Trayan Iliev
– founder and manager of IPT – Intellectual Products & Technologies (http://iproduct.org/) – company for IT trainings and consultancy, specialized in Java, Fullstack JavaScipt, web and mobile technologies
– 15+ years training and consulting experience
– lecturer on the conferences, organized by BGJUG and BGOUG – 9 presentations
– organizer of hackathons on Java robotics & IoT in Sofia and Plovdiv
– presenter on international developer conferences: jPrime, jPofessionals, Voxxed Days
OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs
AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead?
At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them.
In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.
Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...confluent
Watch this talk here: https://www.confluent.io/online-talks/scaling-security-on-100s-of-millions-of-mobile-devices-using-kafka-and-scylla-on-demand
Join mobile cybersecurity leader Lookout as they talk through their data ingestion journey.
Lookout enables enterprises to protect their data by evaluating threats and risks at post-perimeter endpoint devices and providing access to corporate data after conditional security scans. Their continuous assessment of device health creates a massive amount of telemetry data, forcing new approaches to data ingestion. Learn how Lookout changed its approach in order to grow from 1.5 million devices to 100 million devices and beyond, by implementing Confluent Platform and switching to Scylla.
Big Data Streams Architectures. Why? What? How?Anton Nazaruk
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
Building and deploying LLM applications with Apache AirflowKaxil Naik
Behind the growing interest in Generate AI and LLM-based enterprise applications lies an expanded set of requirements for data integrations and ML orchestration. Enterprises want to use proprietary data to power LLM-based applications that create new business value, but they face challenges in moving beyond experimentation. The pipelines that power these models need to run reliably at scale, bringing together data from many sources and reacting continuously to changing conditions.
This talk focuses on the design patterns for using Apache Airflow to support LLM applications created using private enterprise data. We’ll go through a real-world example of what this looks like, as well as a proposal to improve Airflow and to add additional Airflow Providers to make it easier to interact with LLMs such as the ones from OpenAI (such as GPT4) and the ones on HuggingFace, while working with both structured and unstructured data.
In short, this shows how these Airflow patterns enable reliable, traceable, and scalable LLM applications within the enterprise.
https://airflowsummit.org/sessions/2023/keynote-llm/
Flink vs. Spark: this is the slide deck of my talk at the 2015 Flink Forward conference in Berlin, Germany, on October 12, 2015. In this talk, we tried to compare Apache Flink vs. Apache Spark with focus on real-time stream processing. Your feedback and comments are much appreciated.
Getting real-time analytics for devices/application/business monitoring from trillions of events and petabytes of data like companies Netflix, Uber, Alibaba, Paypal, Ebay, Metamarkets do.
A distributed system in its most simplest definition is a group of computers working together as to
appear as a single computer to the end-user. These machines have a shared state, operate
concurrently and can fail independently without affecting the whole system’s uptime.
This is in line with ever-growing technological expansion of the world, distributed systems are
becoming more and more widespread. Take a look at the increasing number of available
computer technologies/innovation around, this is sporadically increasing, and this result in
intense computational requirement.
Yeah, Moore’s law proposed more computing power by fitting more transistors (which
approximately doubles every two years) into a simple chip using cost-efficient approach - cool,
but over the past 5 years, there has been little deviation from this - ability to scale horizontally
and not just vertically alone.
UnConference for Georgia Southern Computer Science March 31, 2015Christopher Curtin
I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...GeeksLab Odessa
4.6.16 AI&BigData Lab
Upcoming events: goo.gl/I2gJ4H
Как устроить анализ данных 40 млн. человек за 5 лет так, чтобы это выглядело почти в реальном времени.
International Journal of Engineering Research and Applications (IJERA) is an open access online peer reviewed international journal that publishes research and review articles in the fields of Computer Science, Neural Networks, Electrical Engineering, Software Engineering, Information Technology, Mechanical Engineering, Chemical Engineering, Plastic Engineering, Food Technology, Textile Engineering, Nano Technology & science, Power Electronics, Electronics & Communication Engineering, Computational mathematics, Image processing, Civil Engineering, Structural Engineering, Environmental Engineering, VLSI Testing & Low Power VLSI Design etc.
Proactive ops for container orchestration environmentsDocker, Inc.
Break -> inspect -> fix is the Ops workflow for infrastructure stacks of the past. Distributed infrastructure and applications claim to be the new generation, but why is it so much more painful to maintain and troubleshoot them? Much of the pain comes from outdated operational models relying on reactive or, worse yet, manual monitoring and Ops.
This talk lays out a proactive Ops model for container infrastructure. By focusing on event monitoring, infrastructure state monitoring, trend analysis, and distributed log collection, a proactive Ops model delivers observability for distributed apps that was not possible before. Using real-world examples from Swarm and Kubernetes, we'll demonstrate the tools used and how we relieve Ops pain in container orchestration.
Learning Programming Using Robots - Sofia University Conference 2018 Trayan Iliev
Learn information technologies by creating your own robots and IoT projects. Robotics and IoT offer rich opportunities for practical and active learning of core information technologies, programming languages and software architectures. Presentation includes examples of teaching practices and robotics projects, and offers suggestions why and how to use them to achieve better students' motivation, engagement, creativity, and connection between theory and practice.
Active Learning Using Connected Things - 2018 (in Bulgarian)Trayan Iliev
Learn about active learning methods and practices using Robotics, IoT, and "smart things" projects. Includes examples of teaching practices and robotics projects, and offers suggestions why and how to use them to achieve better students' motivation, engagement, creativity, and connection between theory and practice. Several blended learning models are compared - Flipped Classroom, Stations/Labs Rotation, Flex model. Project support for individual learning styles is discussed.
Fog Computing – between IoT Devices and The Cloud presentation covers following topics:
- Edge, Fog, Mist & Cloud Computing
- Fog domains and fog federation, wireless sensor networks, - multi-layer IoT architecture
- Fog computing standards and specifications
- Practical use-case scenarios & advantages of fog
- Fog analytics and intelligence on the edge
- Technologies for distributed asynchronous event processing - and analytics in real time
- Lambda architecture – Spark, Storm, Kafka, Apex, Beam, Spring - Reactor & WebFlux
- Eclipse IoT platform
Presentation discusses the best practices when writing higher order components (HOCs), and presents examples how to write them according to React recommendations.
Topics include: maximizing HOC factories composability, using ES7 decorators, wrapping context DI in HOCs, handling async state changes using RxJS Observanels and separation of concerns between presentation and business logic components. Examples (@GitHub projects - referenced in the presentation) are given using react, redux, react-router-redux, redux-observable, recompose, and reselect.
IPT presentation @ jProfessionals 2016 on Java and JavaScipt Reactive Robotics and IoT including: Domain Driven Design (DDD), high-performance reactive micro-services development using Spring Reactor, state-of-the-art component-based client side MVVM implementation with Angular 2, ngrx (Redux pattern), TypeScript and reactive WebSockets.
Presentation is highlighting novelties in SPA development with Angular 2 (+Ionic 2 demo) with real code examples.
We created together simple Ng2 application with Angular CLI.
All the code is available on GitHub (link to demos is at the end of presentation).
Prerequisites:
1. Install NodeJS. It is better to install version 6 or 4x. Read about NPM.
2. Install TypeScript + editor (Visual Studio Code or Sublime 3).
3. Install Angular 2 Command Line Interface (Angular CLI):
npm install -g angular-cli
MVC 1.0 is an action-oriented framework building on experience with previous frameworks such as Struts, Spring MVC, VRaptor etc. It is based on JAX-RS, CDI and BeanValidation JavaEE technologies and provides a standard, view specification neutral way to build web applications. Among supported view template frameworks are: JSP, Facelets, Freemarker, Handlebars, Jade, Mustache, Velocity, Thymeleaf, etc.
NOTE: MVC 1.0 JavaEE 8 API Specification is in early draft stage, and is subject to change based on open community process.
IPT High Performance Reactive Programming with JAVA 8 and JavaScriptTrayan Iliev
Presentation @ jProfessionals BGJUG Conference
Sofia, November 22, 2015 by IPT – IT Education Evolved, High Performance Reactive Programming Workshop - Dec 15-17,2015 http://iproduct.org/en/course-reactive-java-js/
You are welcome to join us!
Low-latency, high-throughput reactive and functional programming in Java using Spring Reactor, RxJava, RxJS, Facebook React, Angular 2, Reactive Streams, Disruptor (ring buffer), Reactor & Proactor design patterns, benchmarking & comparison of concurrency implementations. December 15 - 17, 2015 - Workshop: High Performance Reactive Programming with JAVA 8 and JavaScript - http://iproduct.org/en/course-reactive-java-js/
IPT Workshops on Java Robotics and IoTTrayan Iliev
Learn how to build your own robot & how to program it in Java with IPT workshops & hackathons. Two small robots: LeJaRo - "the leJOS java robot" & IPTPI - Raspberry Pi 2 enabled robot with higher processing capabilities + distance US and optical sensors, line follow sensor array, cameras, encoders and more. Get first hand experience and jump start in JAVA robotics. Find more on http://robolearn.org/. Children of ALL AGES are Welcome :)
The presentation shows some code examples for programing two small robots (Lego® & Raspberry Pi®) in Java ™ – http://iproduct.org/en/, https://github.com/iproduct/course-social-robotics/wiki/Lectures
Novelties in Java EE 7: JAX-RS 2.0 + IPT REST HATEOAS Polling Demo @ BGOUG Co...Trayan Iliev
Presentation shows by example (IPT Polling Demo JAXRS20 HATEOAS, https://github.com/iproduct/IPT-Polling-Demo-JAXRS20-HATEOAS/wiki) the novelties in JAX-RS 2.0 and REST HATEOAS:
- Standardized REST Client API;
- Client and server-side asynchronous HTTP request processing;
- Integration of declarative validation using JSR 349: Bean Validation 1.1;
- Improved server-suggested content negotiation;
- Aspect-oriented extensibility of request/response processing using Filters and Interceptors;
- Dynamic extension registration using DynamicFeature interface;
- Hypermedia As The Engine Of Application State (HATEOAS) REST architectural constraint support using state transition links (support for new HTTP Link header as well as JAXB serialization of resource links).
[IPT, http://iproduct.org]
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeAftab Hussain
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
In the ever-evolving landscape of technology, enterprise software development is undergoing a significant transformation. Traditional coding methods are being challenged by innovative no-code solutions, which promise to streamline and democratize the software development process.
This shift is particularly impactful for enterprises, which require robust, scalable, and efficient software to manage their operations. In this article, we will explore the various facets of enterprise software development with no-code solutions, examining their benefits, challenges, and the future potential they hold.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Atelier - Innover avec l’IA Générative et les graphes de connaissancesNeo4j
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Understanding Nidhi Software Pricing: A Quick Guide 🌟
Choosing the right software is vital for Nidhi companies to streamline operations. Our latest presentation covers Nidhi software pricing, key factors, costs, and negotiation tips.
📊 What You’ll Learn:
Key factors influencing Nidhi software price
Understanding the true cost beyond the initial price
Tips for negotiating the best deal
Affordable and customizable pricing options with Vector Nidhi Software
🔗 Learn more at: www.vectornidhisoftware.com/software-for-nidhi-company/
#NidhiSoftwarePrice #NidhiSoftware #VectorNidhi
2. 2
Trademarks
Oracle®, Java™ and JavaScript™ are trademarks or registered
trademarks of Oracle and/or its affiliates.
Other names may be trademarks of their respective owners.
3. 3
Disclaimer
All information presented in this document and all supplementary
materials and programming code represent only my personal opinion
and current understanding and has not received any endorsement or
approval by IPT - Intellectual Products and Technologies or any third
party. It should not be taken as any kind of advice, and should not be
used for making any kind of decisions with potential commercial impact.
The information and code presented may be incorrect or incomplete. It is
provided "as is", without warranty of any kind, express or implied,
including but not limited to the warranties of merchantability, fitness for a
particular purpose and non-infringement. In no event shall the author or
copyright holders be liable for any claim, damages or other liability,
whether in an action of contract, tort or otherwise, arising from, out of or
in connection with the information, materials or code presented or the
use or other dealings with this information or programming code.
5. 5
Stream Processing with JAVA 9
Stream based data / event / message processing for
real-time distributed SOA / microservice / database
architectures.
PUSH (hot) and PULL (cold) event streams in Java
CompletableFuture & CompletionStage non-blocking,
asynchronous hot event stream composition
Reactive programming. Design patterns. Reactive
Streams (java.util.concurrent.Flow)
Novelties in Java 9 CompletableFuture
Examples for (reactive) hot event streams processing
6. Where to Find the Demo Code?
6
CompletableFuture and Flow demos are available
@ GitHub:
https://github.com/iproduct/reactive-demos-java-9
7. Data / Event / Message Streams
7
“Conceptually, a stream is a (potentially never-ending)
flow of data records, and a transformation is an
operation that takes one or more streams as input,
and produces one or more output streams as a
result.”
Apache Flink: Dataflow Programming Model
8. Data Stream Programming
8
The idea of abstracting logic from execution is hardly
new -- it was the dream of SOA. And the recent
emergence of microservices and containers shows that
the dream still lives on.
For developers, the question is whether they want to
learn yet one more layer of abstraction to their coding.
On one hand, there's the elusive promise of a common
API to streaming engines that in theory should let you
mix and match, or swap in and swap out.
Tony Baer (Ovum) @ ZDNet - Apache Beam and Spark:
New coopetition for squashing the Lambda Architecture?
9. Lambda Architecture - I
9
https://commons.wikimedia.org/w/index.php?curid=34963986, By Textractor - Own work, CC BY-SA 4
10. Lambda Architecture - II
10
https://commons.wikimedia.org/w/index.php?curid=34963987, By Textractor - Own work, CC BY-SA 4
11. Lambda Architecture - III
11
Data-processing architecture designed to handle
massive quantities of data by using both batch- and
stream-processing methods
Balances latency, throughput, fault-tolerance, big data,
real-time analytics, mitigates the latencies of map-
reduce
Data model with an append-only, immutable data
source that serves as a system of record
Ingesting and processing timestamped events that are
appended to existing events. State is determined from
the natural time-based ordering of the data.
12. Druid Distributed Data Store (Java)
12
https://commons.wikimedia.org/w/index.php?curid=33899448 By Fangjin Yang - sent to me personally, GFDL
Apache
ZooKeeper
MySQL /
PostgreSQL
HDFS /
Amazon S3
13. Lambda Architecture: Projects - I
13
Apache Spark is an open-source
cluster-computing framework. Spark
Streaming leverages Spark Core's
fast scheduling capability to perform
streaming analytics. Spark MLlib - a
distributed machine learning lib.
Apache Storm is a distributed
stream processing computation
framework – uses streams as DAG
Apache Apex™ unified stream and
batch processing engine.
17. Example: Internet of Things (IoT)
17
CC BY 2.0, Source:
https://www.flickr.com/photos/wilgengebroed/8249565455/
Radar, GPS, lidar for navigation and obstacle
avoidance ( 2007 DARPA Urban Challenge )
20. 20
Performance is about 2 things (Martin Thompson –
http://www.infoq.com/articles/low-latency-vp ):
– Throughput – units per second, and
– Latency – response time
Real-time – time constraint from input to response
regardless of system load.
Hard real-time system if this constraint is not honored then
a total system failure can occur.
Soft real-time system – low latency response with little
deviation in response time
100 nano-seconds to 100 milli-seconds. [Peter Lawrey]
What's High Performance?
21. 21
Low garbage by reusing existing objects + infrequent GC
when application not busy – can improve app 2 - 5x
JVM generational GC startegy – ideal for objects living very
shortly (garbage collected next minor sweep) or be immortal
Non-blocking, lockless coding or CAS
Critical data structures – direct memory access using
DirectByteBuffers or Unsafe => predictable memory layout
and cache misses avoidance
Busy waiting – giving the CPU to OS kernel slows program
2-5x => avoid context switches
Amortize the effect of expensive IO - blocking
Low Latency: Things to Remember
22. 22
Non-blocking (synchronous) implementation is 2 orders of
magnitude better then synchronized
We should try to avoid blocking and especially contended
blocking if want to achieve low latency
If blocking is a must we have to prefer CAS and optimistic
concurrency over blocking (but have in mind it always
depends on concurrent problem at hand and how much
contention do we experience – test early, test often,
microbenchmarks are unreliable and highly platform dependent
– test real application with typical load patterns)
The real question is: HOW is is possible to build concurrency
without blocking?
Mutex Comparison => Conclusions
23. 23
Queues typically use either linked-lists or arrays for the
underlying storage of elements. Linked lists are not
„mechanically sympathetic” – there is no predictable
caching “stride” (should be less than 2048 bytes in each
direction).
Bounded queues often experience write contention on
head, tail, and size variables. Even if head and tail
separated using CAS, they usually are in the same cache-
line.
Queues produce much garbage.
Typical queues conflate a number of different concerns –
producer and consumer synchronization and data storage
Blocking Queues Disadvantages
[http://lmax-exchange.github.com/disruptor/files/Disruptor-1.0.pdf]
24. 24
CPU Cache – False Sharing
Core 2 Core NCore 1 ...
Registers
Execution Units
L1 Cache A | | B |
L2 Cache A | | B |
L3 Cache A | | B |
DRAM Memory
A | | B |
Registers
Execution Units
L1 Cache A | | B |
L2 Cache A | | B |
25. Tracking Complexity
25
We need tools to cope with all that complexity inherent in
robotics and IoT domains.
Simple solutions are needed – cope with problems through
divide and concur on different levels of abstraction:
Domain Driven Design (DDD) – back to basics:
domain objects, data and logic.
Described by Eric Evans in his book:
Domain Driven Design: Tackling Complexity in the Heart of
Software, 2004
26. Domain Driven Design
26
Main concepts:
Entities, value objects and modules
Aggregates and Aggregate Roots [Haywood]:
value < entity < aggregate < module < BC
Aggregate Roots are exposed as Open Host Services
Repositories, Factories and Services:
application services <-> domain services
Separating interface from implementation
27. Microservices and DDD
27
Actually DDD require additional efforts (as most other
divide and concur modeling approaches :)
Ubiquitous language and Bounded Contexts
DDD Application Layers:
Infrastructure, Domain, Application, Presentation
Hexagonal architecture :
OUTSIDE <-> transformer <->
( application <-> domain )
[A. Cockburn]
28. Imperative and Reactive
28
We live in a Connected Universe
... there is hypothesis that all
the things in the Universe are
intimately connected, and you
can not change a bit without
changing all.
Action – Reaction principle is
the essence of how Universe
behaves.
29. Imperative and Reactive
Reactive Programming: using static or dynamic data
flows and propagation of change
Example: a := b + c
Functional Programming: evaluation of mathematical
functions,
➢ Avoids changing-state and mutable data, declarative
programming
➢ Side effects free => much easier to understand and
predict the program behavior.
Example: books.stream().filter(book -> book.getYear() > 2010)
.forEach( System.out::println )
30. Functional Reactive (FRP)
30
According to Connal Elliot's (ground-breaking paper @
Conference on Functional Programming, 1997), FRP is:
(a) Denotative
(b) Temporally continuous
32. 32
Message Driven – asynchronous message-passing allows
to establish a boundary between components that ensures
loose coupling, isolation, location transparency, and
provides the means to delegate errors as messages
[Reactive Manifesto].
The main idea is to separate concurrent producer and
consumer workers by using message queues.
Message queues can be unbounded or bounded (limited
max number of messages)
Unbounded message queues can present memory
allocation problem in case the producers outrun the
consumers for a long period → OutOfMemoryError
Scalable, Massively Concurrent
35. Reactive Streams Spec.
35
Reactive Streams – provides standard for
asynchronous stream processing with non-blocking
back pressure.
Minimal set of interfaces, methods and protocols for
asynchronous data streams
April 30, 2015: has been released version 1.0.0 of
Reactive Streams for the JVM (Java API,
Specification, TCK and implementation examples)
Java 9: java.util.concurrent.Flow
36. Reactive Streams Spec.
36
Publisher – provider of potentially unbounded number
of sequenced elements, according to Subscriber(s)
demand.
Publisher.subscribe(Subscriber) => onSubscribe onNext*
(onError | onComplete)?
Subscriber – calls Subscription.request(long) to
receive notifications
Subscription – one-to-one Subscriber ↔ Publisher,
request data and cancel demand (allow cleanup).
Processor = Subscriber + Publisher
37. FRP = Async Data Streams
37
FRP is asynchronous data-flow programming using the
building blocks of functional programming (e.g. map,
reduce, filter) and explicitly modeling time
Used for GUIs, robotics, and music. Example (RxJava):
Observable.from(
new String[]{"Reactive", "Extensions", "Java"})
.take(2).map(s -> s + " : on " + new Date())
.subscribe(s -> System.out.println(s));
Result:
Reactive : on Wed Jun 17 21:54:02 GMT+02:00 2015
Extensions : on Wed Jun 17 21:54:02 GMT+02:00 2015
38. Project Reactor
38
Reactor project allows building high-performance (low
latency high throughput) non-blocking asynchronous
applications on JVM.
Reactor is designed to be extraordinarily fast and can
sustain throughput rates on order of 10's of millions of
operations per second.
Reactor has powerful API for declaring data
transformations and functional composition.
Makes use of the concept of Mechanical Sympathy
built on top of Disruptor / RingBuffer.
45. Hot and Cold Event Streams
45
PULL-based (Cold Event Streams) – Cold streams (e.g.
RxJava Observable / Flowable or Reactor Flow / Mono)
are streams that run their sequence when and if they
are subscribed to. They present the sequence from the
start to each subscriber.
PUSH-based (Hot Event Streams) – Hot streams emit
values independent of individual subscriptions. They
have their own timeline and events occur whether
someone is listening or not. An example of this is
mouse events. A mouse is generating events regardless
of whether there is a subscription. When subscription is
made observer receives current events as they happen.
52. Example: IPTPI - RPi + Ardunio Robot
52
Raspberry Pi 2 (quad-core ARMv7
@ 900MHz) + Arduino Leonardo
cloneA-Star 32U4 Micro
Optical encoders (custom), IR
optical array, 3D accelerometers,
gyros, and compass MinIMU-9 v2
IPTPI is programmed in Java
using Pi4J, Reactor, RxJava, Akka
More information about IPTPI:
http://robolearn.org/iptpi-robot/
53. IPTPI Hot Event Streams Example
53
Encoder
Readings
ArduinoData
Flux
Arduino
SerialData
Position
Flux
Robot
Positions
Command
Movement
Subscriber
RobotWSService
(using Reactor)
Angular 2 /
TypeScript
MovementCommands
54. Futures in Java 8 - I
54
Future (implemented by FutureTask) – represents the
result of an cancelable asynchronous computation.
Methods are provided to check if the computation is
complete, to wait for its completion, and to retrieve the
result of the computation (blocking till its ready).
RunnableFuture – a Future that is Runnable.
Successful execution of the run method causes Future
completion, and allows access to its results.
ScheduledFuture – delayed cancelable action that
returns result. Usually a scheduled future is the result
of scheduling a task with a ScheduledExecutorService
55. Future Use Example
55
Future<String> future = executor.submit(
new Callable<String>() {
public String call() {
return searchService.findByTags(tags);
}
}
);
DoSomethingOther();
try {
showResult(future.get()); // use future result
} catch (ExecutionException ex) { cleanup(); }
56. Futures in Java 8 - II
56
CompletableFuture – a Future that may be explicitly
completed (by setting its value and status), and may
be used as a CompletionStage, supporting dependent
functions and actions that trigger upon its completion.
CompletionStage – a stage of possibly asynchronous
computation, that is triggered by completion of
previous stage or stages (CompletionStages form
Direct Acyclic Graph – DAG). A stage performs an
action or computes value and completes upon
termination of its computation, which in turn triggers
next dependent stages. Computation may be Function
(apply), Consumer (accept), or Runnable (run).
57. CompletableFuture Example - I
57
private CompletableFuture<String>
longCompletableFutureTask(int i, Executor executor) {
return CompletableFuture.supplyAsync(() -> {
try {
Thread.sleep(1000); // long computation :)
} catch (InterruptedException e) {
e.printStackTrace();
}
return i + "-" + "test";
}, executor);
}
59. CompletableFuture Example - III
59
try {
System.out.println(results.get(10, TimeUnit.SECONDS));
} catch (ExecutionException | TimeoutException
| InterruptedException e) {
e.printStackTrace();
}
executor.shutdown();
}
// OR just:
System.out.println(results.join());
executor.shutdown();
Which is better?
60. CompletionStage
60
Computation may be Function (apply), Consumer
(accept), or Runnable (run) – e.g.:
completionStage.thenApply( x -> x * x )
.thenAccept(System.out::print )
.thenRun( System.out::println )
Stage computation can be triggered by completion of
1 (then), 2 (combine), or either 1 of 2 (either)
Functional composition can be applied to stages
themselves instead to their results using compose
handle & whenComplete – support unconditional
computation – both normal or exceptional triggering