Real Time Communication APIs workshop, Elyran Kogan, LivepersonAlan Quayle
Real Time Communication APIs workshop: agent bot / SMS connector. Elyran Kogan, Technical Leader, Liveperson If someone wants to use the LivePerson APIs in order to integrate other product to the LivePerson platform, then this is the session. We'll do an APIs workshop step by step how to build agent bot / SMS connector.
Presented at TADSummit 2016, 15-16 Nov, Lisbon in the afternoon plenary.
Learn about the basics of Postman and APIs. If you're brand new to Postman, or new to APIs, this workshop is the first step towards becoming a proficient API user.
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent
Whether you are deploying a new application in Microservices or transitioning from a monolithic database application to a cloud-ready architecture, you will inevitably face the decision of either creating a service mesh of API’s – or – using an event bus for better durability, reliability and extensibility of your application. If you choose to go the event bus route, Kafka is an excellent choice for several reasons. One key technology not to overlook is Avro Schemas. They provide a definition for your event payload, just like an API, to ensure all of the event consumers can reliably consume the events. They also handle schema evolution as requirements change and much, much more.
In this talk we will discuss all the nuances and considerations around using Avro Schemas for your JSON event payloads. From developer tools, to DevOps approaches, versioning, governance and some “gotchas” we found when working with Avro Schemas and the Confluent Schema Registry.
Kafka Connect: Real-time Data Integration at Scale with Apache Kafka, Ewen Ch...confluent
Many companies are adopting Apache Kafka to power their data pipelines, including LinkedIn, Netflix, and Airbnb. Kafka’s ability to handle high throughput real-time data makes it a perfect fit for solving the data integration problem, acting as the common buffer for all your data and bridging the gap between streaming and batch systems.
However, building a data pipeline around Kafka today can be challenging because it requires combining a wide variety of tools to collect data from disparate data systems. One tool streams updates from your database to Kafka, another imports logs, and yet another exports to HDFS. As a result, building a data pipeline can take significant engineering effort and has high operational overhead because all these different tools require ongoing monitoring and maintenance. Additionally, some of the tools are simply a poor fit for the job: the fragmented nature of the data integration tools ecosystem lead to creative but misguided solutions such as misusing stream processing frameworks for data integration purposes.
We describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector.
eventbrite_kafka_summit_event_logo_v3-035858-edited.png
Real Time Communication APIs workshop, Elyran Kogan, LivepersonAlan Quayle
Real Time Communication APIs workshop: agent bot / SMS connector. Elyran Kogan, Technical Leader, Liveperson If someone wants to use the LivePerson APIs in order to integrate other product to the LivePerson platform, then this is the session. We'll do an APIs workshop step by step how to build agent bot / SMS connector.
Presented at TADSummit 2016, 15-16 Nov, Lisbon in the afternoon plenary.
Learn about the basics of Postman and APIs. If you're brand new to Postman, or new to APIs, this workshop is the first step towards becoming a proficient API user.
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...HostedbyConfluent
Whether you are deploying a new application in Microservices or transitioning from a monolithic database application to a cloud-ready architecture, you will inevitably face the decision of either creating a service mesh of API’s – or – using an event bus for better durability, reliability and extensibility of your application. If you choose to go the event bus route, Kafka is an excellent choice for several reasons. One key technology not to overlook is Avro Schemas. They provide a definition for your event payload, just like an API, to ensure all of the event consumers can reliably consume the events. They also handle schema evolution as requirements change and much, much more.
In this talk we will discuss all the nuances and considerations around using Avro Schemas for your JSON event payloads. From developer tools, to DevOps approaches, versioning, governance and some “gotchas” we found when working with Avro Schemas and the Confluent Schema Registry.
Kafka Connect: Real-time Data Integration at Scale with Apache Kafka, Ewen Ch...confluent
Many companies are adopting Apache Kafka to power their data pipelines, including LinkedIn, Netflix, and Airbnb. Kafka’s ability to handle high throughput real-time data makes it a perfect fit for solving the data integration problem, acting as the common buffer for all your data and bridging the gap between streaming and batch systems.
However, building a data pipeline around Kafka today can be challenging because it requires combining a wide variety of tools to collect data from disparate data systems. One tool streams updates from your database to Kafka, another imports logs, and yet another exports to HDFS. As a result, building a data pipeline can take significant engineering effort and has high operational overhead because all these different tools require ongoing monitoring and maintenance. Additionally, some of the tools are simply a poor fit for the job: the fragmented nature of the data integration tools ecosystem lead to creative but misguided solutions such as misusing stream processing frameworks for data integration purposes.
We describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector.
eventbrite_kafka_summit_event_logo_v3-035858-edited.png
Artsem Semianenko (Adform) - "Flink in action или как приручить белочку"
Slides for presentation: https://www.youtube.com/watch?v=YSI5_RFlcPE
Source: https://github.com/art4ul/flink-demo
Микросервисы со Spring Boot & Spring CloudVitebsk DSC
Spring Framework - один из наиболее часто используемых фреймворков для разработки корпоративных приложений. Множество высокопроизводительных решений уже построено на его основе. Если вы начинаете новый проект на Java, то, вероятнее всего, он также будет использовать Spring Framework.Использование микросервисного подхода позволяет реагировать на изменения требований быстрее за счет упрощения отдельных компонентов и возможности их параллельной разработки. Однако, использование этого подхода также сопряжено и с дополнительными проблемами - развертывание и отладка существенно усложнились, а для совместной работы сервисов необходимы дополнительные инфраструктурные компоненты, такие как, централизованная конфигурации, возможность повторной отправки сообщений или балансировка нагрузки между несколькими запущенными инстансами.Spring Boot изменил подход к разработке приложений, основанных на Spring Framework. Автоконфигурации, предоставляемые стартерами, позволяют сразу приступить к реализации основной функциональности и не тратить время на настройку инфраструктурных компонентов. Spring Cloud развил эту идею и предоставляет готовые стартеры для реализации микросервисных паттернов.
Презентация подготовлена по материалам выступления Александра Бармина на витебской конференции “Developer's Software Conference” (30.11.2019).
Confluent building a real-time streaming platform using kafka streams and k...Thomas Alex
Jeremy Custenborder from Confluent talked about how Kafka brings an event-centric approach to building streaming applications, and how to use Kafka Connect and Kafka Streams to build them.
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server PushLucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and Kafka Streams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used to analyze the real time events in real time and publish events with the live findings.
It covers a brief introduction to Apache Kafka Connect, giving insights about its benefits,use cases, motivation behind building Kafka Connect.And also a short discussion on its architecture.
PHP Conference Japan 2019 Track6-5 Aurimas Niekis - How to Supercharge your PHP Web API
https://phpcon.php.gr.jp/2019/
https://www.youtube.com/watch?v=ZtTvUQCDDTM
Kotlin is a language from the tool gurus at JetBrains. In 2016, after about six years of development, Kotlin reached version 1.0. In 2017 it won the hearts of developers and became an officially supported language for Android.
Kotlin, like Java, is for more than creating Android applications. It can replace or enhance Java most places it is used today including on AWS. AWS Lambda functions sometimes called Serverless Computing, is a service which lets us developers build web services without worrying about configuring servers.
In this session, we will create a lambda service on AWS using Kotlin. Along the way, we will learn what a makes Kotlin an excellent replacement for Java and how simple it is to construct an AWS Lambda function.
A one-hour, intermediate-level Postman learning session geared specifically for developers and testers. We’ll walk you through strategies and tactics for debugging more efficiently. Whether you're just exploring new APIs or developing rigorous API workflows, learn how to work smarter while debugging.
Building Event Streaming Applications with Pac-Man (Ricardo Ferreira, Conflue...HostedbyConfluent
Since Pac-Man was originally released in the '80s, it has been a beacon of fun and joy for people of all ages. What few people know is that this game can also be used to inspire developers on how to build event streaming applications. In this near-zero-slides talk, attendees will get to play the game to generate events. As they play, the presenter will write from scratch a scoreboard using ksqlDB -- an open-source event streaming database built for Apache Kafka.
After building the scoreboard, it will be discussed the different strategies to make the data available elsewhere so any interested service could leverage it with ease. Examples of these services will be provided to monitor in near real-time the scoreboard, revealing whoever is the most proficient Pac-Man player in the room.
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...HostedbyConfluent
Having started with classic monolith applications in the late 90s and adopting a new microservice architecture in 2015, our organization needed a convenient, reliable, and low-cost way to push changes back and forth between them. One that preferably utilized technology already on hand and could exchange information between multiple data stores.
In this session we will explore how Kafka Connect and its various connectors satisfied this need. We will review the two disparate tech stacks we needed to integrate, and the strategies and connectors we used to exchange information. Finally, we will cover some enhancements we made to our own processes including integrating Kafka Connect and its connectors into our CI/CD pipeline and writing tools to monitor connectors in our production environment.
APIdays Paris 2018 - Secure & Manage APIs with GraphQL, Ozair Sheikh, Directo...apidays
Secure & Manage APIs with GraphQL
Ozair Sheikh, Director of API Product Management, IBM
Apply to be a speaker here - https://apidays.typeform.com/to/J1snsg
"In this comprehensive workshop Johnny Tu, Senior Trainer at ServiceRocket, will be covering all the basics of Postman.
1. Introduction to Postman app
2. Postman Concepts & Postman UI
3. Creating & Sending API Requests
4. Organizing Requests into Postman Collections
5. Configuring Postman Variables & Environments
6. Performing Basic API Testing with JavaScript
7. Collaborating through Postman Workspaces"
Fundamentals and Architecture of Apache KafkaAngelo Cesaro
Fundamentals and Architecture of Apache Kafka.
This presentation explains Apache Kafka's architecture and internal design giving an overview of Kafka internal functions, including:
Brokers, Replication, Partitions, Producers, Consumers, Commit log, comparison over traditional message queues.
ClojuTRE2015: Kekkonen - making your Clojure web APIs more awesomeMetosin Oy
Thanks to REST and Swagger, we can build beautiful apis to feed both our browser front-ends and external applications. But, wrapping your Clojure code into resources mostly for your ClojureScript front-end doesn't feel right? Just use RPC? Meet in the middle?
Kekkonen is a small library for managing your (web) apis as commands and queries. No magic, data-driven, un-restful and non-rpc. It's goals are to be small, explicit, extendable and to help enforce your business rules both on the server side and on the ClojureScript frontend. Besides Swagger, it provides run-time context-aware apidocs for Clojure(Script).
Artsem Semianenko (Adform) - "Flink in action или как приручить белочку"
Slides for presentation: https://www.youtube.com/watch?v=YSI5_RFlcPE
Source: https://github.com/art4ul/flink-demo
Микросервисы со Spring Boot & Spring CloudVitebsk DSC
Spring Framework - один из наиболее часто используемых фреймворков для разработки корпоративных приложений. Множество высокопроизводительных решений уже построено на его основе. Если вы начинаете новый проект на Java, то, вероятнее всего, он также будет использовать Spring Framework.Использование микросервисного подхода позволяет реагировать на изменения требований быстрее за счет упрощения отдельных компонентов и возможности их параллельной разработки. Однако, использование этого подхода также сопряжено и с дополнительными проблемами - развертывание и отладка существенно усложнились, а для совместной работы сервисов необходимы дополнительные инфраструктурные компоненты, такие как, централизованная конфигурации, возможность повторной отправки сообщений или балансировка нагрузки между несколькими запущенными инстансами.Spring Boot изменил подход к разработке приложений, основанных на Spring Framework. Автоконфигурации, предоставляемые стартерами, позволяют сразу приступить к реализации основной функциональности и не тратить время на настройку инфраструктурных компонентов. Spring Cloud развил эту идею и предоставляет готовые стартеры для реализации микросервисных паттернов.
Презентация подготовлена по материалам выступления Александра Бармина на витебской конференции “Developer's Software Conference” (30.11.2019).
Confluent building a real-time streaming platform using kafka streams and k...Thomas Alex
Jeremy Custenborder from Confluent talked about how Kafka brings an event-centric approach to building streaming applications, and how to use Kafka Connect and Kafka Streams to build them.
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server PushLucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and Kafka Streams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used to analyze the real time events in real time and publish events with the live findings.
It covers a brief introduction to Apache Kafka Connect, giving insights about its benefits,use cases, motivation behind building Kafka Connect.And also a short discussion on its architecture.
PHP Conference Japan 2019 Track6-5 Aurimas Niekis - How to Supercharge your PHP Web API
https://phpcon.php.gr.jp/2019/
https://www.youtube.com/watch?v=ZtTvUQCDDTM
Kotlin is a language from the tool gurus at JetBrains. In 2016, after about six years of development, Kotlin reached version 1.0. In 2017 it won the hearts of developers and became an officially supported language for Android.
Kotlin, like Java, is for more than creating Android applications. It can replace or enhance Java most places it is used today including on AWS. AWS Lambda functions sometimes called Serverless Computing, is a service which lets us developers build web services without worrying about configuring servers.
In this session, we will create a lambda service on AWS using Kotlin. Along the way, we will learn what a makes Kotlin an excellent replacement for Java and how simple it is to construct an AWS Lambda function.
A one-hour, intermediate-level Postman learning session geared specifically for developers and testers. We’ll walk you through strategies and tactics for debugging more efficiently. Whether you're just exploring new APIs or developing rigorous API workflows, learn how to work smarter while debugging.
Building Event Streaming Applications with Pac-Man (Ricardo Ferreira, Conflue...HostedbyConfluent
Since Pac-Man was originally released in the '80s, it has been a beacon of fun and joy for people of all ages. What few people know is that this game can also be used to inspire developers on how to build event streaming applications. In this near-zero-slides talk, attendees will get to play the game to generate events. As they play, the presenter will write from scratch a scoreboard using ksqlDB -- an open-source event streaming database built for Apache Kafka.
After building the scoreboard, it will be discussed the different strategies to make the data available elsewhere so any interested service could leverage it with ease. Examples of these services will be provided to monitor in near real-time the scoreboard, revealing whoever is the most proficient Pac-Man player in the room.
Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...HostedbyConfluent
Having started with classic monolith applications in the late 90s and adopting a new microservice architecture in 2015, our organization needed a convenient, reliable, and low-cost way to push changes back and forth between them. One that preferably utilized technology already on hand and could exchange information between multiple data stores.
In this session we will explore how Kafka Connect and its various connectors satisfied this need. We will review the two disparate tech stacks we needed to integrate, and the strategies and connectors we used to exchange information. Finally, we will cover some enhancements we made to our own processes including integrating Kafka Connect and its connectors into our CI/CD pipeline and writing tools to monitor connectors in our production environment.
APIdays Paris 2018 - Secure & Manage APIs with GraphQL, Ozair Sheikh, Directo...apidays
Secure & Manage APIs with GraphQL
Ozair Sheikh, Director of API Product Management, IBM
Apply to be a speaker here - https://apidays.typeform.com/to/J1snsg
"In this comprehensive workshop Johnny Tu, Senior Trainer at ServiceRocket, will be covering all the basics of Postman.
1. Introduction to Postman app
2. Postman Concepts & Postman UI
3. Creating & Sending API Requests
4. Organizing Requests into Postman Collections
5. Configuring Postman Variables & Environments
6. Performing Basic API Testing with JavaScript
7. Collaborating through Postman Workspaces"
Fundamentals and Architecture of Apache KafkaAngelo Cesaro
Fundamentals and Architecture of Apache Kafka.
This presentation explains Apache Kafka's architecture and internal design giving an overview of Kafka internal functions, including:
Brokers, Replication, Partitions, Producers, Consumers, Commit log, comparison over traditional message queues.
ClojuTRE2015: Kekkonen - making your Clojure web APIs more awesomeMetosin Oy
Thanks to REST and Swagger, we can build beautiful apis to feed both our browser front-ends and external applications. But, wrapping your Clojure code into resources mostly for your ClojureScript front-end doesn't feel right? Just use RPC? Meet in the middle?
Kekkonen is a small library for managing your (web) apis as commands and queries. No magic, data-driven, un-restful and non-rpc. It's goals are to be small, explicit, extendable and to help enforce your business rules both on the server side and on the ClojureScript frontend. Besides Swagger, it provides run-time context-aware apidocs for Clojure(Script).
Full recorded presentation at https://www.youtube.com/watch?v=2UfAgCSKPZo for Tetrate Tech Talks on 2022/05/13.
Envoy's support for Kafka protocol, in form of broker-filter and mesh-filter.
Contents:
- overview of Kafka (usecases, partitioning, producer/consumer, protocol);
- proxying Kafka (non-Envoy specific);
- proxying Kafka with Envoy;
- handling Kafka protocol in Envoy;
- Kafka-broker-filter for per-connection proxying;
- Kafka-mesh-filter to provide front proxy for multiple Kafka clusters.
References:
- https://adam-kotwasinski.medium.com/deploying-envoy-and-kafka-8aa7513ec0a0
- https://adam-kotwasinski.medium.com/kafka-mesh-filter-in-envoy-a70b3aefcdef
Apache Kafka - Scalable Message-Processing and more !Guido Schmutz
Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target.
This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.
Reducing Microservice Complexity with Kafka and Reactive Streamsjimriecken
My talk from ScalaDays 2016 in New York on May 11, 2016:
Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to:
- Extract non-critical processing from the critical path of your application to reduce request latency
- Provide back-pressure to handle both slow and fast producers/consumers
- Maintain high availability, high performance, and reliable messaging
- Evolve message payloads while maintaining backwards and forwards compatibility.
In this presentation Guido Schmutz talks about Apache Kafka, Kafka Core, Kafka Connect, Kafka Streams, Kafka and "Big Data"/"Fast Data Ecosystems, Confluent Data Platform and Kafka in Architecture.
Matteo Merli and Sijie Guo from Streamlio gave a hands-on workshop on Apache Pulsar. #fast #durable #pubsub #messaging system. A low latency alternative to #kafka.
We present Spark Serving, a new spark computing mode that enables users to deploy any Spark computation as a sub-millisecond latency web service backed by any Spark Cluster. Attendees will explore the architecture of Spark Serving and discover how to deploy services on a variety of cluster types like Azure Databricks, Kubernetes, and Spark Standalone. We will also demonstrate its simple yet powerful API for RESTful SparkSQL, SparkML, and Deep Network deployment with the same API as batch and streaming workloads. In addition, we will explore the "dual architecture": HTTP on Spark. This architecture converts any spark cluster into a distributed web client with the familiar and pipelinable SparkML API. These two contributions provide the fundamental spark communication primitives to integrate and deploy any computation framework into the Spark Ecosystem. We will explore how Microsoft has used this work to leverage Spark as a fault-tolerant microservice orchestration engine in addition to an ETL and ML platform. And will walk through two examples drawn from Microsoft's ongoing work on Cognitive Service composition, and unsupervised object detection for Snow Leopard recognition.
Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.
This is a talk that I gave at the Data Council Berlin Meetup on May 16th, 2019
Abstract:
Stream processing is being rapidly adopted by the enterprise. While in the past, stream processing frameworks mostly provided Java- or Scala-based APIs, stream processing with SQL is growing increasingly popular because it makes stream processing accessible to non-programmers and significantly reduces the effort to solve common tasks.
About three years ago, the Apache Flink community started adding SQL support to process static and streaming data in a unified fashion. Today, Flink SQL powers production systems at Alibaba, Huawei, Lyft, and Uber. Fabian Hueske discusses the current state of Flink’s SQL support and explains the importance of Flink’s unified approach to process static and streaming data. After covering the basics, he shares common real-world use cases ranging from low-latency ETL to pattern detection and demonstrates how easily they can be addressed with Flink SQL.
Flink's Journey from Academia to the ASFFabian Hueske
Apache Flink is a project with a very active, supportive, and continuously growing community. Last year, Flink was among the top ten projects of the Apache Software Foundation with the most traffic on user and development mailing lists. Looking back, Flink started as a research prototype developed by three PhD students at TU Berlin in 2009. In 2014, the developers donated the code base to the ASF and joined the newly founded Apache Flink incubator project. Within three years, Flink grew into a healthy project and gained a lot of momentum.
In my presentation, I will discuss Flink's journey from an academic research project to one of the most active projects of the Apache Software Foundation. I will talk about the academic roots of the project, how the original developers got introduced to the ASF, Flink's incubation phase, and how its community evolved after it graduated and became an ASF top-level project. My talk will focus on the decisions, efforts, and circumstances that helped to grow a vital and welcoming open source community.
Why and how to leverage the power and simplicity of SQL on Apache FlinkFabian Hueske
SQL is the lingua franca of data processing and everybody working with data knows SQL. Apache Flink provides SQL support for querying and processing batch and streaming data. Flink’s SQL support powers large-scale production systems at Alibaba, Huawei, and Uber. Based on Flink SQL, these companies have built systems for their internal users as well as publicly offered services for paying customers. In my talk, I will discuss why you should and how you can (not being Alibaba or Uber) leverage the simplicity and power of SQL on Flink.
I will start exploring the use cases that Flink SQL was designed for and present real-world problems that it can solve. In particular, I'll explain why unified batch and stream processing is important and what it means to run SQL queries on streams of data. After discussing why and when you should use Flink SQL, I will show how to leverage its full potential. Since recently, the Flink community is working on a service that integrates a query interface, (external) table catalogs, and result serving functionality for static, appending, and updating result sets. I will discuss the design and planned features of this query service and how it will enable exploratory batch and streaming queries, ETL pipelines, and live updating query results that serve applications, such as real-time dashboards.
Streaming SQL to unify batch and stream processing: Theory and practice with ...Fabian Hueske
SQL is the lingua franca for querying and processing data. To this day, it provides non-programmers with a powerful tool for analyzing and manipulating data. But with the emergence of stream processing as a core technology for data infrastructures, can you still use SQL and bring real-time data analysis to a broader audience?
The answer is yes, you can. SQL fits into the streaming world very well and forms an intuitive and powerful abstraction for streaming analytics. More importantly, you can use SQL as an abstraction to unify batch and streaming data processing. Viewing streams as dynamic tables, you can obtain consistent results from SQL evaluated over static tables and streams alike and use SQL to build materialized views as a data integration tool.
Fabian Hueske and Shuyi Chen explore SQL’s role in the world of streaming data and its implementation in Apache Flink and cover fundamental concepts, such as streaming semantics, event time, and incremental results. They also share their experience using Flink SQL in production at Uber, explaining how Uber leverages Flink SQL to solve its unique business challenges and how the unified stream and batch processing platform enables both technical or nontechnical users to process real-time and batch data reliably using the same SQL at Uber scale.
Stream Analytics with SQL on Apache FlinkFabian Hueske
Apache Flink's DataStream API is very expressive and gives users precise control over time and state. However, many applications do not require this level of expressiveness and can be implemented more concisely and easily with a domain-specific API.
SQL is undoubtedly the most widely used language for data processing but usually applied in the domain of batch processing. Apache Flink features two relational APIs for unified stream and batch processing, the Table API, a language-integrated relational query API for Scala and Java, and SQL. A Table API or SQL query computes the same result regardless whether it is evaluated on a static file or on a Kafka topic. While Flink evaluates queries on batch input like a conventional query engine, queries on streaming input are continuously processed and their results constantly updated and refined.
In this talk we present Flink’s unified relational APIs, show how streaming SQL queries are processed, and discuss exciting new use-cases.
Stream Analytics with SQL on Apache FlinkFabian Hueske
This presentation describes the semantics of Flink's relational APIs (SQL & Table API) on data streams. The core concept are "Dynamic Tables" which can be queried with regular SQL queries and are constantly and automatically updated.
Taking a look under the hood of Apache Flink's relational APIs.Fabian Hueske
Apache Flink features two APIs which are based on relational algebra, a SQL interface and the so-called Table API, which is a LINQ-style API available for Scala and Java. Relational APIs are interesting because they are easy to use and queries can be automatically optimized and translated into efficient runtime code. Flink offers both APIs for streaming and batch data sources. This talk takes a look under the hood of Flink’s relational APIs. The presentation shows the unified architecture to handle streaming and batch queries and explain how Flink translates queries of both APIs into the same representation, leverages Apache Calcite to optimize them, and generates runtime code for efficient execution. Finally, the slides discuss potential improvements and give an outlook for future extensions and features.
Data Stream Processing with Apache FlinkFabian Hueske
This talk is an introduction into Stream Processing with Apache Flink. I gave this talk at the Madrid Apache Flink Meetup at February 25th, 2016.
The talk discusses Flink's features, shows it's DataStream API and explains the benefits of Event-time stream processing. It gives an outlook on some features that will be added after the 1.0 release.
Quantitative Data AnalysisReliability Analysis (Cronbach Alpha) Common Method...2023240532
Quantitative data Analysis
Overview
Reliability Analysis (Cronbach Alpha)
Common Method Bias (Harman Single Factor Test)
Frequency Analysis (Demographic)
Descriptive Analysis
Data Centers - Striving Within A Narrow Range - Research Report - MCG - May 2...pchutichetpong
M Capital Group (“MCG”) expects to see demand and the changing evolution of supply, facilitated through institutional investment rotation out of offices and into work from home (“WFH”), while the ever-expanding need for data storage as global internet usage expands, with experts predicting 5.3 billion users by 2023. These market factors will be underpinned by technological changes, such as progressing cloud services and edge sites, allowing the industry to see strong expected annual growth of 13% over the next 4 years.
Whilst competitive headwinds remain, represented through the recent second bankruptcy filing of Sungard, which blames “COVID-19 and other macroeconomic trends including delayed customer spending decisions, insourcing and reductions in IT spending, energy inflation and reduction in demand for certain services”, the industry has seen key adjustments, where MCG believes that engineering cost management and technological innovation will be paramount to success.
MCG reports that the more favorable market conditions expected over the next few years, helped by the winding down of pandemic restrictions and a hybrid working environment will be driving market momentum forward. The continuous injection of capital by alternative investment firms, as well as the growing infrastructural investment from cloud service providers and social media companies, whose revenues are expected to grow over 3.6x larger by value in 2026, will likely help propel center provision and innovation. These factors paint a promising picture for the industry players that offset rising input costs and adapt to new technologies.
According to M Capital Group: “Specifically, the long-term cost-saving opportunities available from the rise of remote managing will likely aid value growth for the industry. Through margin optimization and further availability of capital for reinvestment, strong players will maintain their competitive foothold, while weaker players exit the market to balance supply and demand.”
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
2. Problems of the old RPC
service
• Proprietary server/client architecture to
wrap RPCs
• Shortcomings:
– Blocking calls without timeouts
– Poor exception handling
– Error-prone programming abstraction
– Limited scalability
flink.apache.org 1
3. Akka – Actor-based
concurrency
• Framework to write distributed and highly
reactive applications
• Actor-based concurrency inspired by Erlang’s
actor model
• Open source, Apache License 2.0
2flink.apache.org
5. How can Akka help us?
• Nice programming abstraction
• Asynchronous messages with callbacks
• Location transparency of actors
• Fault-tolerant and self-healing
• High throughput and scalability
• Smaller code base to maintain
flink.apache.org 5
6. Akka’s integration with Flink
• Job/TaskManager and JobClient are
actors
• Old RPC are replaced by messages
• Easy to extend functionality by defining
new messages
flink.apache.org 6
JobClient
JobManag
er
TaskManag
er
TaskManag
er
Job
7. Possible improvements
• Make more components an actor to
increase parallelism
– Scheduler/InstanceManager
– ExecutionGraph/Execution
• Make the system more reactive by
asynchronous calls
• Use Akka persistence to recover state of
failed actors
flink.apache.org 7
----- Meeting Notes (28/01/15 14:44) -----
Presentation is about why we replaced the old RPC service with Akka.
----- Meeting Notes (28/01/15 14:44) -----
- Communication between distributed components was realized by a proprietary RPC service
- Distributed components: JobManager, TaskManager, JobClient
- Server/Client architecture
- No callbacks --> polling + high latencies
- Lost exceptions
-