Data / Streaming / Microservices Platform with Devops
•
1 like•125 views
This document describes a data/streaming/microservices platform with DevOps capabilities. It includes components for data collection, a unified log store, batch and stream processing, management services, and interactive querying. Typical applications are described along with the platform layers and DevOps perspective. Potential future additions are also outlined.
Report
Share
Report
Share
Download to read offline
Recommended
Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processi...
Watch the webcast here: https://videos.confluent.io/watch/yaWXJTKELvUGfPx9cnmhZ8
Speaker: Kenneth Cheung
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したか
岡田遥来 ジャコ(@ocadaruma) LINE株式会社
Apache Kafka Meetup Japan #5 @LINEでの発表資料です
https://kafka-apache-jp.connpass.com/event/104465/
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...
As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.
Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryso...
KSQL is an easy-to-use and easy-to-understand streaming SQL engine for Apache Kafka built on top of Kafka Streams. The ability to write streaming applications using only SQL makes Apache Kafka available to a whole range of new developers and potential use cases, either as a stand-alone solution, or as a single component to a broader Kafka Streams implementation. Inspired by a customer project now in production, experience the lifecycle of a streaming application developed using KSQL and Kafka Streams. With Apache Gradle as our build framework, we’ll explore the open-source Gradle plugin we built during this project to improve developer efficiency and automate the deployment of KSQL pipelines, user-defined functions, and Kafka Streams microservices.
We’ll demonstrate the deployment process live, and discuss design decisions around incorporating SQL-based processes into an overall streaming application.
Key Takeaways
1. KSQL is a natural choice for expressing data-driven applications, but it may not naturally fit into established DevOps processes and automations.
2. We built an open-source Gradle plugin to handle all aspects of deploying a Kafka-based streaming application: KSQL pipelines, KSQL user-defined functions, and Kafka Streams microservices.
3. KSQL pipelines can be deployed using either a server start script, or the KSQL REST API, and our Gradle plugin fully supports both options.
Recommended
Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processi...
Watch the webcast here: https://videos.confluent.io/watch/yaWXJTKELvUGfPx9cnmhZ8
Speaker: Kenneth Cheung
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したか
岡田遥来 ジャコ(@ocadaruma) LINE株式会社
Apache Kafka Meetup Japan #5 @LINEでの発表資料です
https://kafka-apache-jp.connpass.com/event/104465/
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...
As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.
Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryso...
KSQL is an easy-to-use and easy-to-understand streaming SQL engine for Apache Kafka built on top of Kafka Streams. The ability to write streaming applications using only SQL makes Apache Kafka available to a whole range of new developers and potential use cases, either as a stand-alone solution, or as a single component to a broader Kafka Streams implementation. Inspired by a customer project now in production, experience the lifecycle of a streaming application developed using KSQL and Kafka Streams. With Apache Gradle as our build framework, we’ll explore the open-source Gradle plugin we built during this project to improve developer efficiency and automate the deployment of KSQL pipelines, user-defined functions, and Kafka Streams microservices.
We’ll demonstrate the deployment process live, and discuss design decisions around incorporating SQL-based processes into an overall streaming application.
Key Takeaways
1. KSQL is a natural choice for expressing data-driven applications, but it may not naturally fit into established DevOps processes and automations.
2. We built an open-source Gradle plugin to handle all aspects of deploying a Kafka-based streaming application: KSQL pipelines, KSQL user-defined functions, and Kafka Streams microservices.
3. KSQL pipelines can be deployed using either a server start script, or the KSQL REST API, and our Gradle plugin fully supports both options.
Demystifying Event-Driven Architectures with Apache Kafka | Bogdan Sucaciu, P...
Event-Driven Architectures (EDA ) are perceived as mythical objects that instantly transform your systems into ""real-time"" ones! BUT, come to think of it, aren't they already ""real-time""? I mean, adding an item to the cart is pretty much instant in ( most ) webshops.
In fact, EDA solves an entirely different set of problems and with the help of Apache Kafka, we will walk through the (re)evolution path. Microservices are easy to get started with, but once we do, we keep stumbling across the same issues: data access, consistency, and failures ( sounds familiar? ).
The solution? Patterns, patterns, patterns … You’ve probably heard about terms such as “Event Notification”, “Event-carried State Transfer”, or even “Event Sourcing”, but how can they be used to solve our problems? And more importantly, how can we use Apache Kafka to take advantage of these patterns?
I guess we will find out soon!
The API Journey: from REST to GraphQL
This document discusses the evolution of APIs from REST to GraphQL. It begins by describing RESTful microservices and their limitations in needing multiple requests to retrieve related data. Next, it introduces GraphQL as providing a better solution through allowing clients to request specific data in a single query using a defined schema. The document provides an example GraphQL query and describes how Apollo GraphQL can be used to create a GraphQL server and support federated data. It concludes by noting GraphQL addresses issues with REST by being faster, requiring less data transfer and hosting costs, and through integrated development tools.
Getting started with Azure Event Hubs and Stream Analytics services
The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.
Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...
Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better.
The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka.
We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices.
Some potential takeaways for the discerning audience:
1. Opportunities in a platform approach to Event Driven Architecture in the enterprise
2. Adopting a product mindset around Data & Event Streams
3. Seeking harmony with allied enterprise applications
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...
This document discusses testing event-driven architectures. It begins by defining common event-driven architecture patterns like event notifications and event sourcing. It then discusses brokering the complexity of event-driven architectures by describing how events are communicated between producers and consumers via channels. The document outlines what information should be included in events like payloads and headers. It also discusses the difference between orchestration and choreography in event-driven systems. It provides an example of how events can be used to mediate changes within a system using order validation. Finally, it demonstrates how to test event-driven architectures using specifications and discusses accelerating API quality through testing tools that support multiple protocols and definitions.
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.
Westpac Bank Tech Talk 2: Introduction to Streaming Data and Stream Processin...
Watch the webcast here: https://videos.confluent.io/watch/P5up2YQX9QdVMhmYfsXy7Q
Speaker: Brett Randall
Logging in The World of DevOps
This document summarizes Cecep Mahbub's experience working on logging infrastructure for various companies and discusses best practices for centralized logging. It outlines Cecep's background maintaining logging systems at KMK and BBM, and their current role setting up Bukalapak's logging infrastructure on Google Cloud Platform. Key topics covered include common uses of logs, benefits of a centralized logging server, examples of logging servers and SaaS options, and integrating logs into DevOps processes.
The API Journey: GraphQL Specification and Implementation
The document discusses GraphQL, including its specification and implementation. It describes GraphQL's basic schema definition (1a), additional features like directives, scalars, and enums (1b), and mutations and subscriptions (1c, 1d). It then covers implementing a simple GraphQL server (2a), using federation for multi-tier architectures (2b), and code generation for speeding development (2c).
Simplifying Event Streaming: Tools for Location Transparency and Data Evoluti...
At Under Armour Connected Fitness, we’ve built an event streaming platform on top of Kafka and the Confluent stack that makes it easy for developers to produce and consume schema-based events without requiring direct knowledge of Kafka. We are constantly trying to improve the developer experience. The platform consists of multiple federated Kafka clusters, a schema registry, a topology service, an archiver and specialized client libraries and Web / CLI tools that assist developers with producer and consumer workflows.
In this talk, we will take a deeper dive into the design and implementation of a Scala/Java implementation of our client library that allows developers to produce or consume events without worrying about the underlying infrastructure and their location while enjoying the benefits of data compatibility through schemas. We’ll also look at an HTTP based client proxy that exposes the same API but for languages without our native support. Finally, we’ll walk through Web and CLI tools we built to make working with the platform easier.
The content of this talk will be primarily aimed at software developers looking for ideas on how to build Kafka client tools that allow producer/consumer interactions protected by schema-based event definitions while hiding details of the underlying infrastructure.
Distributed Data Quality - Technical Solutions for Organizational Scaling
The document discusses Yelp's distributed data architecture and quality solutions for organizational scaling. It describes how Yelp connects over 500 engineers across many services through shared data stored in databases like MySQL, Cassandra and Elasticsearch. The data is ingested through Kafka and processed using tools like Flink. Schematizer provides documentation, discovery and ownership of data. It also enables data lineage tracking and auditing to ensure quality as the data is transformed and loaded into data lakes and warehouses. The goal is to provide reliable, up-to-date shared data to align teams and enable autonomy through self-service data access.
Elastic Stack Basic - All The Capabilities in 6.3!
In Elastic Stack 6.3 we have taken a bold step and opened the X-Pack code for viewing, comment, and bug tracking. In addition, we’ve now included all of the FREE X-Pack Features in the default 6.3 distribution. Learn how you can benefit from these changes to accelerate your projects and get started with the Elastic Stack today!
From logging to monitoring to reactive insights - C Schneider
OSGi Community Event 2017 Presentation by Christian Schneider [Adobe]
In a highly distributed world it is crucial to have the best possible insights into your application.
In the old days we achieved this using plain logging and jmx. This does not scale well to highly distributed processing like in microservices though.
This talk shows how to pimp you OSGi based systems with top notch monitoring and tracing capabilities. Using Apache Karaf Decanter we let karaf instances log into a common elastic search instance with almost zero effort and get insights using kibana. The Apache CXF REST and SOAP integrations allow to log and correlate service requests and replies.
Building on top of this we use the OSGi based reactive components framework and reactor.io streams to provide complex event processing capabilties. A typical use case for this is to describe and monitor service level agreements.
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
This document discusses Airbnb's use of Kafka as the foundation for its highly reliable logging system. It describes the types of data Airbnb collects, including product events, database exports, service events, and derived data. Airbnb uses a simple logging pipeline where events are delivered reliably through Kafka in real time. Key components of its production logging pipeline include Jitney for standardized messaging, a central schema repository, client SDKs, producer and consumer agents, and a self-service portal. The pipeline provides continuous integration through schema authoring, deployment, implementation, processing and storage, and monitoring. It handles large volumes of data reliably with 150 brokers processing 1 million messages per second and 10 billion events collected daily with very low
Event Sourcing, Stream Processing and Serverless (Ben Stopford, Confluent) K...
In this talk we'll look at the relationship between three of the most disruptive software engineering paradigms: event sourcing, stream processing and serverless. We'll debunk some of the myths around event sourcing. We'll look at the inevitability of event-driven programming in the serverless space and we'll see how stream processing links these two concepts together with a single 'database for events'. As the story unfolds we'll dive into some use cases, examine the practicalities of each approach-particularly the stateful elements-and finally extrapolate how their future relationship is likely to unfold. Key takeaways include: The different flavors of event sourcing and where their value lies. The difference between stream processing at application- and infrastructure-levels. The relationship between stream processors and serverless functions. The practical limits of storing data in Kafka and stream processors like KSQL.
Building Microservices with Apache Kafka by Colin McCabe
Abstract:- Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics.
ksqlDB Workshop
Watch the webcast here: https://videos.confluent.io/watch/woobbdupdrwpJCHZKEwF5E?
Speaker: Patrick Druley
Developing custom transformation in the Kafka connect to minimize data redund...
Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost.
in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.
Down the event-driven road: Experiences of integrating streaming into analyti...
The requirements of many modern data platforms develop along two directions: (1) Low latency, i.e. the shift from batch-oriented to event-driven processes, which facilitate much more timely and reactive insights; and (2) complex analytics, i.e. the ability to efficiently apply analytic functions or models to the incoming data streams. However, many companies don't start from scratch, and already have well-established data infrastructure and processes with various degrees of affinity and compatibility to these novel paradigms. Based on extensive experience of building data platforms with customers, we describe in this talk some key challenges and aspects of introducing streaming-based approaches in real-world productive environments. These include e.g. integrating existing batch-oriented data sources and APIs, checking consistency when using event sourcing to exchange data, and building realtime analytical visualizations. For all cases, architectural options are discussed, and the final solution is explained, including technologies like Apache Nifi, Airflow, Phoenix, Druid and the Confluent Platform. We close the talk by describing non-technical aspects like building up an event-driven mindset among analysts.
Speaker: Dr. Dominik Benz, inovex
Event: Confluent Meetup, 08.10.2018
Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/
Mehr Tech-Artikel: https://www.inovex.de/blog/
Apache Kafka as Event Streaming Platform for Microservice Architectures
This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The Confluent Platform adds further components such as a Schema Registry, REST Proxy, KSQL, Clients for different programming languages and Connectors for different technologies.
The session discusses how tech giants like LinkedIn, Ebay or Airbnb leverage Apache Kafka as event streaming platform to solve various different business problems and how to create a scalable, flexible microservice architecture. A live demo shows how you can easily process and analyze streams of events using Apache Kafka and KSQL.
More Related Content
What's hot
Demystifying Event-Driven Architectures with Apache Kafka | Bogdan Sucaciu, P...
Event-Driven Architectures (EDA ) are perceived as mythical objects that instantly transform your systems into ""real-time"" ones! BUT, come to think of it, aren't they already ""real-time""? I mean, adding an item to the cart is pretty much instant in ( most ) webshops.
In fact, EDA solves an entirely different set of problems and with the help of Apache Kafka, we will walk through the (re)evolution path. Microservices are easy to get started with, but once we do, we keep stumbling across the same issues: data access, consistency, and failures ( sounds familiar? ).
The solution? Patterns, patterns, patterns … You’ve probably heard about terms such as “Event Notification”, “Event-carried State Transfer”, or even “Event Sourcing”, but how can they be used to solve our problems? And more importantly, how can we use Apache Kafka to take advantage of these patterns?
I guess we will find out soon!
The API Journey: from REST to GraphQL
This document discusses the evolution of APIs from REST to GraphQL. It begins by describing RESTful microservices and their limitations in needing multiple requests to retrieve related data. Next, it introduces GraphQL as providing a better solution through allowing clients to request specific data in a single query using a defined schema. The document provides an example GraphQL query and describes how Apollo GraphQL can be used to create a GraphQL server and support federated data. It concludes by noting GraphQL addresses issues with REST by being faster, requiring less data transfer and hosting costs, and through integrated development tools.
Getting started with Azure Event Hubs and Stream Analytics services
The total amount of data in the world almost doubles every 2 years. Storing data for offline processing is no longer a viable business model. In the past few years, new technologies for real-time data processing emerged. Microsoft Azure offers a comprehensive set of tools to ingest and process data in motion. In this presentation we will go over and learn how to collect data from devices, how to process data in real time using Azure Stream Analytic jobs, and how to produce and handle actionable insights.
Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...
Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better.
The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka.
We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices.
Some potential takeaways for the discerning audience:
1. Opportunities in a platform approach to Event Driven Architecture in the enterprise
2. Adopting a product mindset around Data & Event Streams
3. Seeking harmony with allied enterprise applications
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...
This document discusses testing event-driven architectures. It begins by defining common event-driven architecture patterns like event notifications and event sourcing. It then discusses brokering the complexity of event-driven architectures by describing how events are communicated between producers and consumers via channels. The document outlines what information should be included in events like payloads and headers. It also discusses the difference between orchestration and choreography in event-driven systems. It provides an example of how events can be used to mediate changes within a system using order validation. Finally, it demonstrates how to test event-driven architectures using specifications and discusses accelerating API quality through testing tools that support multiple protocols and definitions.
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.
Westpac Bank Tech Talk 2: Introduction to Streaming Data and Stream Processin...
Watch the webcast here: https://videos.confluent.io/watch/P5up2YQX9QdVMhmYfsXy7Q
Speaker: Brett Randall
Logging in The World of DevOps
This document summarizes Cecep Mahbub's experience working on logging infrastructure for various companies and discusses best practices for centralized logging. It outlines Cecep's background maintaining logging systems at KMK and BBM, and their current role setting up Bukalapak's logging infrastructure on Google Cloud Platform. Key topics covered include common uses of logs, benefits of a centralized logging server, examples of logging servers and SaaS options, and integrating logs into DevOps processes.
The API Journey: GraphQL Specification and Implementation
The document discusses GraphQL, including its specification and implementation. It describes GraphQL's basic schema definition (1a), additional features like directives, scalars, and enums (1b), and mutations and subscriptions (1c, 1d). It then covers implementing a simple GraphQL server (2a), using federation for multi-tier architectures (2b), and code generation for speeding development (2c).
Simplifying Event Streaming: Tools for Location Transparency and Data Evoluti...
At Under Armour Connected Fitness, we’ve built an event streaming platform on top of Kafka and the Confluent stack that makes it easy for developers to produce and consume schema-based events without requiring direct knowledge of Kafka. We are constantly trying to improve the developer experience. The platform consists of multiple federated Kafka clusters, a schema registry, a topology service, an archiver and specialized client libraries and Web / CLI tools that assist developers with producer and consumer workflows.
In this talk, we will take a deeper dive into the design and implementation of a Scala/Java implementation of our client library that allows developers to produce or consume events without worrying about the underlying infrastructure and their location while enjoying the benefits of data compatibility through schemas. We’ll also look at an HTTP based client proxy that exposes the same API but for languages without our native support. Finally, we’ll walk through Web and CLI tools we built to make working with the platform easier.
The content of this talk will be primarily aimed at software developers looking for ideas on how to build Kafka client tools that allow producer/consumer interactions protected by schema-based event definitions while hiding details of the underlying infrastructure.
Distributed Data Quality - Technical Solutions for Organizational Scaling
The document discusses Yelp's distributed data architecture and quality solutions for organizational scaling. It describes how Yelp connects over 500 engineers across many services through shared data stored in databases like MySQL, Cassandra and Elasticsearch. The data is ingested through Kafka and processed using tools like Flink. Schematizer provides documentation, discovery and ownership of data. It also enables data lineage tracking and auditing to ensure quality as the data is transformed and loaded into data lakes and warehouses. The goal is to provide reliable, up-to-date shared data to align teams and enable autonomy through self-service data access.
Elastic Stack Basic - All The Capabilities in 6.3!
In Elastic Stack 6.3 we have taken a bold step and opened the X-Pack code for viewing, comment, and bug tracking. In addition, we’ve now included all of the FREE X-Pack Features in the default 6.3 distribution. Learn how you can benefit from these changes to accelerate your projects and get started with the Elastic Stack today!
From logging to monitoring to reactive insights - C Schneider
OSGi Community Event 2017 Presentation by Christian Schneider [Adobe]
In a highly distributed world it is crucial to have the best possible insights into your application.
In the old days we achieved this using plain logging and jmx. This does not scale well to highly distributed processing like in microservices though.
This talk shows how to pimp you OSGi based systems with top notch monitoring and tracing capabilities. Using Apache Karaf Decanter we let karaf instances log into a common elastic search instance with almost zero effort and get insights using kibana. The Apache CXF REST and SOAP integrations allow to log and correlate service requests and replies.
Building on top of this we use the OSGi based reactive components framework and reactor.io streams to provide complex event processing capabilties. A typical use case for this is to describe and monitor service level agreements.
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
This document discusses Airbnb's use of Kafka as the foundation for its highly reliable logging system. It describes the types of data Airbnb collects, including product events, database exports, service events, and derived data. Airbnb uses a simple logging pipeline where events are delivered reliably through Kafka in real time. Key components of its production logging pipeline include Jitney for standardized messaging, a central schema repository, client SDKs, producer and consumer agents, and a self-service portal. The pipeline provides continuous integration through schema authoring, deployment, implementation, processing and storage, and monitoring. It handles large volumes of data reliably with 150 brokers processing 1 million messages per second and 10 billion events collected daily with very low
Event Sourcing, Stream Processing and Serverless (Ben Stopford, Confluent) K...
In this talk we'll look at the relationship between three of the most disruptive software engineering paradigms: event sourcing, stream processing and serverless. We'll debunk some of the myths around event sourcing. We'll look at the inevitability of event-driven programming in the serverless space and we'll see how stream processing links these two concepts together with a single 'database for events'. As the story unfolds we'll dive into some use cases, examine the practicalities of each approach-particularly the stateful elements-and finally extrapolate how their future relationship is likely to unfold. Key takeaways include: The different flavors of event sourcing and where their value lies. The difference between stream processing at application- and infrastructure-levels. The relationship between stream processors and serverless functions. The practical limits of storing data in Kafka and stream processors like KSQL.
Building Microservices with Apache Kafka by Colin McCabe
Abstract:- Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics.
ksqlDB Workshop
Watch the webcast here: https://videos.confluent.io/watch/woobbdupdrwpJCHZKEwF5E?
Speaker: Patrick Druley
Developing custom transformation in the Kafka connect to minimize data redund...
Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost.
in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.
What's hot (20)
Demystifying Event-Driven Architectures with Apache Kafka | Bogdan Sucaciu, P...
Demystifying Event-Driven Architectures with Apache Kafka | Bogdan Sucaciu, P...
Getting started with Azure Event Hubs and Stream Analytics services
Getting started with Azure Event Hubs and Stream Analytics services
Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...
Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Westpac Bank Tech Talk 2: Introduction to Streaming Data and Stream Processin...
Westpac Bank Tech Talk 2: Introduction to Streaming Data and Stream Processin...
The API Journey: GraphQL Specification and Implementation
The API Journey: GraphQL Specification and Implementation
Simplifying Event Streaming: Tools for Location Transparency and Data Evoluti...
Simplifying Event Streaming: Tools for Location Transparency and Data Evoluti...
Distributed Data Quality - Technical Solutions for Organizational Scaling
Distributed Data Quality - Technical Solutions for Organizational Scaling
Elastic Stack Basic - All The Capabilities in 6.3!
Elastic Stack Basic - All The Capabilities in 6.3!
From logging to monitoring to reactive insights - C Schneider
From logging to monitoring to reactive insights - C Schneider
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
Event Sourcing, Stream Processing and Serverless (Ben Stopford, Confluent) K...
Event Sourcing, Stream Processing and Serverless (Ben Stopford, Confluent) K...
Building Microservices with Apache Kafka by Colin McCabe
Building Microservices with Apache Kafka by Colin McCabe
Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...
Similar to Data / Streaming / Microservices Platform with Devops
Down the event-driven road: Experiences of integrating streaming into analyti...
The requirements of many modern data platforms develop along two directions: (1) Low latency, i.e. the shift from batch-oriented to event-driven processes, which facilitate much more timely and reactive insights; and (2) complex analytics, i.e. the ability to efficiently apply analytic functions or models to the incoming data streams. However, many companies don't start from scratch, and already have well-established data infrastructure and processes with various degrees of affinity and compatibility to these novel paradigms. Based on extensive experience of building data platforms with customers, we describe in this talk some key challenges and aspects of introducing streaming-based approaches in real-world productive environments. These include e.g. integrating existing batch-oriented data sources and APIs, checking consistency when using event sourcing to exchange data, and building realtime analytical visualizations. For all cases, architectural options are discussed, and the final solution is explained, including technologies like Apache Nifi, Airflow, Phoenix, Druid and the Confluent Platform. We close the talk by describing non-technical aspects like building up an event-driven mindset among analysts.
Speaker: Dr. Dominik Benz, inovex
Event: Confluent Meetup, 08.10.2018
Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/
Mehr Tech-Artikel: https://www.inovex.de/blog/
Apache Kafka as Event Streaming Platform for Microservice Architectures
This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The Confluent Platform adds further components such as a Schema Registry, REST Proxy, KSQL, Clients for different programming languages and Connectors for different technologies.
The session discusses how tech giants like LinkedIn, Ebay or Airbnb leverage Apache Kafka as event streaming platform to solve various different business problems and how to create a scalable, flexible microservice architecture. A live demo shows how you can easily process and analyze streams of events using Apache Kafka and KSQL.
Serverless and Streaming: Building ‘eBay’ by ‘Turning the Database Inside Out’
(Neil Avery, Confluent) Kafka Summit SF 2018
Apache Kafka has an immense capability for building data-intensive applications. The ability to model and scale to thousands of parallel real-time streams bring with it incredible fundamentals upon which you can build scalable systems of almost any nature. We also have the brave new world of serverless that causes much confusion as to how to use Functions as a Service (FaaS) and streaming together.
This talk builds a horizontally scalable auction platform with real-time marketplace analytics to work at the scale of eBay. The design uses the concepts of "turning the database inside out” to build a model of how to make a stream platform works synergistically with FaaS/AWS Lambda. First, we explore the founding principles of the log, producer/consumer topics, partitions and events. We then build up to Kafka Streams stateless and stateful stream processing and KSQL. These principles are mapped onto simple use cases in order to establish how to build higher order functionality. These use cases are combined to develop an architecture that provides the design semantics required for a real-time auction system and marketplace intelligence. The architecture is composed of a queryable data fabric using Kafka Streams state stores, a high-throughput worker-queue using exactly-once semantics (EoS) Kafka consumers, and a queue-worker hook to drive AWS Lambda functions.
The end result is a real-time system that scales elastically to service millions of auction events and provide live, marketplace analytics. The audience will learn how to compose large scale applications using all of the Apache Kafka stack as well as how we view “turning the database inside out” when used in conjunction with serverless architectures.
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
An overview of building scalable data pipelines and specific technologies including Apache #Kafka, #Spark, Spark Streaming, #Cassandra, and #FiloDB.
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and limitations of traditional middleware with a single, scalable event streaming platform. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, using traditional tools for specific integrations while relying on Kafka for scalable event collection and processing. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" by facilitating a gradual migration from old middleware to a modern event streaming architecture centered around Kafka.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and provides a single scalable platform for messaging, storage, and processing. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, allowing certain use cases to still leverage existing tools. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" - replacing some functions but integrating with other existing technologies to provide a bridge to new architectures.
Introduction to apache kafka, confluent and why they matter
This is a short and introductory presentation on Apache Kafka (including Kafka Connect APIs, Kafka Streams APIs, both part of Apache Kafka) and other open source components part of the Confluent platform (such as KSQL).
This was the first Kafka Meetup in South Africa.
OSDC 2019 | Democratizing Data at Go-JEK by Maulik Soneji
Having grown 6666x in the last 3 years; the data generated has grown exponentially. As a Data Engineer at GoJEK, we faced the issue with having our complete team managing infrastructure requests. This led us to create an internal portal for other teams to self-provision their data. This talk is divided into two parts. In the first part, I will cover details about how we have scaled our data engineering infrastructure to manage the scale of more than 40 million messages per day. I will explain the data consumption, aggregation, monitoring and cold storage. This will also cover details about how we scaled our infrastructure to achieve the scale that we are at today. In the second part, I will cover how we created our internal portal for infrastructure orchestration. The infrastructure backed by kubernetes enables teams to self-provision data infrastructure without any supervision.
Chti jug - 2018-06-26
This document contains an agenda and overview of Confluent and streaming with Kafka. The agenda includes introductions to Confluent, streaming, KSQL, and a demo. Confluent is presented as the company founded by the creators of Apache Kafka to develop streaming platforms based on Kafka. Key concepts of streaming, the Confluent platform, and Kafka Streams, Kafka Connect, and KSQL are summarized. The document concludes with resources and time for questions.
BBL KAPPA Lesfurets.com
Lesfurest.com invited me to talk about the KAPPA Architecture style during a BBL.
Kappa architecture is a style for real-time processing of large volumes of data, combining stream processing, storage, and serving layers into a single pipeline. It's different from the Lambda architecture, uses separate batch and stream processing pipelines.
Jug - ecosystem
This document provides an overview and agenda for a presentation on Confluent, streaming, and KSQL. The presentation includes: an introduction to Confluent and Apache Kafka; an explanation of why streaming platforms are useful; an overview of the Confluent Platform and its components; key concepts in streaming and Kafka; a demonstration of Kafka Streams, Kafka Connect, and KSQL; and resources for further information. The presentation aims to explain streaming concepts, demonstrate Confluent tools, and allow for a question and answer session.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Learn the differences between an event-driven streaming platform and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.
Extract-Transform-Load (ETL) is still a widely-used pattern to move data between different systems via batch processing. Due to its challenges in today’s world where real time is the new standard, an Enterprise Service Bus (ESB) is used in many enterprises as integration backbone between any kind of microservice, legacy application or cloud service to move data via SOAP / REST Web Services or other technologies. Stream Processing is often added as its own component in the enterprise architecture for correlation of different events to implement contextual rules and stateful analytics. Using all these components introduces challenges and complexities in development and operations.
This session discusses how teams in different industries solve these challenges by building a native streaming platform from the ground up instead of using ETL and ESB tools in their architecture. This allows to build and deploy independent, mission-critical streaming real time application and microservices. The architecture leverages distributed processing and fault-tolerance with fast failover, no-downtime rolling deployments and the ability to reprocess events, so you can recalculate output when your code changes. Integration and Stream Processing are still key functionality but can be realized in real time natively instead of using additional ETL, ESB or Stream Processing tools.
Introduction to Apache Kafka and why it matters - Madrid
This document provides an introduction to Apache Kafka and discusses why it is an important distributed streaming platform. It outlines how Kafka can be used to handle streaming data flows in a reliable and scalable way. It also describes the various Apache Kafka APIs including Kafka Connect, Streams API, and KSQL that allow organizations to integrate Kafka with other systems and build stream processing applications.
Data platform evolution
We will show a case study of moving from SQL Server DWH to Hadoop and Vertica. In this case study you will see implementation of Lambda Architecture with Hadoop, Vertica and MongoDB for real time statistics. We will start from showing the Legacy system and describe the problems we encountered. From there we will cover all the decision making on technology choosing for the current solution. We will finish by presenting the next steps of our Data Platform solution.
Current and Future of Apache Kafka
Apache Kafka is a distributed streaming platform. It provides a high-throughput distributed messaging system that can handle trillions of events daily. Many large companies use Kafka for application logging, metrics collection, and powering real-time analytics. The current version is 0.8.2 and upcoming versions will include a new consumer, security features, and support for transactions.
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
We will explore the strengths and limitations of Hadoop for analyzing large data sets and review the growing ecosystem of tools for augmenting, extending, or replacing Hadoop MapReduce. We will introduce the Amazon Elastic MapReduce (EMR) platform as the big data foundation for Hadoop and beyond by providing specific examples of running Machine Learning (Mahout), Graph Analytics (Giraph), and Statistical Analysis (R) on EMR. We will discuss also big data analytics and visualization of results with Amazon Redshift + third party business intelligence tools, as well as typical end-to-end Big Data workflow on AWS.
We will conclude with real-world examples from ICAO of Big Data analytics for aviation safety data on AWS. The integrated Safety Trend Analysis and Reporting System (iSTARS) is a web based system linking a collection of safety datasets and related web application to perform online safety and risk analysis. It uses AWS EC2, S3, EMR and related partner tools for continuous data aggregation and filtering.
apidays LIVE India - REST the Events - REST APIs for Event-Driven Architectur...
apidays LIVE India 2021 - Connecting 1.3 billion digital innovators
May 20, 2021
REST the Events - REST APIs for Event-Driven Architecture
Mark Teehan, Principal Solution Engineer at Confluent APAC
Moving From Actions & Behaviors to Microservices
My DevCon 2019 talk discusses how to make it easier to integrate Alfresco with other systems using an event-based approach. Two real world examples are discussed and demonstrated. The first is about reporting against Alfresco metadata. The second is about enriching metadata by running content through a Natural Language Processing (NLP) model. Both solutions work by listening to generic events generated by Alfresco and placed on an Apache Kafka queue. For the reporting example, the Spring Boot consumer subscribes to Kafka events, then fetches metadata via CMIS and indexes that into Elasticsearch. For the NLP example, a separate Spring Boot consumer subscribes to the same events, but in this case, fetches the content, extracts text using Apache Tika, runs the text through Apache OpenNLP, then writes back extracted entities to Alfresco via CMIS. These are relatively simple examples, but illustrate how a de-coupled, asynchronous, event-based approach can make integrating Alfresco with other systems easier.
Leverage Kafka to build a stream processing platform
Leverage Kafka to build a stream processing platform, Staff Software Engineer - Frontend, Meroxa
Meetup link: https://www.meetup.com/boston-apache-kafka-meetup/events/291189903/
Confluent and Elastic
Un'introduzione ad Apache Kafka e Kafka Connect APIs (part of Apache Kafka), in particolare come Kafka possa essere usato assieme ad Elasticsearch.
Grazie a Seacom per averci invitato all'evento a Roma.
Similar to Data / Streaming / Microservices Platform with Devops (20)
Down the event-driven road: Experiences of integrating streaming into analyti...
Down the event-driven road: Experiences of integrating streaming into analyti...
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
Serverless and Streaming: Building ‘eBay’ by ‘Turning the Database Inside Out’
Serverless and Streaming: Building ‘eBay’ by ‘Turning the Database Inside Out’
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Introduction to apache kafka, confluent and why they matter
Introduction to apache kafka, confluent and why they matter
OSDC 2019 | Democratizing Data at Go-JEK by Maulik Soneji
OSDC 2019 | Democratizing Data at Go-JEK by Maulik Soneji
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Introduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - Madrid
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
apidays LIVE India - REST the Events - REST APIs for Event-Driven Architectur...
apidays LIVE India - REST the Events - REST APIs for Event-Driven Architectur...
Leverage Kafka to build a stream processing platform
Leverage Kafka to build a stream processing platform
Recently uploaded
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
This case study explores designing a scalable e-commerce platform, covering key requirements, system components, and best practices.
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on automated letter generation for Bonterra Impact Management using Google Workspace or Microsoft 365.
Interested in deploying letter generation automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
A Comprehensive Guide to DeFi Development Services in 2024
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Finale of the Year: Apply for Next One!
Presentation for the event called "Finale of the Year: Apply for Next One!" organized by GDSC PJATK
Recently uploaded (20)
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Data / Streaming / Microservices Platform with Devops
- 1. Data / Streaming / Microservices Platform with DevOps Kidong Lee mykidong@gmail.com
- 2. Typical User Behavior Event Processing Platform Collection Collector Data Bus Unified Log Store Batch Stream Data Workflow Sink - Hive - Tez - Kafka Connect Management Service API Platform Monitoring - InfluxDB - JMXTrans - Kafka Elasticsearch - Netty - Grafana - Coda hale Metrics - Spark - SparkMLLib Admin - Tomcat HDFS Parquet Interactive Query - Drill - Vert.x Resource Management - Nomad - YARN Configuration Management - Ansible Service Discovery - Consul Scheduler - Azkaban Streaming - Kafka Streams gRPC
- 3. eCommerce Recommendation Service: User Events(PageView, Cart, Order Event etc) from the commerce site, collected, realtime- processed and batch-processed with recommendation algorithm to get recommended items. Algorithm: Collaboriative Filtering, Item Similarity, etc are used. Automated Keyword Search Bidding Service: User Events(PageView, Contact, Cart, Order, KeywordSearch Event etc) from the ad site, collected, realtime-processed and batch-processed to get conversions and conversion values. Typical Services
- 4. Collection Layer: Collect User Events and validate Invalid Messages. Push the events to Unified Log System, Kafka. Stream Layer: In Steaming, Kafka Streams converts Json Events to Avro messages which will be sent to another topics in Kafka. In Sink, the converted avro messages from the topics are saved as parquet onto HDFS and Elasticsearch with Kafka Connect. Batch Layer: Spark processes Parquet data to build data model. Final results will be loaded onto Elasticsearch to expose API. Service Layer: API exposes the results from Elasticsearch. Admin calls API via gRPC. Store Layer: HDFS saves all the data as Parquet. Elasticsearch saves the final results. Management Layer: Monitoring, Service Discovery, Configuration Management, Resource Management, Scheduler, etc. Platform Layers
- 5. DevOps Perspecitve of this Platform Data Platform Hadoop DevOps Jenkins as CI / CD Ansible as Configuration Management Nexus as Docker Private Registry Spark Hive Drill Streaming Platform Kafka Kafka Streams Kafka Connect Microservices Platform Consul as Service Discovery Nomad as Container Orchestrator NGINX as Proxy Docker Git as Source Control
- 6. Streaming Platform: KSQL can be added. Microservices Platform: OpenShift Origin can be used as a container orchestrator instead of Nomad. Istio can be added as a service mesh. Additional Components in Future
- 7. DevOps Perspecitve of this Platform in Future Data Platform Hadoop DevOps Jenkins as CI / CD Ansible as Configuration Management Nexus as Docker Private Registry Spark Hive Drill Streaming Platform Kafka Kafka Streams Kafka Connect Microservices Platform Istio as Service Mesh OpenShift Origin as Container Orchestrator Docker Git as Source Control KSQL