Kafka and Kafka Streams in the Global Schibsted Data Platform
•
0 likes•613 views
(Fredrik Vraalsen, City of Oslo) Kafka Summit SF 2018 In this talk we will present how we in Schibsted have set up a new global streaming data platform using Kafka and Kafka Streams, replacing a homegrown solution based on Kinesis and micro batches in Amazon S3. Schibsted is a global media and marketplaces company with over 200 million users across the world. With our new streaming platform, we aim to deliver better performance and enable new features, such as self-serve for our data consumers. In this talk, we will present some of the ways this new platform enables collaboration across Schibsted, as well as some of the challenges we have faced. A number of collaborations with various teams in Schibsted are underway to build projects on top of this new platform. Examples include building tools for experimentation on our sites, visualization of user behavior and data quality checks. Our long-term goal is to provide a self-serve platform for real-time processing of data, enabling our data users to quickly create new data-driven applications. Data and analytics is a central part of Schibsted’s strategy, and we believe the streaming platform will play a significant role in building a global data-driven organization.
Report
Share
Report
Share
Download to read offline
Recommended
Creating an Elastic Platform Using Kafka and Microservices in OpenShift
(Pradeep Chintam, American Express Global Business Travel) Kafka Summit SF 2018
When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project.
The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge.
After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform.
Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...
As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions
Real-Time Market Data Analytics Using Kafka Streams
(Lei Chen, Bloomberg, L.P.) Kafka Summit SF 2018
At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing.
We’ll cover:
-Our system architecture
-Best practices of using the Processor API and State Store API
-Dynamic gap session implementation
-Historical data re-processing practice in KStreams app
-Chaining multiple KStreams apps with Spark Streaming job
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
Recommended
Creating an Elastic Platform Using Kafka and Microservices in OpenShift
(Pradeep Chintam, American Express Global Business Travel) Kafka Summit SF 2018
When a new project, Global Trip Record was launched at American Express GBT and we were looking for a robust, scalable and fault-tolerant middleware to handle all the orchestration and connectivity needs of the project.
The existing solution was monolithic, and we wanted to convert that to a microservices framework, but the biggest challenge was managing the increasing number of external applications that are connected to the platform. Any slow external application or partner system connected to the platform was slowing down the entire platform. There is always a need for partner systems to go offline or a need to resend the entire day’s data, especially with a system like our data lake where the data volumes are huge.
After evaluating multiple solutions, we settled on Apache Kafka, and started with a simple implementation of around 100,000 messages to just decouple one partner system and the core platform.
Today, we are running our microservices (Docker) running in OpenShift (Kubernetes) processing Kafka Streams, running real-time anomaly detection using Kafka Streams, powering our data lake through Kafka, feeding our distributed caching layer (Apache Ignite) and connecting all internal and external systems using Kafka. With a total of more than 10 million messages per day, i.e., 1.5TB of data with just a small three-node cluster, we are one happy platform for over a year now. With the kind of stability, flexibility and success in our project, a lot of other teams started and will soon be in production with Kafka Steams. The powerful combination of Kafka and OpenShift has proven to be an easily scalable model with great elasticity to the entire platform.
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Real-Time Dynamic Data Export Using the Kafka Ecosystem
(Preston Thompson, Braze) Kafka Summit SF 2018
If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy.
We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors.
In this talk, I will discuss:
-How we started our journey in designing this large-scale streaming architecture
-Why streaming technologies were necessary to solve our technology and business issues
-The lessons we learned along the way that can help you with your Kafka-based architecture
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...
As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions
Real-Time Market Data Analytics Using Kafka Streams
(Lei Chen, Bloomberg, L.P.) Kafka Summit SF 2018
At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing.
We’ll cover:
-Our system architecture
-Best practices of using the Processor API and State Store API
-Dynamic gap session implementation
-Historical data re-processing practice in KStreams app
-Chaining multiple KStreams apps with Spark Streaming job
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
A Walkthrough of InfluxCloud 2.0 by Tim Hall
Tim Hall, VP of Products at InfluxData, will demonstrate how to setup and use the next InfluxCloud 2.0 in this InfluxDays NYC 2019 presentation. He provides a brief history of InfluxCloud followed by an overview of InfluxDB 2.0 and a demo.
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
As a 120 year-old company, Nordstrom was facing numerous challenges as a result of an aging, service-oriented, architecture. Developers needing to implement reporting for analytics separately from core functionality resulted in questionable data quality for analytical purposes. Scaling dependent services in harmony to not overwhelm each other was a struggle faced by many, if not most, teams. Several years into a company-wide transition to an event-sourced architecture, Nordstrom has solved these and various other problems. By leveraging the capabilities of Apache Kafka and Confluent, combined with a deep organizational focus on well-defined business event schemas, a singular event can be used for analytical, functional, operational, and model building purposes. This session will describe this architecture and the lessons learned while building it, with a focus on the internally built, multi-tenant, multi-cluster, Kafka-as-a-Service platform that enables it.
Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...
We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Speaker: Yupeng Fu, Staff Engineer, Uber
High availability and reliability are important requirements to Uber services, and the services shall tolerate datacenter failures in a region and fail over to another region. In this talk, we will present the active-active Apache Kafka® at Uber and how it facilitates disaster discovery across regions for Uber services. In particular, we will highlight the key components including topic replication, topic aggregation, offsets sync and then walk through several use cases of their disaster recovery strategy using active-active Kafka. Lastly, we will present several interesting challenges and the future work planned.
Yupeng Fu is a staff engineer in Uber Data Org leading the streaming data platform. Previously, he worked at Alluxio and Palantir, building distributed data analysis and storage platforms. Yupeng holds a B.S. and an M.S. from Tsinghua University and did his Ph.D. research on databases at UCSD.
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...
(Bruno Simic, Solutions Engineer, Couchbase)
Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream processing allows analyzing streaming data in real-time using tools like KSQL. KSQL builds on Kafka Streams to provide a SQL-like interface for transforming, filtering, and creating derived streams from immutable event data ingested into Kafka. Jupiter, a digital banking platform, uses KSQL to reconcile transactions from different interfaces in real-time and check rewards eligibility by analyzing consolidated transaction streams. While powerful, KSQL deployments require monitoring, richer SQL support, and integration with big data platforms for long-term storage.
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...
Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
How a Data Mesh is Driving our Platform | Trey Hicks, Gloo
At Gloo.us, we face a challenge in providing platform data to heterogeneous applications in a way that eliminates access contention, avoids high latency ETLs, and ensures consistency for many teams. We're solving this problem by adopting Data Mesh principles and leveraging Kafka, Kafka Connect, and Kafka streams to build an event driven architecture to connect applications to the data they need. A domain driven design keeps the boundaries between specialized process domains and singularly focused data domains clear, distinct, and disciplined. Applying the principles of a Data Mesh, process domains assume the responsibility of transforming, enriching, or aggregating data rather than relying on these changes at the source of truth -- the data domains. Architecturally, we've broken centralized big data lakes into smaller data stores that can be consumed into storage managed by process domains.
This session covers how we’re applying Kafka tools to enable our data mesh architecture. This includes how we interpret and apply the data mesh paradigm, the role of Kafka as the backbone for a mesh of connectivity, the role of Kafka Connect to generate and consume data events, and the use of KSQL to perform minor transformations for consumers.
Navigating the obdervability storm with Kafka | Jose Manuel Cristobal, Adidas
When all your stores are closed, e-commerce becomes your bigger store… and the most challenging. That means a myriad of systems orchestrated to make it happen, all of them scaling out accordingly and implementing Observability and SRE practices to support this growth, preserving stability and reliability.
How can we detect problems, root causes and react? How can we predict those problems?
HOLMES is the adidas solution to accelerate problem detection, giving a holistic view of technical systems through metrics and logs democratisation.
In this talk, we'll show how Kafka technology stack allows adidas to support the ingestion and offload of all logs and metrics of the company. A platform which adoption has skyrocketed during 2020, supporting 100 Billion messages per day.
The main takeaway will be the explanation of a cutting-edge solution based on kafka technology stack (kafka, Kafka Streams and Kafka Connect) for demanding throughput ecosystem.
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
This document discusses Airbnb's use of Kafka as the foundation for its highly reliable logging system. It describes the types of data Airbnb collects, including product events, database exports, service events, and derived data. Airbnb uses a simple logging pipeline where events are delivered reliably through Kafka in real time. Key components of its production logging pipeline include Jitney for standardized messaging, a central schema repository, client SDKs, producer and consumer agents, and a self-service portal. The pipeline provides continuous integration through schema authoring, deployment, implementation, processing and storage, and monitoring. It handles large volumes of data reliably with 150 brokers processing 1 million messages per second and 10 billion events collected daily with very low
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik
Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations.
When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce.
In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.
Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...
This document summarizes Jan Svoboda's presentation on building event streaming microservices with Spring Boot and Apache Kafka. Jan discusses his motivation for learning Kafka and microservices development. He outlines a typical journey from strangling monoliths to building event-driven architectures using Kafka. Jan explains how Spring and Kafka integrate well together. The remainder of the presentation demonstrates building a sample application using these techniques and dives into common architecture patterns like event sourcing, CQRS, scaling stateful services, and reverse proxy routing.
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services.
At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure.
This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
This document discusses KIP-633, which proposes changing the default grace period for windowed operations in Kafka Streams from 24 hours to 0. The motivation is that the 24-hour default was causing confusion for users and suppressing results. The KIP was worked on by a team of contributors and introduces new APIs in Kafka Streams 3.0 that give developers better control over grace periods. It provides resources for migrating code to the new APIs and understanding the compatibility changes.
Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...
Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability.
This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment:
kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability.
kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters.
We'll also present challenges we encountered along the way and share our learnings with the audience.
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018
Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.
PLNOG 8: Kazimierz Jantas - Innowacyjne rozwiązania dla IT
This document summarizes Zycko Polska's IT solutions and Riverbed technology products. It discusses how Riverbed solutions such as Steelhead, Granite, Aptimize, and Stingray can optimize application delivery and WAN performance through technologies like WAN optimization, application acceleration, network monitoring and more. The document also provides information on specific Riverbed products and how they can help with challenges like consolidation, disaster recovery, security and more.
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Scientist at First Derivatives plc.
More Related Content
What's hot
A Walkthrough of InfluxCloud 2.0 by Tim Hall
Tim Hall, VP of Products at InfluxData, will demonstrate how to setup and use the next InfluxCloud 2.0 in this InfluxDays NYC 2019 presentation. He provides a brief history of InfluxCloud followed by an overview of InfluxDB 2.0 and a demo.
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
As a 120 year-old company, Nordstrom was facing numerous challenges as a result of an aging, service-oriented, architecture. Developers needing to implement reporting for analytics separately from core functionality resulted in questionable data quality for analytical purposes. Scaling dependent services in harmony to not overwhelm each other was a struggle faced by many, if not most, teams. Several years into a company-wide transition to an event-sourced architecture, Nordstrom has solved these and various other problems. By leveraging the capabilities of Apache Kafka and Confluent, combined with a deep organizational focus on well-defined business event schemas, a singular event can be used for analytical, functional, operational, and model building purposes. This session will describe this architecture and the lessons learned while building it, with a focus on the internally built, multi-tenant, multi-cluster, Kafka-as-a-Service platform that enables it.
Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...
We often need to build applications that analyze Kafka data to unlock the most value from event streams, so how can organizations build these real-time analytics applications? In this talk, we examine an indexing approach that enables fast SQL analytics on data from Kafka, without data flattening or denormalization. Rockset is the real-time indexing database that builds an inverted index, a columnar index and a row index on all fields of your Kafka messages, including nested fields and arrays. This Converged Index accelerates various types of analytic queries–search, aggregations and joins–without the need to denormalize or transform data for performance reasons. With indexing delivering significant gains in query performance, we also need to index new data in a timely manner. We discuss several strategies used for efficient ingestion and indexing from Kafka, including rollups, write optimizations on the underlying RocksDB storage engine, and the disaggregation of ingest and query compute.
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Speaker: Yupeng Fu, Staff Engineer, Uber
High availability and reliability are important requirements to Uber services, and the services shall tolerate datacenter failures in a region and fail over to another region. In this talk, we will present the active-active Apache Kafka® at Uber and how it facilitates disaster discovery across regions for Uber services. In particular, we will highlight the key components including topic replication, topic aggregation, offsets sync and then walk through several use cases of their disaster recovery strategy using active-active Kafka. Lastly, we will present several interesting challenges and the future work planned.
Yupeng Fu is a staff engineer in Uber Data Org leading the streaming data platform. Previously, he worked at Alluxio and Palantir, building distributed data analysis and storage platforms. Yupeng holds a B.S. and an M.S. from Tsinghua University and did his Ph.D. research on databases at UCSD.
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...
(Bruno Simic, Solutions Engineer, Couchbase)
Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream processing allows analyzing streaming data in real-time using tools like KSQL. KSQL builds on Kafka Streams to provide a SQL-like interface for transforming, filtering, and creating derived streams from immutable event data ingested into Kafka. Jupiter, a digital banking platform, uses KSQL to reconcile transactions from different interfaces in real-time and check rewards eligibility by analyzing consolidated transaction streams. While powerful, KSQL deployments require monitoring, richer SQL support, and integration with big data platforms for long-term storage.
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...
Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
How a Data Mesh is Driving our Platform | Trey Hicks, Gloo
At Gloo.us, we face a challenge in providing platform data to heterogeneous applications in a way that eliminates access contention, avoids high latency ETLs, and ensures consistency for many teams. We're solving this problem by adopting Data Mesh principles and leveraging Kafka, Kafka Connect, and Kafka streams to build an event driven architecture to connect applications to the data they need. A domain driven design keeps the boundaries between specialized process domains and singularly focused data domains clear, distinct, and disciplined. Applying the principles of a Data Mesh, process domains assume the responsibility of transforming, enriching, or aggregating data rather than relying on these changes at the source of truth -- the data domains. Architecturally, we've broken centralized big data lakes into smaller data stores that can be consumed into storage managed by process domains.
This session covers how we’re applying Kafka tools to enable our data mesh architecture. This includes how we interpret and apply the data mesh paradigm, the role of Kafka as the backbone for a mesh of connectivity, the role of Kafka Connect to generate and consume data events, and the use of KSQL to perform minor transformations for consumers.
Navigating the obdervability storm with Kafka | Jose Manuel Cristobal, Adidas
When all your stores are closed, e-commerce becomes your bigger store… and the most challenging. That means a myriad of systems orchestrated to make it happen, all of them scaling out accordingly and implementing Observability and SRE practices to support this growth, preserving stability and reliability.
How can we detect problems, root causes and react? How can we predict those problems?
HOLMES is the adidas solution to accelerate problem detection, giving a holistic view of technical systems through metrics and logs democratisation.
In this talk, we'll show how Kafka technology stack allows adidas to support the ingestion and offload of all logs and metrics of the company. A platform which adoption has skyrocketed during 2020, supporting 100 Billion messages per day.
The main takeaway will be the explanation of a cutting-edge solution based on kafka technology stack (kafka, Kafka Streams and Kafka Connect) for demanding throughput ecosystem.
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
This document discusses Airbnb's use of Kafka as the foundation for its highly reliable logging system. It describes the types of data Airbnb collects, including product events, database exports, service events, and derived data. Airbnb uses a simple logging pipeline where events are delivered reliably through Kafka in real time. Key components of its production logging pipeline include Jitney for standardized messaging, a central schema repository, client SDKs, producer and consumer agents, and a self-service portal. The pipeline provides continuous integration through schema authoring, deployment, implementation, processing and storage, and monitoring. It handles large volumes of data reliably with 150 brokers processing 1 million messages per second and 10 billion events collected daily with very low
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik
Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations.
When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce.
In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.
Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...
This document summarizes Jan Svoboda's presentation on building event streaming microservices with Spring Boot and Apache Kafka. Jan discusses his motivation for learning Kafka and microservices development. He outlines a typical journey from strangling monoliths to building event-driven architectures using Kafka. Jan explains how Spring and Kafka integrate well together. The remainder of the presentation demonstrates building a sample application using these techniques and dives into common architecture patterns like event sourcing, CQRS, scaling stateful services, and reverse proxy routing.
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services.
At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure.
This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
This document discusses KIP-633, which proposes changing the default grace period for windowed operations in Kafka Streams from 24 hours to 0. The motivation is that the 24-hour default was causing confusion for users and suppressing results. The KIP was worked on by a team of contributors and introduces new APIs in Kafka Streams 3.0 that give developers better control over grace periods. It provides resources for migrating code to the new APIs and understanding the compatibility changes.
Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...
Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability.
This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment:
kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability.
kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters.
We'll also present challenges we encountered along the way and share our learnings with the audience.
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018
Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.
What's hot (20)
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...
Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...
Sub-Second SQL Search, Aggregations and Joins with Kafka and Rockset | Dhruba...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...
Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
How a Data Mesh is Driving our Platform | Trey Hicks, Gloo
How a Data Mesh is Driving our Platform | Trey Hicks, Gloo
Navigating the obdervability storm with Kafka | Jose Manuel Cristobal, Adidas
Navigating the obdervability storm with Kafka | Jose Manuel Cristobal, Adidas
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
Kafka Summit NYC 2017 - Every Message Counts: Kafka as a Foundation for Highl...
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik
Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...
Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...
Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...
Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard
Similar to Kafka and Kafka Streams in the Global Schibsted Data Platform
PLNOG 8: Kazimierz Jantas - Innowacyjne rozwiązania dla IT
This document summarizes Zycko Polska's IT solutions and Riverbed technology products. It discusses how Riverbed solutions such as Steelhead, Granite, Aptimize, and Stingray can optimize application delivery and WAN performance through technologies like WAN optimization, application acceleration, network monitoring and more. The document also provides information on specific Riverbed products and how they can help with challenges like consolidation, disaster recovery, security and more.
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Scientist at First Derivatives plc.
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...
Kx Technology is an in-memory columnar database and programming system that is highly optimized for real-time streaming and historical time-series data analytics. It provides extreme performance at low latency and can scale to process massive data volumes without significant infrastructure. Kx has been widely adopted over two decades in the financial services industry for applications like market surveillance, risk management, and quantitative research.
Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...
The Ohio Department of Transportation has adopted Confluent as the event driven enabler of DriveOhio, a modern Intelligent Transportation System. DriveOhio digitally links sensors, cameras, speed monitoring equipment, and smart highway assets in real time, to dynamically adjust the surface road network to maximize the safety and efficiency for travelers. Over the past 24 months the team has increased the number and types of devices within the DriveOhio environment, while also working to see their vendors adopt Kafka to better participate in data sharing.
World’s Fastest Image Serving Technology
This document discusses ERDAS APOLLO Essentials' image serving technology. It highlights that (1) managing geospatial imagery is challenging due to increasing data sizes, resolutions, and user demands for speed; (2) ERDAS APOLLO Essentials helps organizations reduce storage needs and accelerate delivery through techniques like ECW compression and tile caching; and (3) case studies demonstrate the technology can compress large imagery by over 90% while maintaining visual quality and enabling faster access.
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
This talk was delivered at the Serverless Conference in New York City in 2017. Deloitte and Amtrak built a Serverless Cloud-Native solution on AWS for real-time operational datastore and near real-time reporting data mart that modernized Amtrak's legacy systems & applications. With Serverless solutions, we are able leapfrog over several rungs of computing evolution.
Gary Arora is a Cloud Solutions Architect at Deloitte Consulting, specializing on Azure & AWS.
Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration
Patience with Apache Cassandra’s volatile latencies was wearing thin at Rakuten, a global online retailer serving 1.5B worldwide members. The Rakuten Catalog Platform team architected an advanced data platform – with Cassandra at its core – to normalize, validate, transform, and store product data for their global operations. However, while the business was expecting this platform to support extreme growth with exceptional end-user experiences, the team was battling Cassandra’s instability, inconsistent performance at scale, and maintenance overhead. So, they decided to migrate.
Join this webinar to hear a firsthand account of:
How specific Cassandra challenges were impacting the team and their product
How they determined whether migration would be worth the effort
What processes they used to evaluate alternative databases
What their migration required from a technical perspective
Strategies (and lessons learned) for your own database migration
HBase Meetup @ Cask HQ 09/25
The document summarizes an agenda for an HBase Meetup at Cask HQ. The agenda includes announcements about Cask's newly open sourced projects - CDAP (Cask Data Application Platform), Coopr (cluster provisioning), and Tigon (real-time streaming on YARN and HBase). It also lists talks on using HBase at Flipboard and master topologies after HBase 1.0. Cask is now fully open source and aims to build communities around these projects to help more developers build applications on Hadoop platforms.
Building real time data-driven products
This presentation will describe how to go beyond a "Hello world" stream application and build a real-time data-driven product. We will present architectural patterns, go through tradeoffs and considerations when deciding on technology and implementation strategy, and describe how to put the pieces together. We will also cover necessary practical pieces for building real products: testing streaming applications, and how to evolve products over time.
Presented at highloadstrategy.com 2016 by Øyvind Løkling (Schibsted Products & Technology), joint work with Lars Albertsson (independent, www.mapflat.com).
What is the Oracle PaaS Cloud for Developers (Oracle Cloud Day, The Netherlan...
The promise of the cloud is substantial. Oracle's public cloud promise goes beyond the generic promise. This presentation describes the promise of the Oracle Public Cloud specifically for developers. It describes the current state of the PaaS Platform, the actual and coming services and what they could mean to a developer. From same platform, different location (DBaaS, JCS) to cloud native stack (ICS, MCS) and services for Citizen Developers, the presentation touches upon virtually all services relevant to developers. The presentation concludes with first the steps enterprises can start taking to move to the cloud and second the steps individual developers could and perhaps should take in order to conquer the clouds.
NEW LAUNCH! Introducing AWS Snowball Edge and AWS Snowmobile
At re:Invent 2016 AWS announced several revolutionary ways to move data into and out of the AWS Cloud in large batches, so you can retire servers, racks, and data centers and embark on new projects faster. Two of these new services are AWS Snowmobile and AWS Snowball Edge. Snowmobile cuts cloud data migration projects from years to weeks in time with a 45 foot truck with 100PB of capacity. AWS Snowball Edge is a petabyte-scale data transfer service with portable storage and compute capabilities. It transports up to 100TB of data in a ruggedized, weather resistant, rack mountable, shippable device with the same embedded cryptography and security as the original Snowball. Learn more about each of these services and hear from two customers on the impact these services have made on their data center operations.
Calum McCrea, Software Engineer at Kx Systems, "Kx: How Wall Street Tech can ...
Calum McCrea, Software Engineer at Kx Systems speaking at Big Data, Paris v 6.0 on April 27, 2017
"Kx: How Wall Street Tech can Speed up the World"
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
The document describes a managed Oracle database as a service (DBaaS) that is jointly offered by Swisscom and Trivadis. It provides concise summaries of the key components and benefits of the service, including:
1) The service leverages the best of both Swisscom and Trivadis - Swisscom provides the cloud infrastructure and security while Trivadis provides database expertise and management.
2) Customers benefit from high availability, security within Swiss data centers, cost savings from outsourced management, and scalability.
3) Automation is a key part of the solution, allowing the service to be scaled through orchestration of virtual infrastructure,
What's New in Upcoming Apache Spark 2.3
Apache Spark 2.0 set the architectural foundations of Structure in Spark, Unified high-level APIs, Structured Streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2.
Continuing forward in that spirit, the upcoming release of Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of soon to be released Spark 2.3 features:
• Kubernetes Scheduler Backend
• PySpark Performance and Enhancements
• Continuous Structured Streaming Processing
• DataSource v2 APIs
• Structured Streaming v2 APIs
Accelerate Big Data Application Development with Cascading
Accelerate Big Data Application Development with Cascading
Supreet Oberoi's Presentation at the SF Bay Area DevBrill Meetup
Red Hat Storage Day LA - Performance and Sizing Software Defined Storage
This document summarizes a presentation given by Kyle Bader of Red Hat on software defined storage and performance testing of MySQL on Red Hat Ceph Storage compared to AWS EBS. Some key points:
- Performance testing showed Red Hat Ceph Storage could provide over 78 IOPS/GB for MySQL workloads, meeting and exceeding the 30 IOPS/GB target of AWS EBS provisioned IOPS.
- The price per IOP of Red Hat Ceph Storage on a Supermicro cluster was $0.78, well below the $2.50 target cost of AWS EBS provisioned IOPS storage.
- Different hardware configurations, especially core-to-flash ratios, impacted Ceph Storage performance
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
This document describes BBVA's implementation of a Big Data Lake using Apache Spark for log collection, storage, and analytics. It discusses:
1) Using Syslog-ng for log collection from over 2,000 applications and devices, distributing logs to Kafka.
2) Storing normalized logs in HDFS and performing analytics using Spark, with outputs to analytics, compliance, and indexing systems.
3) Choosing Spark because it allows interactive, batch, and stream processing with one system using RDDs, SQL, streaming, and machine learning.
Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data Stores
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data Stores
Kamil Bajda-Pawlikowski, CTO, Starburst Data
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Wasp2 - IoT and Streaming Platform
WASP is a framework to develop big data pipelines, working with streaming analytics, multi model storages and machine learning models. Everything is in real time.
XDF 2019 Xilinx Accelerated Database and Data Analytics Ecosystem
Xilinx Accelerated Database and Data Analytics Ecosystem presentation at Xilinx Developer Forum 2019 San Jose on 10/2/2019.
Similar to Kafka and Kafka Streams in the Global Schibsted Data Platform (20)
PLNOG 8: Kazimierz Jantas - Innowacyjne rozwiązania dla IT
PLNOG 8: Kazimierz Jantas - Innowacyjne rozwiązania dla IT
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...
"An introduction to Kx Technology - a Big Data solution", Kyra Coyne, Data Sc...
Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...
Safer Commutes & Streaming Data | George Padavick, Ohio Department of Transpo...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Leapfrog into Serverless - a Deloitte-Amtrak Case Study | Serverless Confere...
Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration
Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration
What is the Oracle PaaS Cloud for Developers (Oracle Cloud Day, The Netherlan...
What is the Oracle PaaS Cloud for Developers (Oracle Cloud Day, The Netherlan...
NEW LAUNCH! Introducing AWS Snowball Edge and AWS Snowmobile
NEW LAUNCH! Introducing AWS Snowball Edge and AWS Snowmobile
Calum McCrea, Software Engineer at Kx Systems, "Kx: How Wall Street Tech can ...
Calum McCrea, Software Engineer at Kx Systems, "Kx: How Wall Street Tech can ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
Accelerate Big Data Application Development with Cascading
Accelerate Big Data Application Development with Cascading
Red Hat Storage Day LA - Performance and Sizing Software Defined Storage
Red Hat Storage Day LA - Performance and Sizing Software Defined Storage
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)
Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data Stores
Presto: Fast SQL-on-Anything Across Data Lakes, DBMS, and NoSQL Data Stores
XDF 2019 Xilinx Accelerated Database and Data Analytics Ecosystem
XDF 2019 Xilinx Accelerated Database and Data Analytics Ecosystem
More from confluent
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the
Real-time Streaming and AI Era
Andrew Foo
Customer Solutions @ Confluent
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
In our exclusive webinar, you'll learn why event-driven architecture is the key to unlocking cost efficiency, operational effectiveness, and profitability. Gain insights on how this approach differs from API-driven methods and why it's essential for your organization's success.
Santander Stream Processing with Apache Flink
Flink is becoming the de facto standard for stream processing due to its scalability, performance, fault tolerance, and language flexibility. It supports stream processing, batch processing, and analytics through one unified system. Developers choose Flink for its robust feature set and ability to handle stream processing workloads at large scales efficiently.
Unlocking the Power of IoT: A comprehensive approach to real-time insights
In today's data-driven world, the Internet of Things (IoT) is revolutionizing industries and unlocking new possibilities. Join Data Reply, Confluent, and Imply as we unveil a comprehensive solution for IoT that harnesses the power of real-time insights.
Workshop híbrido: Stream Processing con Flink
El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real.
Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real.
En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace.
In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms.
You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes.
Don't miss out on this opportunity to learn from industry experts and take your business to the next level.
AWS Immersion Day Mapfre - Confluent
La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.
Eventos y Microservicios - Santander TechTalk
Durante esta sesión examinaremos cómo el mundo de los eventos y los microservicios se complementan y mejoran explorando cómo los patrones basados en eventos nos permiten descomponer monolitos de manera escalable, resiliente y desacoplada.
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
This document discusses networking options and best practices for Confluent Cloud. It provides an overview of public endpoints, private link, and peering options. It then discusses best practices for private networking architectures on Azure using hub-and-spoke and private link designs. Finally, it addresses networking considerations and challenges for Kafka Connect managed connectors, as well as planned enhancements for DNS peering and outbound private link support.
Citi TechTalk Session 2: Kafka Deep Dive
Purpose of the session is to have a dive into Apache, Kafka, Data Streaming and Kafka in the cloud
- Dive into Apache Kafka
- Data Streaming
- Kafka in the cloud
Build real-time streaming data pipelines to AWS with Confluent
Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.
Q&A with Confluent Professional Services: Confluent Service Mesh
No matter whether you are migrating your Kafka cluster to Confluent Cloud, running a cloud-hybrid environment or are in a different situation where data protection and encryption of sensitive information is required, Confluent Service Mesh allows you to transparently encrypt your data without the need to make code changes to you existing applications.
Citi Tech Talk: Event Driven Kafka Microservices
Microservices have become a dominant architectural paradigm for building systems in the enterprise, but they are not without their tradeoffs. Learn how to build event-driven microservices with Apache Kafka
Confluent & GSI Webinars series - Session 3
An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities.
It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks.
This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.
Citi Tech Talk: Messaging Modernization
This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.
Citi Tech Talk: Data Governance for streaming and real time data
Getting to grips with governance on realtime data: Data Discovery, Data Catalogues, Schema Management, Lineage
Confluent & GSI Webinars series: Session 2
This session will show why the old paradigm does not work and that a new approach to the data strategy needs to be taken. It aims to show how a Data Streaming Platform is integral to the evolution of a company’s data strategy and how Confluent is not just an integration layer but the central nervous system for an organisation
Data In Motion Paris 2023
Vous apprendrez également à :
• Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données
• Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience
• Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés
Confluent Partner Tech Talk with Synthesis
A discussion on the arduous planning process, and deep dive into the design/architectural decisions.
Learn more about the networking, RBAC strategies, the automation, and the deployment plan.
More from confluent (20)
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
Recently uploaded
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Project Management Semester Long Project - Acuity
Acuity is an innovative learning app designed to transform the way you engage with knowledge. Powered by AI technology, Acuity takes complex topics and distills them into concise, interactive summaries that are easy to read & understand. Whether you're exploring the depths of quantum mechanics or seeking insight into historical events, Acuity provides the key information you need without the burden of lengthy texts.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Recently uploaded (20)
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Kafka and Kafka Streams in the Global Schibsted Data Platform
- 2. •Data Platform Architect City of Oslo •Former Data Engineer Data Platform FredrikVraalsen WHOAMI?
- 3. SCHIBSTED
- 4. SCHIBSTED 22 countries 200 million users/month 20 billion pageviews/month
- 10. THEISSUES • “High” latency (~30 seconds) • Delivery guarantees • On-boarding • Manual configuration • Homegrown solution
- 11. VISION • Data-driven applications • Self-serve • GDPR • Performance • State of the art
- 12. WHY ?
- 13. STREAMPROCESSING
- 14. • Lightweight library • Streams and Tables • High-level DSL • Low level API KAFKASTREAMS
- 15. YGGDRASILWASBORN
- 22. DATAQUALITY
- 24. GROWINGPAINS
- 25. SCALINGUP
- 29. SELFSERVE
- 30. SELFSERVE • Challenge: Transformations and routing • Multiple configurations • Who maintains? • Required Scala knowledge
- 31. { "time": round(parse-time(.published, "yyyy-MM-dd'T'HH:mm:ssX") * 1000), "device_manufacturer": .device.manufacturer, "device_model": .device.model, "language": .device.acceptLanguage, "os_name": .device.osType, "os_version": .device.osVersion, "platform": .device.platformType, "user_properties": { "is_logged_in" : boolean(.actor."spt:userId") } } SELFSERVE • JSLT – DSL for JSON transformation & queries https://github.com/schibsted/jslt
- 32. SELFSERVE
- 35. Data Platform – Oslo Origo
- 36. Oslo Origo • Digital transformation • Smarter services • Data driven
- 38. Data Platform – Oslo Origo https://www.flickr.com/photos/boaski/8079390195
- 39. Data Platform – Oslo Origo • Creating value from our data • Awareness of opportunities • Simple and safe data access • Insights and decision making • Collaboration and sharing