Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi

•Download as PPTX, PDF•

0 likes•37 views

Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.

Technology

1
Building a Codeless Log
Pipeline with Confluent
Sink Connector
2021

Pollyanna R. Valente
Site Reliability Engineer at Loggi which, among other things, takes
care of the company’s data pipeline. Holding a Computer Science
Degree with one year of experience working with Kafka and is an
active member of Kafka BR (Brazilians’ community Kafka).
Site Reliability Engineer at Loggi, focused on maintaining the
production environment and building a data pipeline.
Daniel Sousa

Agenda
● Loggi
● Motivation
● Our Pipeline
3

4
We are an unicorn
startup, making
millions of deliveries
throughout Brazil.

Loggi is a logistics company enabled by
technology to connect Brazil.
5

Connecting Brazil means delivering
everything to everyone everywhere.
6

Sources: Ebit (Brazil), NBC (USA), Xinhua (China), Italian Trade Agency (China) , US Department of Commerce (USA)
7
1. Client pickup 2. Cross-docking 3. Mini hub 4. Last-mile
● Loggi One mobile app for senders
● Loggi One web app for senders
● API integration for senders
● Driver mobile app
● XD App mobile app for operators
● Warehouse Management System (WMS)
● Integration with sorters and IOT
● XD App mobile app for operators
● Integration with sorters and IOT
● Driver mobile app
● Loggi One web app for recipients
● Proprietary route optimization algos
● Independent contractor for pickup
● Loggi Leve: franchisees with dedicated
fleets
● Cross-docking warehouses leased on
Loggi’ balance sheet
● Operators are mainly employees
● Some urbans hubs are leased and
operated directly by Loggi
● Loggi Leve: franchisees own and operate
their hubs
● Independent contractors
● Loggi Leve: franchises with dedicated fleet
Building from scratch the next generation delivery network
Harnessing the power of available technologies at every step: mobile, machine learning, automation, IOT and cloud computing
Tech
Stack
Asset
Intensity
HIGH
LOW
3
HIGH
LOW
9
HIGH
LOW
6
HIGH
LOW
2

8
Motivation
Recently, some teams came up with a new
demand to perform analytical explorations
joining structured log messages with business
data, stored in relational databases. This would
allow them to answer questions such as, What
is the percentage of packages with an
unsuccessful first delivery, for example.
So, we need to find a way to put the two
information on the same place to the team be
allowed do consume easily both information
and work with them together.

(Joseph deBuzna + Zulfikar Quereshi, HVR) Kafka Summit SF 2018 This presentation is a customer story about France-based regional airline HOP! and their need to make better use of data that was contained in various applications. They also needed this information to be available in real time. As one can imagine, airlines manage a wide variety of information such as weather, customer information, flight plans, sensor data from planes and much more. In this presentation, Joe will discuss how HOP! was delivering their data before and the limitations associated with delivering this data. Joe will then talk about HOP!’s selection of Kafka and HVR as a solution to enabling data availability and real-time information for analysis and action. In this session, attendees will learn: -How Kafka was selected and chosen as a solution for HOP!’s complex challenges -Architecture and capabilities implemented that enabled data feeding from multiple sources to Kafka -Considerations and challenges with this approach -Business results and future plans

Real-Time Market Data Analytics Using Kafka Streams

confluent

(Lei Chen, Bloomberg, L.P.) Kafka Summit SF 2018 At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing. We’ll cover: -Our system architecture -Best practices of using the Processor API and State Store API -Dynamic gap session implementation -Historical data re-processing practice in KStreams app -Chaining multiple KStreams apps with Spark Streaming job

Kafka Vienna Meetup 020719

Patrik Kleindl

Comparing three data ingestion approaches where Apache Kafka integrates with ...

HostedbyConfluent

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.

Kafka & InfluxDB: BFFs for Enterprise Data Applications | Russ Savage, Influx...

HostedbyConfluent

Modern data processing applications built on Kafka and InfluxDB deliver the performance, reliability, and flexibility that customers need for robust real-time data pipeline solutions. As the saying goes, the pipeline is greater than the sum of its Kafka and InfluxDB parts. In this session, Russ Savage, Director of Product Management at InfluxData will discuss basic concepts of integrating Kafka and InfluxDB while highlighting how companies are creating fault-tolerant, scalable and fast data pipelines with the power of InfluxDB and Kafka.

Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...

confluent

(Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018 Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications. This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures. Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.

Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay V...

confluent

After migrating much of our tech stack to Kafka Streams and Kafka Connect, some of our newer streaming applications encountered the same problem: while their primary input data was available in Kafka and their static side input data could be ingested using traditional connectors, they still needed access to semi-static data that has historically been acquired through in-line synchronous requests and responses. External service calls made from within a Kafka Streams application are highly discouraged, as they make error-handling difficult and typically slow down the application. To solve this problem, we used Kafka Connect components to build a large-scale data subscription service which subscribes to, polls, and disseminates auxiliary data to Kafka topics for immediate consumption by our streaming applications. This talk will cover: - An overview of our subscription service architecture - A deep-dive into our Kafka Connector and each of its custom components - Design considerations and various pitfalls that we encountered along the way

TBD Data Governance | David Araujo and Michael Agnich, Confluent

HostedbyConfluent

In this session Mike & David will walk you through how to navigate and govern your data in motion with a new suite of products from Confluent. Have you ever been in endless meetings to try to find out what, where and who is responsible for the data or searching through chaotic spreadsheets and wikis for this information? Confluent is excited to show you their vision to democratize your data in motion.

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

confluent

Should we manage events like APIs? | Kim Clark, IBM

HostedbyConfluent

APIs have become ubiquitous as a way of exposing the capabilities of the enterprise both internally and externally. However, are APIs alone enough? There is a strong resurgence in interest in asynchronous communication and event driven architecture. Applications want to receive events immediately so they can respond in real time, and furthermore they also want the benefit of being decoupled from the availability and performance characteristics of the systems providing that data. However, whilst the way that APIs are socialized, exposed, versioned etc. is well matured in the form of API management technology. We are now on the cusp of seeing first class support for event endpoint management to provide the same sophistication for discovering, exposing and consuming events.

Data governance and discoverability at AO.com | Jon Vines, AO.com and Christo...

HostedbyConfluent

One challenge of widespread adoption of any technology within an organisation is balancing organic growth and maintaining standards and best practice. At AO.com - one of the UK's largest online electrical retailers - we’ve invested in tooling to simplify application onboarding into the event processing platform. This includes creating topics, defining access to the platform and supporting our governance functions. A key part of any data function is data governance and discoverability. Through standardised definitions of Kafka topics and use of Avro schemas, we can map which topics exist, what data they contain and who has access to them. This allows us to support multiple cross-functional teams using automatic data gathering. In this session, AO.com and Confluent Professional Services will share how we tackled the challenge of platform adoption and provide hands-on examples of the open-source tools, ""Kafka Clusterstate Tools"" and ""Kafka Streams Inspector"", we developed.

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

confluent

(Bob Lehmann, Bayer) Kafka Summit SF 2018 You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why? In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform. In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...

HostedbyConfluent

Connected and Automated Vehicle education (CAVe)-in-a-box is an educational tool developed under the FHWA Workforce Development project. CAVe-in-a-box is an interconnected set of Intelligent transportation system equipment that was developed to serve as training and educational resources for the emerging Intelligent transportation system workforce. (CAVe)-in-a-box uses Kafka for real-time data streaming, data processing, visualization, and data collection. In this talk, we explore how Kafka is being used in cutting-edge connected and automated vehicle research.

Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...

HostedbyConfluent

Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...

HostedbyConfluent

Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more. In looking at the tradeoffs behind a system like Kafka, we learn to reason more clearly about distributed systems and to make high-stakes technology adoption decisions more effectively. These are skills we all want to improve! In this talk, we'll examine trade-offs on which our favorite distributed messaging system takes opinionated positions: - Whether to store data contiguously or using an index - How many storage tiers are best? - Where should metadata live? - And more. It's always useful to dissect a modern distributed system with the goal of understanding it better, and it's even better to learn to deeper architectural principles in the process. Come to this talk for a generous helping of both.

Confluent Cloud Networking | Rajan Sundaram, Confluent

HostedbyConfluent

Introduction to networking options available in Confluent Cloud Self Serve provisioning of confluent Kafka clusters. VPC Peering, VNet Peering, Transit Gateway and Private Link Options for AWS, GCP, Azure networking offering. Caveats of confluent's cloud networking solutions customers should be aware of. Details of two major pieces of the architecture of Confluent Cloud - Data Plane Network and Control Plane.

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

Introducing Events and Stream Processing into Nationwide Building Society

confluent

Watch this talk here: https://www.confluent.io/online-talks/introducing-events-and-stream-processing-nationwide-building-society Open Banking regulations compel the UK’s largest banks, and building societies to enable their customers to share personal information with other regulated companies securely. As a result companies such as Nationwide Building Society are re-architecting their processes and infrastructure around customer needs to reduce the risk of losing relevance and the ability to innovate. In this online talk, you will learn why, when facing Open Banking regulation and rapidly increasing transaction volumes, Nationwide decided to take load off their back-end systems through real-time streaming of data changes into Apache Kafka®. You will hear how Nationwide started their journey with Apache Kafka®, beginning with the initial use case of creating a real-time data cache using Change Data Capture, Confluent Platform and Microservices. Rob Jackson, Head of Application Architecture, will also cover how Confluent enabled Nationwide to build the stream processing backbone that is being used to re-engineer the entire banking experience including online banking, payment processing and mortgage applications. View now to: -Explore the technologies used by Nationwide to meet the challenges of Open Banking -Understand how Nationwide is using KSQL and Kafka Streams Framework to join topics and process data. -Learn how Confluent Platform can enable enterprises such as Nationwide to embrace the event streaming paradigm -See a working demo of the Nationwide system and what happens when the underlying infrastructure breaks.

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Closing Keynote: The Physics of Streaming | Tim Berglund, Confluent | Kafka S...

HostedbyConfluent

Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...

HostedbyConfluent

Your mainframe and IBM i platforms do hard work for your business, supporting essential computing transactions every day. However, mainframe data does not easily integrate with the cloud platforms driving data-driven, real-time, analytics-focused business processes. Integrating data from this critical technology often results in high costs, missed deadlines, and unhappy customers. So, what can you do? Join us to hear how Precisely Connect can help use the power of Apache Kafka to eliminate data silos and make cloud-based, event-driven data architectures a reality. Start your cloud transformation journey today, knowing you don’t need to leave essential transaction data behind! Learn more about: • Where to begin your cloud transformation journey using mainframe and IBM i data and Apache Kafka • What you need to move mainframe and IBM i data to the cloud while reducing costs, modernizing architectures, and using the staff you have today • How Precisely Connect customers are using change data capture and Apache Kafka to deliver real-time insights to the cloud

Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...

HostedbyConfluent

Server-Sent Events (SSE) is a server push technology where clients receive automatic server updates through the secure http connection. SSE can be used in apps like live stock updates, that use one way data communications and also helps to replace long polling by maintaining a single connection and keeping a continuous event stream going through it. We used a simple Kafka producer to publish messages onto Kafka topics and developed a reactive Kafka consumer by leveraging Spring Webflux to read data from Kafka topic in non-blocking manner and send data to clients that are registered with Kafka consumer without closing any http connections. This implementation allows us to send data in a fully asynchronous & non-blocking manner and allows us to handle a massive number of concurrent connections. We’ll cover: •Push data to external or internal apps in near real time •Push data onto the files and securely copy them to any cloud services •Handle multiple third-party apps integrations

Government Track Welcome Address

HostedbyConfluent

Kafka Summit NYC 2017 - The Rise of the Streaming Platform

confluent

Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...

confluent

Watch this talk here: https://www.confluent.io/online-talks/using-apache-kafka-to-optimize-real-time-analytics-financial-services-iot-applications When it comes to the fast-paced nature of capital markets and IoT, the ability to analyze data in real time is critical to gaining an edge. It’s not just about the quantity of data you can analyze at once, it’s about the speed, scale, and quality of the data you have at your fingertips. Modern streaming data technologies like Apache Kafka and the broader Confluent platform can help detect opportunities and threats in real time. They can improve profitability, yield, and performance. Combining Kafka with Panopticon visual analytics provides a powerful foundation for optimizing your operations. Use cases in capital markets include transaction cost analysis (TCA), risk monitoring, surveillance of trading and trader activity, compliance, and optimizing profitability of electronic trading operations. Use cases in IoT include monitoring manufacturing processes, logistics, and connected vehicle telemetry and geospatial data. This online talk will include in depth practical demonstrations of how Confluent and Panopticon together support several key applications. You will learn: -Why Apache Kafka is widely used to improve performance of complex operational systems -How Confluent and Panopticon open new opportunities to analyze operational data in real time -How to quickly identify and react immediately to fast-emerging trends, clusters, and anomalies -How to scale data ingestion and data processing -Build new analytics dashboards in minutes

Mercado Libre Case Study

VMware Tanzu

With RabbitMQ, Redis, and Groovy on Grails, Mercado Libre has a set of open source capabilities and a surrounding community to help them improve development lifecycles and scale both applications and data services, ultimately improving the customer experience. As they look to future growth, they have a solid foundation with a cloud-based architecture they can rely on. To learn more, visit pivotal.io/platform-as-a-service/pivotal-cf.

Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly

SolarWinds Loggly

What's hot

Elastically Scaling Kafka Using Confluent

confluent

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

HostedbyConfluent

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

confluent

Should we manage events like APIs? | Kim Clark, IBM

HostedbyConfluent

Data governance and discoverability at AO.com | Jon Vines, AO.com and Christo...

HostedbyConfluent

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

confluent

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...

HostedbyConfluent

Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...

HostedbyConfluent

Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...

HostedbyConfluent

Confluent Cloud Networking | Rajan Sundaram, Confluent

HostedbyConfluent

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

Introducing Events and Stream Processing into Nationwide Building Society

confluent

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Closing Keynote: The Physics of Streaming | Tim Berglund, Confluent | Kafka S...

HostedbyConfluent

Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...

HostedbyConfluent

Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...

HostedbyConfluent

Government Track Welcome Address

HostedbyConfluent

Kafka Summit NYC 2017 - The Rise of the Streaming Platform

confluent

Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...

confluent

What's hot (20)

Elastically Scaling Kafka Using Confluent

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...

Should we manage events like APIs? | Kim Clark, IBM

Data governance and discoverability at AO.com | Jon Vines, AO.com and Christo...

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...

Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...

Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...

Confluent Cloud Networking | Rajan Sundaram, Confluent

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

Introducing Events and Stream Processing into Nationwide Building Society

Death of the dumb pipes: Using Apache Kafka® for Integration projects

Closing Keynote: The Physics of Streaming | Tim Berglund, Confluent | Kafka S...

Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...

Server Sent Events using Reactive Kafka and Spring Web flux | Gagan Solur Ven...

Government Track Welcome Address

Kafka Summit NYC 2017 - The Rise of the Streaming Platform

Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...

Similar to Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi

Mercado Libre Case Study

VMware Tanzu

Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly

SolarWinds Loggly

JCommerce – success stories

JCommerce

Cloud Native Application Integration With APIs

Nirmal Fernando

Cloud native application architectures focus on building applications as microservices and running them on containers that run on dynamic orchestration platforms and utilize cloud computing functionalities. Agile DevOps and continuous delivery pipelines ensure agility and speed of application development and faster time to market. These systems follow a number of design principles to ensure they are built as loosely coupled services designed for cloud scale and performance. A core design principle is the use of APIs for application integration. Underlying cloud orchestration layers provide certain functionalities for integration via APIs - be it RESTful or internal formats such as Protbuf, Thrift, gRPC, NATS, etc. APIs thus play an important role for both internal services communications as well as integration between composite apps. A cloud-native API gateway that also provides features of a full lifecycle API Management is key. In this deep dive workshop, we look at the concepts of cloud-native app integration via APIs which utilize cloud-native API management. We focus on the architecture, design concepts followed by the implementation of API led microservices and then look at the runtime component which includes DevOps, CICD and hybrid clouds.

WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...

WSO2

Smart Device Link Integration into Linux systems by Jeremiah Foster

Luxoft

Global Open Source Development 2011-2014 Review and 2015 Forecast

Sammy Fung

Ecosystem WG

TinaBregovi

Fighting legacy with hexagonal architecture and frameworkless php

Fabio Pellegrini

Molto spesso capita di venire a contatto con applicazioni legacy piuttosto datate, i classici monoliti che sono cresciuti a dismisura nel tempo accumulando debito tecnologico. A causa delle priorità di business delle aziende, non sempre si riesce ad allocare il budget e il tempo necessario per iniziare subito il processo di ristrutturazione architetturale e di rimodellazione dei dati che servirebbe. In questo talk presenterò una soluzione che mi è capitato di adottare recentemente per iniziare a ridefinire la struttura di un progetto legacy, utilizzando un approccio basato su Domain Driven Design, architettura esagonale e l’utilizzo di PHP senza framework. Vedremo come è stato creato un nuovo servizio “satellite” da zero, come sono state implementate le componenti principali, come si è tenuto il codice legacy ai margini dell’applicazione, come si è approcciato il testing, il tutto nell’ottica di poter spacchettare il monolite in microservizi in un secondo momento.

Introduction to Serverless. Oracle Fn Project.

Rolando Carrasco

Acid Tango | Corporate Brochure

Enrique Sánchez Sumozas

Alpha Recycling Case Study By PGS Software Ltd

PGS Software S.A.

Alpha Recycling is one of the fastest growing east coast recycling providers and the largest catalytic converter processor in United States characterized by huge volumes, an own fleet of freight vehicles and dedicated teams of field buyers. Processing over 150 000 converters monthly, its crucial advantage over competition is a modern business model that relies on innovative IT solutions. By being able to instantly and accurately assess the profit range for each converter thanks to a specialized application with a mobile client, the Alpha Recycling buyers can offer highest market prices to individual and institutional scrap sellers. In the meantime, a state of the art back-end ERP takes care of financial transactions, price lists, accounting and reports while communicating through several commercial API’s. PGS Software was proud to build this very specialized and comprehensive solution.

Acid tango | Corporate Brochure

Enrique Sánchez Sumozas

Next Generation Service Platform Summary 2015

Alan Quayle

V1 info tech portfolio

V1InfoTech

MF_OM_logistics_FinalS K Goel

SoftTeco - Software Development Company Profile

akrivarotava

SoftTeco is a software development company that was founded in 2008 providing custom software development services. Currently, we are providing professional services to customers located in the USA, Europe and Canada. The core strength of the company comes from our highly professional team of software engineers. We have successfully completed over 300 projects in various industrial domains with different technology platforms with the highest level of process quality with transparency development processes.

resumeGiri Bhavana

BitCraft 2017 general presentation

Michal Wasilewski

Platform governance, gestire un ecosistema di microservizi a livello enterprise

Giulio Roggero

Similar to Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi (20)

Mercado Libre Case Study

Loggly - Case Study - Datami Keeps Developer Productivity High with Loggly

JCommerce – success stories

Cloud Native Application Integration With APIs

WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...

Smart Device Link Integration into Linux systems by Jeremiah Foster

Global Open Source Development 2011-2014 Review and 2015 Forecast

Ecosystem WG

Fighting legacy with hexagonal architecture and frameworkless php

Introduction to Serverless. Oracle Fn Project.

Acid Tango | Corporate Brochure

Alpha Recycling Case Study By PGS Software Ltd

Acid tango | Corporate Brochure

Next Generation Service Platform Summary 2015

V1 info tech portfolio

MF_OM_logistics_Final

SoftTeco - Software Development Company Profile

resume

BitCraft 2017 general presentation

Platform governance, gestire un ecosistema di microservizi a livello enterprise

More from HostedbyConfluent

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Inflectra

In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring. Learn about: • The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks. • Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective. • Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification. • Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process. Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

DevOps and Testing slides at DASA Connect

Kari Kakkonen

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

UiPath Test Automation using UiPath Test Suite series, part 4

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap. The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies. Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques What will you get from this session? 1. Insights into SAP testing best practices 2. Heatmap utilization for testing 3. Optimization of testing processes 4. Demo Topics covered: Execution from the test manager Orchestrator execution result Defect reporting SAP heatmap example with demo Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

The Future of Platform Engineering

Jemma Hussein Allen

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Recently uploaded (20)

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Designing Great Products: The Power of Design and Leadership by Chief Designe...

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Elevating Tactical DDD Patterns Through Object Calisthenics

DevOps and Testing slides at DASA Connect

GraphRAG is All You need? LLM & Knowledge Graph

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

The Art of the Pitch: WordPress Relationships and Sales

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Securing your Kubernetes cluster_ a step-by-step guide to success !

Accelerate your Kubernetes clusters with Varnish Caching

UiPath Test Automation using UiPath Test Suite series, part 4

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

The Future of Platform Engineering

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Connector Corner: Automate dynamic content and events by pushing a button

Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi

1. 1 Building a Codeless Log Pipeline with Confluent Sink Connector 2021

2. Pollyanna R. Valente Site Reliability Engineer at Loggi which, among other things, takes care of the company’s data pipeline. Holding a Computer Science Degree with one year of experience working with Kafka and is an active member of Kafka BR (Brazilians’ community Kafka). Site Reliability Engineer at Loggi, focused on maintaining the production environment and building a data pipeline. Daniel Sousa

3. Agenda ● Loggi ● Motivation ● Our Pipeline 3

4. 4 We are an unicorn startup, making millions of deliveries throughout Brazil.

5. Loggi is a logistics company enabled by technology to connect Brazil. 5

6. Connecting Brazil means delivering everything to everyone everywhere. 6

7. Sources: Ebit (Brazil), NBC (USA), Xinhua (China), Italian Trade Agency (China) , US Department of Commerce (USA) 7 1. Client pickup 2. Cross-docking 3. Mini hub 4. Last-mile ● Loggi One mobile app for senders ● Loggi One web app for senders ● API integration for senders ● Driver mobile app ● XD App mobile app for operators ● Warehouse Management System (WMS) ● Integration with sorters and IOT ● XD App mobile app for operators ● Integration with sorters and IOT ● Driver mobile app ● Loggi One web app for recipients ● Proprietary route optimization algos ● Independent contractor for pickup ● Loggi Leve: franchisees with dedicated fleets ● Cross-docking warehouses leased on Loggi’ balance sheet ● Operators are mainly employees ● Some urbans hubs are leased and operated directly by Loggi ● Loggi Leve: franchisees own and operate their hubs ● Independent contractors ● Loggi Leve: franchises with dedicated fleet Building from scratch the next generation delivery network Harnessing the power of available technologies at every step: mobile, machine learning, automation, IOT and cloud computing Tech Stack Asset Intensity HIGH LOW 3 HIGH LOW 9 HIGH LOW 6 HIGH LOW 2

8. 8 Motivation Recently, some teams came up with a new demand to perform analytical explorations joining structured log messages with business data, stored in relational databases. This would allow them to answer questions such as, What is the percentage of packages with an unsuccessful first delivery, for example. So, we need to find a way to put the two information on the same place to the team be allowed do consume easily both information and work with them together.

9. 9 Our Log Pipeline

10. 10 Thank you!

Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi

Similar to Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Valente and Daniel Sousa, Loggi