Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022

•

0 likes•490 views

Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022 Despite the plentiful hype, blockchain technology hasn't solved world hunger (yet). Nor has it dealt with the issues of low data throughput, deployment complexity and high costs. We believe that bringing in another immutable distributed event store in the form of Apache Kafka presents an opportunity to leverage the benefits of both systems to create a more scalable and accessible solution. This session will explain how slow data on the blockchain can be joined together with fast data in Kafka and published out to other systems. Jan and Alex (two of Confluent’s resident crypto fans) will walk through a prototype of a distributed blockchain application that will help with electric car battery management. Car battery identity data will be stored on a blockchain whereas the actual charge state data will be streamed to Kafka as part of the electric car charging process. Bringing the data together will allow us to have an improved and secure battery lifecycle management. This session is targeted for anyone who is interested in learning about blockchain technology, event streaming, and interoperability of blockchain data with a wider business. Demo application code will be available to participants.

Technology

Off the Chain
Enriching blockchain data with real time event streaming

GM Current 22
Alex Stuart
Conﬂuent Advisory Solutions Engineer, UK
Previously: Splunk, Experian
Terrible Crypto Investor, slightly better
blockchain community builder
Jan Svoboda
Conﬂuent Senior Solutions Engineer, DE
Previously: Pivotal, IBM, HPE
Built the Solidity Stuff

What are we covering today
3
1. The State of Blockchain
3. Building with Blockchains and Kafka
2. Why Blockchain vs Kafka?
4. What we Built
5. Learnings + Q&A

Blockchain in 2022
4
$8.3B Market Opportunity, 34% CAGR
EU and US Legislation on digital assets
$17.5 B of Project and VC Funding in 2022
“The Merge”, September 15th
2022

A tale of two logs
“The humble log is an
abstraction that lies at the
heart of many systems,
from NoSQL databases to
cryptocurrencies.”
-Jay Kreps, I ❤ Logs
Immutable
Distributed
Highly Available
Real time

So where do they differ?
7
Blockchains
• Increase the data security
Tamper “proof”, vastly more
difﬁcult to change
• Truly distributed
Cross organizations, borders,
business
• Encrypted Payloads
Apache Kafka
• Ease of Integration
• High Volume processing of data
Millions of messages a second
• Use case ﬂexibility
• Scalability

How does this drive differences in our architectural
design?
8
Immutability Trust Real Scale Real Time

Understanding the Build - Use Cases
10
id address
origin_country string
year_built unit
cycles unit
blocked bool
Ethereum Smart Contract
Charging Station Master Data
Battery Charging
Transactions
Battery Status
Charging Analytics Data
Get Battery Info
Increase Battery Cycles
Battery Production
Battery Owner
id address
origin_country string
year_built unit
cycles unit
blocked bool
id address
origin_country string
year_built unit
cycles unit
blocked bool
Create Battery Info
Lock Battery

Why We Chose…
13
Solidity KSQLDB Kafka
Who owns the data?
Who can see the data?
Who can change against my
details?
What questions do we need to
ask the streams?
How easily can we abstract
away complexity for building a
prototype?
How can ingesting real time
data be accelerated?
What does the fan-out of our
data look like?

Any Questions? Please Ask!
Alex Stuart
astuart@conﬂuent.io
ajfstuart
Jan Svoboda
jsvoboda@conﬂuent.io

Similar to Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022

Introduction to blockchain

Krzysztof Bury

Back in 2017 I joined the world of #Blockchain and presented my first ever Blockchain presentation on 14th November 2017 at a Tech meetup in Penticton, BC (Canada) A few key highlights: * The fundamentals of Blockchain technology have not changed (Trust, Transparency, Security, Quality/Certainty, Reduced costs) * We were talking about Web 3.0 back in 2017 😉 * People still miss define Blockchain * Janet Yellen and Christine Lagarde were commenting on Blockchain back in 2017 * Checkout the Dapps section – I was ahead of my time there! * And the last three slides on the Market cap and increases – I was excited to tell people the market cap was $200bn, BTC was $8,000 and ETH was nearly $400 ……… Ahhh the good old days 😉 And finally – my “leaders comment” back in 2017 still stands “Blockchain is the next disruptive technology to transform the lives of our customers and our business operations” What a journey so far…and guess what…its only just begun 👍👍👍

Blockchain Introduction - Canada Nov 2017.pptx

Antony Welfare

Scaling Up, Scaling Out Blockchain Beyond 2024

Edward Tsang

blockchain-161025100639.pptx

Praveenkumar155694

This is a presentation that the CEO of HyperTrends Global Inc. Anup Marwadi (https://www.hypertrends.com) gave to the San Diego's RMA Chapter (https://www.sandiegorma.org/product/san-diego-chapter-dinner-01092019/). In this presentation, Anup goes over the fundamentals of Blockchain, Hashing, Cryptocurrencies as well as various different applications of the Blockchain as it pertains to the banking sector. Feel free to share this around.

Blockchain and Banking

HyperTrends Global Inc.

Blockchain

Software Infrastructure

Introduction to Blockchain

Jordan Harris

Blockchain as a new cyber strategy for your business

David Joao Vieira Carvalho

Abstract -- GIST Blockchain-Economy Center (BEC, Director Heung-No Lee) aims to introduce the Decentralized Secure(DeSecure) blockchains it has been developing since 2018. They aim to resolve the re-centralization problem of today’s mining market. One of the key ideas is to have the proof-of-work (PoW) puzzle time-varying from block-to-block, using the error-correction-codes (ECC). Two new blockchains based on Bitcoin and Ethereum are to be developed using new consensus algorithm based on this new ECC-PoW. Time-varying puzzles make it very difficult to develop an ASIC mining chips. As the result, with the size of network growing, the difficulty level needs not be growing as well. As such, energy spent for mining can be controlled. The proposed ECC-PoW mechanism is to be explained in details. In addition, our plan to hardfork Bitcoin and Ethereum, by replacing the SHA based PoW with the proposed ECC-PoW, and by developing two new DeSecure blockchains, i.e. BTC-ECC and ETH-ECC, is discussed. The two DeSecure blockchains will be openly shared under an open source license at Github. We address how DeSecure blockchains can be used to resolving the issue of scalability. Our schedule to release the cores (C++ and Go) and technical meet-ups will be addressed.

Ethcon seoul 2019 presentation final

Heung-No Lee

blockchain-introduction-public.pptx

AyushVerma584677

Litecoin Genesis Date - October 7, 2011 Founder Charlie Lee, a former Google and Coinbase employee. Litecoin reached a $1 billion marketcap in November 2013.[ In May 2017, Litecoin became the first of the top-5 (by market cap) cryptocurrencies to adopt Segregated Witness . Later in May of the same year, the first Lightning Network transaction was completed through litecoin, transferring 0.00000001 LTC from Zurich to San Francisco in under one second.

Litecoin Crypto Currency Bootcamp

Joseph Holbrook, Chief Learning Officer (CLO)

Implementing Commerce in the 21st century with Blockchain and Cryptocurrencies

Payson Johnston

Доклад Владимира Бичева на третьем митапе сообщества блокчейн-разработчиков С...

Дмитрий Плахов

Blockchain Security and Privacy

Anil John

V SYSTEMS Whitepaper_EN

V SYSTEMS

Blockchain ecosystem and evolution

Chandra Sekhar AKNR

Blockchain Explained

Anthony De Monroy

Paradigm Capital Blockchain, Cryptocurrency & Apps Report

Daniel Lee

Blockchain

Sai Nath

2018 SAI workshop blockchain Kristof Verslype

Smals

Similar to Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022 (20)

Introduction to blockchain

Blockchain Introduction - Canada Nov 2017.pptx

Scaling Up, Scaling Out Blockchain Beyond 2024

blockchain-161025100639.pptx

Blockchain and Banking

Blockchain

Introduction to Blockchain

Blockchain as a new cyber strategy for your business

Ethcon seoul 2019 presentation final

blockchain-introduction-public.pptx

Litecoin Crypto Currency Bootcamp

Implementing Commerce in the 21st century with Blockchain and Cryptocurrencies

Доклад Владимира Бичева на третьем митапе сообщества блокчейн-разработчиков С...

Blockchain Security and Privacy

V SYSTEMS Whitepaper_EN

Blockchain ecosystem and evolution

Blockchain Explained

Paradigm Capital Blockchain, Cryptocurrency & Apps Report

Blockchain

2018 SAI workshop blockchain Kristof Verslype

More from HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard37

In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method. We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy. Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations. Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today. Join us to discover how Zero-ETL can elevate your organization's data strategy.

The Zero-ETL Approach: Enhancing Data Agility and Insight

Safe Software

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Design and Development of a Provenance Capture Platform for Data Science

Paolo Missier

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

caitlingebhard1

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

In today's digital world, trust is key to customer relationships, but keeping it is a huge challenge. Customers are well-informed and empowered, quick to change brands if their trust is broken, even if it costs them more. This puts a lot of pressure on organizations to handle trust and safety issues with great care and transparency. The challenge, however, is real. Fragmented solutions have left privacy, legal, and security teams in a perpetual cycle of catch-up, struggling to update privacy notices, manage customer data rights, and answer lengthy security questionnaires—all while trying to prove ROI to the business. It's a thankless job, filled with repetition, tedious tasks, and constant interdepartmental coordination. Combine this with fast regulatory changes and the quick evolution of AI, and it becomes overwhelming. Join this webinar to learn more about TrustArc's new innovative solution Trust Center, the only unified, no-code online hub for trust and safety information built for privacy, security, compliance, and legal teams. Trust Center streamlines your path to compliance, shortens the pre-sales cycle, and reduces both legal and regulatory risks, saving time, effort, and cost. This webinar will review: - Why companies are building unified Trust Centers for a robust privacy program. - How unified Trust Centers streamline sales cycles, ensure regulatory compliance, and reduce operational bottlenecks. - How compliance, legal, security, GRC, and privacy teams benefit from a unified Trust Center in terms of needs, pains, and outcomes. - How TrustArc Trust Center saves time and work while reducing legal, reputational, and compliance risk by effectively managing policies, notices, terms, and disclosures, and providing real-time updates on subprocessors.

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

TrustArc

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

Understanding the FAA Part 107 License ..

Christopher Logan Kennedy

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...

WSO2

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

Recently uploaded (20)

JohnPollard-hybrid-app-RailsConf2024.pptx

The Zero-ETL Approach: Enhancing Data Agility and Insight

Six Myths about Ontologies: The Basics of Formal Ontology

Design and Development of a Provenance Capture Platform for Data Science

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Platformless Horizons for Digital Adaptability

Introduction to Multilingual Retrieval Augmented Generation (RAG)

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Understanding the FAA Part 107 License ..

Why Teams call analytics are critical to your entire business

WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022

1. Off the Chain Enriching blockchain data with real time event streaming

2. GM Current 22 Alex Stuart Conﬂuent Advisory Solutions Engineer, UK Previously: Splunk, Experian Terrible Crypto Investor, slightly better blockchain community builder Jan Svoboda Conﬂuent Senior Solutions Engineer, DE Previously: Pivotal, IBM, HPE Built the Solidity Stuff

3. What are we covering today 3 1. The State of Blockchain 3. Building with Blockchains and Kafka 2. Why Blockchain vs Kafka? 4. What we Built 5. Learnings + Q&A

4. Blockchain in 2022 4 $8.3B Market Opportunity, 34% CAGR EU and US Legislation on digital assets $17.5 B of Project and VC Funding in 2022 “The Merge”, September 15th 2022

5. A tale of two logs

6. A tale of two logs “The humble log is an abstraction that lies at the heart of many systems, from NoSQL databases to cryptocurrencies.” -Jay Kreps, I ❤ Logs Immutable Distributed Highly Available Real time

7. So where do they differ? 7 Blockchains • Increase the data security Tamper “proof”, vastly more difﬁcult to change • Truly distributed Cross organizations, borders, business • Encrypted Payloads Apache Kafka • Ease of Integration • High Volume processing of data Millions of messages a second • Use case ﬂexibility • Scalability

8. How does this drive differences in our architectural design? 8 Immutability Trust Real Scale Real Time

9. Hands On Demo 9

10. Understanding the Build - Use Cases 10 id address origin_country string year_built unit cycles unit blocked bool Ethereum Smart Contract Charging Station Master Data Battery Charging Transactions Battery Status Charging Analytics Data Get Battery Info Increase Battery Cycles Battery Production Battery Owner id address origin_country string year_built unit cycles unit blocked bool id address origin_country string year_built unit cycles unit blocked bool Create Battery Info Lock Battery

11. Understanding the Build - Architecture

12.

13. Why We Chose… 13 Solidity KSQLDB Kafka Who owns the data? Who can see the data? Who can change against my details? What questions do we need to ask the streams? How easily can we abstract away complexity for building a prototype? How can ingesting real time data be accelerated? What does the fan-out of our data look like?

14. Learnings 14

15. Any Questions? Please Ask! Alex Stuart astuart@conﬂuent.io ajfstuart Jan Svoboda jsvoboda@conﬂuent.io

Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022

Recommended

Recommended

More Related Content

Similar to Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022

Similar to Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022 (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Off the Chain: Scaling Blockchain Data With Kafka With Jan Svoboda and Alex Stuart | Current 2022