The New York Times stores every piece of content it has ever published in Apache Kafka. It takes a log-based approach where all content is published as events to a central "Monolog" topic in Kafka. This log becomes the single source of truth, and all applications consume from the log. The content is stored in Kafka as Protobuf messages with a well-defined schema. A separate "Skinny Log" topic contains notifications that content was processed, to enable caching and track service level objectives. This log-based architecture aims to decouple producers and consumers of content and make the system more flexible and scalable.
Grokking Techtalk #39: How to build an event driven architecture with Kafka ...Grokking VN
Bài talk chia sẻ về quá trình 2 năm ứng dụng Kafka và Kafka Connect để chuyển đổi mô hình hệ thống của Vexere từ Monolithic thành Microservice, Event Driven:
+ Event driven architecture là gì?
+ Làm thế nào để xây dựng 1 hệ thống event driven architecture một cách hiệu qủa bằng Kafka và Kafka Connect
+ Các use case hữu ích với Kafka & Kafka Connect
+ Kinh nghiệm thực tế và các bài học rút ra
- Về diễn giả: Anh Nguyễn Văn Lợi là một kỹ sư phần mềm với hơn 9 năm kinh nghiệm thực tế từ các công ty có hệ thống lớn trong các mảng VoIP, Ecommerce, Big Data, Logistics. Tại Vexere, anh luôn đề cao tinh thần tự học hỏi, phát triển và chia sẻ để team member liên tục tích lũy kiến thức, kỹ năng, nhằm tăng hiệu quả công việc và mang lại sản phẩm có trải nghiệm tốt nhất cho người dùng.
Writing Continuous Applications with Structured Streaming in PySparkDatabricks
We are in the midst of a Big Data Zeitgeist in which data comes at us fast, in myriad forms and formats at intermittent intervals or in a continuous stream, and we need to respond to streaming data immediately. This need has created a notion of writing a streaming application that reacts and interacts with data in real-time. We call this a continuous application. In this talk we will explore the concepts and motivations behind continuous applications and how Structured Streaming Python APIs in Apache Spark 2.x enables writing them. We also will examine the programming model behind Structured Streaming and the APIs that support them. Through a short demo and code examples, Jules will demonstrate how to write an end-to-end Structured Streaming application that reacts and interacts with both real-time and historical data to perform advanced analytics using Spark SQL, DataFrames, and Datasets APIs.
Developing event-driven microservices with event sourcing and CQRS (phillyete)Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC. In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
Grokking Techtalk #39: How to build an event driven architecture with Kafka ...Grokking VN
Bài talk chia sẻ về quá trình 2 năm ứng dụng Kafka và Kafka Connect để chuyển đổi mô hình hệ thống của Vexere từ Monolithic thành Microservice, Event Driven:
+ Event driven architecture là gì?
+ Làm thế nào để xây dựng 1 hệ thống event driven architecture một cách hiệu qủa bằng Kafka và Kafka Connect
+ Các use case hữu ích với Kafka & Kafka Connect
+ Kinh nghiệm thực tế và các bài học rút ra
- Về diễn giả: Anh Nguyễn Văn Lợi là một kỹ sư phần mềm với hơn 9 năm kinh nghiệm thực tế từ các công ty có hệ thống lớn trong các mảng VoIP, Ecommerce, Big Data, Logistics. Tại Vexere, anh luôn đề cao tinh thần tự học hỏi, phát triển và chia sẻ để team member liên tục tích lũy kiến thức, kỹ năng, nhằm tăng hiệu quả công việc và mang lại sản phẩm có trải nghiệm tốt nhất cho người dùng.
Writing Continuous Applications with Structured Streaming in PySparkDatabricks
We are in the midst of a Big Data Zeitgeist in which data comes at us fast, in myriad forms and formats at intermittent intervals or in a continuous stream, and we need to respond to streaming data immediately. This need has created a notion of writing a streaming application that reacts and interacts with data in real-time. We call this a continuous application. In this talk we will explore the concepts and motivations behind continuous applications and how Structured Streaming Python APIs in Apache Spark 2.x enables writing them. We also will examine the programming model behind Structured Streaming and the APIs that support them. Through a short demo and code examples, Jules will demonstrate how to write an end-to-end Structured Streaming application that reacts and interacts with both real-time and historical data to perform advanced analytics using Spark SQL, DataFrames, and Datasets APIs.
Developing event-driven microservices with event sourcing and CQRS (phillyete)Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC. In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
This is the presentation I did at Apache Asia Roadshow 2009 held at Colombo, Sri Lanka. My talk was titled "Introduction to Apache Synapse". In this presentation, I attempt to address areas like enterprise integration problems, ESB pattern, Synapse architecture, features and the configuration model.
The talk cover concepts and internal mechanisms of how PostgreSQL, a popular open-source database, operates. While doing so, I'll also draw similarities to other RDBMS like Oracle, MySQL or SQL Server.
Some topics to touch during this presentation:
- PostgreSQL internal concepts: table, index, page, heap, vacuum, toast, etc.
- MVCC and relational transactions
- Indexes and how they affect performance
- Discuss on Uber's blog post about moving from PostgreSQL to MySQL
The talk is suitable for technical audience who has worked with databases before (software engineers/data analysts) and want to learn about its internal mechanism.
Speaker: Huy Nguyen, CTO & Cofounder, Holistics Software
Huy's currently CTO of Holistics, a Business Intelligence (BI) and Data Infrastructure product. Holistics helps customers generate reports and insights from their data. Holistics customers include tech companies like Grab, Traveloka, The Coffee House, Tech In Asia and e27.
Before Holistics, Huy worked at Viki, helping build their end-to-end data platform that scale to over 100M records a day. Previously, Huy spent a year writing medical simulation in Europe, and did an internship with Facebook HQ working for their growth team.
Huy's proudest achievement is 251 scores on Flappy Bird.
Language: Vietnamese, with slides in English.
Building Cloud-Native App Series - Part 5 of 11
Microservices Architecture Series
Microservices Architecture,
Monolith Migration Patterns
- Strangler Fig
- Change Data Capture
- Split Table
Infrastructure Design Patterns
- API Gateway
- Service Discovery
- Load Balancer
Building Event-Driven (Micro) Services with Apache KafkaGuido Schmutz
This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Introducing some strategies in Database Planning for Software-as-a-Service (SaaS) company. With helping of Microservices architecture, we can design a low-price and safe storage engine for SaaS platform.
Apache Kafka 0.8 basic training - VerisignMichael Noll
Apache Kafka 0.8 basic training (120 slides) covering:
1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka
2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers
3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning
4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps
5. Playing with Kafka using Wirbelsturm
Audience: developers, operations, architects
Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/
Verisign is a global leader in domain names and internet security.
Tools mentioned:
- Wirbelsturm (https://github.com/miguno/wirbelsturm)
- kafka-storm-starter (https://github.com/miguno/kafka-storm-starter)
Blog post at:
http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/
Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!
Real-Time Stream Processing with KSQL and Apache Kafkaconfluent
Real Time Stream Processing with KSQL and Kafka
David Peterson, Confluent APAC
APIdays Melbourne 2018
Unordered, unbounded and massive datasets are increasingly common in day-to-day business. Using this to your advantage is incredibly difficult with current system designs. We are stuck in a model where we can only take advantage of this *after* it has happened. Many times, this is too late to be useful in the enterprise.
KSQL is a streaming SQL engine for Apache Kafka. KSQL lowers the entry bar to the world of stream processing, providing a simple and completely interactive SQL interface for processing data in Kafka. KSQL (like Kafka) is open-source, distributed, scalable, and reliable.
A real time Kafka platform moves your data up the stack, closer to the heart of your business, allowing you to build scalable, mission-critical services by quickly deploying SQL-like queries in a severless pattern.
This talk will highlight key use cases for real time data, and stream processing with KSQL: Real time analytics, security and anomaly detection, real time ETL / data integration, Internet of Things, application development, and deploying Machine Learning models with KSQ.
Real time data and stream processing means that Kafka is just as important to the disrupted as it is to the disruptors.
Grokking TechTalk #33: High Concurrency Architecture at TIKIGrokking VN
- Speaker: Nguyễn Hoàng Bách - Senior Principal Engineer @ TIKI
Trải qua 9 năm xây dựng và phát triển hệ thống, đội ngũ engineer TIKI lần lượt phải giải quyết từng bài toán kỹ thuật khó khăn để hệ thống phát triển theo kịp tốc độ tăng trưởng của business. Đặc thù của hệ thống Ecommerce có một thách thức lớn là phải đảm bảo tính chính xác của dữ liệu nhưng đồng thời vẫn phải đáp ứng lượng truy cập lớn. Do đó High Concurrency Architecture có vai trò quan trọng trong kiến trúc tổng thể của TIKI. Nó cũng là bước tiến lớn của các kỹ sư TIKI trong 6 tháng qua.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
Rainbird: Realtime Analytics at Twitter (Strata 2011)Kevin Weil
Introducing Rainbird, Twitter's high volume distributed counting service for realtime analytics, built on Cassandra. This presentation looks at the motivation, design, and uses of Rainbird across Twitter.
Achieving Lakehouse Models with Spark 3.0Databricks
It’s very easy to be distracted by the latest and greatest approaches with technology, but sometimes there’s a reason old approaches stand the test of time. Star Schemas & Kimball is one of those things that isn’t going anywhere, but as we move towards the “Data Lakehouse” paradigm – how appropriate is this modelling technique, and how can we harness the Delta Engine & Spark 3.0 to maximise it’s performance?
This is the presentation I did at Apache Asia Roadshow 2009 held at Colombo, Sri Lanka. My talk was titled "Introduction to Apache Synapse". In this presentation, I attempt to address areas like enterprise integration problems, ESB pattern, Synapse architecture, features and the configuration model.
The talk cover concepts and internal mechanisms of how PostgreSQL, a popular open-source database, operates. While doing so, I'll also draw similarities to other RDBMS like Oracle, MySQL or SQL Server.
Some topics to touch during this presentation:
- PostgreSQL internal concepts: table, index, page, heap, vacuum, toast, etc.
- MVCC and relational transactions
- Indexes and how they affect performance
- Discuss on Uber's blog post about moving from PostgreSQL to MySQL
The talk is suitable for technical audience who has worked with databases before (software engineers/data analysts) and want to learn about its internal mechanism.
Speaker: Huy Nguyen, CTO & Cofounder, Holistics Software
Huy's currently CTO of Holistics, a Business Intelligence (BI) and Data Infrastructure product. Holistics helps customers generate reports and insights from their data. Holistics customers include tech companies like Grab, Traveloka, The Coffee House, Tech In Asia and e27.
Before Holistics, Huy worked at Viki, helping build their end-to-end data platform that scale to over 100M records a day. Previously, Huy spent a year writing medical simulation in Europe, and did an internship with Facebook HQ working for their growth team.
Huy's proudest achievement is 251 scores on Flappy Bird.
Language: Vietnamese, with slides in English.
Building Cloud-Native App Series - Part 5 of 11
Microservices Architecture Series
Microservices Architecture,
Monolith Migration Patterns
- Strangler Fig
- Change Data Capture
- Split Table
Infrastructure Design Patterns
- API Gateway
- Service Discovery
- Load Balancer
Building Event-Driven (Micro) Services with Apache KafkaGuido Schmutz
This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Introducing some strategies in Database Planning for Software-as-a-Service (SaaS) company. With helping of Microservices architecture, we can design a low-price and safe storage engine for SaaS platform.
Apache Kafka 0.8 basic training - VerisignMichael Noll
Apache Kafka 0.8 basic training (120 slides) covering:
1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka
2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers
3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning
4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps
5. Playing with Kafka using Wirbelsturm
Audience: developers, operations, architects
Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/
Verisign is a global leader in domain names and internet security.
Tools mentioned:
- Wirbelsturm (https://github.com/miguno/wirbelsturm)
- kafka-storm-starter (https://github.com/miguno/kafka-storm-starter)
Blog post at:
http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/
Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!
Real-Time Stream Processing with KSQL and Apache Kafkaconfluent
Real Time Stream Processing with KSQL and Kafka
David Peterson, Confluent APAC
APIdays Melbourne 2018
Unordered, unbounded and massive datasets are increasingly common in day-to-day business. Using this to your advantage is incredibly difficult with current system designs. We are stuck in a model where we can only take advantage of this *after* it has happened. Many times, this is too late to be useful in the enterprise.
KSQL is a streaming SQL engine for Apache Kafka. KSQL lowers the entry bar to the world of stream processing, providing a simple and completely interactive SQL interface for processing data in Kafka. KSQL (like Kafka) is open-source, distributed, scalable, and reliable.
A real time Kafka platform moves your data up the stack, closer to the heart of your business, allowing you to build scalable, mission-critical services by quickly deploying SQL-like queries in a severless pattern.
This talk will highlight key use cases for real time data, and stream processing with KSQL: Real time analytics, security and anomaly detection, real time ETL / data integration, Internet of Things, application development, and deploying Machine Learning models with KSQ.
Real time data and stream processing means that Kafka is just as important to the disrupted as it is to the disruptors.
Grokking TechTalk #33: High Concurrency Architecture at TIKIGrokking VN
- Speaker: Nguyễn Hoàng Bách - Senior Principal Engineer @ TIKI
Trải qua 9 năm xây dựng và phát triển hệ thống, đội ngũ engineer TIKI lần lượt phải giải quyết từng bài toán kỹ thuật khó khăn để hệ thống phát triển theo kịp tốc độ tăng trưởng của business. Đặc thù của hệ thống Ecommerce có một thách thức lớn là phải đảm bảo tính chính xác của dữ liệu nhưng đồng thời vẫn phải đáp ứng lượng truy cập lớn. Do đó High Concurrency Architecture có vai trò quan trọng trong kiến trúc tổng thể của TIKI. Nó cũng là bước tiến lớn của các kỹ sư TIKI trong 6 tháng qua.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
Rainbird: Realtime Analytics at Twitter (Strata 2011)Kevin Weil
Introducing Rainbird, Twitter's high volume distributed counting service for realtime analytics, built on Cassandra. This presentation looks at the motivation, design, and uses of Rainbird across Twitter.
Achieving Lakehouse Models with Spark 3.0Databricks
It’s very easy to be distracted by the latest and greatest approaches with technology, but sometimes there’s a reason old approaches stand the test of time. Star Schemas & Kimball is one of those things that isn’t going anywhere, but as we move towards the “Data Lakehouse” paradigm – how appropriate is this modelling technique, and how can we harness the Delta Engine & Spark 3.0 to maximise it’s performance?
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022HostedbyConfluent
Knock Knock, Who’s There? With Justin Chen and Dhruv Jauhar | Current 2022
When managing multi-tenant Kafka clusters, the ability to identify individual clients is crucial but can be challenging to achieve.
Previously at Shopify, a single SSL certificate was used by nearly all clients to connect to our Kafka clusters. As Kafka distinguishes users based on their certificate’s subject, all clients were masked as the same user, and thus we were unable to identify who was connecting and what operations they were enacting. As a result, ensuring proper data usage and ownership was not possible, as the producers and consumers of each Kafka topic were unknown. Without this insight, protective measures such as request quotas and access-control were rendered impractical.
We set out to provide our thousands of Kafka clients with their own unique identity, by automating certificate management within our Kubernetes platform via controllers. In providing our clients with their own identities, we subsequently improved observability around data usage/ownership by implementing a custom Kafka authorizer to gather client request metrics. Lastly, we enhanced our Kafka infrastructure’s resiliency by introducing request quotas and protected our clusters from unintended operations with user access-control.
A little presentation about how we squeezed Symfony to implement some sort of SOA
better see here
https://docs.google.com/presentation/d/1TnDjbERj8F6UgqbZ7fJQsAL6yfg5G56G1iTqSzeTUE8/pub?start=false&loop=false&delayms=3000
Monoliths are so 2001 – What you need is ModularityGraham Charters
Presentation given at IBM InterConnect 2015 conference. Describes:
- the motivation for modularity
- issues with modularity in Java
- introduction to OSGi and WebSphere OSGi Applications
- strategy for adopting OSGi with existing Java EE applications, using a sample (AcmeAir) as a use case
Having used apache pulsar in production for an year for our pub sub use cases such as stream analytics, event sourcing etc, this slide deck presents the lesson learned per se understanding the architecture, tuning the cluster, managing to keep it highly available and fault tolerant and much more.
While the slides are presented in terms of apache pulsar, a lot of the concepts can be easily extended to a lot of distributed systems.
The views here are my own and do not represent the view of nutanix corporation.
Resilient Event Driven Systems With KafkaIccha Sethi
Talk at Craft conf 2018 on how to build resilient event driven systems with Kafka. Walks through an example of building notifications feature for chat app Stride.
It is official: MySQL listens to HTTP and speaks JSON. MySQL got a new plugin that lets HTTP clients and JavaScript users connect to MySQL using HTTP. The development preview brings three APIs: key-document for nested JSON documents, CRUD for JSON mapped SQL tables and plain SQL with JSON replies. More so: MySQL 5.7.4 has SQL functions for modifying JSON, for searching documents and new indexing methods!
Boilerplates: Step up your Web Development ProcessFibonalabs
Having a boilerplate for your project, whether you are a developer or a designer, is always a good idea. It saves a lot of time from going back and forth between setting up the scaffolding and getting your hands dirty with code; it also helps in sharing your code with others. You can use it to get a head start on new projects and work on the core functionality. Using custom-designed React boilerplates shortened our initial sprints from 2 weeks to 3 days!
ELK Stack workshop covers real-world use cases and works with the participants to - implement them. This includes Elastic overview, Logstash configuration, creation of dashboards in Kibana, guidelines and tips on processing custom log formats, designing a system to scale, choosing hardware, and managing the lifecycle of your logs.
Lessons from managing a Pulsar cluster (Nutanix)StreamNative
In this presentation, we will cover:
- How to performance test and optimize a Pulsar cluster. We will present how we load tested Pulsar with locust and, following this, how we tuned our configurations for our use cases.
- Event sourcing pattern with Apache Pulsar. Avro schema usage, compatibility choices and schema evolution on pulsar topics that worked for us.
- Bonus: How we source Apache Flink from apache pulsar and run our workflows.
By attending this webinar, you can expect to come away with:
- How to performance test a Pulsar cluster for your use case.
- How to leverage the highly configurable broker and Bookkeeper to suit your needs.
- Event sourcing patterns on top of Apache Pulsar.
- Avro schema usage, compatibility choices, and evolution.
- Familiarise with pulsar connector for Flink and possible use cases.
Ever wonder how some applications are built? Ever wonder how to combine components of the Windows Azure platform? Stop wondering and learn how we’ve built MyGet.org, a multi-tenant software-as-a-service. In this session we’ll discuss architecture, commands, events, access control, multi tenancy and how to mix and match those things together. Learn about the growing pains and misconceptions we had on the Windows Azure platform. The result just may be a reliable, cost-effective solution that scales.
This slide is translated version. Originally it was written in Korean. (http://www.slideshare.net/saltynut/how-do-we-drive-tech-changes )
It describes how do we drive technical changes onto our organizations had used old-fashioned java combinations(Java 1.6+Spring 3.x+MyBatis) and monolithic architecture.
Key point is what we need to do to drive changes, and I'll discuss what we did during Phase1 and what we are doing at Phase 2 for architecture, frontend, backend, methodologies/process.
Phase1
- Architecture : Frontend / Backend Separation
- Frontend : Angular.js, Grunt, Bower
- Backend : Java 1.7/Spring4, ORM
- Methodology/Process : Scrum, Git
Phase2
- Architecture : Micro-Service Architecture(MSA)
- Frontend : Content Router, E2E Test
- Backend : Polyglot, Multi-Framework
- Methodology/Process : Scrum+JIRA, Git Branch Policy, Pair Programming, Code Workshop
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
In our exclusive webinar, you'll learn why event-driven architecture is the key to unlocking cost efficiency, operational effectiveness, and profitability. Gain insights on how this approach differs from API-driven methods and why it's essential for your organization's success.
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
In today's data-driven world, the Internet of Things (IoT) is revolutionizing industries and unlocking new possibilities. Join Data Reply, Confluent, and Imply as we unveil a comprehensive solution for IoT that harnesses the power of real-time insights.
Workshop híbrido: Stream Processing con Flinkconfluent
El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real.
Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real.
En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent
Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace.
In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms.
You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes.
Don't miss out on this opportunity to learn from industry experts and take your business to the next level.
La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.
Eventos y Microservicios - Santander TechTalkconfluent
Durante esta sesión examinaremos cómo el mundo de los eventos y los microservicios se complementan y mejoran explorando cómo los patrones basados en eventos nos permiten descomponer monolitos de manera escalable, resiliente y desacoplada.
Purpose of the session is to have a dive into Apache, Kafka, Data Streaming and Kafka in the cloud
- Dive into Apache Kafka
- Data Streaming
- Kafka in the cloud
Build real-time streaming data pipelines to AWS with Confluentconfluent
Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.
Q&A with Confluent Professional Services: Confluent Service Meshconfluent
No matter whether you are migrating your Kafka cluster to Confluent Cloud, running a cloud-hybrid environment or are in a different situation where data protection and encryption of sensitive information is required, Confluent Service Mesh allows you to transparently encrypt your data without the need to make code changes to you existing applications.
Citi Tech Talk: Event Driven Kafka Microservicesconfluent
Microservices have become a dominant architectural paradigm for building systems in the enterprise, but they are not without their tradeoffs. Learn how to build event-driven microservices with Apache Kafka
Confluent & GSI Webinars series - Session 3confluent
An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities.
It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks.
This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.
Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud
How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?
This session will show why the old paradigm does not work and that a new approach to the data strategy needs to be taken. It aims to show how a Data Streaming Platform is integral to the evolution of a company’s data strategy and how Confluent is not just an integration layer but the central nervous system for an organisation
Vous apprendrez également à :
• Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données
• Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience
• Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés
Confluent Partner Tech Talk with Synthesisconfluent
A discussion on the arduous planning process, and deep dive into the design/architectural decisions.
Learn more about the networking, RBAC strategies, the automation, and the deployment plan.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
By Design, not by Accident - Agile Venture Bolzano 2024
Apache Kafka® Delivers a Single Source of Truth for The New York Times
1. The Source of Truth
2018-05-09
The New York Times
Why The New York Times
Stores Every Piece of Content
Ever Published
in Kafka
2. 1
Boerge Svingen was a founder of Fast Search &
Transfer (alltheweb.com, FAST ESP). He was later a
founder and CTO of Open AdExchange, doing
contextual advertising for online news. He is now
working on search and backend platforms at The
New York Times.
Boerge Svingen
Director of Engineering, The New York Times
3. 2
Housekeeping Items
" This session will last about an hour.
● This session will be recorded.
" You can submit your questions by entering them into the GoToWebinar panel.
" The last 10-15 minutes will consist of Q&A.
● The slides and recording will be available after the talk.
42. Why not PubSub or AWS SNS/SQS/
Kinesis?
For most use cases, Kafka is used as a
message broker, and the log is an
implementation detail.
43. Why not PubSub or AWS SNS/SQS/
Kinesis?
For this use case, the log is the point.
44. Why not PubSub or AWS SNS/SQS/
Kinesis?
Two requirements:
1. We most retain the log forever.
2. Since messages have causal relationships,
consumption must be ordered.
45. Why not PubSub or AWS SNS/SQS/
Kinesis?
Only Kafka gives us an
ordered log with infinite
retention.
46. Agenda
1. A little history
2. How things used to work
3. Log-based architectures
4. The schema
5. The Monolog
6. The Skinny Log
7. Some challenges
61. message Article {
PublicationProperties publicationProperties = 1; // Publication metadata
CreativeWork creativeWork = 2; // Editorial metadata
PromotionalProperties promotionalProperties = 3; // Promotional metadata
PrintInformation printInformation = 4; // Print metadata
Dateline dateline = 5; // Where reporting for the article occurred
int32 wordCount = 6; // The number of words of body text.
DocumentBlock body = 7; // The body of the article
}
81. The Skinny Log
Example: A list service consumes a publish
for an asset, updates the list, and posts a
notification saying that list has been updated.