EDA Governance Model: a multicloud approach based on GitOps | Alejandro Alija and Marco Laucelli, Galeo Tech
•Download as PPTX, PDF•
0 likes•372 views
This document discusses Galeo's EDA governance model based on GitOps for multi-cloud environments. The model defines a standardized platform and service catalogue using a unified documentation model consolidated in Git repositories. This allows governance procedures like project approval, infrastructure provisioning, and component deployment to be automated through a single CI/CD pipeline without additional documentation overhead. The catalogue is kept up-to-date based on what is actually deployed, facilitating reuse and standardization across projects.
Report
Share
Report
Share
Recommended
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
A fast and efficient integration of end device data into data processing systems is becoming increasingly important in the Internet of Things. Factors such as secure and reliable data transmission, real-time data processing and the analysis of huge amounts of data afterwards play a major role. In order to implement an architecture that enables smooth communication between end devices and systems, tools are needed that are designed for a type of communication that takes these factors into account.
This presentation shows the strengths and application areas of MQTT, the de-facto standard communication protocol for the Internet of Things, and Apache Kafka, which is often used for data streaming in the Internet of Things, and explains how and why the technologies complement each other ideally.
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...
As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...
Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
This document summarizes BNY Mellon's implementation of a data distribution hub to improve data management across the enterprise. Key points:
1) BNY Mellon developed a data distribution hub using a microservices architecture and Kafka to decouple data sources and consumers, centralize data transformation and security, and improve data lineage management.
2) The hub scales horizontally using three dimensions - functional decomposition of microservices, horizontal scaling within an internal cloud, and data sharding.
3) Kafka is used extensively to integrate microservices, maintain processing offsets, and allow idempotent execution.
4) Both batch and real-time data processing are supported, with the latter using Storm on Kafka and loading data into
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services.
At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure.
This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.
0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche
The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...
This document discusses testing event-driven architectures. It begins by defining common event-driven architecture patterns like event notifications and event sourcing. It then discusses brokering the complexity of event-driven architectures by describing how events are communicated between producers and consumers via channels. The document outlines what information should be included in events like payloads and headers. It also discusses the difference between orchestration and choreography in event-driven systems. It provides an example of how events can be used to mediate changes within a system using order validation. Finally, it demonstrates how to test event-driven architectures using specifications and discusses accelerating API quality through testing tools that support multiple protocols and definitions.
Recommended
IoT Data Streaming - Why MQTT and Kafka are a match made in heaven | Dominik ...
A fast and efficient integration of end device data into data processing systems is becoming increasingly important in the Internet of Things. Factors such as secure and reliable data transmission, real-time data processing and the analysis of huge amounts of data afterwards play a major role. In order to implement an architecture that enables smooth communication between end devices and systems, tools are needed that are designed for a type of communication that takes these factors into account.
This presentation shows the strengths and application areas of MQTT, the de-facto standard communication protocol for the Internet of Things, and Apache Kafka, which is often used for data streaming in the Internet of Things, and explains how and why the technologies complement each other ideally.
How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...
As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.
Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...
Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
Kafka Summit NYC 2017 - Achieving Predictability and Compliance with BNY Mell...
This document summarizes BNY Mellon's implementation of a data distribution hub to improve data management across the enterprise. Key points:
1) BNY Mellon developed a data distribution hub using a microservices architecture and Kafka to decouple data sources and consumers, centralize data transformation and security, and improve data lineage management.
2) The hub scales horizontally using three dimensions - functional decomposition of microservices, horizontal scaling within an internal cloud, and data sharding.
3) Kafka is used extensively to integrate microservices, maintain processing offsets, and allow idempotent execution.
4) Both batch and real-time data processing are supported, with the latter using Storm on Kafka and loading data into
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services.
At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure.
This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.
0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche
The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.
Testing Event Driven Architectures: How to Broker the Complexity | Frank Kilc...
This document discusses testing event-driven architectures. It begins by defining common event-driven architecture patterns like event notifications and event sourcing. It then discusses brokering the complexity of event-driven architectures by describing how events are communicated between producers and consumers via channels. The document outlines what information should be included in events like payloads and headers. It also discusses the difference between orchestration and choreography in event-driven systems. It provides an example of how events can be used to mediate changes within a system using order validation. Finally, it demonstrates how to test event-driven architectures using specifications and discusses accelerating API quality through testing tools that support multiple protocols and definitions.
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream processing allows analyzing streaming data in real-time using tools like KSQL. KSQL builds on Kafka Streams to provide a SQL-like interface for transforming, filtering, and creating derived streams from immutable event data ingested into Kafka. Jupiter, a digital banking platform, uses KSQL to reconcile transactions from different interfaces in real-time and check rewards eligibility by analyzing consolidated transaction streams. While powerful, KSQL deployments require monitoring, richer SQL support, and integration with big data platforms for long-term storage.
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more. In looking at the tradeoffs behind a system like Kafka, we learn to reason more clearly about distributed systems and to make high-stakes technology adoption decisions more effectively. These are skills we all want to improve!
In this talk, we'll examine trade-offs on which our favorite distributed messaging system takes opinionated positions:
- Whether to store data contiguously or using an index
- How many storage tiers are best?
- Where should metadata live?
- And more.
It's always useful to dissect a modern distributed system with the goal of understanding it better, and it's even better to learn to deeper architectural principles in the process. Come to this talk for a generous helping of both.
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
(Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018
Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications.
This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures.
Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions
Developing custom transformation in the Kafka connect to minimize data redund...
Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost.
in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.
Digital transformation: Highly resilient streaming architecture and strategie...
Failure is inevitable in any distributed system but anticipating failures and building systems to recover from failures instantaneous makes the system highly resilient. At Capital One we process billions of events everyday and we leverage cloud, microservices, streaming and machine learning technologies to solve customer problems and provide the best customer experience.
As part of this session I will be talking about highly resilient streaming architecture that is supporting processing of billions of events every day then some of the strategies & best practices to build highly available and fault-tolerant systems utilizing Kafka and Cloud environments.
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Guru Sattanathan, Confluent, Senior Solutions Engineer
Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats!
Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts?
If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today.
In this session, we will go in this order:
-Recap on Enterprise Integration technologies
-What are the key flaws & What needs improvement?
-What is Apache Kafka?
-Rethinking Integration using Apache Kafka
https://www.meetup.com/KafkaMelbourne/events/280590162/
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
Joins in Kafka Streams and ksqlDB are a killer-feature for data processing and basic join semantics are well understood. However, in a streaming world records are associated with timestamps that impact the semantics of joins: welcome to the fabulous world of _temporal_ join semantics. For joins, timestamps are as important as the actual data and it is important to understand how they impact the join result.
In this talk we want to deep dive on the different types of joins, with a focus of their temporal aspect. Furthermore, we relate the individual join operators to the overall ""time engine"" of the Kafka Streams query runtime and explain its relationship to operator semantics. To allow developers to apply their knowledge on temporal join semantics, we provide best practices, tip and tricks to ""bend"" time, and configuration advice to get the desired join results. Last, we give an overview of recent, and an outlook to future, development that improves joins even further.
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
When the University of California, San Diego launched its largest investment in tech in 2018, they planned to future proof their business processes and systems. Unexpectedly, it also prepared them to handle a global pandemic that changed every norm for the campus. With shelter-in-place orders taking immediate effect, they needed to quickly set up a robust online learning platform - one with powerful analytics to track student success. And, for the times students and staff are on campus, a contact tracing application was essential for their safety. We’d like to offer a conversation with Scott Lee to tell you more about UC San Diego’s rapid transformation from a traditional, on-campus institution to one of the leading examples of remote learning, and the critical role data connectivity played in making this possible.
From legacy systems to microservices and back | Andera Gioia, Quantyca
Legacy systems are the kings of our IT architecture. They rule the evolution of the technology ecosystem that hosts them thanks to the control they have acquired over time on core data and key business processes. Legacy systems however pose severe limits in addressing critical business needs, as well as seizing opportunities for future growth.
Digital Integration Hub (DIH) is a quite popular architecture to modernize legacy systems allowing the IT to evolve faster and better integrate modern technologies.
In this talk we’ll:
- Review DIH general architecture
- Illustrate different patterns to offload data from legacy systems to Kafka with pros and cons in term of consistency and latency
- Present different ways to consume data offloaded from legacy system to Kafka and their fit with different use cases
- Show how to close the loop and use Kafka and CQRS pattern to handle all the data moving backward from applications to legacy systems
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven
When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.
Belgium & Luxembourg dedicated online Data Virtualization discovery workshop
Watch full webinar here: https://bit.ly/33yYuQm
Data virtualization has become an essential part of enterprise data architectures, bridging the gap between IT and business users and delivering significant cost and time savings. This technology revolutionizes the way data is accessed, delivered, consumed and governed regardless of its format and location.
This 1.5 hour discovery session will show help you identify the benefits of this modern and agile data integration and management technology for your organisation.
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformation
The document introduces an evolutionary event-driven architecture called the Enterprise Digital Transformation Platform (EDTP) for accelerating digital transformation. The EDTP is a 4-tier platform based on cloud, containers, microservices, events and streaming. It addresses challenges of data integration and decoupling through architectural concepts like event-driven design, microservices and templates. The EDTP provides full-stack deployment automation and microservice templating to accelerate development. Use cases from Toyota Financial Services are presented to demonstrate the EDTP's capabilities.
More Related Content
What's hot
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream processing allows analyzing streaming data in real-time using tools like KSQL. KSQL builds on Kafka Streams to provide a SQL-like interface for transforming, filtering, and creating derived streams from immutable event data ingested into Kafka. Jupiter, a digital banking platform, uses KSQL to reconcile transactions from different interfaces in real-time and check rewards eligibility by analyzing consolidated transaction streams. While powerful, KSQL deployments require monitoring, richer SQL support, and integration with big data platforms for long-term storage.
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more. In looking at the tradeoffs behind a system like Kafka, we learn to reason more clearly about distributed systems and to make high-stakes technology adoption decisions more effectively. These are skills we all want to improve!
In this talk, we'll examine trade-offs on which our favorite distributed messaging system takes opinionated positions:
- Whether to store data contiguously or using an index
- How many storage tiers are best?
- Where should metadata live?
- And more.
It's always useful to dissect a modern distributed system with the goal of understanding it better, and it's even better to learn to deeper architectural principles in the process. Come to this talk for a generous helping of both.
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
(Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018
Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications.
This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures.
Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
Confluent Cloud makes Devops engineers lives a lot more easier.
Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge.
In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge.
These lessons include:
1. Automation, Automation, Automation - all the process has to be completely automated at such scale
2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad
3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions
Developing custom transformation in the Kafka connect to minimize data redund...
Compacted topics grow over time and are often utilizing high performance, low latency and relatively expensive storage solutions. Reducing duplicated data plays a critical role in the size of compacted topics. with less data on the topics, the Kafka cluster consumes less disk space which in turn it leads to lower operation cost.
in this use case-driven talk, we are going to demonstrate how our team at UnitedHealth Group leveraged existing transformers to extract data from the message metadata in the topic as well as how we developed our customized transformers to minimize the amount of duplicated data in each message in the topic.
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.
Digital transformation: Highly resilient streaming architecture and strategie...
Failure is inevitable in any distributed system but anticipating failures and building systems to recover from failures instantaneous makes the system highly resilient. At Capital One we process billions of events everyday and we leverage cloud, microservices, streaming and machine learning technologies to solve customer problems and provide the best customer experience.
As part of this session I will be talking about highly resilient streaming architecture that is supporting processing of billions of events every day then some of the strategies & best practices to build highly available and fault-tolerant systems utilizing Kafka and Cloud environments.
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions.
Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions.
In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Guru Sattanathan, Confluent, Senior Solutions Engineer
Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats!
Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts?
If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today.
In this session, we will go in this order:
-Recap on Enterprise Integration technologies
-What are the key flaws & What needs improvement?
-What is Apache Kafka?
-Rethinking Integration using Apache Kafka
https://www.meetup.com/KafkaMelbourne/events/280590162/
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP""
Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal.
In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
Joins in Kafka Streams and ksqlDB are a killer-feature for data processing and basic join semantics are well understood. However, in a streaming world records are associated with timestamps that impact the semantics of joins: welcome to the fabulous world of _temporal_ join semantics. For joins, timestamps are as important as the actual data and it is important to understand how they impact the join result.
In this talk we want to deep dive on the different types of joins, with a focus of their temporal aspect. Furthermore, we relate the individual join operators to the overall ""time engine"" of the Kafka Streams query runtime and explain its relationship to operator semantics. To allow developers to apply their knowledge on temporal join semantics, we provide best practices, tip and tricks to ""bend"" time, and configuration advice to get the desired join results. Last, we give an overview of recent, and an outlook to future, development that improves joins even further.
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
When the University of California, San Diego launched its largest investment in tech in 2018, they planned to future proof their business processes and systems. Unexpectedly, it also prepared them to handle a global pandemic that changed every norm for the campus. With shelter-in-place orders taking immediate effect, they needed to quickly set up a robust online learning platform - one with powerful analytics to track student success. And, for the times students and staff are on campus, a contact tracing application was essential for their safety. We’d like to offer a conversation with Scott Lee to tell you more about UC San Diego’s rapid transformation from a traditional, on-campus institution to one of the leading examples of remote learning, and the critical role data connectivity played in making this possible.
From legacy systems to microservices and back | Andera Gioia, Quantyca
Legacy systems are the kings of our IT architecture. They rule the evolution of the technology ecosystem that hosts them thanks to the control they have acquired over time on core data and key business processes. Legacy systems however pose severe limits in addressing critical business needs, as well as seizing opportunities for future growth.
Digital Integration Hub (DIH) is a quite popular architecture to modernize legacy systems allowing the IT to evolve faster and better integrate modern technologies.
In this talk we’ll:
- Review DIH general architecture
- Illustrate different patterns to offload data from legacy systems to Kafka with pros and cons in term of consistency and latency
- Present different ways to consume data offloaded from legacy system to Kafka and their fit with different use cases
- Show how to close the loop and use Kafka and CQRS pattern to handle all the data moving backward from applications to legacy systems
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven
When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.
What's hot (20)
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Stream Processing with Kafka and KSQL in Jupiter | Namit Mahuvakar, Jupiter
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
Developing custom transformation in the Kafka connect to minimize data redund...
Developing custom transformation in the Kafka connect to minimize data redund...
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Low-latency real-time data processing at giga-scale with Kafka | John DesJard...
Digital transformation: Highly resilient streaming architecture and strategie...
Digital transformation: Highly resilient streaming architecture and strategie...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
How a distributed graph analytics platform uses Apache Kafka for data ingesti...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
One Click Streaming Data Pipelines & Flows | Leveraging Kafka & Spark | Ido F...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
Transformation During a Global Pandemic | Ashish Pandit and Scott Lee, Univer...
From legacy systems to microservices and back | Andera Gioia, Quantyca
From legacy systems to microservices and back | Andera Gioia, Quantyca
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven
The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Similar to EDA Governance Model: a multicloud approach based on GitOps | Alejandro Alija and Marco Laucelli, Galeo Tech
Belgium & Luxembourg dedicated online Data Virtualization discovery workshop
Watch full webinar here: https://bit.ly/33yYuQm
Data virtualization has become an essential part of enterprise data architectures, bridging the gap between IT and business users and delivering significant cost and time savings. This technology revolutionizes the way data is accessed, delivered, consumed and governed regardless of its format and location.
This 1.5 hour discovery session will show help you identify the benefits of this modern and agile data integration and management technology for your organisation.
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformation
The document introduces an evolutionary event-driven architecture called the Enterprise Digital Transformation Platform (EDTP) for accelerating digital transformation. The EDTP is a 4-tier platform based on cloud, containers, microservices, events and streaming. It addresses challenges of data integration and decoupling through architectural concepts like event-driven design, microservices and templates. The EDTP provides full-stack deployment automation and microservice templating to accelerate development. Use cases from Toyota Financial Services are presented to demonstrate the EDTP's capabilities.
Business and IT agility through DevOps and microservice architecture powered ...
IT needs to run in production in order to generate business value. DevOps is among other things a way of thinking focusing on production software. A business application requires a tailor made platform to generate business value. The combination of application and its platform is a DevOps product. The DevOps team has full responsibility for that product through its entire lifecycle.
The microservices architecture promises flexibility, scalability, and optimal use of compute resources. Via independent components with well-defined scope and responsibility, interface, and ownership that are evolved and managed in an automated DevOps process, this architecture leverages current technologies and hard-learned insights from past decades.
This session defines the objectives of Business with IT, of microservices and DevOps and introduces Containers and the container platform Kubernetes as crucial ingredients for making DevOps happen.
IBM Blockchain Platform - Architectural Good Practices v1.0
This document discusses architectural good practices for blockchains and Hyperledger Fabric performance. It provides an overview of key concepts like transaction processing in Fabric and performance metrics. It also covers optimizing different parts of the Fabric network like client applications, peers, ordering service, and chaincode. The document recommends using tools like Hyperledger Caliper and custom test harnesses for performance testing and monitoring Fabric deployments. It highlights lessons learned from real projects around reusing connections and load balancing requests.
DevOps Digital Transformation: A real life use case enabled by Alien4Cloud
View the presentation that went along with the September 5th Webinar on DevOps Digital Transformation with Alien4Cloud based on Cloudify.
Data platform modernization with Databricks.pptx
The document discusses modernizing a healthcare organization's data platform from version 1.0 to 2.0 using Azure Databricks. Version 1.0 used Azure HDInsight (HDI) which was challenging to scale and maintain. It presented performance issues and lacked integrations. Version 2.0 with Databricks will provide improved scalability, cost optimization, governance, and ease of use through features like Delta Lake, Unity Catalog, and collaborative notebooks. This will help address challenges faced by consumers, data engineers, and the client.
Re-Platforming Applications for the Cloud
Are you planning to move existing applications to the cloud and want to avoid setbacks? These slides are from a webinar jointly presented by Atmosera and iTrellis, LLC. The webinar can help you find out how to assess your needs, plan out a migration and successfully operate your applications in a modern cloud environment. The webinar will provide the following answers:
* What re-platforming means and why you need to think about it
* How to take full advantage of a cloud such as Azure: agility, flexibility, and cost savings
* Lessons learned and best practices for planning a successful move to a modern cloud.
The full webinar playback URL is at https://www.atmosera.com/webinar-replatforming-application-cloud/
Neil-Leslie-CV
This CV summarizes Neil Leslie's experience as a database administrator and technical leader over 14 years at GE and previous roles. He has extensive experience leading teams and managing complex database migrations, from on-premise to cloud environments. His skills include Oracle, SQL Server, Linux, and AWS technologies. He is currently the CTO Cloud and Database Leader at GE Oil and Gas, where he helped insource technical resources and reduce costs through various initiatives.
Chugging Our Own "Craft Brew” – HPE’s Journey Towards Containers-as-a-Service...
Hewlett Packard Enterprise has a vast IT organization that consists of 15k Dev + Ops professionals, operating in 11 countries, with hundreds of development teams working together on over 1000 projects. Millions of lines of code get changed every day, generating 20,000+ change request in an average year. And now, we’re on a journey to building a Docker environment for this massive organization, to serve the needs of our own multi-billion dollar enterprise. This session showcases our own learnings across multiple phases of our containerization project, with real life experiences from both the Dev and Ops perspectives. We’ll also talk about shared experiences from some of HPE’s customers. “Coding” our infrastructure with Docker, one application image, one deployment command, multiple deployment models – this is an “extreme” tale of how one of the world’s largest enterprises is fully embracing Docker.
Production-Ready Kubernetes: It's Not About Technology
Adopting cloud-native technologies can seem like a technological challenge, but it's far from being the only theme.
The organization, culture and people aspects is the main challenge to overcome resistance to change, drive value creation and accelerate the transformation.
This talk shares our journey in adopting a Cloud-native application stack with a Kubernetes PaaS, focusing on the transformation journey.
DITA Interoperability
Session at tcworld 2016. Organized by Kristen James Eberlein (Eberlein Consulting LLC); other participants were Joe Gollner (Gnostyx), George Bina (SyncroSoft), Jean-François Ameye (IXIASOFT), and Eliot Kimber (Contrext).
Oracle Forms Modernization Roadmap
How can Oracle Forms (or other legacy) applications be modernized to fit in a contemporary IT architecture? Trends, concepts and technologies are discussed.
Software Architecture and Architectors: useless VS valuable
The document discusses several cases where architecture decisions hurt software teams. It provides lessons learned from each case:
- Overly sharing code across domains can cause dependencies and fragility. Architectural boundaries should separate domains.
- Attempting to satisfy all use cases with a single highly configurable solution often satisfies no one. Specific solutions may be preferable.
- Microservices are not a universal solution and will not improve a flawed design alone. Modules should only be split across microservices at domain boundaries.
- Systems need to be designed to evolve dynamically over time as requirements change. Centralized control hinders this.
DevOps Operations Challenges
The document discusses operations challenges in adopting DevOps practices. It notes that DevOps requires breaking down barriers between development and operations teams through collaboration, communication, and shared tools and processes. Some challenges include disagreements over tool ownership, pressure to adopt new technologies quickly, integrating legacy systems with cloud technologies, testing approaches for microservices, and aligning global teams with different time zones and priorities. Solutions proposed include automating delivery pipelines, deploying and testing changes frequently, monitoring all environments, and embracing changes in culture, processes, and technologies.
Implementing dev ops to face a two speed it architecture
The document discusses implementing DevOps to address challenges of a "two speed IT" architecture with both innovative and industrialized parts. It proposes adopting a DevOps methodology to break down silos, address execution challenges, and bring startup flexibility to the enterprise. This includes cultural, architectural and DevOps transformations to balance agility and stability across edge applications, core applications and shared services. It provides an example roadmap for a phased DevOps adoption with initial proofs of concept and incremental implementations.
Coud computing
Cloud Computing is a term used to refer to a model of network computing where a program or application runs on a connected server or servers rather than a local computing device such as a PC, tablet or Smartphone.
www.ipsrglobal.com
Ravi namboori-cloud computing
Cloud computing relies on sharing of resources to achieve coherence and economies of scale, similar to a utility (like the electricity grid) over a network.[1] At the foundation of cloud computing is the broader concept of converged infrastructure and shared services.
Cloud computing, or in simpler shorthand just "the cloud", also focuses on maximizing the effectiveness of the shared resources. Cloud resources are usually not only shared by multiple users but are also dynamically reallocated per demand. This can work for allocating resources to users. For example, a cloud computer facility that serves European users during European business hours with a specific application (e.g., email) may reallocate the same resources to serve North American users during North America's business hours with a different application (e.g., a web server). This approach should maximize the use of computing power thus reducing environmental damage as well since less power, air conditioning, rackspace, etc. are required for a variety of functions. With cloud computing, multiple users can access a single server to retrieve and update their data without purchasing licenses for different applications.
Ravi namboori ppt
Cloud computing, or in simpler shorthand just "the cloud", also focuses on maximizing the effectiveness of the shared resources. Cloud resources are usually not only shared by multiple users but are also dynamically reallocated per demand. This can work for allocating resources to users. For example, a cloud computer facility that serves European users during European business hours with a specific application (e.g., email) may reallocate the same resources to serve North American users during North America's business hours with a different application (e.g., a web server). This approach should maximize the use of computing power thus reducing environmental damage as well since less power, air conditioning, rackspace, etc. are required for a variety of functions. With cloud computing, multiple users can access a single server to retrieve and update their data without purchasing licenses for different applications.
Amit_Kumar_CV
This document contains a summary of Amit Kumar's professional experience and qualifications. He has over 9 years of IT experience, including 8 years of data warehouse and business intelligence experience. Currently he works as a data architect at Capgemini, leading a team of 12 on an oil and gas equipment install base project. He has extensive experience designing and developing data integration solutions using tools like Informatica, Hadoop, and SQL.
SCM Migration Webinar - English
In this informative webinar, learn how migrating from a proprietary SCM solution such as Rational® ClearCase®, Serena PVCS®, CA® Harvest, etc., to Subversion or Git will make an impact on your organization and/or enterprise.
Join us as we take the lessons we've learned from successfully migrating thousands of users to today's market leading SCM solutions, and provide you with best practices in building an actionable business case and conducting a smooth transition.
Key Takeaways:
Build a business case to adopt Git and/or Subversion in your organization
How CollabNet's TeamForge platform can provide the enterprise capabilities to enable Git and/or Subversion in your enterprise
Our recommended migration strategy proven with thousands of users
Considerations for extending your SCM solution
Similar to EDA Governance Model: a multicloud approach based on GitOps | Alejandro Alija and Marco Laucelli, Galeo Tech (20)
Belgium & Luxembourg dedicated online Data Virtualization discovery workshop
Belgium & Luxembourg dedicated online Data Virtualization discovery workshop
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformation
Evolutionary evnt-driven-architecture-for-accelerated-digital-transformation
Business and IT agility through DevOps and microservice architecture powered ...
Business and IT agility through DevOps and microservice architecture powered ...
IBM Blockchain Platform - Architectural Good Practices v1.0
IBM Blockchain Platform - Architectural Good Practices v1.0
DevOps Digital Transformation: A real life use case enabled by Alien4Cloud
DevOps Digital Transformation: A real life use case enabled by Alien4Cloud
Chugging Our Own "Craft Brew” – HPE’s Journey Towards Containers-as-a-Service...
Chugging Our Own "Craft Brew” – HPE’s Journey Towards Containers-as-a-Service...
Production-Ready Kubernetes: It's Not About Technology
Production-Ready Kubernetes: It's Not About Technology
Software Architecture and Architectors: useless VS valuable
Software Architecture and Architectors: useless VS valuable
Implementing dev ops to face a two speed it architecture
Implementing dev ops to face a two speed it architecture
More from HostedbyConfluent
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way.
Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks.
Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case.
Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."
Renaming a Kafka Topic | Kafka Summit London
"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics.
Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"
Evolution of NRT Data Ingestion Pipeline at Trendyol
"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights.
We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string.
Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring.
The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service.
While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications.
The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection.
Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."
Exactly-once Stream Processing with Arroyo and Kafka
"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output.
Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing.
In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."
Fish Plays Pokemon | Kafka Summit London
"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium.
This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption.
By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."
Tiered Storage 101 | Kafla Summit London
What is tiered storage and what is it good for? After this session you will know how to leverage the tiered storage feature to enable longer retention than the storage attached to brokers allows. You will get acquainted with the different configuration options and know what to expect when you enable the feature, like for example when will the first upload to the remote object storage take place.
Building a Self-Service Stream Processing Portal: How And Why
"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work.
However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business.
You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want.
Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware.
What you’ll take away from our talk:
- What were the pain points in the previous environment
- How we transitioned to Confluent without service downtime
- Creating a self-service stream processing portal built on top of Connect and ksqlDB
- Use case of stream process portal"
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics.
Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application.
At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."
Navigating Private Network Connectivity Options for Kafka Clusters
"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option!
In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met.
I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines.
Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"
Explaining How Real-Time GenAI Works in a Noisy Pub
"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many.
LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data.
Large language models for your streaming application - explained with a little maths and a lot of pub stories"
TL;DR Kafka Metrics | Kafka Summit London
"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them.
This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."
A Window Into Your Kafka Streams Tasks | KSL
Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance.
- Introduction to Kafka producer internal and workflow
- Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance
- Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient.
- Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees.
- Q&A session with attendees to address specific questions and concerns."
Data Contracts Management: Schema Registry and Beyond
"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element.
In this talk, we’ll:
1. see why data contracts are so important but also difficult to implement;
2. identify the characteristics of a well-designed data contract:
discuss the anatomy of a data contract, its main elements and, how to formally describe them;
3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."
Code-First Approach: Crafting Efficient Flink Apps
"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications.
We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications.
Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."
Debezium vs. the World: An Overview of the CDC Ecosystem
"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you!
[1] https://debezium.io/"
Beyond Tiered Storage: Serverless Kafka with No Local Disks
"Separation of compute and storage has become the de-facto standard in the data industry for batch processing.
The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world.
In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.
Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it.
This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."
More from HostedbyConfluent (20)
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Recently uploaded
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Recently uploaded (20)
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
EDA Governance Model: a multicloud approach based on GitOps | Alejandro Alija and Marco Laucelli, Galeo Tech
- 1. Kafka Summit 2021 EDA Governance Model: A multi-cloud approach based on GitOps Alejandro Alija & MarcoLaucelli
- 2. Galeo in one sentence We build digital platforms to be the agile fabric where digital products are conceived, developed, and scaled. We are hands-on. We deliver.
- 3. Define and govern the new digital architecture for the future. The challenge Standardized platform 01 Cost Control & allocation 03 Service Catalogue 05 Asset reuse 04 Project Flexibility & Shared nothing 02 Governed lifecycle 06
- 4. As promoters of a new architecture, the team put a lot of focus on simplifying developer’s experience. • Is EDA approach suitable for my project? • How should the infrastructure for my project be defined and provisioned? • How should the EDA components of my system be described and deployed? • How can I access and reuse existing assets? The challenge: DX
- 5. Everything together: governance 1. Bureaucraticoverhead: mostGovernance documentationmodelsrequireadditional documentationeffortsfromdevelopersthatusually don’tpay back fortheir specificduties. 2. Cataloguesdiffer from reality: independentdocumentationfor governanceand forgovernance andcatalogues end updifferingfromwhatis actuallybeingdeployed. Reuseand standardizationgetimpacted.Apart fromgovernanceitself. Based on the customer experiences two main problems needed to be avoided
- 6. The governance approach. The underlyingidea isthatevery governancestep will beconsideredasapartofthewhole deploymentprocesses,andso theyshouldbe integrated into a single orchestrated pipeline thatwill be ableto covergovernance,supervision, approval anddeploymenttasks asa partofa unifiedprocess. The orchestrationofthe processis basedon aunified,machinereadable documentation modelthatcontainsthe requiredinformationtovalidateservice design andcomponent specifications.Thisunderlying unifieddocumentationmodel allows executing theoverall catalogueanddeploymenttasksinasynchronised anduniform way.
- 7. We defined a EDA Governance Framework based on an operational support for the whole lifecycle management. EDA Governance framework Theprocessisminimallyintermediated,sohumaninterventionisonlyrequiredforsupervision andapprovaltasks.Nomanualinterventionon thecataloguesortheactualEDA environmentis required.
- 8. The Catalogue. EDA Governance framework Theprocessisminimallyintermediated,sohumaninterventionisonlyrequiredforsupervision andapprovaltasks.Nomanualinterventionon thecataloguesortheactualEDA environment isrequired. Key component of the EDA Gov: • Keeps record of the deployed asset and services. • Acts as theoverall orchestrator of the deployment process. For each asset it keeps trackof: • Infrastructure • Topologies and Topics • Metadata • Security and access configuration Consolidated in the corporate Git Repos.
- 9. The documental model. Based on standardized description templates that are readable for both the actual deployement pipelines and the catalogue reporter. Non-functionalcomponent descriptionsand infrastructure specification. Descriptionfothe wholetopology includingproducers,consumersand othereventualcomponents.
- 10. EDA Governance procedures EDA Infrastructure & Project Procedures • Project sign-up and approval procedure: • Project approval procedure • Request a Confluent Environment • Request a Confluent Cluster EDA Service & Component Procedures • Request access to an existing Confluent resource: topic, ksqlDB table, … • Request modification to an existing resource: topic, ksqlDB table, … • Accept Access Request by a third-party procedure • Request and Create a new topology
- 11. Catalogue & Procedures Based on corporate Git repositories and governance CI/CD procedures, that allowexecute and synchonyze development and governance procedures witouth incrementing bureaucratic efforts.
- 12. Demo Developer My NewProject - Topology_description.yaml CI/CDPipeline - ACLscreationinConfluentCloud Webhook –Pull request Trigger Confluent CLI– ACLs creation Producers Consumers
- 13. Conclusions 1. GitOps&CI/CDfor topologies: is a very convenient modelfor OperationalGovernance 2. Cataloguesstaysup todate: based on independentlawyers 3. Platform evolution andcompletion (API,MS) is easilyintegratedwiththesame Governance model 4. Reuse is facilitated: as theinformationis up2dateand theprocedures are very lightweight procedures are very lightweight
- 14. EDA Governance Model: Thanks alejandro@galeo.tech marco@galeo.tech www.galeo.tech WE ARE HIRING