Watch this talk here: https://www.confluent.io/online-talks/building-an-enterprise-eventing-framework-on-demand
Learn how Centene improved their ability to interact and engage with healthcare providers in real time with MongoDB and Confluent Platform.
Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of their architecture is the development of a universal eventing framework designed to accommodate transformation into an event-driven architecture (EDA).
The business requirements within Centene's claims adjudication domain were solved leveraging the Kafka Stream DSL, Confluent Platform and MongoDB. Most importantly, Centene discusses how they plan on leveraging this framework to change their culture from batch processing to real-time stream processing.
Workshop on Google Cloud Data PlatformGoDataDriven
The document provides an agenda and information about a GoDataFest workshop on Google Cloud Platform for data. The agenda includes an introduction to GCP for data, a session on roles and tools on GCP for different data roles, and a session where participants will build projects on GCP in mixed workgroups. It outlines the goals and tools used by different roles like data engineer, analytics engineer, and Looker user. It also provides information on Google Cloud technologies like BigQuery, Dataform, Looker, and how they fit into the modern data lifecycle and platform. Participants are then divided into mixed workgroups based on their preferred role and given insights to explore in their projects.
RPA is a business process automation solution that uses software "robots" to automate repetitive tasks, improving speed, accuracy, and freeing up employees to focus on more strategic work. The document discusses how RPA can read emails and documents, extract and validate data, generate reports, and connect different systems without complex integration. It provides examples of how RPA has helped businesses in various industries automate accounting, data migration, customer billing, and other processes, reducing costs, errors and turnaround times while improving compliance. The company Signity Solutions offers RPA consulting, development, implementation and management services to help organizations adopt this technology.
Grafana is an open source analytics and monitoring tool that uses InfluxDB to store time series data and provide visualization dashboards. It collects metrics like application and server performance from Telegraf every 10 seconds, stores the data in InfluxDB using the line protocol format, and allows users to build dashboards in Grafana to monitor and get alerts on metrics. An example scenario is using it to collect and display load time metrics from a QA whitelist VM.
Integrating Google Cloud Dataproc with Alluxio for faster performance in the ...Alluxio, Inc.
Google Dataproc is Google Cloud's fully managed Apache Spark and Apache Hadoop service. Alluxio is an open source data orchestration platform that can be used with Dataproc to accelerate analytics workloads. With a single initialization action, Alluxio can be installed on a Dataproc cluster to cache data from Cloud Storage for faster queries. Alluxio also enables "zero-copy bursting" of workloads to the cloud by allowing frameworks to access data directly from remote HDFS without needing to copy it. This provides elastic compute capacity while avoiding high network latency and bandwidth costs of copying large datasets.
Apache Flink Crash Course by Slim Baltagi and Srini PalthepuSlim Baltagi
In this hands-on Apache Flink presentation, you will learn in a step-by-step tutorial style about:
• How to setup and configure your Apache Flink environment: Local/VM image (on a single machine), cluster (standalone), YARN, cloud (Google Compute Engine, Amazon EMR, ... )?
• How to get familiar with Flink tools (Command-Line Interface, Web Client, JobManager Web Interface, Interactive Scala Shell, Zeppelin notebook)?
• How to run some Apache Flink example programs?
• How to get familiar with Flink's APIs and libraries?
• How to write your Apache Flink code in the IDE (IntelliJ IDEA or Eclipse)?
• How to test and debug your Apache Flink code?
• How to deploy your Apache Flink code in local, in a cluster or in the cloud?
• How to tune your Apache Flink application (CPU, Memory, I/O)?
The document discusses strategies for executing a large-scale migration to AWS. It outlines establishing a cloud enablement team and AWS landing zone to provide a secure, scalable multi-account environment. Application migration strategies discussed include discovery, determining the migration path, rehosting/lift and shift, and replatforming/lift and reshape. Specific migration tools and services mentioned include AWS Application Discovery Service, VMware HCX, AWS Server Migration Service, and AWS Database Migration Service.
다양한 하둡에코 소프트웨어 성능을 검증하려는 목적으로 성능 테스트 환경을 구성해보았습니다. ELK, JMeter를 활용해 구성했고 Kafka에 적용해 보았습니다.
프로젝트에서 요구되는 성능요건을 고려해 다양한 옵션을 조정해 시뮬레이션 해볼수 있습니다.
처음 적용한 뒤 2년 정도가 지났지만, kafka 만이 아니다 다른 Hadoop eco 및 Custom Solution에도 유용하게 활용 가능하겠습니다.
Workshop on Google Cloud Data PlatformGoDataDriven
The document provides an agenda and information about a GoDataFest workshop on Google Cloud Platform for data. The agenda includes an introduction to GCP for data, a session on roles and tools on GCP for different data roles, and a session where participants will build projects on GCP in mixed workgroups. It outlines the goals and tools used by different roles like data engineer, analytics engineer, and Looker user. It also provides information on Google Cloud technologies like BigQuery, Dataform, Looker, and how they fit into the modern data lifecycle and platform. Participants are then divided into mixed workgroups based on their preferred role and given insights to explore in their projects.
RPA is a business process automation solution that uses software "robots" to automate repetitive tasks, improving speed, accuracy, and freeing up employees to focus on more strategic work. The document discusses how RPA can read emails and documents, extract and validate data, generate reports, and connect different systems without complex integration. It provides examples of how RPA has helped businesses in various industries automate accounting, data migration, customer billing, and other processes, reducing costs, errors and turnaround times while improving compliance. The company Signity Solutions offers RPA consulting, development, implementation and management services to help organizations adopt this technology.
Grafana is an open source analytics and monitoring tool that uses InfluxDB to store time series data and provide visualization dashboards. It collects metrics like application and server performance from Telegraf every 10 seconds, stores the data in InfluxDB using the line protocol format, and allows users to build dashboards in Grafana to monitor and get alerts on metrics. An example scenario is using it to collect and display load time metrics from a QA whitelist VM.
Integrating Google Cloud Dataproc with Alluxio for faster performance in the ...Alluxio, Inc.
Google Dataproc is Google Cloud's fully managed Apache Spark and Apache Hadoop service. Alluxio is an open source data orchestration platform that can be used with Dataproc to accelerate analytics workloads. With a single initialization action, Alluxio can be installed on a Dataproc cluster to cache data from Cloud Storage for faster queries. Alluxio also enables "zero-copy bursting" of workloads to the cloud by allowing frameworks to access data directly from remote HDFS without needing to copy it. This provides elastic compute capacity while avoiding high network latency and bandwidth costs of copying large datasets.
Apache Flink Crash Course by Slim Baltagi and Srini PalthepuSlim Baltagi
In this hands-on Apache Flink presentation, you will learn in a step-by-step tutorial style about:
• How to setup and configure your Apache Flink environment: Local/VM image (on a single machine), cluster (standalone), YARN, cloud (Google Compute Engine, Amazon EMR, ... )?
• How to get familiar with Flink tools (Command-Line Interface, Web Client, JobManager Web Interface, Interactive Scala Shell, Zeppelin notebook)?
• How to run some Apache Flink example programs?
• How to get familiar with Flink's APIs and libraries?
• How to write your Apache Flink code in the IDE (IntelliJ IDEA or Eclipse)?
• How to test and debug your Apache Flink code?
• How to deploy your Apache Flink code in local, in a cluster or in the cloud?
• How to tune your Apache Flink application (CPU, Memory, I/O)?
The document discusses strategies for executing a large-scale migration to AWS. It outlines establishing a cloud enablement team and AWS landing zone to provide a secure, scalable multi-account environment. Application migration strategies discussed include discovery, determining the migration path, rehosting/lift and shift, and replatforming/lift and reshape. Specific migration tools and services mentioned include AWS Application Discovery Service, VMware HCX, AWS Server Migration Service, and AWS Database Migration Service.
다양한 하둡에코 소프트웨어 성능을 검증하려는 목적으로 성능 테스트 환경을 구성해보았습니다. ELK, JMeter를 활용해 구성했고 Kafka에 적용해 보았습니다.
프로젝트에서 요구되는 성능요건을 고려해 다양한 옵션을 조정해 시뮬레이션 해볼수 있습니다.
처음 적용한 뒤 2년 정도가 지났지만, kafka 만이 아니다 다른 Hadoop eco 및 Custom Solution에도 유용하게 활용 가능하겠습니다.
Apache Kafka in the Airline, Aviation and Travel IndustryKai Wähner
Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. Coronavirus is just a piece of the challenge.
This presentation explores use cases, architectures, and references for Apache Kafka as event streaming technology in the aviation industry, including airline, airports, global distribution systems (GDS), aircraft manufacturers, and more.
Examples include Lufthansa, Singapore Airlines, Air France Hop, Amadeus, and more. Technologies include Kafka, Kafka Connect, Kafka Streams, ksqlDB, Machine Learning, Cloud, and more.
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...StreamNative
This document summarizes a presentation about Apache Pulsar multi-tenancy and security features at Verizon Media. It discusses how Pulsar implements tenant and namespace isolation through storage quotas, throttling policies, broker and bookie isolation. It also covers authentication, authorization, encryption in transit and at rest, and how Pulsar proxy supports SNI routing for hybrid cloud deployments and cross-organization replication. Future plans include tenant-based broker virtualization and hybrid cloud deployments with geo-replication.
What do you mean by “API as a Product”?Nordic APIs
You may have heard the term “API Product.” But what does it mean? In this talk I will introduce the concept and explain the benefits and challenges of transforming your organization to view your APIs as measurable products that expose your companies capabilities creating agility, autonomy, and acceleration. Traditional product manufacturers create new product and launch them into the marketplace and then measure value; we will teach you to view your APIs in the same way. Concepts covered in this presentation will be designing APIs with Design Thinking, funding your product, building teams, marketing your API, managing your marketplace, and measuring success.
Edge Services as a Critical AWS Infrastructure Component - August 2017 AWS On...Amazon Web Services
This document discusses edge services from Amazon Web Services (AWS) as a critical component of AWS infrastructure. It defines edge services as services like AWS CloudFront, AWS Shield, AWS WAF, and Amazon Route 53 that control access to core application resources through the edge to secure, scale, and optimize applications. The document reviews the benefits of edge services like improved performance, security, and cost optimization. It provides overviews of specific edge services like CloudFront, Shield, WAF, and Route 53 and how they can be used to start leveraging edge services.
This session will begin with an introduction to non-relational (NoSQL) databases and compare them with relational (SQL) databases. Learn the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service, and see the DynamoDB console first-hand. See a walk-through demo of building a serverless web application using this high-performance key-value and JSON document store.
This document describes the platform architecture for an omnichannel retail company. It discusses the challenges of legacy architectures and outlines a new platform architecture with key components. The platform exposes primitive APIs and services that represent core retail processes and data. Tenants can then build applications that utilize and extend these platform services. All data changes are handled as asynchronous events to ensure eventual consistency across caching layers. The architecture aims to decouple tenants, improve scalability, and facilitate innovation while maintaining enterprise governance over core platform components.
Transforming Consumer Banking with a 100% Cloud-Based Bank (FSV204) - AWS re:...Amazon Web Services
Customer demands for higher levels of service and value, constantly evolving technology capabilities, and stringent regulatory requirements are all powerful forces reshaping retail banking. Built exclusively on AWS, Starling Bank’s 100% cloud-based, mobile-only banking solution satisfies regulators in terms of its resilience, security, and reliability. It also satisfies consumers by giving them greater control over their data, streamlining the account opening process, accelerating payments, and providing access to innovative new services developed from scratch with open APIs, a developer platform, integration with Apple Pay, Google Pay, and Fitbit Pay and a custom backend ledger and payments integrations. Starling Bank is leading the open banking revolution. In this session, learn how Starling Bank delivers value to their customers and innovates at a very fast pace in a sector that can be slow to evolve.
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...Amazon Web Services
AWS has a large and growing portfolio of big data management and analytics services, designed to be integrated into solution architectures that meet the needs of your business. In this session, we look at analytics through the eyes of a business intelligence analyst, a data scientist, and an application developer, and we explore how to quickly leverage Amazon Redshift, Amazon QuickSight, RStudio, and Amazon Machine Learning to create powerful, yet straightforward, business solutions.
This document provides an introduction to microservices. It begins by outlining the challenges of monolithic architecture such as long build/release cycles and difficulty scaling. It then introduces microservices as a way to decompose monolithic applications into independently deployable services. Key benefits of microservices include improved agility, scalability, and innovation. The document discusses microservice design principles like communicating over APIs, using the right tools for each service, securing services, and being a good citizen in the ecosystem. It provides examples of how to implement a restaurant microservice using AWS services like API Gateway, Lambda, DynamoDB and containers.
The 'macro view' on Big Query:
We started with an overview, some typical uses and moved to project hierarchy, access control and security.
In the end we touch about tools and demos.
BIAN Applied to Open Banking - Thoughts on Architecture and ImplementationBiao Hao
At the BIAN Open Day in NYC November 12, 2019, we shared our thoughts on how BIAN Value Chain business areas, Channels, Customers, Products and Operations, provide a context for addressing Open Banking capabilities in a more systematic way, and the implications the decoupled Value Chain have on business models and reference architecture. Sample use cases such as account information and account aggregation, their mapping to related BIAN service domains, and implementation using microservices and pattern for performance are also discussed.
by Kashif Imran, Sr. Solutions Architect, AWS
Serverless computing allows you to build and run applications without the need for provisioning or managing servers. With serverless computing, you can build web, mobile, and IoT backends; run stream processing or big data workloads; run chatbots, and more. In this session, you’ll learn how to get started with serverless computing with AWS Lambda, which lets you run code without provisioning or managing servers. We’ll introduce you to the basics of building with Lambda and how you can benefit from features such as continuous scaling, built-in high availability, integrations with AWS and third-party apps, and subsecond metering pricing. We’ll also introduce you to the broader portfolio of AWS services that help you build serverless applications with Lambda, including Amazon API Gateway, Amazon DynamoDB, AWS Step Functions, and more.
In this session we’ll take a high-level overview of AWS Lambda, a serverless compute platform that has changed the way that developers around the world build applications. We’ll explore how Lambda works under the hood, the capabilities it has, and how it is used. By the end of this talk you’ll know how to create Lambda based applications and deploy and manage them easily.
Speaker: Chris Munns - Principal Developer Advocate, AWS Serverless Applications, AWS
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
API Management Solution Powerpoint Presentation SlidesSlideTeam
Select this API Management Solution PowerPoint Presentation Slides and study the needs of app developers. Display your company’s objectives like the expansion of the market base, building a platform ecosystem, and improving the digital outreach company through this application gateway PPT templates. Highlight the structure of architectural components of API with the help of this computing interface management PPT slide. You can easily introduce your services of API portal like documentation, registration, and analysis in a well-organized manner by taking the aid of our invigorating software management PPT designs. Take advantage of our professionally designed network administration PPT themes to exhibit various components like API design, deployment, security, analytics, and monetization in an appropriate color-coded fashion. You can take the assistance of this API solution PPT presentation to provide a report on API management in a well-organized format. Click the download button and make this open-source management PowerPoint presentation your source to educate prospective clients about attractive opportunities in the API management market. https://bit.ly/3tOpgMa
Amazon Athena is a serverless query service that allows users to run interactive SQL queries on data stored in Amazon S3 without having to load the data into a database. It uses Presto to allow ANSI SQL queries on data in formats like CSV, JSON, and columnar formats like Parquet and ORC. For a dataset of 1 billion rows of sales data stored in S3, Athena queries performed comparably to a basic Redshift cluster and much faster than loading and querying the data in Redshift, making Athena a cost-effective solution for ad-hoc queries on data in S3.
SpringOne Platform 2017
Stéphane Maldini, Pivotal; Simon Basle, Pivotal
"In 2016, Project Reactor was the foundation before Spring Reactive story, in particular with Reactor Core 3.0 fueling our initial Spring Framework 5 development.
2017 and 2018 are the years Project Reactor empowers the final Spring Framework 5 GA and an entire ecosystem, thus including further refinement, feedbacks and incredible new features. In fact, the new Reactor Core 3.1 and Reactor Netty 0.7 are the very major versions used by the like of Spring Boot 2.0, and they have dramatically consolidated around a simple but yet coherent API.
Discover those changes and the new Reactor capabilities including support for Reactive AOP, Observability, Tracing, Error Strategies for long-running streams, new Netty driver, improved test support, community driven initiatives and much more
Finally, the first java framework & ecosystem gets the reactive library it needs !"
This presentation explains the three layer API design which organisations can use to get most out of there systems with less development and maintenance time spent on fixing issues as a whole in org.
The document describes LlamaIndex, a toolkit that provides indices over unstructured and structured data to augment language models with private data through in-context learning. It discusses how LlamaIndex solves data ingestion and indexing problems to perform language model data augmentation in an efficient manner. Examples of how LlamaIndex can be used with different data connectors like Gmail, audio files, and Notion pages are also provided.
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing...confluent
Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of our architecture is the development of a universal eventing framework to accommodate transformation into an event-driven architecture (EDA). Our application provides a representational state transfer (REST) and remote procedure call (gRPC) interface that allows development teams to publish and consume events with a simple Noun-Verb-Object (NVO) syntax. Embedded within the framework are structured schema evolutions with Confluent Schema Registry and AVRO, configurable (self-service) event-routing with K-Tables, dynamic event-aggregation with Kafka Streams, distributed event-tracing with Jaeger, and event querying against a MongoDB event-store hydrated by Kafka Connect. Lastly, we developed techniques to handle long-term event storage within Kafka; specifically surrounding the automated deletion of expired events and re-hydration of missing events. In Centene's first business use case, events related to claim processing of provider reconsiderations was used to provide real-time updates to providers on the status of their claim appeals. To satisfy the business requirement, multiple monolith systems independently leveraged the event framework, to stream status updates for display on the Centene Provider Portal instantly. This provided a capability that was brand new to Centene: the ability to interact and engage with our providers in real-time through the use of event streams. In this presentation, we will walk you through the architecture of the eventing framework and showcase how our business requirements within our claims adjudication domain were able to be solved leveraging the Kafka Stream DSL and the Confluent Platform. And more importantly, how Centene plans on leveraging this framework, written on-top of Kafka Streams, to change our culture from batch processing to real-time stream processing.
This document discusses using event streams as the system of record for data, rather than traditional databases. It argues that streams can serve as the single source of truth for data, providing benefits like data lineage, auditing, and integrity. It also describes how healthcare company Liaison uses a streaming platform from MapR to power their data integration platform, gaining the advantages of streams while meeting various compliance requirements.
Apache Kafka in the Airline, Aviation and Travel IndustryKai Wähner
Aviation and travel are notoriously vulnerable to social, economic, and political events, as well as the ever-changing expectations of consumers. Coronavirus is just a piece of the challenge.
This presentation explores use cases, architectures, and references for Apache Kafka as event streaming technology in the aviation industry, including airline, airports, global distribution systems (GDS), aircraft manufacturers, and more.
Examples include Lufthansa, Singapore Airlines, Air France Hop, Amadeus, and more. Technologies include Kafka, Kafka Connect, Kafka Streams, ksqlDB, Machine Learning, Cloud, and more.
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...StreamNative
This document summarizes a presentation about Apache Pulsar multi-tenancy and security features at Verizon Media. It discusses how Pulsar implements tenant and namespace isolation through storage quotas, throttling policies, broker and bookie isolation. It also covers authentication, authorization, encryption in transit and at rest, and how Pulsar proxy supports SNI routing for hybrid cloud deployments and cross-organization replication. Future plans include tenant-based broker virtualization and hybrid cloud deployments with geo-replication.
What do you mean by “API as a Product”?Nordic APIs
You may have heard the term “API Product.” But what does it mean? In this talk I will introduce the concept and explain the benefits and challenges of transforming your organization to view your APIs as measurable products that expose your companies capabilities creating agility, autonomy, and acceleration. Traditional product manufacturers create new product and launch them into the marketplace and then measure value; we will teach you to view your APIs in the same way. Concepts covered in this presentation will be designing APIs with Design Thinking, funding your product, building teams, marketing your API, managing your marketplace, and measuring success.
Edge Services as a Critical AWS Infrastructure Component - August 2017 AWS On...Amazon Web Services
This document discusses edge services from Amazon Web Services (AWS) as a critical component of AWS infrastructure. It defines edge services as services like AWS CloudFront, AWS Shield, AWS WAF, and Amazon Route 53 that control access to core application resources through the edge to secure, scale, and optimize applications. The document reviews the benefits of edge services like improved performance, security, and cost optimization. It provides overviews of specific edge services like CloudFront, Shield, WAF, and Route 53 and how they can be used to start leveraging edge services.
This session will begin with an introduction to non-relational (NoSQL) databases and compare them with relational (SQL) databases. Learn the fundamentals of Amazon DynamoDB, a fully managed NoSQL database service, and see the DynamoDB console first-hand. See a walk-through demo of building a serverless web application using this high-performance key-value and JSON document store.
This document describes the platform architecture for an omnichannel retail company. It discusses the challenges of legacy architectures and outlines a new platform architecture with key components. The platform exposes primitive APIs and services that represent core retail processes and data. Tenants can then build applications that utilize and extend these platform services. All data changes are handled as asynchronous events to ensure eventual consistency across caching layers. The architecture aims to decouple tenants, improve scalability, and facilitate innovation while maintaining enterprise governance over core platform components.
Transforming Consumer Banking with a 100% Cloud-Based Bank (FSV204) - AWS re:...Amazon Web Services
Customer demands for higher levels of service and value, constantly evolving technology capabilities, and stringent regulatory requirements are all powerful forces reshaping retail banking. Built exclusively on AWS, Starling Bank’s 100% cloud-based, mobile-only banking solution satisfies regulators in terms of its resilience, security, and reliability. It also satisfies consumers by giving them greater control over their data, streamlining the account opening process, accelerating payments, and providing access to innovative new services developed from scratch with open APIs, a developer platform, integration with Apple Pay, Google Pay, and Fitbit Pay and a custom backend ledger and payments integrations. Starling Bank is leading the open banking revolution. In this session, learn how Starling Bank delivers value to their customers and innovates at a very fast pace in a sector that can be slow to evolve.
Easy Analytics on AWS with Amazon Redshift, Amazon QuickSight, and Amazon Mac...Amazon Web Services
AWS has a large and growing portfolio of big data management and analytics services, designed to be integrated into solution architectures that meet the needs of your business. In this session, we look at analytics through the eyes of a business intelligence analyst, a data scientist, and an application developer, and we explore how to quickly leverage Amazon Redshift, Amazon QuickSight, RStudio, and Amazon Machine Learning to create powerful, yet straightforward, business solutions.
This document provides an introduction to microservices. It begins by outlining the challenges of monolithic architecture such as long build/release cycles and difficulty scaling. It then introduces microservices as a way to decompose monolithic applications into independently deployable services. Key benefits of microservices include improved agility, scalability, and innovation. The document discusses microservice design principles like communicating over APIs, using the right tools for each service, securing services, and being a good citizen in the ecosystem. It provides examples of how to implement a restaurant microservice using AWS services like API Gateway, Lambda, DynamoDB and containers.
The 'macro view' on Big Query:
We started with an overview, some typical uses and moved to project hierarchy, access control and security.
In the end we touch about tools and demos.
BIAN Applied to Open Banking - Thoughts on Architecture and ImplementationBiao Hao
At the BIAN Open Day in NYC November 12, 2019, we shared our thoughts on how BIAN Value Chain business areas, Channels, Customers, Products and Operations, provide a context for addressing Open Banking capabilities in a more systematic way, and the implications the decoupled Value Chain have on business models and reference architecture. Sample use cases such as account information and account aggregation, their mapping to related BIAN service domains, and implementation using microservices and pattern for performance are also discussed.
by Kashif Imran, Sr. Solutions Architect, AWS
Serverless computing allows you to build and run applications without the need for provisioning or managing servers. With serverless computing, you can build web, mobile, and IoT backends; run stream processing or big data workloads; run chatbots, and more. In this session, you’ll learn how to get started with serverless computing with AWS Lambda, which lets you run code without provisioning or managing servers. We’ll introduce you to the basics of building with Lambda and how you can benefit from features such as continuous scaling, built-in high availability, integrations with AWS and third-party apps, and subsecond metering pricing. We’ll also introduce you to the broader portfolio of AWS services that help you build serverless applications with Lambda, including Amazon API Gateway, Amazon DynamoDB, AWS Step Functions, and more.
In this session we’ll take a high-level overview of AWS Lambda, a serverless compute platform that has changed the way that developers around the world build applications. We’ll explore how Lambda works under the hood, the capabilities it has, and how it is used. By the end of this talk you’ll know how to create Lambda based applications and deploy and manage them easily.
Speaker: Chris Munns - Principal Developer Advocate, AWS Serverless Applications, AWS
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
API Management Solution Powerpoint Presentation SlidesSlideTeam
Select this API Management Solution PowerPoint Presentation Slides and study the needs of app developers. Display your company’s objectives like the expansion of the market base, building a platform ecosystem, and improving the digital outreach company through this application gateway PPT templates. Highlight the structure of architectural components of API with the help of this computing interface management PPT slide. You can easily introduce your services of API portal like documentation, registration, and analysis in a well-organized manner by taking the aid of our invigorating software management PPT designs. Take advantage of our professionally designed network administration PPT themes to exhibit various components like API design, deployment, security, analytics, and monetization in an appropriate color-coded fashion. You can take the assistance of this API solution PPT presentation to provide a report on API management in a well-organized format. Click the download button and make this open-source management PowerPoint presentation your source to educate prospective clients about attractive opportunities in the API management market. https://bit.ly/3tOpgMa
Amazon Athena is a serverless query service that allows users to run interactive SQL queries on data stored in Amazon S3 without having to load the data into a database. It uses Presto to allow ANSI SQL queries on data in formats like CSV, JSON, and columnar formats like Parquet and ORC. For a dataset of 1 billion rows of sales data stored in S3, Athena queries performed comparably to a basic Redshift cluster and much faster than loading and querying the data in Redshift, making Athena a cost-effective solution for ad-hoc queries on data in S3.
SpringOne Platform 2017
Stéphane Maldini, Pivotal; Simon Basle, Pivotal
"In 2016, Project Reactor was the foundation before Spring Reactive story, in particular with Reactor Core 3.0 fueling our initial Spring Framework 5 development.
2017 and 2018 are the years Project Reactor empowers the final Spring Framework 5 GA and an entire ecosystem, thus including further refinement, feedbacks and incredible new features. In fact, the new Reactor Core 3.1 and Reactor Netty 0.7 are the very major versions used by the like of Spring Boot 2.0, and they have dramatically consolidated around a simple but yet coherent API.
Discover those changes and the new Reactor capabilities including support for Reactive AOP, Observability, Tracing, Error Strategies for long-running streams, new Netty driver, improved test support, community driven initiatives and much more
Finally, the first java framework & ecosystem gets the reactive library it needs !"
This presentation explains the three layer API design which organisations can use to get most out of there systems with less development and maintenance time spent on fixing issues as a whole in org.
The document describes LlamaIndex, a toolkit that provides indices over unstructured and structured data to augment language models with private data through in-context learning. It discusses how LlamaIndex solves data ingestion and indexing problems to perform language model data augmentation in an efficient manner. Examples of how LlamaIndex can be used with different data connectors like Gmail, audio files, and Notion pages are also provided.
Building an Enterprise Eventing Framework (Bryan Zelle, Centene; Neil Buesing...confluent
Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of our architecture is the development of a universal eventing framework to accommodate transformation into an event-driven architecture (EDA). Our application provides a representational state transfer (REST) and remote procedure call (gRPC) interface that allows development teams to publish and consume events with a simple Noun-Verb-Object (NVO) syntax. Embedded within the framework are structured schema evolutions with Confluent Schema Registry and AVRO, configurable (self-service) event-routing with K-Tables, dynamic event-aggregation with Kafka Streams, distributed event-tracing with Jaeger, and event querying against a MongoDB event-store hydrated by Kafka Connect. Lastly, we developed techniques to handle long-term event storage within Kafka; specifically surrounding the automated deletion of expired events and re-hydration of missing events. In Centene's first business use case, events related to claim processing of provider reconsiderations was used to provide real-time updates to providers on the status of their claim appeals. To satisfy the business requirement, multiple monolith systems independently leveraged the event framework, to stream status updates for display on the Centene Provider Portal instantly. This provided a capability that was brand new to Centene: the ability to interact and engage with our providers in real-time through the use of event streams. In this presentation, we will walk you through the architecture of the eventing framework and showcase how our business requirements within our claims adjudication domain were able to be solved leveraging the Kafka Stream DSL and the Confluent Platform. And more importantly, how Centene plans on leveraging this framework, written on-top of Kafka Streams, to change our culture from batch processing to real-time stream processing.
This document discusses using event streams as the system of record for data, rather than traditional databases. It argues that streams can serve as the single source of truth for data, providing benefits like data lineage, auditing, and integrity. It also describes how healthcare company Liaison uses a streaming platform from MapR to power their data integration platform, gaining the advantages of streams while meeting various compliance requirements.
The document discusses the importance of data integration and some signs that an organization has poor data integration. It notes that data is distributed across disparate systems and integrating data brings value by combining related information. Poor integration can result in incomplete or inconsistent data, inability to get a single view of the truth, and high maintenance costs. The document advocates providing integrated solutions to avoid these issues.
Presentation given to the BCS Data Management Specialist Group by Steve Higgins of CSC on healthcare data management
A video of the presentation is available at http://youtu.be/Fqm4XDyA6fI
Billions of Rows, Millions of Insights, Right NowRob Winters
Presentation from Tableau Customer Conference 2013 on building a real time reporting/analytics platform. Topics discussed include definitions of big data and real time, technology choices and rationale, use cases for real time big data, architecture, and pitfalls to avoid.
This document discusses how organizations can use big data and operational analytics to transform IT operations. It outlines how taking a data-driven approach that combines machine data and wire data can provide real-time visibility across networks, applications, databases and other systems. This approach overcomes limitations of using individual monitoring tools by silo. The document also covers key considerations for implementing IT big data solutions such as data gravity, improving the signal-to-noise ratio, and understanding when data needs to be accessed in real-time. It provides an example of how healthcare company McKesson used network traffic analysis to improve Citrix application performance and reduce IT costs.
Real time data integration best practices and architectureBui Kiet
This document discusses best practices and architectures for real-time data integration. It outlines how traditional integration approaches are no longer sufficient due to business demands for more timely, accurate information. Real-time integration can reduce decision latency and improve responsiveness. The document describes different real-time integration patterns like transactional data processing, data replication, and event-driven architectures. It provides examples of how to implement real-time integration through a data integration hub and event processing. The key benefits of real-time integration are also summarized.
Events Everywhere: Enabling Digital Transformation in the Public Sectorconfluent
Events are driving a paradigm shift in application development from commands to reactions to events. Event-driven architectures using streaming platforms allow building real-time, scalable applications. Apache Kafka is an event streaming platform that can be used to build event-driven microservices and data pipelines to integrate various data sources. This enables building applications that deliver value to customers through real-time processing of event streams. The public sector can leverage these approaches to enable digital transformation through software-defined, event-driven applications across various domains.
How to Restructure Active Directory with ZeroIMPACTQuest
We’ll explore best practices for reducing risk and avoiding disruption during AD migrations, ways to improve security, ensure compliance and simplify AD consolidations, and integration processes that can help carefully manage your project before, during and after the actual merger.
DevOps in the Amazon Cloud – Learn from the pioneersNetflix suroGaurav "GP" Pal
DevOps helps accelerate the delivery of software applications through automation and by removing Development & Operations silos. The Netflix Platform Engineering team has developed a robust data pipeline solution called SURO that has been open sourced. Come learn from the experiences of pioneers like Netflix how they are leveraging the data pipeline for new and innovative use cases. This is the presentation by Danny Yuan, Netflix Platform Engineering Team on operational and monitoring aspects of applications on cloud platforms.
How to Restructure and Modernize Active DirectoryQuest
In this presentation, you’ll learn how to apply best practices for reducing migration risk and avoiding disruption, improve security, ensure compliance and simplify your consolidation, and carefully manage your project before, during and after the active directory merger. You can listen to the presentation here: http://bit.ly/2gowzqI.
The process of streaming real-time data from a wide variety of machine data sources and entities can be very complex and unwieldy. Using an agent-based approach, Informatica has invented a new technique and open access product that makes this process much more user friendly and efficient, even when dealing with multiple environments such as Hadoop, Cassandra, Storm, Amazon Kinesis and Complex Event Processing.
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method.
We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy.
Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations.
Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today.
Join us to discover how Zero-ETL can elevate your organization's data strategy.
Pragmatics Driven Issues in Data and Process Integrity in EnterprisesAmit Sheth
Keynote/Invited Talk
IFIP TC-11 First Working Conference on
Keynote/Invited Talk at the IFIP TC-11 First Working Conference on
Integrity and Internal Control in Information Systems
Zurich, Switzerland, December 4-5, 1997
Empowering Real Time Patient Care Through Spark StreamingDatabricks
Takeda’s Plasma Derived Therapies (PDT) business unit has recently embarked on a project to use Spark Streaming on Databricks to empower how they deliver value to their Plasma Donation centers. As patients come in and interface without clinics, we store and track all of the patient interactions in real time and deliver outputs and results based on said interactions. The current problem with our existing architecture is that it is very expensive to maintain and has an unsustainable number of failure points. Spark Streaming is essential for allowing this use case because it allows for a more robust ETL pipeline. With Spark Streaming, we are able to replace our existing ETL processes (that are based on Lamdbas, step functions, triggered jobs, etc) into a purely stream driven architecture.
Data is brought into our s3 raw layer as a large set of CSV files through AWS DMS and Informatica IICS as these services bring data from on-prem systems into our cloud layer. We have a stream currently running which takes these raw files up and merges them into Delta tables established in the bronze/stage layer. We are using AWS Glue as the metadata provider for all of these operations. From the stage layer, we have another set of streams using the stage Delta tables as their source, which transform and conduct stream to stream lookups before writing the enriched records into RDS (silver/prod layer). Once the data has been merged into RDS we have a DMS task which lifts the data back into S3 as CSV files. We have a small intermediary stream which merge these CSV files into corresponding delta tables, from which we have our gold/analytic streams. The on-prem systems are able to speak to the silver layer and allow for the near real-time latency that our patient care centers require.
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
In the ever-evolving landscape of data management, Zero-ETL is an approach that is reshaping how businesses handle and integrate their data. This webinar explores Zero-ETL, a paradigm shift from the traditional Extract, Transform, Load (ETL) process, offering a more streamlined, efficient, and real-time data integration method.
We will begin with an introduction to the concept of Zero-ETL, including how it allows direct access to data in its native environment and real-time data transformation, providing up-to-date information with significantly reduced data redundancy.
Next, we'll take you through several demonstrations showing how Zero-ETL can deliver real-time data and enable the free movement of data between systems. We will also discuss the various tools that support all aspects of Zero-ETL, providing attendees with an understanding of how they can adopt this innovative approach in their organizations.
Lastly, the session will conclude with an interactive Q&A segment, allowing participants to gain deeper insights into how Zero-ETL can be tailored to their specific business needs and how they can get started today.
Join us to discover how Zero-ETL can elevate your organization's data strategy.
This talk was given by Jun Rao (Staff Software Engineer at LinkedIn) and Sam Shah (Senior Engineering Manager at LinkedIn) at the Analytics@Webscale Technical Conference (June 2013).
Mitigating One Million Security Threats With Kafka and Spark With Arun Janart...HostedbyConfluent
Mitigating One Million Security Threats With Kafka and Spark With Arun Janarthnam | Current 2022
Citrix Analytics (Security), a user behavior analytics service, protects 100’s of companies from risks and threats posed by users. The service processes 3 billion events per day and can identify security threats in under a minute.
Kafka is the backbone of our real-time platform. It seamlessly glues the numerous stages required for ETL, Feature Extraction, Model Training & Serving, data access etc and enables us to develop new products faster.
In this session, we will talk about how, in the last 6 months, 7M risk indicators were triggered and 1M threat mitigating actions were taken, and the integral role Kafka played in achieving it. We would also like to share some interesting ways Kafka is used at Citrix. Like, how topics are auto provisioned, and security is handled in a multi-tenant, public facing “northbound” Kafka cluster and the Kafka + Spark optimizations that reduced the cost of running 100’s of streaming jobs.
Massive amounts of data are being generated from various sources like cell phones, sensors, web logs etc. This ambient data needs to be processed in real-time to enable scenarios like fraud detection, manufacturing process control, network monitoring etc. SQL Server StreamInsight provides a platform to process data streams with low latency queries, enabling near real-time analytics and action. Key capabilities include filtering, correlating, aggregating events over windows using a LINQ-like declarative query language.
Real Time Business Platform by Ivan Novick from PivotalVMware Tanzu Korea
This document discusses Pivotal's real time business platform for maximizing the value of data investments. It recommends identifying business problems with high ROI potential, then focusing data solutions on high-speed ingestion, consolidation, real-time queries, and analytics to drive real-time insights. The platform combines Gemfire for fast transactions with Greenplum for analytics. Use cases discussed include predictive maintenance, fraud detection, and recommendation engines. The platform provides a complete solution from data capture and analytics to application integration.
Similar to Building an Enterprise Eventing Framework (20)
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
In our exclusive webinar, you'll learn why event-driven architecture is the key to unlocking cost efficiency, operational effectiveness, and profitability. Gain insights on how this approach differs from API-driven methods and why it's essential for your organization's success.
Santander Stream Processing with Apache Flinkconfluent
Flink is becoming the de facto standard for stream processing due to its scalability, performance, fault tolerance, and language flexibility. It supports stream processing, batch processing, and analytics through one unified system. Developers choose Flink for its robust feature set and ability to handle stream processing workloads at large scales efficiently.
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
In today's data-driven world, the Internet of Things (IoT) is revolutionizing industries and unlocking new possibilities. Join Data Reply, Confluent, and Imply as we unveil a comprehensive solution for IoT that harnesses the power of real-time insights.
Workshop híbrido: Stream Processing con Flinkconfluent
El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real.
Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real.
En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent
Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace.
In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms.
You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes.
Don't miss out on this opportunity to learn from industry experts and take your business to the next level.
La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.
Eventos y Microservicios - Santander TechTalkconfluent
Durante esta sesión examinaremos cómo el mundo de los eventos y los microservicios se complementan y mejoran explorando cómo los patrones basados en eventos nos permiten descomponer monolitos de manera escalable, resiliente y desacoplada.
Q&A with Confluent Experts: Navigating Networking in Confluent Cloudconfluent
This document discusses networking options and best practices for Confluent Cloud. It provides an overview of public endpoints, private link, and peering options. It then discusses best practices for private networking architectures on Azure using hub-and-spoke and private link designs. Finally, it addresses networking considerations and challenges for Kafka Connect managed connectors, as well as planned enhancements for DNS peering and outbound private link support.
Purpose of the session is to have a dive into Apache, Kafka, Data Streaming and Kafka in the cloud
- Dive into Apache Kafka
- Data Streaming
- Kafka in the cloud
Build real-time streaming data pipelines to AWS with Confluentconfluent
Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.
Q&A with Confluent Professional Services: Confluent Service Meshconfluent
No matter whether you are migrating your Kafka cluster to Confluent Cloud, running a cloud-hybrid environment or are in a different situation where data protection and encryption of sensitive information is required, Confluent Service Mesh allows you to transparently encrypt your data without the need to make code changes to you existing applications.
Citi Tech Talk: Event Driven Kafka Microservicesconfluent
Microservices have become a dominant architectural paradigm for building systems in the enterprise, but they are not without their tradeoffs. Learn how to build event-driven microservices with Apache Kafka
Confluent & GSI Webinars series - Session 3confluent
An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities.
It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks.
This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.
This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.
This session will show why the old paradigm does not work and that a new approach to the data strategy needs to be taken. It aims to show how a Data Streaming Platform is integral to the evolution of a company’s data strategy and how Confluent is not just an integration layer but the central nervous system for an organisation
Vous apprendrez également à :
• Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données
• Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience
• Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés
Confluent Partner Tech Talk with Synthesisconfluent
A discussion on the arduous planning process, and deep dive into the design/architectural decisions.
Learn more about the networking, RBAC strategies, the automation, and the deployment plan.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
1. 1
Building an Enterprise Eventing
Framework
Bryan Zelle, IT Manager, Centene
Robert Walters, Dir. of IoT, MongoDB
Jeff Bean, Solution Architect, Confluent
How Centene Improved their ability to interact and engage with healthcare
providers in real time with MongoDB and Confluent Platform
2. 2
Speakers
Bryan Zelle, IT Manager,
Centene
Jeff Bean,
Partner Solution Architect,
Confluent
Robert Walters,
Dir. of IoT,
MongoDB
Lisa Sensmeier,
Partner Marketing,
Confluent
3. 3
About Centene
Challenges with data integration and migration
Decision process
Centene architecture, use case and data flow
Confluent Platform
MongoDB
Q and A
Agenda
5. Centene Introduction
Mission Statement:
Transforming the health of the community, one person at a time
Medicaid:
Medicare (Part D):
Marketplace:
Medicare:
Other:
Total:
12,700,000
4,000,000
2,000,000
1,000,000
3,700,000
23,400,000
30 States
50 States
21 States
28 States
33 States
50 States
Membership Composition:
Industry:
Largest Medicaid and Medicare Managed Care Provider
0
5
10
15
20
25
Centene United Health
Group
Humana Anthem CVS
Membership(Millions)
Largest Managed Care Organizations
Medicaid Medicare & Medicare PDP OtherGovernment Marketplace
6. $-
$10,000
$20,000
$30,000
$40,000
$50,000
$60,000
$70,000
$80,000
$90,000
$100,000
2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005
TotalRevenus(millions)
Centene Yearly Revenue
Centene Revenue WellCare Revenue
Summary of Centene’s
Key Challenges in one
word…
Growth
$4.1 Billion Revenue to $96.9 Billion in 10 Years
$80.4 Billion in growth in past 5 years
$48.6 Billion in growth in past 2½ years
Envolve
Jan 2015
Wellcare
Mar 2019
Fidelis
Sep 2017
HealthNet
Mar 2016
?
?
Cause of the growth…
Mergers & Acquisitions
By the numbers:
7. Medicare
Medicaid
International
Federal
Marketplace
Addressable Market
Federal Medicare$860 B
40%
State Medicaid
International Market
Federal Services
Health Insurance Marketplace
$2,000,000,000,000 +
Centene Revenue
$97,000,000,000 +
Centene
Revenue
4%
Addressable
Market
96%
Additional Growth
Opportunities
$710 B
33%
$260 B
12%
$120B
6%
$115 B
5%
Centene Growth Outlook
Targeted
Pipeline
($270 Billion)
8. Medicare
Medicaid
International
Federal
Marketplace
Addressable Market
Federal Medicare$860 B
40%
State Medicaid
International Market
Federal Services
Health Insurance Marketplace
$2,000,000,000,000 +
Centene Revenue
$97,000,000,000 +
Centene
Revenue
4%
Addressable
Market
96%
Additional Growth
Opportunities
$710 B
33%
$260 B
12%
$120B
6%
$115 B
5%
Centene Growth Outlook
Targeted
Pipeline
($270 Billion)
Mergers
&
Acquisitions
Data Integration
&
Data Migration
9. Data Integration & Data Migration
1
Shared
Database
• Application Refactor
• Direct Schema Coupling
• Scaling Challenges
• Single Point of Failure
10. Data Integration & Data Migration
Shared
Database
Export
Import
• Application Refactor
• Direct Schema Coupling
• Scaling Challenges
• Single Point of Failure
File
2
File Transfer
(Batch ETL)
• Latent Data
• Direct Database Load
• Consistency Challenges
11. Data Integration & Data Migration
Export
Import
Shared
Database
File Transfer
(Batch ETL)
• Application Refactor
• Direct Schema Coupling
• Scaling Challenges
• Single Point of Failure
• Latent Data
• Direct Database Load
• Consistency Challenges
File
API
API
Function Call
Response
3
• Direct Coupling
• Application Refactor
• Availability Concerns
• Scaling Concerns
Remote Procedure
Invocation
12. Data Integration & Data Migration
Shared
Database
File Transfer
(Batch ETL)
Export
Import
• Application Refactor
• Direct Schema Coupling
• Scaling Challenges
• Single Point of Failure
File
• Latent Data
• Direct Database Load
• Consistency Challenges
API
API
Function Call
Response
• Direct Coupling
• Application Refactor
• Availability Concerns
• Scaling Concerns
Remote Procedure
Invocation
4
Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor
• Highly Availability
• Highly Scalable
• Real-Time Data
13. Data Integration & Data Migration
Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor
• Highly Availability
• Highly Scalable
• Real-Time Data
Shared
Database
File Transfer
(Batch ETL)
Export
Import
• Application Refactor
• Direct Schema Coupling
• Scaling Challenges
• Single Point of Failure
File
• Latent Data
• Direct Database Load
• Consistency Challenges
API
API
Function Call
Response
• Direct Coupling
• Application Refactor
• Availability Concerns
• Scaling Concerns
Remote Procedure
Invocation
*
What is a Event?
Definition: “A significant change in state”
• Statement of fact (immutable)
• Expects no response (or call to action)
• Has a defined “timepoint”
Persistence
• Stateless: Notification Event
• Stateful: Event-Carried State Transfer
Synthesized / Composite Events
E1 E2 E3+• Combination
of Events
E1 E3+• Absence of
an Event
14. Data Integration & Data Migration
Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor
• Highly Availability
• Highly Scalable
• Real-Time Data
Shared
Database
File Transfer
(Batch ETL)
Export
Import
• Application Refactor
• Direct Schema Coupling
• Scaling Challenges
• Single Point of Failure
File
• Latent Data
• Direct Database Load
• Consistency Challenges
API
API
Function Call
Response
• Direct Coupling
• Application Refactor
• Availability Concerns
• Scaling Concerns
Remote Procedure
Invocation
*
What is a Event?
Definition: “A significant change in state”
• Statement of fact (immutable)
• Expects no response (no call to action)
• Has a defined “timepoint”
How do we
publish / consume
meaningful
events?
15. Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor*
• Highly Availability
• Highly Scalable
• Real-Time Data
Change Data Capture (CDC)
Broker Topology
Subscribe to Event Channels (Topics)
Self-Defined Event Routing
Partial Coupling of Event Channels
Reduced Complexity at cost of reduced
coordinating of event execution
Advantages:
• Mature 3rd Party Products / Tooling
• Limited Database Load
• Fast Implementation
• No refactoring of source system
Disadvantages:
• No consistent data structure
• No data governance
• Direct technology coupling
16. Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor*
• Highly Availability
• Highly Scalable
• Real-Time Data
Example Event Payload (Informatica CDC Publisher v1.2)
{
"INFA_SEQUENCE":{"string":"2,PWX_GENERIC,1,,2,3,C7084816514A5D260"}
,"DTL__CAPXUSER":{"string":"USER1"}
,"DTL__CAPXTIMESTAMP":{"string":"201803051315400000000000"}
,"INFA_OP_TYPE":{"string":"UPDATE_EVENT"}
,"INFA_TABLE_NAME":{"string":"d8amisou6p.MEMBER_CONTACT"}
,"MEMBER_PCP":{"string":"Dr. Bryan Zelle"}
,"MEMBER_PCP_Present":true
,"MEMBER_PCP_BeforeImage":{"string":"Dr. John Smith"}
,"MEMBER_PCP_BeforeImage_Present":true
}
Transaction
Metadata
Event
Body
Who - Who changed the data ?*
What - What data changed ?
When - When the data changed ?
Where - Where was the data changed ?
What Event
information are we
capturing?
Change Data Capture (CDC)
17. Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor*
• Highly Availability
• Highly Scalable
• Real-Time Data
Mediated (Orchestrated) Eventing
Mediator Topology
Mediator transfers events to assigned
event channel (Topic)
Centrally Coordinated Event Routing
Complete Decoupling of Event
Channels
Increased Complexity at cost of
increased coordination of event
execution
Advantages:
• Consistent / Common Framework
• Enforce Data governance
• Economy of Scale Advantage
• Technology abstraction / decoupling
Disadvantages:
• External bottleneck (Mediator Owner)
• Single Point of Failure
• Duplicative data storage
18. Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor*
• Highly Availability
• Highly Scalable
• Real-Time Data
Mediated (Orchestrated) Eventing
Example Event Payload (JSON vis REST)
“Metadata” : {
“Transaction ID” : “C7084816514A5D260”,
“User ID” : “USER1”,
“Time Stamp” : “201803051315400000000000”,
“Transaction Type” : “UPDATE”,
“Source System” : “d8amisou6p.MEMBER_CONTACT” } ,
“Event Body” : {
“Event Type” : “Member-PCP-Change”,
“Previous Value” : “Dr. John Smith”,
“Updated Value” : “Dr. Bryan Zelle”,
“Event Source” : “Inbound-Member-Call”,
“Caller Information” : {
“Name” : “Jane Doe”,
“Inbound Number” : “1-614-847-0982”,
“Call Resolution Status” : “5 - Highly Satisfied”,
“First Call Resolution” : “Success”,
“Internal Representative” : “CN-10238381”,
”Call Duration (Minutes)” : “8:19” }
Transaction
Metadata
Who - Who changed the data ?*
What - What data changed ?
When - When the data changed ?
Where - Where was the data changed ?
Why - Why was the data changed ?
What Event
information are we
capturing?
Event
Body
*
19. Pub / Sub Messaging
(Streaming ETL)
Event
MessageBus
• Loosely Coupled
• No Application Refactor*
• Highly Availability
• Highly Scalable
• Real-Time Data
Why is the “Why”
so important?
“Event Body” : {
“Event Type” : “Member-PCP-Change”,
“Previous Value” : “Dr. John Smith”,
“Updated Value” : “Dr. Bryan Zelle”,
“Event Source” : “Inbound-Member-Call”,
“Caller Information” : {
“Name” : “Jane Doe”,
“Inbound Number” : “1-614-847-0982”,
“Internal Representative” : “CN-10238381”,
“Call Resolution Status” : “5 - Highly Satisfied”,
“First Call Resolution” : “Success”,
”Call Duration (Minutes)” : “8:19”
}
If Inbound Number not currently associated
with Member -> Create event to add it
to the Member’s Profile
New Events can be
Created or Derived
If Resolution Status >= 4 -> Create event to
assign call rep to member for call-back
If two separate “Member-PCP-Change”
events happen within 2 weeks window ->
Create event for Case Manager to Review
If first call resolution < > “Success” and call
duration > 15 min -> Create event to
escalate call to Supervisor for Audit
20. Recap Recap
1
Centene’s Core Challenge is Growth
cause by Mergers & Acquisitions;
causing us to revaluate our Enterprise
Data Integration and Data Migration
Strategies…
Event
MessageBus
2
Async Pub / Sub Eventing through
Kafka provides us valuable capabilities:
- Highly Scalable
- High Autonomy / Decoupling
- High Availability & Data Resiliency
- Real Time Data Transfer
- Complex Steam Processing
“Metadata” : {
“Transaction ID” : “C7084816514A5D260”,
“User ID” : “USER1”,
“Time Stamp” : “201803051315400000000000”,
“Transaction Type” : “UPDATE”,
“Source System” : “d8amisou6p.MEMBER_CONTACT” } ,
“Event Body” : {
“Event Type” : “Member-PCP-Change”,
“Previous Value” : “Dr. John Smith”,
“Updated Value” : “Dr. Bryan Zelle”,
“Event Source” : “Inbound-Member-Call”,
“Caller Information” : {
“Name” : “Jane Doe”,
“Inbound Number” : “1-614-847-0982”,
“Call Resolution Status” : “5 - Highly Satisfied”,
“First Call Resolution” : “Success”,
“Internal Representative” : “CN-10238381”,
”Call Duration (Minutes)” : “8:19” }
3
Leveraging a Mediator Topology
enables the creation of meaningful
events; which provide insight into why
things are happening, so we can react
to them in real time…
27. K-Streams Application Code
1) Fetch the Event Key --> Key = N:V:O
2) Join Hashed Routing Rules: maps N:V:O to Consumers that have Subscribed
3) Discard Events if no Consumer has Subscribed
4) Generate Duplicate Event message; one for each subscribed Consumers
5) Validate Consumers effective date is within Event date
6) Validate the Event contains tags the Consumer specifies
7) Validate the Consumers PHI permissions are appropriate
8) Place the Event destination topic onto the Kafka message header (metadata)
9) Send the event to the destination topic specified in the message header
28. VM Provisioning / Management
Infrastructure as Code (IaC)
Configuration Management
Service Administration API
Service Dashboards / Monitoring
Incident / Change Management
Service Alerting / Triggers
Service Logging (Audit / Troubleshooting)
Self-Service Portal
Automated Performance / Load Tests (SRE)
Disaster Recovery / Backup Strategy
SLA / SLO (Defined and Captured)
Binary Repository Management
Incident Response Management (IROC)
Ongoing Production Support
Eventing as a Service
- CentEvent
Incident Response
Management
IROC
Binary Repository
Management
Artifactory
Service Dashboards
/ Monitoring
Mongo Charts
Logging - Audit /
Troubleshooting
ELK Stack
Internal
Customers
External
Customers
Gateway DMZ
Unified Self-Service
Portal
Vue JavaScript
Service Alerting /
Triggers
Pager Duty
Incident / Change
Management
ServiceNow
Performance / Load
Testing (SRE)
Java Micro-Service
Disaster Recover /
Backup Strategy
Mongo Replication
SLA / SLO
Java MS -> Mongo
Configuration
Management
Configuration
Management
Configuration
Management
Internal Cloud
Infrastructure
Traditional
Infrastructure
Cloud
Infrastructure
Centene Cloud
Platform (CCP)
Traditional
VMs (Manual)
Amazon Web
Services (AWS)
Unified Service Administration API
IaC IaC IaC
Ongoing
Production Support
CentEvent Team
Internet Download Ansible
Java Admin API
Axway
Complete
Incomplete
In-progress
AWS Fargate
29. SLA-SLO (Mongo Charts)
Event RTR – Connector + Pipeline
IROC – Run Books + APIs
Regression Testing - Producer
ELK Dashboards - Phase I
Consumer Notifications / Alerts
AWS POC - Phase I
Lambda Function POC – Phase I
Self-Service Portal MVP – Phase I
Event Sourcing – Hydration Pipeline
Oct Nov Dec Jan Feb Mar Apr May June
Q4 - 2019 Q1 - 2020 Q2 - 2020
Operations Product Maturity DemandCentEvent Roadmap
Phase II : Consumer
ELK Dashboard - Phase II
AWS MVP - Phase II AWS Deployment - Phase III
Lambda Deployment - Phase II
Phase II : Deployment
31. Application Refactor
Use Case
Member Insight Platform Migration
- Refactor from self-managed Kafka to KaaS
- Refactor from Member Insight API to CentEvent
Total Events
Volume
191,000,000
YTD Inbound
Calls
16,000,000
YTD
Correspondence
10,250,000
YTD Member
Services
4,350,000
YTD Outbound
Calls
4,070,000
Member Event Volume continues to grow significantly -
increasing the importance of a stable and scalable
underlying infrastructure of services
36. 6
Confluent Platform
Operations and Security
Development & Stream Processing
Support,Services,Training&Partners
Apache Kafka
Security plugins | Role-Based Access Control
Control Center | Replicator | Auto Data Balancer | Operator
Connectors
Clients | REST Proxy
MQTT Proxy | Schema Registry
KSQL
Connect Continuous Commit Log Streams
Complete Event
Streaming Platform
Mission-critical
Reliability
Freedom of Choice
Datacenter Public Cloud Confluent Cloud
Self-Managed Software Fully-Managed Service
37. 77
Apache Kafka™ Connect API – Streaming Data Capture
JDBC
Mongo
MySQL
Elastic
Cassandra
HDFS
Kafka Connect API
Kafka Pipeline
Connector
Connector
Connector
Connector
Connector
Connector
Sources Sinks
Fault tolerant
Manage hundreds of
data sources and sinks
Preserves data schema
Part of Apache Kafka
project
Integrated within
Confluent Platform’s
Control Center
38. 8
Schema Registry: Make Data Backwards
Compatible and Future-Proof
Deploy with reliability
● Validate data compatibility and get warnings
● Let developers focus on deploying apps
App 1
!
Schema
Registry
Kafka
topic
Scale with confidence
● Store a versioned history of all schemas
● Enable evolution of schemas while
preserving backwards compatibility for
existing consumers
!
Serializer
App 1
Serializer
39. Intelligent Operational Data Platform
Best way to work
with data
Intelligently put data
where you want it
Freedom to run
anywhere
40. MongoDB Connector for Apache Kafka
Build robust data pipelines for Microservices and Event Driven Architectures
Developed with the community and supported by MongoDB engineers, verified by Confluent
Supports MongoDB as a sink and a source for Kafka
Integrate with Change Streams and Atlas triggers to create fully reactive, event driven pipelines
Available on GitHub - https://github.com/mongodb/mongo-kafka
Confluent Hub - https://www.confluent.io/hub/mongodb/kafka-connect-mongodb
https://www.mongodb.com/kafka-connector