SnowCamp - Adding search to a legacy application
•Download as PPTX, PDF•
0 likes•105 views
Most applications evolve to a point where they need to provide search capabilities. But updating an application is always a risk. Plus, sometimes, you don’t have access to the source code. The easiest way to access the data is by getting them directly from the database. The initial load is the easiest step. However, how do you keep the search index in sync with the database? How do you keep the latency between the search store and the source of truth, so your users don’t have to wait for the next run of the batch to access the newest changes? In this live coding session, we will show you how you can solve this issue by connecting Elasticsearch to the database with a touch of Hazelcast.
Report
Share
Report
Share
Recommended
JOnConf - A CDC use-case: designing an Evergreen Cache
This document discusses using change data capture (CDC) and Hazelcast Jet to build an evergreen cache that remains in sync with a database. It covers alternatives to cache invalidation like polling and triggers, introduces CDC and the Debezium implementation, and proposes a Jet job that watches database change events, analyzes them, and updates the cache accordingly to solve the cache freshness problem.
Zero-Downtime Deployment with Kubernetes, SpringBoot & Flyway
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day.
You need to think about application backward compatibility, possible rollback, database schema migration, etc. I believe the later is one of the biggest pain point. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema migration with the help of Flyway, while keeping the service up during the entire update process.
Bringing olap fully online analyze changing datasets in mem sql and spark wi...
As the world moves from batch to online data processing, real-time data pipelines will supercede siloed data warehouse and transaction processing systems as core infrastructure.
While many analytics solutions tout query execution speed, this is only half of the equation.
For real time workloads, stale data renders query speed irrelevant when results and insights are out of date.
Beyond just “online queries,” real-time enterprises need “online datasets” that continuously update and make data accessible across the organization.
This session will cover approaches to building real-time pipelines with MemSQL, Hadoop, and Spark. Topics will include:
Key industry trends and the move to real-time data pipelines
How MemSQL customer Novus built the premier financial portfolio management platform using MemSQL as a real-time data store and query engine.
Operationalizing Spark for Advanced Analytics
Demonstration of how Pinterest is using the MemSQL Spark Connector to derive real-time insights on interesting and meaningful user activity with MemSQL and Spark.
Introduction to the MemSQL Spark Connector
Strategies for integrating Spark and Hadoop with real-time systems for transaction processing and operational analytics.
Presenters include MemSQL CEO Eric Frenkiel, Novus CTO Robert Stepeck, and Pinterest Software Engineer Yu Yang.
In a world of web portals and push notifications, users have developed demanding expectations for a real-time experience. Continuous updates, a responsive interface, and short loading times have become the norm. Most business analysts and data scientists, whose workflows remain bound by legacy tools and complex data pipelines, lack this fast, simple user experience.
From a business perspective, latency and complexity impede revenue by preventing access to the right data at the right time. Businesses that recognize the value of access to real-time data now have options to meet stringent objectives. They understand that serving “always up to date” data for analysis requires converging transactions and analytics in a real-time system. This session will highlight these architectures and customer achievements.
Zero-downtime deployment on Kubernetes with Hazelcast
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day because we need to handle state. Though it would be much easier to have stateless apps, and despite our best efforts in this direction, state is found in (at least) two places: sessions and databases.
You need to think keeping the state while stopping and starting application nodes. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema change with the help of Hazelcast, while keeping the service up during the entire update process.
Common Strategies for Improving Performance on Your Delta Lakehouse
The Delta Architecture pattern has made the lives of data engineers much simpler, but what about improving query performance for data analysts? What are some common places to look at for tuning query performance? In this session we will cover some common techniques to apply to our delta tables to make them perform better for data analysts queries. We will look at a few examples of how you can analyze a query, and determine what to focus on to deliver better performance results.
The Future of ETL Isn't What It Used to Be
Speaker: Gwen Shapira, Principal Data Architect, Confluent
Join Gwen Shapira, Apache Kafka® committer and co-author of ""Kafka: The Definitive Guide,"" as she presents core patterns of modern data engineering and explains how you can use microservices, event streams and a streaming platform like Apache Kafka to build scalable and reliable data pipelines designed to evolve over time.
This is part 1 of 3 in Streaming ETL - The New Data Integration series.
Watch the recording: https://videos.confluent.io/watch/q7roRtNZBnjiT9C3ii88fo?.
Open Source Big Data Ingestion - Without the Heartburn!
Big Data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Upstream data sources can 'drift' due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail, inducing heartburn in even the most resilient data scientist. This session will survey the big data ingestion landscape, focusing on how open source tools such as Sqoop, Flume, Nifi and StreamSets can keep the data pipeline flowing.
Change Data Feed in Delta
This document discusses Delta Change Data Feed (CDF), which allows capturing changes made to Delta tables. It describes how CDF works by storing change events like inserts, updates and deletes. It also outlines how CDF can be used to improve ETL pipelines, unify batch and streaming workflows, and meet regulatory needs. The document provides examples of enabling CDF, querying change data and storing the change events. It concludes by offering a demo of CDF in Jupyter notebooks.
Recommended
JOnConf - A CDC use-case: designing an Evergreen Cache
This document discusses using change data capture (CDC) and Hazelcast Jet to build an evergreen cache that remains in sync with a database. It covers alternatives to cache invalidation like polling and triggers, introduces CDC and the Debezium implementation, and proposes a Jet job that watches database change events, analyzes them, and updates the cache accordingly to solve the cache freshness problem.
Zero-Downtime Deployment with Kubernetes, SpringBoot & Flyway
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day.
You need to think about application backward compatibility, possible rollback, database schema migration, etc. I believe the later is one of the biggest pain point. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema migration with the help of Flyway, while keeping the service up during the entire update process.
Bringing olap fully online analyze changing datasets in mem sql and spark wi...
As the world moves from batch to online data processing, real-time data pipelines will supercede siloed data warehouse and transaction processing systems as core infrastructure.
While many analytics solutions tout query execution speed, this is only half of the equation.
For real time workloads, stale data renders query speed irrelevant when results and insights are out of date.
Beyond just “online queries,” real-time enterprises need “online datasets” that continuously update and make data accessible across the organization.
This session will cover approaches to building real-time pipelines with MemSQL, Hadoop, and Spark. Topics will include:
Key industry trends and the move to real-time data pipelines
How MemSQL customer Novus built the premier financial portfolio management platform using MemSQL as a real-time data store and query engine.
Operationalizing Spark for Advanced Analytics
Demonstration of how Pinterest is using the MemSQL Spark Connector to derive real-time insights on interesting and meaningful user activity with MemSQL and Spark.
Introduction to the MemSQL Spark Connector
Strategies for integrating Spark and Hadoop with real-time systems for transaction processing and operational analytics.
Presenters include MemSQL CEO Eric Frenkiel, Novus CTO Robert Stepeck, and Pinterest Software Engineer Yu Yang.
In a world of web portals and push notifications, users have developed demanding expectations for a real-time experience. Continuous updates, a responsive interface, and short loading times have become the norm. Most business analysts and data scientists, whose workflows remain bound by legacy tools and complex data pipelines, lack this fast, simple user experience.
From a business perspective, latency and complexity impede revenue by preventing access to the right data at the right time. Businesses that recognize the value of access to real-time data now have options to meet stringent objectives. They understand that serving “always up to date” data for analysis requires converging transactions and analytics in a real-time system. This session will highlight these architectures and customer achievements.
Zero-downtime deployment on Kubernetes with Hazelcast
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day because we need to handle state. Though it would be much easier to have stateless apps, and despite our best efforts in this direction, state is found in (at least) two places: sessions and databases.
You need to think keeping the state while stopping and starting application nodes. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema change with the help of Hazelcast, while keeping the service up during the entire update process.
Common Strategies for Improving Performance on Your Delta Lakehouse
The Delta Architecture pattern has made the lives of data engineers much simpler, but what about improving query performance for data analysts? What are some common places to look at for tuning query performance? In this session we will cover some common techniques to apply to our delta tables to make them perform better for data analysts queries. We will look at a few examples of how you can analyze a query, and determine what to focus on to deliver better performance results.
The Future of ETL Isn't What It Used to Be
Speaker: Gwen Shapira, Principal Data Architect, Confluent
Join Gwen Shapira, Apache Kafka® committer and co-author of ""Kafka: The Definitive Guide,"" as she presents core patterns of modern data engineering and explains how you can use microservices, event streams and a streaming platform like Apache Kafka to build scalable and reliable data pipelines designed to evolve over time.
This is part 1 of 3 in Streaming ETL - The New Data Integration series.
Watch the recording: https://videos.confluent.io/watch/q7roRtNZBnjiT9C3ii88fo?.
Open Source Big Data Ingestion - Without the Heartburn!
Big Data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Upstream data sources can 'drift' due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail, inducing heartburn in even the most resilient data scientist. This session will survey the big data ingestion landscape, focusing on how open source tools such as Sqoop, Flume, Nifi and StreamSets can keep the data pipeline flowing.
Change Data Feed in Delta
This document discusses Delta Change Data Feed (CDF), which allows capturing changes made to Delta tables. It describes how CDF works by storing change events like inserts, updates and deletes. It also outlines how CDF can be used to improve ETL pipelines, unify batch and streaming workflows, and meet regulatory needs. The document provides examples of enabling CDF, querying change data and storing the change events. It concludes by offering a demo of CDF in Jupyter notebooks.
Gobblin meetup-whats new in 0.7
Gobblin provides a data ingestion and lifecycle management platform for LinkedIn's Hadoop clusters. It supports ingesting data from various sources into HDFS, and provides additional capabilities like replication, retention, optimization and compliance. Gobblin treats each dataset independently and orchestrates operators like ingestion, replication and retention through shared metadata. This allows for flexible and extensible management of LinkedIn's large and growing volume of datasets and data flows through their entire lifecycle.
Live Coding a KSQL Application
Join us as we build a complete streaming application with KSQL. There will be plenty of hands-on action, plus a description of our thought process and design choices along the way. Look out for advice on best practices and handy tips and tricks as we go. This is part 2 out of 3 in the Empowering Streams through KSQL series.
The code from the talk is available here: https://gist.github.com/rmoff/7efa882dfd808dbab4eb7b8e6f9eda16.
MemSQL 201: Advanced Tips and Tricks Webcast
This document summarizes a webinar on advanced tips and tricks for MemSQL. It discusses the differences between rowstore and columnstore storage models and when each is best used. It also covers data ingestion using MemSQL Pipelines for real-time loading, data sharding and query tuning techniques like using reference tables. Additionally, it discusses monitoring memory usage, workload management using management views, and query optimization tools like analyzing and optimizing tables.
Not Your Father's Database: How to Use Apache Spark Properly in Your Big Data...
This session will cover a series of problems that are adequately solved with Apache Spark, as well as those that are require additional technologies to implement correctly. Here’s an example outline of some of the topics that will be covered in the talk: Problems that are perfectly solved with Apache Spark: 1) Analyzing a large set of data files. 2) Doing ETL of a large amount of data. 3) Applying Machine Learning & Data Science to a large dataset. 4) Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally.
By Vida Ha at Spark Summit East 2016.
How Microsoft Built and Scaled Cosmos
Cosmos is a large-scale data processing system used by thousands at Microsoft to process exabytes of data across clusters of over 50,000 servers. It provides a SQL-like language and allows teams to easily share and join data. This drives huge scalability requirements. The Apollo scheduler was developed to maximize cluster utilization while minimizing latency for heterogeneous workloads at cloud scale. Later, JetScope was created to support lower latency interactive queries through intermediate result streaming and gang scheduling while maintaining fault tolerance.
Apache flink
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
Live Coding a KSQL Application
Join us as we build a complete streaming application with KSQL. There will be plenty of hands-on action, plus a description of our thought process and design choices along the way. Look out for advice on best practices and handy tips and tricks as we go. This is part 2 out of 3 in the Empowering Streams through KSQL series.
Deploying and Operating KSQL
This document provides an overview of deploying and operating KSQL. It introduces Nick Dearden and Hojjat Jafarpour who work on KSQL at Confluent. The agenda includes discussing deployment, configuration, scaling, and monitoring of KSQL. Specific topics covered are getting started with KSQL, starting the KSQL server, connecting to Kafka and Schema Registry, using the KSQL CLI, deployment patterns and log files. The presentation also demonstrates viewing metrics for KSQL servers, queries, input and output topics through JMX.
Practical examples of using extended events
Many presentations I have seen about SQL Server extended events focus on the mechanics of setting them up and querying the definitions back from the DMVs. While useful, I had always been missing the points about when and how to use extended events. What are they and why should you be using them? This presentation will show three real-world examples of using extended events. Each example will include demos on configuring the event and using the data and you will learn:
• Use extended events rather than queries against DMVs or tracing
• Collect query information
• Catch and examine deadlocks
• Collect actual plans
Flink Community Update December 2015: Year in Review
This document summarizes the Berlin Apache Flink Meetup #12 that took place in December 2015. It discusses the key releases and improvements to Flink in 2015, including the release of versions 0.10.0 and 0.10.1, and new features that were added to the master branch, such as improvements to the Kafka connector. It also lists pending pull requests, recommended reading, and provides statistics on Flink's growth in 2015 in terms of GitHub activity, meetup groups, organizations at Flink Forward, and articles published.
Building Continuous Application with Structured Streaming and Real-Time Data ...
This document summarizes a presentation about building a structured streaming connector for continuous applications using Azure Event Hubs as the streaming data source. It discusses key design considerations like representing offsets, implementing the getOffset and getBatch methods required by structured streaming sources, and challenges with testing asynchronous behavior. It also outlines issues contributed back to the Apache Spark community around streaming checkpoints and recovery.
Spark SQL Beyond Official Documentation
Implementing efficient Spark application with the goal of having maximal performance often requires knowledge that goes beyond official documentation. Understanding Spark’s internal processes and features may help to design the queries in alignment with internal optimizations and thus achieve high efficiency during execution. In this talk we will focus on some internal features of Spark SQL which are not well described in official documentation with a strong emphasis on explaining these features on some basic examples while sharing some performance tips along the way.
Exploring KSQL Patterns
Watch this talk here: http://videos.confluent.io/watch/Rgd5r8oV1ToDpcFfenMQrF
This session covers the patterns and techniques of using KSQL. Tim Berglund discusses the various building blocks that you can use in your own applications, starting with the language syntax itself and covering how and when to use its powerful capabilities like a pro. This is part 1 out of 3 in the Empowering Streams through KSQL series.
BigData conference - Introduction to stream processing
This document discusses stream processing and summarizes a presentation about the topic. It introduces Hazelcast Jet as a stream processing engine and covers open data standards like GTFS. It also describes a demo that uses GTFS data to enrich public transit vehicle position updates in real-time using Hazelcast Jet. The presentation discusses streaming approaches, benefits over batch processing, and provides an overview of stream processing concepts.
Reactive Database Access With Slick 3
Slick is a database query and access library for Scala that provides a functional-relational mapping to connect Scala code to databases. It allows type-safe database queries to be written using a Scala collection-like API and generates SQL, providing full control over queries while avoiding impedance mismatch issues. Slick supports reactive streams processing and various database operations including joins, transactions, plain SQL, and pre-compiled queries across many database types.
Data platform architecture principles - ieee infrastructure 2020
This document discusses principles for building a healthy data platform, including:
1. Establishing explicit contracts between teams to define dependencies and service level agreements.
2. Abstracting the data platform into services for ingesting, storing, and processing data in motion and at rest.
3. Enabling observability of data pipelines through metadata collection and integration with tools like Marquez to provide lineage, availability, and change management visibility.
An Engineering Approach to Database Evaluations
This talk will go over a methodical approach for making a decision, dig into interesting tradeoffs, and give tips about what things to look for under the hood and how to evaluate the tech behind the database.
Lessons Learned from Building and Operating Scuba
This document provides an overview of Scuba, Facebook's real-time analytics database. It summarizes Scuba's key features including real-time data ingestion and querying capabilities with simple rollup queries and flexible schemas. It also describes Scuba's architecture with distributed data storage and demand control. Finally, it discusses lessons learned from building and operating Scuba, including common issues and reasons for its success filling a specific niche for analytics.
Delta: Building Merge on Read
We can leverage Delta Lake, structured streaming for write-heavy use cases. This talk will go through a use case at Intuit whereby we built MOR as an architecture to allow for a very low SLA, etc. For MOR, there are different ways to view the fresh data, so we will also go over the methods used to perfTest the various ways that we were able to arrive at the best method for the given use case.
Telco analytics at scale
This document discusses Telco analytics at scale using distributed stream processing. It describes using technologies like Apache Spark Streaming, Kafka, and Hadoop (HDFS, Hive, HBase) to ingest and process large volumes of streaming data from various sources in real-time or near real-time. Example use cases discussed include fraud detection, real-time rating, security information and event management. It also covers strategies for distributed in-memory caching and rule processing to enable low latency analytics at high throughput scales needed for telco data and applications.
Zero-downtime Deployment on Kubernetes
Presented by: Nicolas Fränkel
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day because we need to handle state. Though it would be much easier to have stateless apps, and despite our best efforts in this direction, state is found in (at least) two places: sessions and databases.
You need to think keeping the state while stopping and starting application nodes. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema change with the help of Hazelcast, while keeping the service up during the entire update process.
Vote Early, Vote Often: From Napkin to Canvassing Application in a Single Wee...
The frenetic pace of application development in modern IT organizations means it’s not unusual to demand an application be built with minimal requirement gathering – literally, from a napkin-based sketch – to a working first draft of the app within extremely short time frames – even a weekend! – with production deployment to follow just a few days later.
I'll demonstrate a real-life application development scenario – the creation of a mobile application that gives election canvassers a tool to identify, classify, and inform voters in a huge suburban Chicago voting district – using the latest Oracle application development UI, data modeling tools, and database technology. Along the way, we’ll show how Oracle APEX makes short work of building a working application while the Oracle DBA leverages her newest tools – SQL Developer and Data Modeler – to build a secure, reliable, scalable application for her development team.
More Related Content
What's hot
Gobblin meetup-whats new in 0.7
Gobblin provides a data ingestion and lifecycle management platform for LinkedIn's Hadoop clusters. It supports ingesting data from various sources into HDFS, and provides additional capabilities like replication, retention, optimization and compliance. Gobblin treats each dataset independently and orchestrates operators like ingestion, replication and retention through shared metadata. This allows for flexible and extensible management of LinkedIn's large and growing volume of datasets and data flows through their entire lifecycle.
Live Coding a KSQL Application
Join us as we build a complete streaming application with KSQL. There will be plenty of hands-on action, plus a description of our thought process and design choices along the way. Look out for advice on best practices and handy tips and tricks as we go. This is part 2 out of 3 in the Empowering Streams through KSQL series.
The code from the talk is available here: https://gist.github.com/rmoff/7efa882dfd808dbab4eb7b8e6f9eda16.
MemSQL 201: Advanced Tips and Tricks Webcast
This document summarizes a webinar on advanced tips and tricks for MemSQL. It discusses the differences between rowstore and columnstore storage models and when each is best used. It also covers data ingestion using MemSQL Pipelines for real-time loading, data sharding and query tuning techniques like using reference tables. Additionally, it discusses monitoring memory usage, workload management using management views, and query optimization tools like analyzing and optimizing tables.
Not Your Father's Database: How to Use Apache Spark Properly in Your Big Data...
This session will cover a series of problems that are adequately solved with Apache Spark, as well as those that are require additional technologies to implement correctly. Here’s an example outline of some of the topics that will be covered in the talk: Problems that are perfectly solved with Apache Spark: 1) Analyzing a large set of data files. 2) Doing ETL of a large amount of data. 3) Applying Machine Learning & Data Science to a large dataset. 4) Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally.
By Vida Ha at Spark Summit East 2016.
How Microsoft Built and Scaled Cosmos
Cosmos is a large-scale data processing system used by thousands at Microsoft to process exabytes of data across clusters of over 50,000 servers. It provides a SQL-like language and allows teams to easily share and join data. This drives huge scalability requirements. The Apollo scheduler was developed to maximize cluster utilization while minimizing latency for heterogeneous workloads at cloud scale. Later, JetScope was created to support lower latency interactive queries through intermediate result streaming and gang scheduling while maintaining fault tolerance.
Apache flink
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale.
Live Coding a KSQL Application
Join us as we build a complete streaming application with KSQL. There will be plenty of hands-on action, plus a description of our thought process and design choices along the way. Look out for advice on best practices and handy tips and tricks as we go. This is part 2 out of 3 in the Empowering Streams through KSQL series.
Deploying and Operating KSQL
This document provides an overview of deploying and operating KSQL. It introduces Nick Dearden and Hojjat Jafarpour who work on KSQL at Confluent. The agenda includes discussing deployment, configuration, scaling, and monitoring of KSQL. Specific topics covered are getting started with KSQL, starting the KSQL server, connecting to Kafka and Schema Registry, using the KSQL CLI, deployment patterns and log files. The presentation also demonstrates viewing metrics for KSQL servers, queries, input and output topics through JMX.
Practical examples of using extended events
Many presentations I have seen about SQL Server extended events focus on the mechanics of setting them up and querying the definitions back from the DMVs. While useful, I had always been missing the points about when and how to use extended events. What are they and why should you be using them? This presentation will show three real-world examples of using extended events. Each example will include demos on configuring the event and using the data and you will learn:
• Use extended events rather than queries against DMVs or tracing
• Collect query information
• Catch and examine deadlocks
• Collect actual plans
Flink Community Update December 2015: Year in Review
This document summarizes the Berlin Apache Flink Meetup #12 that took place in December 2015. It discusses the key releases and improvements to Flink in 2015, including the release of versions 0.10.0 and 0.10.1, and new features that were added to the master branch, such as improvements to the Kafka connector. It also lists pending pull requests, recommended reading, and provides statistics on Flink's growth in 2015 in terms of GitHub activity, meetup groups, organizations at Flink Forward, and articles published.
Building Continuous Application with Structured Streaming and Real-Time Data ...
This document summarizes a presentation about building a structured streaming connector for continuous applications using Azure Event Hubs as the streaming data source. It discusses key design considerations like representing offsets, implementing the getOffset and getBatch methods required by structured streaming sources, and challenges with testing asynchronous behavior. It also outlines issues contributed back to the Apache Spark community around streaming checkpoints and recovery.
Spark SQL Beyond Official Documentation
Implementing efficient Spark application with the goal of having maximal performance often requires knowledge that goes beyond official documentation. Understanding Spark’s internal processes and features may help to design the queries in alignment with internal optimizations and thus achieve high efficiency during execution. In this talk we will focus on some internal features of Spark SQL which are not well described in official documentation with a strong emphasis on explaining these features on some basic examples while sharing some performance tips along the way.
Exploring KSQL Patterns
Watch this talk here: http://videos.confluent.io/watch/Rgd5r8oV1ToDpcFfenMQrF
This session covers the patterns and techniques of using KSQL. Tim Berglund discusses the various building blocks that you can use in your own applications, starting with the language syntax itself and covering how and when to use its powerful capabilities like a pro. This is part 1 out of 3 in the Empowering Streams through KSQL series.
BigData conference - Introduction to stream processing
This document discusses stream processing and summarizes a presentation about the topic. It introduces Hazelcast Jet as a stream processing engine and covers open data standards like GTFS. It also describes a demo that uses GTFS data to enrich public transit vehicle position updates in real-time using Hazelcast Jet. The presentation discusses streaming approaches, benefits over batch processing, and provides an overview of stream processing concepts.
Reactive Database Access With Slick 3
Slick is a database query and access library for Scala that provides a functional-relational mapping to connect Scala code to databases. It allows type-safe database queries to be written using a Scala collection-like API and generates SQL, providing full control over queries while avoiding impedance mismatch issues. Slick supports reactive streams processing and various database operations including joins, transactions, plain SQL, and pre-compiled queries across many database types.
Data platform architecture principles - ieee infrastructure 2020
This document discusses principles for building a healthy data platform, including:
1. Establishing explicit contracts between teams to define dependencies and service level agreements.
2. Abstracting the data platform into services for ingesting, storing, and processing data in motion and at rest.
3. Enabling observability of data pipelines through metadata collection and integration with tools like Marquez to provide lineage, availability, and change management visibility.
An Engineering Approach to Database Evaluations
This talk will go over a methodical approach for making a decision, dig into interesting tradeoffs, and give tips about what things to look for under the hood and how to evaluate the tech behind the database.
Lessons Learned from Building and Operating Scuba
This document provides an overview of Scuba, Facebook's real-time analytics database. It summarizes Scuba's key features including real-time data ingestion and querying capabilities with simple rollup queries and flexible schemas. It also describes Scuba's architecture with distributed data storage and demand control. Finally, it discusses lessons learned from building and operating Scuba, including common issues and reasons for its success filling a specific niche for analytics.
Delta: Building Merge on Read
We can leverage Delta Lake, structured streaming for write-heavy use cases. This talk will go through a use case at Intuit whereby we built MOR as an architecture to allow for a very low SLA, etc. For MOR, there are different ways to view the fresh data, so we will also go over the methods used to perfTest the various ways that we were able to arrive at the best method for the given use case.
Telco analytics at scale
This document discusses Telco analytics at scale using distributed stream processing. It describes using technologies like Apache Spark Streaming, Kafka, and Hadoop (HDFS, Hive, HBase) to ingest and process large volumes of streaming data from various sources in real-time or near real-time. Example use cases discussed include fraud detection, real-time rating, security information and event management. It also covers strategies for distributed in-memory caching and rule processing to enable low latency analytics at high throughput scales needed for telco data and applications.
What's hot (20)
Not Your Father's Database: How to Use Apache Spark Properly in Your Big Data...
Not Your Father's Database: How to Use Apache Spark Properly in Your Big Data...
Flink Community Update December 2015: Year in Review
Flink Community Update December 2015: Year in Review
Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...
BigData conference - Introduction to stream processing
BigData conference - Introduction to stream processing
Data platform architecture principles - ieee infrastructure 2020
Data platform architecture principles - ieee infrastructure 2020
Similar to SnowCamp - Adding search to a legacy application
Zero-downtime Deployment on Kubernetes
Presented by: Nicolas Fränkel
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day because we need to handle state. Though it would be much easier to have stateless apps, and despite our best efforts in this direction, state is found in (at least) two places: sessions and databases.
You need to think keeping the state while stopping and starting application nodes. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema change with the help of Hazelcast, while keeping the service up during the entire update process.
Vote Early, Vote Often: From Napkin to Canvassing Application in a Single Wee...
The frenetic pace of application development in modern IT organizations means it’s not unusual to demand an application be built with minimal requirement gathering – literally, from a napkin-based sketch – to a working first draft of the app within extremely short time frames – even a weekend! – with production deployment to follow just a few days later.
I'll demonstrate a real-life application development scenario – the creation of a mobile application that gives election canvassers a tool to identify, classify, and inform voters in a huge suburban Chicago voting district – using the latest Oracle application development UI, data modeling tools, and database technology. Along the way, we’ll show how Oracle APEX makes short work of building a working application while the Oracle DBA leverages her newest tools – SQL Developer and Data Modeler – to build a secure, reliable, scalable application for her development team.
Elasticsearch : petit déjeuner du 13 mars 2014
Elasticsearch est un moteur de recherche Open Source très puissant basé sur
Apache Lucene. Il permet l'indexation de millions de données, leur recherche et leur
analyse en temps réel. Les outils Elascticsearch sont déjà utilisés par des acteurs de
référence tels que FourSquare, GitHub, OpenDataSoft ou encore Dailymotion.
Alter Way et Elasticsearch vous convient à venir découvrir la suite Elasticsearch
enfin disponible en version 1.0 et prête pour la production !
OSCONF Koshi - Zero downtime deployment with Kubernetes, Flyway and Spring Boot
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day. You need to think about application backward compatibility, possible rollback, database schema migration, etc. I believe the later is one of the biggest pain point. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema migration with the help of Flyway, while keeping the service up during the entire update process.
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
This document discusses Elasticsearch and how it can be used to search, analyze, and make sense of large amounts of data. It provides examples of how Elasticsearch is being used by large companies to handle petabytes of data and gain insights. Implementations in France are highlighted. The document concludes by demonstrating how easily Elasticsearch can be deployed and used to ingest and search sample data.
Continuum Analytics and Python
Talk given to the Philly Python Users Group (PUG) on October 1, 2015: http://www.meetup.com/phillypug/ Thanks SIG (http://www.sig.com) for hosting!
JavaDay Istanbul - 3 improvements in your microservices architecture
While a microservices architecture is more scalable than a monolith, it has a direct hit on performance.
To cope with that, one performance improvement is to set up a cache. It can be configured for database access, for REST calls or just to store session state across a cluster of server nodes. In this demo-based talk, I’ll show how Hazelcast In-Memory Data Grid can help you in each one of those areas and how to configure it. Hint: it’s much easier than one would expect.
Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB
Now that you've chosen MongoDB, how should you migrate your deployment? In this webinar, we'll discuss the different ways companies migrate to MongoDB including hybrid solutions, where only new or certain features are migrated, and all-in migrations, where everything is migrated. We'll use real customer examples to discuss the pros and cons of each so you can decide what strategy will work best for your deployment.
Devclub.lv - Introduction to stream processing
While “software is eating the world”, those who are able to best manage the huge mass of data will emerge out on the top.
The batch processing model has been faithfully serving us for decades. However, it might have reached the end of its usefulness for all but some very specific use-cases. As the pace of businesses increases, most of the time, decision-makers prefer slightly wrong data sooner, than 100% accurate data later. Stream processing – or data streaming – exactly matches this usage: instead of managing the entire bulk of data, manage pieces of them as soon as they become available.
Database automated build and test - SQL In The City Cambridge
David Atkinson gives a presentation on automated database build and testing, or continuous integration (CI), for databases. He explains that CI ensures database code is always working by automatically building and testing databases on each change. The demo shows two developers using CI to incrementally keep a test database up to date, run automated tests, and generate deployment scripts. When a fix introduces a breaking change, they address it by adding a custom migration script to the deployment package to safely update data during deployment.
SQL in the Hybrid World
This is a high level presentation I delivered at BIWA Summit. It's just some high level thoughts related to today's NoSQL and Hadoop SQL engines (not deeply technical).
Delivering Changes for Applications and Databases
Presentation used at PortoData (28/07/2016) about manageing database changes in application projects. Demoed Flyway migrations for script packages, and a strategy for using DBUp and localDb for integration testing at the data layer
Tech talks - 3 performance improvements
While a microservices architecture is more scalable than a monolith, it has a direct hit on performance.
To cope with that, one performance improvement is to set up a cache. It can be configured for database access, for REST calls or just to store session state across a cluster of server nodes. In this demo-based talk, I’ll show how Hazelcast In-Memory Data Grid can help you in each one of those areas and how to configure it.
Hint: it’s much easier than one would expect.
XP Days Ukraine - Zero-downtime deployment with Kubernetes, Spring Boot and F...
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day.
You need to think about application backward compatibility, possible rollback, database schema migration, etc. I believe the later is one of the biggest pain point. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema migration with the help of Flyway, while keeping the service up during the entire update process.
JCon - Zero-Downtime-Deployment with Kubernetes, Spring Boot and Flyway
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day.
You need to think about application backward compatibility, possible rollback, database schema migration, etc. I believe the later is one of the biggest pain point. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema migration with the help of Flyway, while keeping the service up during the entire update process.
VoxxedDays Cluj - Zero-downtime deployment with Kubernetes, Spring Boot and F...
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day. You need to think about application backward compatibility, possible rollback, database schema migration, etc. I believe the later is one of the biggest pain point.
In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema migration with the help of Flyway, while keeping the service up during the entire update process.
DevOpsDays Madrid - Zero-downtime deployment with Kubernetes, Spring Boot and...
Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day.
You need to think about application backward compatibility, possible rollback, database schema migration, etc. I believe the later is one of the biggest pain point. In this talk, I’ll demo how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema migration with the help of Flyway, while keeping the service up during the entire update process.
Adversary Driven Defense in the Real World
Talk by Shannon Lietz and James Wickett at DevOps Enterprise Summit 2018, Las Vegas.
Talk covers finding real world adversaries and balancing your effort and defenses to adjust for them.
Simplifying Change Data Capture using Databricks Delta
In this talk, we will present recent enhancements to the techniques previously discussed in this blog: https://databricks.com/blog/2018/10/29/simplifying-change-data-capture-with-databricks-delta.html. We will start by discussing the different CDC architectures that can be deployed in concert with Databricks Delta. We will then use notebooks to demonstrate updated CDC SQL and look at performance tuning considerations for both batch as well as streaming CDC pipelines into Delta.
(Tugdual grall) no sql-hadoop
The document discusses how big data trends are pushing organizations toward NoSQL technologies for operational databases. There are two main reasons for this - first, most new data is unstructured or semi-structured, which relational databases are not suited for, while NoSQL databases can better handle this type of data. Second, application developers need a highly flexible data model to quickly store new types of data and evolve schemas, which fixed schema relational databases do not provide as easily as schema-less NoSQL databases. The document argues that for these reasons relating to big data, NoSQL deployments should be paired with Hadoop for full benefits.
Similar to SnowCamp - Adding search to a legacy application (20)
Vote Early, Vote Often: From Napkin to Canvassing Application in a Single Wee...
Vote Early, Vote Often: From Napkin to Canvassing Application in a Single Wee...
OSCONF Koshi - Zero downtime deployment with Kubernetes, Flyway and Spring Boot
OSCONF Koshi - Zero downtime deployment with Kubernetes, Flyway and Spring Boot
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
Séminaire Big Data Alter Way - Elasticsearch - octobre 2014
JavaDay Istanbul - 3 improvements in your microservices architecture
JavaDay Istanbul - 3 improvements in your microservices architecture
Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB
Webinar: MongoDB Migration Patterns - How Customers Start Using MongoDB
Database automated build and test - SQL In The City Cambridge
Database automated build and test - SQL In The City Cambridge
XP Days Ukraine - Zero-downtime deployment with Kubernetes, Spring Boot and F...
XP Days Ukraine - Zero-downtime deployment with Kubernetes, Spring Boot and F...
JCon - Zero-Downtime-Deployment with Kubernetes, Spring Boot and Flyway
JCon - Zero-Downtime-Deployment with Kubernetes, Spring Boot and Flyway
VoxxedDays Cluj - Zero-downtime deployment with Kubernetes, Spring Boot and F...
VoxxedDays Cluj - Zero-downtime deployment with Kubernetes, Spring Boot and F...
DevOpsDays Madrid - Zero-downtime deployment with Kubernetes, Spring Boot and...
DevOpsDays Madrid - Zero-downtime deployment with Kubernetes, Spring Boot and...
Simplifying Change Data Capture using Databricks Delta
Simplifying Change Data Capture using Databricks Delta
More from Nicolas Fränkel
Un CV de dévelopeur toujours a jour
On dit que GitHub est le CV d'un développeur. Un rapide coup d'œil à votre historique de commit et les recruteurs savent tout de vous. Cette approche comporte quelques problèmes. La plupart des entreprises ne publient même pas leur code sous une licence Open Source. Si vous travaillez pour l'une d'entre elles, et si vous n'êtes pas un développeur Open Source les soirs et les week-ends, alors vous n'avez aucune chance.
Récemment, GitHub a permis un certain degré de personnalisation de son profil. Ainsi, même si votre historique de commit a plus de blanc que de vert, vous pouvez fournir un bon point d'entrée pour les employeurs potentiels. Mais ça ne vaut que l'effort que vous y mettez et les données perdent leur valeur rapidement. Pourtant, avec un peu de travail et l'aide d'outils d'automatisation (tels que GitHub Actions), vous pouvez présenter un profil toujours à jour.
jLove - A Change-Data-Capture use-case: designing an evergreen cache
When one’s app is challenged with poor performances, it’s easy to set up a cache in front of one’s SQL database. It doesn’t fix the root cause (e.g. bad schema design, bad SQL query, etc.) but it gets the job done. If the app is the only component that writes to the underlying database, it’s a no-brainer to update the cache accordingly, so the cache is always up-to-date with the data in the database.
Things start to go sour when the app is not the only component writing to the DB. Among other sources of writes, there are batches, other apps (shared databases exist unfortunately), etc. One might think about a couple of ways to keep data in sync i.e. polling the DB every now and then, DB triggers, etc. Unfortunately, they all have issues that make them unreliable and/or fragile.
You might have read about Change-Data-Capture before. It’s been described by Martin Kleppmann as turning the database inside out: it means the DB can send change events (SELECT, DELETE and UPDATE) that one can register to. Just opposite to Event Sourcing that aggregates events to produce state, CDC is about getting events out of states. Once CDC is implemented, one can subscribe to its events and update the cache accordingly. However, CDC is quite in its early stage, and implementations are quite specific.
In this talk, I’ll describe an easy-to-setup architecture that leverages CDC to have an evergreen cache.
ADDO - Your own Kubernetes controller, not only in Go
In Kubernetes, operators allow the API to be extended to your heart content. If one task requires too much YAML, it’s easy to create an operator to take care of the repetitive cruft, and only require a minimum amount of YAML.
On the other hand, since its beginnings, the Go language has been advertised as closer to the hardware, and is now ubiquitous in low-level programming. Kubernetes has been rewritten from Java to Go, and its whole ecosystem revolves around Go. For that reason, It’s only natural that Kubernetes provides a Go-based framework to create your own operator. While it makes sense, it requires organizations willing to go down this road to have Go developers, and/or train their teams in Go. While perfectly acceptable, this is not the only option. In fact, since Kubernetes is based on REST, why settle for Go and not use your own favorite language?
In this talk, I’ll describe what is an operator, how they work, how to design one, and finally demo a Java-based operator that is as good as a Go one.
TestCon Europe - Mutation Testing to the Rescue of Your Tests
Unit testing ensures your production code is relevant. But what does ensure your testing code is relevant? Come discover mutation testing and make sure your never forget another assert again.
In the realm of testing, the code coverage metrics is the most often talked about. However, it doesn’t mean that the test has been useful or even that an assert has been coded. Mutation testing is a strategy to make sure that the test code is relevant.
In this talk, Nicolas will explain how Code Coverage is computed and what its inherent flaw is. Afterwards, he will describe how Mutation Testing work and how it helps pointing out code that is tested but leave out corner cases. He will also demo PIT, a Java production-grade framework that enables Mutation Testing.
OSCONF Jaipur - A Hitchhiker's Tour to Containerizing a Java application
As “the Cloud” becomes more and more widespread, now is a good time to assess how you can containerize your Java application. I assume you’re able to write a a Dockerfile around the generated JAR. However, each time the application’s code will change, the whole image will need to be rebuilt. If you’re deploying to a local Kubernetes cluster environment, this increases that much the length of the feedback loop.
In this demo-based talk, I’ll present different ways to get your Java app in a container: Dockerfile, Jib, and Cloud Native Buildpacks. We will also have a look at what kind of Docker image they generate, how they layer the images, whether those images are compatible with skaffold, etc.
GeekcampSG 2020 - A Change-Data-Capture use-case: designing an evergreen cache
CDC is a brand new approach that "turns the database inside out": it allows to get events out of the database state. This can be leveraged to get a cache that is never stale.
OSCONF Hyderabad - Shorten all URLs!
At a point in the past, it was forecast that Java would die, but the JVM platform would be its legacy. And in fact, for a long time, the JVM has been tremendously successful. Wikipedia itself lists a bunch of languages that run on it, some of them close to Java e.g. Kotlin, some of them very remote e.g. Clojure.
But nowadays, the Cloud is becoming ubiquitous. Containerization is the way to go to alleviate some of the vendor lock-in issues. Kubernetes is a de facto platform. If a container needs to be killed for whatever reason (resource consumption, unhealthy, etc.), a new one needs to replace it as fast as possible. In that context, the JVM seems to be a dead-end: its startup time is huge in comparison to a native process. Likewise, it consumes a lot of memory that just increase the monthly bill.
What does that mean for us developers? Has all the time spent in learning the JVM ecosystem been invested with no hope of return over investment? Shall we need to invest even more time in new languages, frameworks, libraries, etc.? That is one possibility for sure. But we can also leverage our existing knowledge, and embrace the Cloud and containers ways with the help of some tools.
In this talk, I’ll create a simple URL shortener with a “standard” stack: Kotlin, JAX-RS and Hazelcast. Then, with the help of Quarkus and GraalVM, I’ll turn this application into a native executable with all Cloud/Container related work has been moved to the build process.
London In-Memory Computing Meetup - A Change-Data-Capture use-case: designing...
When one’s app is challenged with poor performances, it’s easy to set up a cache in front of one’s SQL database. It doesn’t fix the root cause (e.g. bad schema design, bad SQL query, etc.) but it gets the job done. If the app is the only component that writes to the underlying database, it’s a no-brainer to update the cache accordingly, so the cache is always up-to-date with the data in the database.
Things start to go sour when the app is not the only component writing to the DB. Among other sources of writes, there are batches, other apps (shared databases exist, unfortunately), etc. One might think about a couple of ways to keep data in sync i.e. polling the DB every now and then, DB triggers, etc. Unfortunately, they all have issues that make them unreliable and/or fragile.
In this talk, I will describe an easy-to-setup architecture that leverages CDC to have an evergreen cache.
JUG Tirana - Introduction to data streaming
This talk will be about the reasons behind the new stream processing model, how it compare to the old batch model, what are their pros and cons, and a list of existing technologies implementing stream processing with their most prominent characteristics. It will contain details of one possible use-case of data streaming that is not possible with batches: display in (near) real-time all trains in Switzerland and their position on a map, beginning with an overview of all the requirements and the design. Finally, using an OpenData endpoint and the Hazelcast platform,showing a working demo implementation of it.
Java.IL - Your own Kubernetes controller, not only in Go!
In Kubernetes, operators allow the API to be extended to your heart content. If one task requires too much YAML, it’s easy to create an operator to take care of the repetitive cruft, and only require a minimum amount of YAML.
On the other hand, since its beginnings, the Go language has been advertised as closer to the hardware, and is now ubiquitous in low-level programming. Kubernetes has been rewritten from Java to Go, and its whole ecosystem revolves around Go. For that reason, It’s only natural that Kubernetes provides a Go-based framework to create your own operator. While it makes sense, it requires organizations willing to go down this road to have Go developers, and/or train their teams in Go. While perfectly acceptable, this is not the only option. In fact, since Kubernetes is based on REST, why settle for Go and not use your own favorite language?
In this talk, I’ll describe what is an operator, how they work, how to design one, and finally demo a Java-based operator that is as good as a Go one.
vJUG - Introduction to data streaming
The document discusses stream processing and provides an overview of Hazelcast Jet. It begins with explaining why streaming is useful and describes different streaming approaches like event-driven programming. It then provides details on Hazelcast Jet, including its concepts of pipelines and jobs. The document also discusses open data standards like GTFS and demonstrates a sample streaming pipeline that enriches public transportation data from open APIs.
London Java Community - An Experiment in Continuous Deployment of JVM applica...
A couple of years ago, continuous integration in the JVM ecosystem meant Jenkins. Since that time, a lot of other tools have been made available. But new tools don’t mean new features, just new ways. Besides that, what about continuous deployment? There’s no tool that allows deploying new versions of a JVM-based application without downtime. The only way to achieve zero downtime is to have multiple nodes deployed on a platform, and let that platform achieve that e.g. Kubernetes.
And yet, achieving true continuous deployment of bytecode on one single JVM instance is possible if one changes one’s way of looking at things. What if the compilation could be seen as changes? What if those changes could be stored in a data store, and a listener on this data store could stream those changes to the running production JVM via the Attach API?
In this talk, we'll demo exactly that using Hazelcast and Hazelcast Jet - but it’s possible to re-use the principles that will be shown using other streaming technologies.
OSCONF - Your own Kubernetes controller: not only in Go
This document discusses creating Kubernetes operators and controllers using different programming languages besides Go. It suggests that a Java-based controller is possible using the GraalVM, which allows creating native executables from Java bytecode. Key points covered include what controllers and operators are, that no specific technology stack is required, and that the JVM could be a good option for controller development with GraalVM's support for polyglot programming and creating native applications.
vKUG - Migrating Spring Boot apps from annotation-based config to Functional
Migrating Spring Boot apps from annotation-based config to Functional with Kotlin - Nicolas Fränkel
In the latest years, there has been some push-back against frameworks, and more specifically annotations: some call them magic. Obviously, they make understanding the flow of the application harder.
Spring and Spring Boot latest versions go along this trend, by offering an additional way to configure beans with explicit code instead of annotations. It’s declarative in the sense it looks like configuration, though it’s based on Domain-Specific Language(s). This talk aims to demo a step-by-step process to achieve that.
AllTheTalks.online - A Streaming Use-Case: And Experiment in Continuous Deplo...
Nicolas Fränkel proposes an experimental use case of continuous deployment of bytecode by dynamically loading Java agents into a running JVM using the Attach API. The agent would use the Instrumentation API to modify bytecode of running applications by redefining classes in order to deploy changes made by a continuous integration pipeline without restarting the JVM. Current limitations include a lack of tagging and an inability to add/remove fields or methods via redefinition. Improvements could include streaming changes directly from the CI pipeline and adding metadata to classes from the Git tag.
ING Meetup - Migrating Spring Boot Config Annotations to Functional with Kotlin
In the latest years, there has been some push-back against frameworks, and more specifically annotations: some call them magic. Obviously, they make understanding the flow of the application harder. Spring and Spring Boot latest versions go along this trend, by offering an additional way to configure beans with explicit code instead of annotations. It's declarative in the sense it looks like configuration, though it's based on Domain-Specific Language(s). This talk aims to demo a step-by-step process to achieve that.
SouJava- 3 easy performance improvements in your microservices architecture
While a microservices architecture is more scalable than a monolith, it has a direct hit on performance.
To cope with that, one performance improvement is to set up a cache. It can be configured for database access, for REST calls or just to store session state across a cluster of server nodes. In this demo-based talk, I'll show how Hazelcast In-Memory Data Grid can help you in each one of those areas and how to configure it. Hint: it's much easier than one would expect.
JUG SF - Introduction to data streaming
While “software is eating the world”, those who are able to best manage the huge mass of data will emerge out on the top.
The batch processing model has been faithfully serving us for decades. However, it might have reached the end of its usefulness for all but some very specific use-cases. As the pace of businesses increases, most of the time, decision makers prefer slightly wrong data sooner, than 100% accurate data later. Stream processing - or data streaming - exactly matches this usage: instead of managing the entire bulk of data, manage pieces of them as soon as they become available.
In this talk, I’ll define the context in which the old batch processing model was born, the reasons that are behind the new stream processing one, how they compare, what are their pros and cons, and a list of existing technologies implementing the latter with their most prominent characteristics. I’ll conclude by describing in detail one possible use-case of data streaming that is not possible with batches: display in (near) real-time all trains in Switzerland and their position on a map. I’ll go through the all the requirements and the design. Finally, using an OpenData endpoint and the Hazelcast platform, I’ll try to impress attendees with a working demo implementation of it.
SCALE - Stream processing and Open Data, a match made in Heaven
While “software is eating the world”, those who are able to best manage the huge mass of data will emerge out on the top.
Some countries in Europe understand the potential there's in existing data that sits behind closed fences, and passed laws to make this data available to everyone.
On the other hand, the batch processing model gets more and more obsolete: users want the information as soon as possible. While there’s a trade-off between correctness of data, and its speed of delivery, most business decisions do not rely on 100% correct data.
In this talk, I’ll explain how one can leverage the data related to public transportation in Switzerland, to display them in (near) real-time on a map.
ConFoo - 3 performance improvements
While a microservices architecture is more scalable than a monolith, it has a direct hit on performance.
To cope with that, one performance improvement is to set up a cache. It can be configured for database access, for REST calls or just to store session state across a cluster of server nodes. In this demo-based talk, I'll show how Hazelcast In-Memory Data Grid can help you in each one of those areas and how to configure it.
More from Nicolas Fränkel (20)
jLove - A Change-Data-Capture use-case: designing an evergreen cache
jLove - A Change-Data-Capture use-case: designing an evergreen cache
ADDO - Your own Kubernetes controller, not only in Go
ADDO - Your own Kubernetes controller, not only in Go
TestCon Europe - Mutation Testing to the Rescue of Your Tests
TestCon Europe - Mutation Testing to the Rescue of Your Tests
OSCONF Jaipur - A Hitchhiker's Tour to Containerizing a Java application
OSCONF Jaipur - A Hitchhiker's Tour to Containerizing a Java application
GeekcampSG 2020 - A Change-Data-Capture use-case: designing an evergreen cache
GeekcampSG 2020 - A Change-Data-Capture use-case: designing an evergreen cache
London In-Memory Computing Meetup - A Change-Data-Capture use-case: designing...
London In-Memory Computing Meetup - A Change-Data-Capture use-case: designing...
Java.IL - Your own Kubernetes controller, not only in Go!
Java.IL - Your own Kubernetes controller, not only in Go!
London Java Community - An Experiment in Continuous Deployment of JVM applica...
London Java Community - An Experiment in Continuous Deployment of JVM applica...
OSCONF - Your own Kubernetes controller: not only in Go
OSCONF - Your own Kubernetes controller: not only in Go
vKUG - Migrating Spring Boot apps from annotation-based config to Functional
vKUG - Migrating Spring Boot apps from annotation-based config to Functional
AllTheTalks.online - A Streaming Use-Case: And Experiment in Continuous Deplo...
AllTheTalks.online - A Streaming Use-Case: And Experiment in Continuous Deplo...
ING Meetup - Migrating Spring Boot Config Annotations to Functional with Kotlin
ING Meetup - Migrating Spring Boot Config Annotations to Functional with Kotlin
SouJava- 3 easy performance improvements in your microservices architecture
SouJava- 3 easy performance improvements in your microservices architecture
SCALE - Stream processing and Open Data, a match made in Heaven
SCALE - Stream processing and Open Data, a match made in Heaven
Recently uploaded
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
Modelling Up - DDDEurope 2024 - Amsterdam
What to do when you have a perfect model for your software but you are constrained by an imperfect business model?
This talk explores the challenges of bringing modelling rigour to the business and strategy levels, and talking to your non-technical counterparts in the process.
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
原版定制【微信:bwp0011】《(hull学位证书)英国赫尔大学毕业证硕士文凭》【微信:bwp0011】成绩单 、雅思、外壳、留信学历认证永久存档查询，采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信bwp0011】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信bwp0011】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
Orca: Nocode Graphical Editor for Container Orchestration
Tool demo on CEDI/SISTEDES/JISBD2024 at A Coruña, Spain. 2024.06.18
"Orca: Nocode Graphical Editor for Container Orchestration"
by Pedro J. Molina PhD. from Metadev
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Located in the bustling city of Bangalore, Manyata Tech Park stands as one of India’s largest and most prominent tech parks, playing a pivotal role in shaping the city’s reputation as the Silicon Valley of India. Established to cater to the burgeoning IT and technology sectors
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
Are you ready to unlock the secrets hidden within Java thread dumps? Join us for a hands-on session where we'll delve into effective troubleshooting patterns to swiftly identify the root causes of production problems. Discover the right tools, techniques, and best practices while exploring *real-world case studies of major outages* in Fortune 500 enterprises. Engage in interactive lab exercises where you'll have the opportunity to troubleshoot thread dumps and uncover performance issues firsthand. Join us and become a master of Java thread dump analysis!
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
In today's fiercely competitive mobile app market, the role of the QA team is pivotal for continuous improvement and sustained success. Effective testing strategies are essential to navigate the challenges confidently and precisely. Ensuring the perfection of mobile apps before they reach end-users requires thoughtful decisions in the testing plan.
一比一原版(USF毕业证)旧金山大学毕业证如何办理
USF硕士毕业证成绩单【微信95270640】一比一伪造旧金山大学文凭@假冒USF毕业证成绩单+Q微信95270640办理USF学位证书@仿造USF毕业文凭证书@购买旧金山大学毕业证成绩单USF真实使馆认证/真实留信认证回国人员证明
#一整套旧金山大学文凭证件办理#—包含旧金山大学旧金山大学本科毕业证成绩单学历认证|使馆认证|归国人员证明|教育部认证|留信网认证永远存档教育部学历学位认证查询办理国外文凭国外学历学位认证#我们提供全套办理服务。
一整套留学文凭证件服务：
一：旧金山大学旧金山大学本科毕业证成绩单毕业证 #成绩单等全套材料从防伪到印刷水印底纹到钢印烫金
二：真实使馆认证（留学人员回国证明）使馆存档
三：真实教育部认证教育部存档教育部留服网站永久可查
四：留信认证留学生信息网站永久可查
国外毕业证学位证成绩单办理方法：
1客户提供办理旧金山大学旧金山大学本科毕业证成绩单信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）。
教育部文凭学历认证认证的用途：
如果您计划在国内发展那么办理国内教育部认证是必不可少的。事业性用人单位如银行国企公务员在您应聘时都会需要您提供这个认证。其他私营 #外企企业无需提供！办理教育部认证所需资料众多且烦琐所有材料您都必须提供原件我们凭借丰富的经验帮您快速整合材料让您少走弯路。
实体公司专业为您服务如有需要请联系我: 微信95270640奈一次次令他失望山娃今年岁上五年级识得很多字从走出小屋开始山娃就知道父亲的家和工地共有一个很动听的名字——天河工地的底层空空荡荡很宽阔很凉爽在地上铺上报纸和水泥袋父亲和工人们中午全睡在地上地面坑坑洼洼山娃曾多次绊倒过也曾有长铁钉穿透凉鞋刺在脚板上但山娃不怕工地上也常有五六个从乡下来的小学生他们的父母亲也是高楼上的建筑工人小伙伴来自不同省份都操着带有浓重口音的普通话可不知为啥山娃不仅很快与他们熟识了
WWDC 2024 Keynote Review: For CocoaCoders Austin
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
🏎️Tech Transformation: DevOps Insights from the Experts 👩💻
Connect with fellow Trailblazers, learn from industry experts Glenda Thomson (Salesforce, Principal Technical Architect) and Will Dinn (Judo Bank, Salesforce Development Lead), and discover how to harness DevOps tools with Salesforce.
Measures in SQL (SIGMOD 2024, Santiago, Chile)
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar
June. 18, 2024
For more Alluxio Events: https://www.alluxio.io/events/
Speaker:
- Jianjian Xie (Staff Software Engineer, Alluxio)
As Trino users increasingly rely on cloud object storage for retrieving data, speed and cloud cost have become major challenges. The separation of compute and storage creates latency challenges when querying datasets; scanning data between storage and compute tiers becomes I/O bound. On the other hand, cloud API costs related to GET/LIST operations and cross-region data transfer add up quickly.
The newly introduced Trino file system cache by Alluxio aims to overcome the above challenges. In this session, Jianjian will dive into Trino data caching strategies, the latest test results, and discuss the multi-level caching architecture. This architecture makes Trino 10x faster for data lakes of any scale, from GB to EB.
What you will learn:
- Challenges relating to the speed and costs of running Trino in the cloud
- The new Trino file system cache feature overview, including the latest development status and test results
- A multi-level cache framework for maximized speed, including Trino file system cache and Alluxio distributed cache
- Real-world cases, including a large online payment firm and a top ridesharing company
- The future roadmap of Trino file system cache and Trino-Alluxio integration
14 th Edition of International conference on computer vision
About the event
14th Edition of International conference on computer vision
Computer conferences organized by ScienceFather group. ScienceFather takes the privilege to invite speakers participants students delegates and exhibitors from across the globe to its International Conference on computer conferences to be held in the Various Beautiful cites of the world. computer conferences are a discussion of common Inventions-related issues and additionally trade information share proof thoughts and insight into advanced developments in the science inventions service system. New technology may create many materials and devices with a vast range of applications such as in Science medicine electronics biomaterials energy production and consumer products.
Nomination are Open!! Don't Miss it
Visit: computer.scifat.com
Award Nomination: https://x-i.me/ishnom
Conference Submission: https://x-i.me/anicon
For Enquiry: Computer@scifat.com
Photoshop Tutorial for Beginners (2024 Edition)
Photoshop Tutorial for Beginners (2024 Edition)
Explore the evolution of programming and software development and design in 2024. Discover emerging trends shaping the future of coding in our insightful analysis."
Here's an overview:Introduction: The Evolution of Programming and Software DevelopmentThe Rise of Artificial Intelligence and Machine Learning in CodingAdopting Low-Code and No-Code PlatformsQuantum Computing: Entering the Software Development MainstreamIntegration of DevOps with Machine Learning: MLOpsAdvancements in Cybersecurity PracticesThe Growth of Edge ComputingEmerging Programming Languages and FrameworksSoftware Development Ethics and AI RegulationSustainability in Software EngineeringThe Future Workforce: Remote and Distributed TeamsConclusion: Adapting to the Changing Software Development LandscapeIntroduction: The Evolution of Programming and Software Development
Photoshop Tutorial for Beginners (2024 Edition)Explore the evolution of programming and software development and design in 2024. Discover emerging trends shaping the future of coding in our insightful analysis."Here's an overview:Introduction: The Evolution of Programming and Software DevelopmentThe Rise of Artificial Intelligence and Machine Learning in CodingAdopting Low-Code and No-Code PlatformsQuantum Computing: Entering the Software Development MainstreamIntegration of DevOps with Machine Learning: MLOpsAdvancements in Cybersecurity PracticesThe Growth of Edge ComputingEmerging Programming Languages and FrameworksSoftware Development Ethics and AI RegulationSustainability in Software EngineeringThe Future Workforce: Remote and Distributed TeamsConclusion: Adapting to the Changing Software Development LandscapeIntroduction: The Evolution of Programming and Software Development
The importance of developing and designing programming in 2024
Programming design and development represents a vital step in keeping pace with technological advancements and meeting ever-changing market needs. This course is intended for anyone who wants to understand the fundamental importance of software development and design, whether you are a beginner or a professional seeking to update your knowledge.
Course objectives:
1. **Learn about the basics of software development:
- Understanding software development processes and tools.
- Identify the role of programmers and designers in software projects.
2. Understanding the software design process:
- Learn about the principles of good software design.
- Discussing common design patterns such as Object-Oriented Design.
3. The importance of user experience (UX) in modern software:
- Explore how user experience can improve software acceptance and usability.
- Tools and techniques to analyze and improve user experience.
4. Increase efficiency and productivity through modern development tools:
- Access to the latest programming tools and languages used in the industry.
- Study live examples of applications
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM 🐰
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
原版一模一样【微信：741003700 】【(sdsu毕业证书)圣地亚哥州立大学毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
办理(sdsu毕业证书)圣地亚哥州立大学毕业证【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。
办理(sdsu毕业证书)圣地亚哥州立大学毕业证【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：
校徽：象征着学校的荣誉和传承。
校名:学校英文全称
授予学位：本部分将注明获得的具体学位名称。
毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。
颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。
其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。
办理(sdsu毕业证书)圣地亚哥州立大学毕业证【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。
综上所述，办理(sdsu毕业证书)圣地亚哥州立大学毕业证【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。
Boost Your Savings with These Money Management Apps
A money management app can transform your financial life by tracking expenses, creating budgets, and setting financial goals. These apps offer features like real-time expense tracking, bill reminders, and personalized insights to help you save and manage money effectively. With a user-friendly interface, they simplify financial planning, making it easier to stay on top of your finances and achieve long-term financial stability.
Superpower Your Apache Kafka Applications Development with Complementary Open...
Kafka Summit talk (Bangalore, India, May 2, 2024, https://events.bizzabo.com/573863/agenda/session/1300469 )
Many Apache Kafka use cases take advantage of Kafka’s ability to integrate multiple heterogeneous systems for stream processing and real-time machine learning scenarios. But Kafka also exists in a rich ecosystem of related but complementary stream processing technologies and tools, particularly from the open-source community. In this talk, we’ll take you on a tour of a selection of complementary tools that can make Kafka even more powerful. We’ll focus on tools for stream processing and querying, streaming machine learning, stream visibility and observation, stream meta-data, stream visualisation, stream development including testing and the use of Generative AI and LLMs, and stream performance and scalability. By the end you will have a good idea of the types of Kafka “superhero” tools that exist, which are my favourites (and what superpowers they have), and how they combine to save your Kafka applications development universe from swamploads of data stagnation monsters!
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Consistent toolbox talks are critical for maintaining workplace safety, as they provide regular opportunities to address specific hazards and reinforce safe practices.
These brief, focused sessions ensure that safety is a continual conversation rather than a one-time event, which helps keep safety protocols fresh in employees' minds. Studies have shown that shorter, more frequent training sessions are more effective for retention and behavior change compared to longer, infrequent sessions.
Engaging workers regularly, toolbox talks promote a culture of safety, empower employees to voice concerns, and ultimately reduce the likelihood of accidents and injuries on site.
The traditional method of conducting safety talks with paper documents and lengthy meetings is not only time-consuming but also less effective. Manual tracking of attendance and compliance is prone to errors and inconsistencies, leading to gaps in safety communication and potential non-compliance with OSHA regulations. Switching to a digital solution like Safelyio offers significant advantages.
Safelyio automates the delivery and documentation of safety talks, ensuring consistency and accessibility. The microlearning approach breaks down complex safety protocols into manageable, bite-sized pieces, making it easier for employees to absorb and retain information.
This method minimizes disruptions to work schedules, eliminates the hassle of paperwork, and ensures that all safety communications are tracked and recorded accurately. Ultimately, using a digital platform like Safelyio enhances engagement, compliance, and overall safety performance on site. https://safelyio.com/
Recently uploaded (20)
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
Orca: Nocode Graphical Editor for Container Orchestration
Orca: Nocode Graphical Editor for Container Orchestration
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
🏎️Tech Transformation: DevOps Insights from the Experts 👩💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩💻
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
Alluxio Webinar | 10x Faster Trino Queries on Your Data Platform
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Boost Your Savings with These Money Management Apps
Boost Your Savings with These Money Management Apps
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
SnowCamp - Adding search to a legacy application
- 1. @nicolas_frankel Adding search to a legacy application without hassle
- 2. @nicolas_frankel 2022 Merci à nos sponsors
- 3. @nicolas_frankel Me, myself and I Developer Developer advocate Interested in system architecture
- 6. @nicolas_frankel The problem with search “All applications evolve to the point that they need to add a search capability” -- Anonymous
- 7. @nicolas_frankel First decision step Do you keep your SQL database? • Queries using LIKE • Dedicated search database
- 8. @nicolas_frankel Second decision step Choosing the search engine datastore • Elasticsearch • SolR • Algolia • Cloud-based search engines
- 9. @nicolas_frankel Third decision step Choosing the “right” architecture • Index data in application code • Index data via ORM framework • Custom • Hibernate Search • Index data from database via a batch job • Something else?
- 10. @nicolas_frankel The problem of dual writes No transaction between the database and Elasticsearch What operation shall we execute first: 1. Update database and hope Elasticsearch succeeds? 2. The other way around?
- 12. @nicolas_frankel Source data from the database to Elasticsearch
- 13. @nicolas_frankel Databas e The problem with batches Elasticsearc h Job INSERT x NOT FOUND SELECT * INDEX x FOUND
- 14. @nicolas_frankel The problem with batches Run too frequently • Waste of resources Run not frequently enough • Data not searchable
- 15. @nicolas_frankel Change-Data-Capture “In databases, Change Data Capture is a set of software design patterns used to determine and track the data that has changed so that action can be taken using the changed data. CDC is an approach to data integration that is based on the identification, capture and delivery of the changes made to enterprise data sources.” -- https://en.wikipedia.org/wiki/Change_data_capture
- 16. @nicolas_frankel CDC implementation options 1. Polling + Timestamps on rows 2. Polling + Version numbers on rows 3. Polling + Status indicators on rows 4. Triggers on tables 5. Log scanners -- https://en.wikipedia.org/wiki/Change_data_capture
- 17. @nicolas_frankel What is a transaction/binary/etc. log? “The binary log contains ‘events’ that describe database changes such as table creation operations or changes to table data.” -- https://dev.mysql.com/doc/refman/8.0/en/binary-log.html
- 18. @nicolas_frankel What if we “hacked” the log?
- 19. @nicolas_frankel Sample MySQL binlog ### UPDATE `test`.`t` ### WHERE ### @1=1 /* INT meta=0 nullable=0 is_null=0 */ ### @2='apple' /* VARSTRING(20) meta=20 nullable=0 is_null=0 */ ### @3=NULL /* VARSTRING(20) meta=0 nullable=1 is_null=1 */ ### SET ### @1=1 /* INT meta=0 nullable=0 is_null=0 */ ### @2='pear' /* VARSTRING(20) meta=20 nullable=0 is_null=0 */ ### @3='2009:01:01' /* DATE meta=0 nullable=1 is_null=0 */ # at 569 #150112 21:40:14 server id 1 end_log_pos 617 CRC32 0xf134ad89 #Table_map: `test`.`t` mapped to number 251 # at 617 #150112 21:40:14 server id 1 end_log_pos 665 CRC32 0x87047106 #Delete_rows: table id 251 flags: STMT_END_F
- 20. @nicolas_frankel But… Implementation-dependent Fragile Who maintains/debugs it?
- 21. @nicolas_frankel Debezium to the rescue Java-based abstraction layer for CDC Provided by Red Hat Apache v2 licensed
- 22. @nicolas_frankel Debezium “Debezium records all row-level changes within each database table in a change event stream” -- https://debezium.io/
- 23. @nicolas_frankel Debezium connector plugins Production-ready • MongoDB • MySQL • PostrgreSQL • SQL Server • DB2 (!) • Oracle Incubating • Cassandra • Vitess
- 24. @nicolas_frankel Stream instead of batch Propagate changes as soon as they occur Still eventual consistency • Because of duplicating data in distributed systems! But gap reduced to the smallest window possible
- 25. @nicolas_frankel Thanks for your attention! https://blog.frankel.ch/ @nicolas_frankel https://bit.ly/legacy-search https://apisix.apache.org/
Editor's Notes
- @startuml database "\nDatabase\n" as db database "\nElasticsearch\n" as es interface API as api interface SQL as sql component " Batch " as batch << job >> << <&clock> scheduled >> db -up-() sql es -up-() api batch .left.> sql: read batch .right.> api: write @enduml