The document discusses asynchronous programming with Spring 4.X and relational database management systems in microservices architectures. It covers asynchronous vs synchronous programming, the C10K problem of handling 10,000 clients simultaneously and its solutions like load balancing, NoSQL databases, and event-driven programming. It provides examples of using Spring's @Async annotation, DeferredResult, and CompletableFuture for asynchronous programming. It also discusses challenges with databases being blocking I/O and solutions like avoiding blocking on database connections, using asynchronous data access with Spring, and transaction management across asynchronous calls.
События, шины и интеграция данных в непростом мире микросервисов / Валентин Г...Ontico
Микросервисы получают все большую популярность в компаниях по всему миру. Какие организационные и технические проблемы они помогают решать? С какого момента монолиты перестают справляться с растущей нагрузкой на ваш сервис? Почему Zalando -- самый большой онлайн-ретейлер в Европе -- выбрал микросервисы в качестве главной архитектуры для новых проектов?
Помогая в решении организационных проблем быстрорастущей компании, микросервисы ставят новые технические задачи, одной из которых, помимо увеличения сложности системы в целом, является проблема безопасного обмена сообщениями между микросервисами, удобной интеграции данных и возможности их корреляции и анализа.
Слушатели узнают, как в Zalando решают эту проблему с использованием централизованной шины передачи данных -- Nakadi. Получат представление о тех проблемах, которые их могут поджидать при выборе похожей архитектуры на примере проблем выбора формата передачи данных, системы версионирования формата сообщений и сложностей эксплуатации высоконагруженных кластеров Kafka в облачной системе AWS.
ApacheCon EU 2016 - Apache Camel the integration libraryClaus Ibsen
This presentation will demonstrate to developers involved with integration how the Apache Camel project can make your life much easier.
We start with an introduction to what Apache Camel is, and how you can use Camel to make integration much easier. Allowing you to focus on your business logic, rather than low level messaging protocols, and transports.
You will hear how Apache Camel is related Enterprise Integration Patterns which you can use in your architectural designs and as well in Java or XML code, running on the JVM with Camel.
You will also hear what other features Camel provides out of the box, which can make integration much easier for you.
События, шины и интеграция данных в непростом мире микросервисов / Валентин Г...Ontico
Микросервисы получают все большую популярность в компаниях по всему миру. Какие организационные и технические проблемы они помогают решать? С какого момента монолиты перестают справляться с растущей нагрузкой на ваш сервис? Почему Zalando -- самый большой онлайн-ретейлер в Европе -- выбрал микросервисы в качестве главной архитектуры для новых проектов?
Помогая в решении организационных проблем быстрорастущей компании, микросервисы ставят новые технические задачи, одной из которых, помимо увеличения сложности системы в целом, является проблема безопасного обмена сообщениями между микросервисами, удобной интеграции данных и возможности их корреляции и анализа.
Слушатели узнают, как в Zalando решают эту проблему с использованием централизованной шины передачи данных -- Nakadi. Получат представление о тех проблемах, которые их могут поджидать при выборе похожей архитектуры на примере проблем выбора формата передачи данных, системы версионирования формата сообщений и сложностей эксплуатации высоконагруженных кластеров Kafka в облачной системе AWS.
ApacheCon EU 2016 - Apache Camel the integration libraryClaus Ibsen
This presentation will demonstrate to developers involved with integration how the Apache Camel project can make your life much easier.
We start with an introduction to what Apache Camel is, and how you can use Camel to make integration much easier. Allowing you to focus on your business logic, rather than low level messaging protocols, and transports.
You will hear how Apache Camel is related Enterprise Integration Patterns which you can use in your architectural designs and as well in Java or XML code, running on the JVM with Camel.
You will also hear what other features Camel provides out of the box, which can make integration much easier for you.
How Typepad changed their architecture without taking down the serviceroyans
Are you pushing the envelope of what your web application can handle? Do your engineers feel the impending need to overhaul and retrofit your system before it unexpectedly keels over on you? Come hear Six Apart talk about the tools, software and most importantly, the process it has been using to completely rewrite the backend of TypePad without disrupting thousands of users and paying customers.
Full stack development with node and NoSQL - All Things Open - October 2017Matthew Groves
What is different about this generation of web applications? A solid development approach must consider latency, throughput, and interactivity demanded by users users across mobile devices, web browsers, and IoT. These applications often use NoSQL to support a flexible data model and easy scalability required for modern development.
A full stack application (composed of Couchbase, WebAPI, Angular2, and ASP.NET/ASP.NET Core) will be demonstrated in this session. The individual parts of a stack may vary, but the overall design is the focus.
We start with an introduction to what Apache Camel is, and how you can use Camel to make integration much easier. Allowing you to focus on your business logic, rather than low level messaging protocols, and transports. You will also hear what other features Camel provides out of the box, which can make integration much easier for you.
We look into web console tooling that allows you to get insight into your running Apache Camel applications, which has among others visual route diagrams with tracing/debugging and profiling capabilities. In addition to the web tooling we will also show you other tools in the making.
Altitude SF 2017: Advanced VCL: Shielding and ClusteringFastly
Shielding and clustering are two techniques that help in various ways, including increasing cache hit ratio and allowing for more effective storage. If used incorrectly, however, both can make your life more difficult. In this session, Fastly Engineer Rogier “Doc” Mulhuijzen will discuss how to deal with these prickly yet critical techniques, ultimately changing the way you think about writing VCL.
Как сделать высоконагруженный сервис, не зная количество нагрузки / Олег Обле...Ontico
Существует множество архитектур и способов масштабирования систем. Сегодня многие компании мигрируют в облачные сервисы или используют контейнеры. Но действительно ли это так необходимо и нужно ли следовать трендам?
В данном докладе мне бы хотелось рассказать об архитектуре, которую я спланировал и внедрил в компании InnoGames. Архитектура, не требующая вмешательства администратора в случае лавинообразного увеличения нагрузки и, что ещё более важно, умеющая редуцироваться в случае отсутствия её для экономии затрат.
Вы узнаете об опыте создания сервиса с очень непростыми критериями и поймёте, что не обязательно платить в 3 раза дороже за AWS или любую подобную систему.
- Что такое CRM. Зачем нам этот сервис.
- Инфраструктура.
-- Graphite. Почему он должен быть надежным и быстрым.
-- Puppet + gitlab.
-- Балансировка нагрузки.
-- Наше облако. Зачем нам openstack, когда есть serveradmin!? Как роль сервера определяется несколькими атрибутами в веб-интерфейсе.
-- Nagios + аггрегаторы. Другой взгляд на то, как мониторить сервисы через Graphite.
-- Мониторинг кластеров. Clusterhc и Grafsy.
-- Brassmonkey. Как мы написали своего сисадмина на python.
-- Бэкапы.
- Архитектура CRM3.
- Autoscaling или как проанализировать кучу данных и принять решения.
2 hour session where I cover what is Apache Camel, latest news on the upcoming Camel v3, and then the main topic of the talk is the new Camel K sub-project for running integrations natively on the cloud with kubernetes. The last part of the talk is about running Camel with GraalVM / Quarkus to archive native compiled binaries that has impressive startup and footprint.
Empowering developers to deploy their own data storesTomas Doran
Empowering developers to deploy their own data stores using Terrafom, Puppet and rage. A talk about automating server building and configuration for Elasticsearch clusters, using Hashicorp and puppet labs tool. Presented at Config Management Camp 2016 in Ghent
Spring Framework 4.0과 4.1의 주요 업데이트 내용을 정리했습니다. 그리고, Spring I/O 2015 | The conference 에서 공개된 4.2의 소식들을 모아보았습니다. :)
ps. 제가 웬만하면 자료는 한글로 바꾸는 편인데... 이번에는 시간이 너무 부족해서 대부분 영어를 그대로 사용했네요. 다음에 기회가 되면 한글화시키는걸로...
2 hour session where I cover what is Apache Camel, latest news on the upcoming Camel v3, and then the main topic of the talk is the new Camel K sub-project for running integrations natively on the cloud with kubernetes. The last part of the talk is about running Camel with GraalVM / Quarkus to archive native compiled binaries that has impressive startup and footprint.
Apache Camel v3, Camel K and Camel QuarkusClaus Ibsen
In this session, we will explore key challenges with function interactions and coordination, addressing these problems using Enterprise Integration Patterns (EIP) and modern approaches with the latest innovations from the Apache Camel community:
Apache Camel is the Swiss army knife of integration, and the most powerful integration framework. In this session you will hear about the latest features in the brand new 3rd generation.
Camel K, is a lightweight integration platform that enables Enterprise Integration Patterns to be used natively on any Kubernetes cluster. When used in combination with Knative, a framework that adds serverless building blocks to Kubernetes, and the subatomic execution environment of Quarkus, Camel K can mix serverless features such as auto-scaling, scaling to zero, and event-based communication with the outstanding integration capabilities of Apache Camel.
- Apache Camel 3
- Camel K
- Camel Quarkus
We will show how Camel K works. We’ll also use examples to demonstrate how Camel K makes it easier to connect to cloud services or enterprise applications using some of the 300 components that Camel provides.
This presentation explains how to deploy and use the Integrated Caching feature on Netscaler. I gave this presentation to Citrix staff, customers and partners in worldwide in 2011. The presentation covers best practices and gotchas :) Integrated Caching is an excellent feature that can greatly improve the performance of your website.
How Typepad changed their architecture without taking down the serviceroyans
Are you pushing the envelope of what your web application can handle? Do your engineers feel the impending need to overhaul and retrofit your system before it unexpectedly keels over on you? Come hear Six Apart talk about the tools, software and most importantly, the process it has been using to completely rewrite the backend of TypePad without disrupting thousands of users and paying customers.
Full stack development with node and NoSQL - All Things Open - October 2017Matthew Groves
What is different about this generation of web applications? A solid development approach must consider latency, throughput, and interactivity demanded by users users across mobile devices, web browsers, and IoT. These applications often use NoSQL to support a flexible data model and easy scalability required for modern development.
A full stack application (composed of Couchbase, WebAPI, Angular2, and ASP.NET/ASP.NET Core) will be demonstrated in this session. The individual parts of a stack may vary, but the overall design is the focus.
We start with an introduction to what Apache Camel is, and how you can use Camel to make integration much easier. Allowing you to focus on your business logic, rather than low level messaging protocols, and transports. You will also hear what other features Camel provides out of the box, which can make integration much easier for you.
We look into web console tooling that allows you to get insight into your running Apache Camel applications, which has among others visual route diagrams with tracing/debugging and profiling capabilities. In addition to the web tooling we will also show you other tools in the making.
Altitude SF 2017: Advanced VCL: Shielding and ClusteringFastly
Shielding and clustering are two techniques that help in various ways, including increasing cache hit ratio and allowing for more effective storage. If used incorrectly, however, both can make your life more difficult. In this session, Fastly Engineer Rogier “Doc” Mulhuijzen will discuss how to deal with these prickly yet critical techniques, ultimately changing the way you think about writing VCL.
Как сделать высоконагруженный сервис, не зная количество нагрузки / Олег Обле...Ontico
Существует множество архитектур и способов масштабирования систем. Сегодня многие компании мигрируют в облачные сервисы или используют контейнеры. Но действительно ли это так необходимо и нужно ли следовать трендам?
В данном докладе мне бы хотелось рассказать об архитектуре, которую я спланировал и внедрил в компании InnoGames. Архитектура, не требующая вмешательства администратора в случае лавинообразного увеличения нагрузки и, что ещё более важно, умеющая редуцироваться в случае отсутствия её для экономии затрат.
Вы узнаете об опыте создания сервиса с очень непростыми критериями и поймёте, что не обязательно платить в 3 раза дороже за AWS или любую подобную систему.
- Что такое CRM. Зачем нам этот сервис.
- Инфраструктура.
-- Graphite. Почему он должен быть надежным и быстрым.
-- Puppet + gitlab.
-- Балансировка нагрузки.
-- Наше облако. Зачем нам openstack, когда есть serveradmin!? Как роль сервера определяется несколькими атрибутами в веб-интерфейсе.
-- Nagios + аггрегаторы. Другой взгляд на то, как мониторить сервисы через Graphite.
-- Мониторинг кластеров. Clusterhc и Grafsy.
-- Brassmonkey. Как мы написали своего сисадмина на python.
-- Бэкапы.
- Архитектура CRM3.
- Autoscaling или как проанализировать кучу данных и принять решения.
2 hour session where I cover what is Apache Camel, latest news on the upcoming Camel v3, and then the main topic of the talk is the new Camel K sub-project for running integrations natively on the cloud with kubernetes. The last part of the talk is about running Camel with GraalVM / Quarkus to archive native compiled binaries that has impressive startup and footprint.
Empowering developers to deploy their own data storesTomas Doran
Empowering developers to deploy their own data stores using Terrafom, Puppet and rage. A talk about automating server building and configuration for Elasticsearch clusters, using Hashicorp and puppet labs tool. Presented at Config Management Camp 2016 in Ghent
Spring Framework 4.0과 4.1의 주요 업데이트 내용을 정리했습니다. 그리고, Spring I/O 2015 | The conference 에서 공개된 4.2의 소식들을 모아보았습니다. :)
ps. 제가 웬만하면 자료는 한글로 바꾸는 편인데... 이번에는 시간이 너무 부족해서 대부분 영어를 그대로 사용했네요. 다음에 기회가 되면 한글화시키는걸로...
2 hour session where I cover what is Apache Camel, latest news on the upcoming Camel v3, and then the main topic of the talk is the new Camel K sub-project for running integrations natively on the cloud with kubernetes. The last part of the talk is about running Camel with GraalVM / Quarkus to archive native compiled binaries that has impressive startup and footprint.
Apache Camel v3, Camel K and Camel QuarkusClaus Ibsen
In this session, we will explore key challenges with function interactions and coordination, addressing these problems using Enterprise Integration Patterns (EIP) and modern approaches with the latest innovations from the Apache Camel community:
Apache Camel is the Swiss army knife of integration, and the most powerful integration framework. In this session you will hear about the latest features in the brand new 3rd generation.
Camel K, is a lightweight integration platform that enables Enterprise Integration Patterns to be used natively on any Kubernetes cluster. When used in combination with Knative, a framework that adds serverless building blocks to Kubernetes, and the subatomic execution environment of Quarkus, Camel K can mix serverless features such as auto-scaling, scaling to zero, and event-based communication with the outstanding integration capabilities of Apache Camel.
- Apache Camel 3
- Camel K
- Camel Quarkus
We will show how Camel K works. We’ll also use examples to demonstrate how Camel K makes it easier to connect to cloud services or enterprise applications using some of the 300 components that Camel provides.
This presentation explains how to deploy and use the Integrated Caching feature on Netscaler. I gave this presentation to Citrix staff, customers and partners in worldwide in 2011. The presentation covers best practices and gotchas :) Integrated Caching is an excellent feature that can greatly improve the performance of your website.
Fighting Against Chaotically Separated Values with EmbulkSadayuki Furuhashi
We created a plugin-based data collection tool that can read any chaotically formatted files called "CSV" by guessing its schema automatically
Talked at csv,conf,v2 in Berlin
http://csvconf.com/
Azure Cosmos DB is Microsoft’s globally distributed, multi-model database service for operational and analytics workloads. It offers a multi-mastering feature by automatically scaling throughput, compute, and storage. You can elastically scale throughput and storage, and take advantage of fast, single-digit-millisecond data access using your favorite API including SQL Core(SQL API), MongoDB, Cassandra, Tables, or Gremlin. Cosmos DB provides comprehensive service level agreements (SLAs) for throughput, latency, availability, and several consistencies.
Degrading Performance? You Might be Suffering From the Small Files SyndromeDatabricks
No matter if your data pipelines are handling real-time event-driven streams, near-real-time streams, or batch processing jobs. When you work with a massive amount of data made out of small files, specifically parquet, your system performance will degrade.
A small file is one that is significantly smaller than the storage block size. Yes, even with object stores such as Amazon S3, Azure Blob, etc., there is minimum block size. Having a significantly smaller object file can result in wasted space on the disk since the storage is optimized to support fast read and write for minimal block size.
To understand why this happens, you need first to understand how cloud storage works with the Apache Spark engine. In this session, you will learn about Parquet, the Storage API calls, how they work together, why small files are a problem, and how you can leverage DeltaLake for a more straightforward, cleaner solution.
Asynchronous Web Programming with HTML5 WebSockets and JavaJames Falkner
(Talk originally given @ KCDC - http://kcdc.info ).
Over the last decade, advances in web computing have removed many of the barriers to entry for developers. New languages, frameworks, and development methodologies have kickstarted new ideas and new ways to develop web applications to make modern life easier and more efficient. WebSockets (introduced as part of HTML5) is one such technology that enables a new class of scalable, super-responsive, collaborative, and real-time web applications with a wide range of uses.
In this talk, we will first cover the basics of asynchronous web programming using WebSockets, including predecessors such as polling and long-polling, applications of WebSockets, its limitations and potential bottlenecks, and potential future improvements.
Next, we will demo and dissect a real-world use case for realtime social data analytics, using the Apache Tomcat implementation of WebSockets and the Java-based Liferay Portal Server. This will include a discussion about development of WebSocket endpoints, its lifecycle within the application container and browser, debugging WebSockets, and scalability topics.
Databases are like languages: it’s very useful to know more than one. NoSQL databases promise better performance, scaling, lower cost of ownership, and flexibility for many use cases. With recent advances in NoSQL including ACID transactions, SQL queries, scopes, collections, and more, making the jump to NoSQL is becoming more straightforward. In this session, you will learn how to automatically migrate a relational database (including tables, data, indexes, users, and even queries) over to a modern NoSQL database.
The three steps that will be covered include:
1. Lifting your legacy data and data structure into a modern database.
2. Shifting your legacy application and clients to use NoSQL
3. Refactoring your legacy data model to improve performance and efficiency
After this short session, you’ll have taken a huge leap to learning a new technology and providing benefits to your team and organization, including the ability to:
- Develop faster with SQL for JSON queries (N1QL), plus multi-modal key-value, full text search, and analytics capabilities
- Deploy everywhere from edge to cloud, wherever and however you want
- Perform optimally at scale with a built-in memory-first architecture for sub-millisecond operations
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaGuido Schmutz
Apache Kafka is a popular distributed streaming data platform. A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. Data sources flowing into Kafka are often native data streams such as social media streams, telemetry data, financial transactions and many others. But these data stream only contain part of the information. A lot of data necessary in stream processing is stored in traditional systems backed by relational databases. To implement new and modern, real-time solutions, an up-to-date view of that information is needed. So how do we make sure that information can flow between the RDBMS and Kafka, so that changes are available in Kafka as soon as possible in near-real-time? This session will present different approaches for integrating relational databases with Kafka, such as Kafka Connect, Oracle GoldenGate and bridging Kafka with Oracle Advanced Queuing (AQ).
Since late 2009 there is Spring 3 published. Some things are new, something keep and something was removed.
Thos talk discuss the changes of the 3rd edition of Spring and introduce Spring Roo, Grails and the SpringSource Toolsuite.
An Approach to Detecting Writing Styles Based on Clustering Techniquesambekarshweta25
An Approach to Detecting Writing Styles Based on Clustering Techniques
Authors:
-Devkinandan Jagtap
-Shweta Ambekar
-Harshit Singh
-Nakul Sharma (Assistant Professor)
Institution:
VIIT Pune, India
Abstract:
This paper proposes a system to differentiate between human-generated and AI-generated texts using stylometric analysis. The system analyzes text files and classifies writing styles by employing various clustering algorithms, such as k-means, k-means++, hierarchical, and DBSCAN. The effectiveness of these algorithms is measured using silhouette scores. The system successfully identifies distinct writing styles within documents, demonstrating its potential for plagiarism detection.
Introduction:
Stylometry, the study of linguistic and structural features in texts, is used for tasks like plagiarism detection, genre separation, and author verification. This paper leverages stylometric analysis to identify different writing styles and improve plagiarism detection methods.
Methodology:
The system includes data collection, preprocessing, feature extraction, dimensional reduction, machine learning models for clustering, and performance comparison using silhouette scores. Feature extraction focuses on lexical features, vocabulary richness, and readability scores. The study uses a small dataset of texts from various authors and employs algorithms like k-means, k-means++, hierarchical clustering, and DBSCAN for clustering.
Results:
Experiments show that the system effectively identifies writing styles, with silhouette scores indicating reasonable to strong clustering when k=2. As the number of clusters increases, the silhouette scores decrease, indicating a drop in accuracy. K-means and k-means++ perform similarly, while hierarchical clustering is less optimized.
Conclusion and Future Work:
The system works well for distinguishing writing styles with two clusters but becomes less accurate as the number of clusters increases. Future research could focus on adding more parameters and optimizing the methodology to improve accuracy with higher cluster values. This system can enhance existing plagiarism detection tools, especially in academic settings.
Hierarchical Digital Twin of a Naval Power SystemKerry Sado
A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesChristina Lin
Traditionally, dealing with real-time data pipelines has involved significant overhead, even for straightforward tasks like data transformation or masking. However, in this talk, we’ll venture into the dynamic realm of WebAssembly (WASM) and discover how it can revolutionize the creation of stateless streaming pipelines within a Kafka (Redpanda) broker. These pipelines are adept at managing low-latency, high-data-volume scenarios.
We have compiled the most important slides from each speaker's presentation. This year’s compilation, available for free, captures the key insights and contributions shared during the DfMAy 2024 conference.
Using recycled concrete aggregates (RCA) for pavements is crucial to achieving sustainability. Implementing RCA for new pavement can minimize carbon footprint, conserve natural resources, reduce harmful emissions, and lower life cycle costs. Compared to natural aggregate (NA), RCA pavement has fewer comprehensive studies and sustainability assessments.
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressionsVictor Morales
K8sGPT is a tool that analyzes and diagnoses Kubernetes clusters. This presentation was used to share the requirements and dependencies to deploy K8sGPT in a local environment.
Water billing management system project report.pdfKamal Acharya
Our project entitled “Water Billing Management System” aims is to generate Water bill with all the charges and penalty. Manual system that is employed is extremely laborious and quite inadequate. It only makes the process more difficult and hard.
The aim of our project is to develop a system that is meant to partially computerize the work performed in the Water Board like generating monthly Water bill, record of consuming unit of water, store record of the customer and previous unpaid record.
We used HTML/PHP as front end and MYSQL as back end for developing our project. HTML is primarily a visual design environment. We can create a android application by designing the form and that make up the user interface. Adding android application code to the form and the objects such as buttons and text boxes on them and adding any required support code in additional modular.
MySQL is free open source database that facilitates the effective management of the databases by connecting them to the software. It is a stable ,reliable and the powerful solution with the advanced features and advantages which are as follows: Data Security.MySQL is free open source database that facilitates the effective management of the databases by connecting them to the software.
16. “그런데 JDBC 는 blocking IO
라며?”
“왜 비동기 프로그래밍을 할까?”“하지만 많은 기능을 구현할 수
있습니다.”
17.
18. “휴지통에 있는 메일을 삭제하
는 API”
- 데이터 베이스에 record 삭제
- 메일 파일(Mime) 삭제
- 타 시스템에 메일이 삭제 되었
음을 알려줌.
검색서버, History 서버
19. 그런데…
휴지통에 1만건이 존재한다면?
- DB 작업 : 300ms- 파일 삭제 작업 : 10ms *
10,000 = 100sec
- 타시스템 연동작업 : 30ms *
10,000 * 3 = 900 sec
Total : 약 1000 sec, 16분
20. @Async
• @Async 를 쓸때는 별도의 ThreadPool 을 사용하자.
• @Async 와 @Transactional 을 쓸때는 주의하자.
– @Transactional 의 원리를 이해하고 사용해야 함.
– 부하
21. @Transactional
• AOP-Proxy 에서는 public method 로 선언
– @EnableTransactionManagement(proxyTargetClass = true)
– <tx:annotation-driven/>
– AspectJ 에서는 상관없음.
• Bean 내부의 다른 method에서 다른 Transaction을 실행할때
– 선언적 Transaction 을 실행한다.
– Self-autowiring in Spring 4.3+
– @Async 에서 많이 놓치는 실수.
34. @Async
@Transactional(readOnly = false)
public void removeByAsync(List<Long> submailIds){
mailRepository.deleteByMailIdIn(subMailIds)
subMailIds.stream.forEach(mailId -> {
storageComponent.deleteByMailId(mailId);
searchComponent.deleteByMailId(mailId);
historyComponent.deleteByMailId(mailId);
);
}
DB Transaction
이 묶일 필요가
있나요?
35. @Transactional(readOnly = false)
public void removeByAsync(List<Long> submailIds){
mailRepository.deleteByMailIdIn(subMailIds)
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronizationAdapter() {
@Override
public void afterCommit() {
subMailIds.stream.forEach(mailId -> {
storageComponent.deleteByMailId(mailId);
searchComponent.deleteByMailId(mailId);
historyComponent.deleteByMailId(mailId);
});
}
}
}
afterCommit() – invoke after transaction commit
afterCompletion() - invoke after transaction commit / rollback
beforeCommit() – invoke before trasnaction commit
beforeCompletion() - invoke before transaction commit / rollback
36. “WAS 스레드, DBPool 갯수 설
정이야기 해볼까요?”
Peek Time 에 비동기로 프로그래
밍된 api 를 호출하면?
DBPool 은 괜찬은가요?
“DataSource 를 분리합니다”
37. DataSource Configuration
• 비동기용와 API 처리용 DataSource를 분리한다.
– Peek time 때, 비동기 프로세스는 늦게 처리되도 된다.
– 장애는 전파하지 말자.
• AbstractRoutingDataSource class
– 일종의 Proxy DataSource
– setTargetDataSources method - key 에 따른 DataSource 등록
– determineCurrentLookupKey method – key 에 따른 DataSource 획득
38. @Slf4j
public class ExecutionRoutingDataSource extends AbstractRoutingDataSource {
@Override
protected Object determineCurrentLookupKey() {
return ExecutionContextHolder.getExecutionType();
}
}
39. public class ExecutionRoutingDataSource extends AbstractRoutingDataSource {
@Override
protected Object determineCurrentLookupKey() {
return ExecutionContextHolder.getExecutionType();
}
}
@Bean("routingDataSource")
public ExecutionRoutingDataSource routingDataSource() {
Map<Object, Object> targetDataSources = new HashMap<>();
targetDataSources.put(ExecutionType.API, apiDataSource());
targetDataSources.put(ExecutionType.BATCH, batchDataSource());
ExecutionRoutingDataSource routingDataSource = new ExecutionRoutingDataSource();
routingDataSource.setTargetDataSources(targetDataSources);
routingDataSource.setDefaultTargetDataSource(apiDataSource());
return routingDataSource;
}
LookUpKey 에 따라 동적으로
DataSource 를 사용
미리 등록된 enum
ExecutionType.API or BATCH
40. @TransactionalExecutionType(
executionType = ExecutionType.BATCH
)
@Transactional(
readOnly = true,
isolation = Isolation.READ_COMMITTED
)
public List<CompletableFuture<List<Long>>> partition(Long taskQueueId) {
// Business logic and query to DB
}
@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface TransactionalExecutionType {
ExecutionType executionType() default ExecutionType.API;
}
42. AsyncRestTemplate 은
RestTemplate 과 같다.
Spring 의 PSA
(Portable Service Abstraction)
Sample : AsyncRestTemplateTest.java
getWords(), asyncGetWords(), asyncCombinedProcess()
44. public class CompletableFutureBuilder {
public static <T> CompletableFuture<T> build(ListenableFuture<T> listenableFuture) {
CompletableFuture<T> completableFuture = new CompletableFuture<T>() {
@Override
public boolean cancel(boolean mayInterruptIfRunning) {
boolean result = listenableFuture.cancel(mayInterruptIfRunning);
super.cancel(mayInterruptIfRunning);
return result;
}
};
listenableFuture.addCallback(new ListenableFutureCallback<T>() {
@Override
public void onFailure(Throwable ex) {
completableFuture.completeExceptionally(ex);
}
@Override
public void onSuccess(T result) {
completableFuture.complete(result);
}
});
return completableFuture;
}
45. public class CompletableFutureBuilder {
public static <T> CompletableFuture<T> build(ListenableFuture<T> listenableFuture) {
CompletableFuture<T> completableFuture = new CompletableFuture<T>() {
@Override
public boolean cancel(boolean mayInterruptIfRunning) {
boolean result = listenableFuture.cancel(mayInterruptIfRunning);
super.cancel(mayInterruptIfRunning);
return result;
}
};
listenableFuture.addCallback(new ListenableFutureCallback<T>() {
@Override
public void onFailure(Throwable ex) {
completableFuture.completeExceptionally(ex);
}
@Override
public void onSuccess(T result) {
completableFuture.complete(result);
}
});
return completableFuture;
}
46. public class CompletableFutureBuilder {
public static <T> CompletableFuture<T> build(ListenableFuture<T> listenableFuture) {
CompletableFuture<T> completableFuture = new CompletableFuture<T>() {
@Override
public boolean cancel(boolean mayInterruptIfRunning) {
boolean result = listenableFuture.cancel(mayInterruptIfRunning);
super.cancel(mayInterruptIfRunning);
return result;
}
};
listenableFuture.addCallback(new ListenableFutureCallback<T>() {
@Override
public void onFailure(Throwable ex) {
completableFuture.completeExceptionally(ex);
}
@Override
public void onSuccess(T result) {
completableFuture.complete(result);
}
});
return completableFuture;
}
47. CompletableFuture
• 비동기 프로그래밍을 위한 클래스
– NonBlocking 으로 처리 가능하다. (vs Future class)
– Blocking 으로도 처리 가능하다.
– 여러개의 작업들을 엮을 수 있다.
– 여러개의 작업들을 합칠 수 있다.
– 예외 사항을 처리할 수 있다.
• Implements CompletionStage<T>
– 테스크를 포함하는 모델을 의미한다.
– 테스크는 체인패턴의 기본 요소이다.
48. CompletableFuture
• 접미사를 확인
– thenApply() vs thenApplyAsync()
• 파라미터를 확인
– thenApplyAsync(Function<? super T, ? extends U> fn)
– thenApplyAsync(Function<? super T, ? extends U> fn, Executor executor)
• 비동기 작업 시작 method
– runAsync(), supplyAsync()
• 작업들을 연결하는 method
– thenApplyAsync(), thenComposeAsync(), anyOf(), allOf()
• 예외처리를 하는 method
– exceptionally()
49. CompletableFuture
Method AsyncMethod Arguments Returns
thenAccept thenAcceptAsync 이전 Stage 의 결과 Noting
thenRun thenRunAsync Noting
thenApply thenApplyAsync 이전 Stage 의 결과 Result of current stage
thenCompose thenComposeAsync 이전 Stage 의 결과 Future result of current
stage
thenCombine thenCombineAsync 이전 두 Stage 의 결과 Result of current stage
whenComplete whenCompleteAsync 이전 두 Stage 의 결과 or Exception Noting
JoinForkPool 같이 Managed 되지 않는 것 보다는 Thread Pool 을 이용하여 관리하라.
JoinForkPool 이 나쁘다는것은 아니다.
TX 전체 시간은 DB Query time + 각각의 컴포넌트들을 삭제하는 시간.
이중에 필요한 시간은??
TX 가 길어지면 뭐가 문제가 될까?
-> DB Pool 고갈
-> Lock wait
-> Timeout
이상황에서 API 는 멀티 스레드이므로 계속해서 부가가 들어오고 장애가 타 시스템으로 전파될 가능성이 높다
@Async 를 이용해서 TX를 짧게 쪼개는 방법.
장점 - Long TX 를 방지 할수 있음
단점 – exception 이 발생하면 일부는 삭제되고 일부는 남아있음
@Async 를 이용해서 TX를 짧게 쪼개는 방법.
단점 – 댕글링 파일이나 정보가 남을 수 있음
장점 – TX 실패시 Async 로 안넘어갈수 있음.
스토리지와 검색과 히스토리는 어디로?
그런데 Exception 이 발생하면?
TransactionSynchronizationAdaptor 인터페이스
DataSource 의 한 일종이다.
determineCurrentLookupKey 에 의해서 runtime 에 어떤 데이터소스를 사용할지 결정된다.
Bean 으로 등록해서 사용한다