Autoscaling of workloads in the Kubernetes environment. A slidedeck about Pod and Node autoscaling and the machinery behind it that makes it happen. Few recommendations for Pod and Node autoscaling while implementing it.
Exactly-once Stream Processing with Kafka StreamsGuozhang Wang
I will present the recent additions to Kafka to achieve exactly-once semantics (0.11.0) within its Streams API for stream processing use cases. This is achieved by leveraging the underlying idempotent and transactional client features. The main focus will be the specific semantics that Kafka distributed transactions enable in Streams and the underlying mechanics to let Streams scale efficiently.
Kafka High Availability in multi data center setup with floating Observers wi...HostedbyConfluent
Enabling High Availability in cluster setup that spawns different data centers is challenging but it is even more if we are using just two data-centers. Not ideal for Kafka HA at all. But this is reality for most organizations as they are using the same data-centers previously used for database HA.
In this presentation we will see how to use Kafka Observer feature to address this challenge with additional tweak to distribute load evenly among Observers and ordinary Brokers and make them floating between data-centers. The whole demo is supported by Infrastructure as a code automation trough Ansible.
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다.
다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다.
[Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안]
Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 …
[Apache Kafka 모니터링을 위한 Metrics 이해]
Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다.
이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다.
모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임
[Apache Kafka 성능 Configuration 최적화]
성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다.
튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.
Autoscaling of workloads in the Kubernetes environment. A slidedeck about Pod and Node autoscaling and the machinery behind it that makes it happen. Few recommendations for Pod and Node autoscaling while implementing it.
Exactly-once Stream Processing with Kafka StreamsGuozhang Wang
I will present the recent additions to Kafka to achieve exactly-once semantics (0.11.0) within its Streams API for stream processing use cases. This is achieved by leveraging the underlying idempotent and transactional client features. The main focus will be the specific semantics that Kafka distributed transactions enable in Streams and the underlying mechanics to let Streams scale efficiently.
Kafka High Availability in multi data center setup with floating Observers wi...HostedbyConfluent
Enabling High Availability in cluster setup that spawns different data centers is challenging but it is even more if we are using just two data-centers. Not ideal for Kafka HA at all. But this is reality for most organizations as they are using the same data-centers previously used for database HA.
In this presentation we will see how to use Kafka Observer feature to address this challenge with additional tweak to distribute load evenly among Observers and ordinary Brokers and make them floating between data-centers. The whole demo is supported by Infrastructure as a code automation trough Ansible.
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다.
다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다.
[Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안]
Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 …
[Apache Kafka 모니터링을 위한 Metrics 이해]
Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다.
이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다.
모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임
[Apache Kafka 성능 Configuration 최적화]
성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다.
튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.
Building Streaming Data Applications Using Apache KafkaSlim Baltagi
Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing.
In this talk you will learn more about:
1. A quick introduction to Kafka Core, Kafka Connect and Kafka Streams: What is and why?
2. Code and step-by-step instructions to build an end-to-end streaming data application using Apache Kafka
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
Decentralized finance with crypto and NFTs is a huge topic these days. It becomes a powerful combination with the coming metaverse platforms across industries. This session explores the relationship between crypto technologies and modern enterprise architecture.
I discuss how data streaming and Apache Kafka help build innovation and scalable real-time applications of a future metaverse. Let's skip the buzz (and NFT bubble) and instead review existing real-world deployments in the crypto and blockchain world powered by Kafka and its ecosystem.
Kafka Connect & Streams - the ecosystem around KafkaGuido Schmutz
After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
An interactive hands-on introduction of autoscaling in Kubernetes featuring Cluster Autoscaling, Horizontal Pod Autoscaling (HPA) with resource metrics, custom (object) metrics and external metrics and Vertical Pod Autoscaling (VPA).
Featuring a demo application showcasing the different scaling approaches for stateful and stateless applications.
Producer Performance Tuning for Apache KafkaJiangjie Qin
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Watch this talk here: https://www.confluent.io/online-talks/how-apache-kafka-works-on-demand
Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.
We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.
This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
Getting up to speed with Kafka Connect: from the basics to the latest feature...HostedbyConfluent
"Kafka Connect is an ideal tool for building data pipelines. It is both reliable and scalable, with a pluggable interface that lets you flow data between Kafka and any system you need. A Connect pipeline is made up of many different components, and understanding how each of these interact together is essential, even for the simplest setup.
In this talk we will introduce the Connect components, from connectors, to transformations to the runtime itself. We will also share some of the new capabilities and best practices that you should be aware of to help you run and manage connectors effectively.
Finally we will talk about some different open source projects that have been built on top of Connect that can help you get the most out of the framework."
In the last few years, Apache Kafka has been used extensively in enterprises for real-time data collecting, delivering, and processing. In this presentation, Jun Rao, Co-founder, Confluent, gives a deep dive on some of the key internals that help make Kafka popular.
- Companies like LinkedIn are now sending more than 1 trillion messages per day to Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
- Many companies (e.g., financial institutions) are now storing mission critical data in Kafka. Learn how Kafka supports high availability and durability through its built-in replication mechanism.
- One common use case of Kafka is for propagating updatable database records. Learn how a unique feature called compaction in Apache Kafka is designed to solve this kind of problem more naturally.
Increasingly, organizations are relying on Kafka for mission critical use-cases where high availability and fast recovery times are essential. In particular, enterprise operators need the ability to quickly migrate applications between clusters in order to maintain business continuity during outages. In many cases, out-of-order or missing records are entirely unacceptable. MirrorMaker is a popular tool for replicating topics between clusters, but it has proven inadequate for these enterprise multi-cluster environments. Here we present MirrorMaker 2.0, an upcoming all-new replication engine designed specifically to provide disaster recovery and high availability for Kafka. We describe various replication topologies and recovery strategies using MirrorMaker 2.0 and associated tooling.
Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019confluent
Cloud migration: it's practically a rite of passage for anyone who's built infrastructure on bare metal. When we migrated our 5-year-old Kafka deployment from the datacenter to GCP, we were faced with the task of making our highly mutable server infrastructure more cloud-friendly. This led to a surprising decision: we chose to run our Kafka cluster on Kubernetes. I'll share war stories from our Kafka migration journey, explain why we chose Kubernetes over arguably simpler options like GCP VMs, and present the lessons we learned while making our way toward a stable and self-healing Kubernetes deployment. I'll also go through some improvements in the more recent Kafka releases that make upgrades crucial for any Kafka deployment on immutable and ephemeral infrastructure. You'll learn what happens when you try to run one complex distributed system on top of another, and come away with some handy tricks for automating cloud cluster management, plus some migration pitfalls to avoid. And if you're not sure whether running Kafka on Kubernetes is right for you, our experiences should provide some extra data points that you can use as you make that decision.
Building Streaming Data Applications Using Apache KafkaSlim Baltagi
Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing.
In this talk you will learn more about:
1. A quick introduction to Kafka Core, Kafka Connect and Kafka Streams: What is and why?
2. Code and step-by-step instructions to build an end-to-end streaming data application using Apache Kafka
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
Decentralized finance with crypto and NFTs is a huge topic these days. It becomes a powerful combination with the coming metaverse platforms across industries. This session explores the relationship between crypto technologies and modern enterprise architecture.
I discuss how data streaming and Apache Kafka help build innovation and scalable real-time applications of a future metaverse. Let's skip the buzz (and NFT bubble) and instead review existing real-world deployments in the crypto and blockchain world powered by Kafka and its ecosystem.
Kafka Connect & Streams - the ecosystem around KafkaGuido Schmutz
After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL.
Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go.
Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.
An interactive hands-on introduction of autoscaling in Kubernetes featuring Cluster Autoscaling, Horizontal Pod Autoscaling (HPA) with resource metrics, custom (object) metrics and external metrics and Vertical Pod Autoscaling (VPA).
Featuring a demo application showcasing the different scaling approaches for stateful and stateless applications.
Producer Performance Tuning for Apache KafkaJiangjie Qin
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Watch this talk here: https://www.confluent.io/online-talks/how-apache-kafka-works-on-demand
Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.
We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.
This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
Getting up to speed with Kafka Connect: from the basics to the latest feature...HostedbyConfluent
"Kafka Connect is an ideal tool for building data pipelines. It is both reliable and scalable, with a pluggable interface that lets you flow data between Kafka and any system you need. A Connect pipeline is made up of many different components, and understanding how each of these interact together is essential, even for the simplest setup.
In this talk we will introduce the Connect components, from connectors, to transformations to the runtime itself. We will also share some of the new capabilities and best practices that you should be aware of to help you run and manage connectors effectively.
Finally we will talk about some different open source projects that have been built on top of Connect that can help you get the most out of the framework."
In the last few years, Apache Kafka has been used extensively in enterprises for real-time data collecting, delivering, and processing. In this presentation, Jun Rao, Co-founder, Confluent, gives a deep dive on some of the key internals that help make Kafka popular.
- Companies like LinkedIn are now sending more than 1 trillion messages per day to Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
- Many companies (e.g., financial institutions) are now storing mission critical data in Kafka. Learn how Kafka supports high availability and durability through its built-in replication mechanism.
- One common use case of Kafka is for propagating updatable database records. Learn how a unique feature called compaction in Apache Kafka is designed to solve this kind of problem more naturally.
Increasingly, organizations are relying on Kafka for mission critical use-cases where high availability and fast recovery times are essential. In particular, enterprise operators need the ability to quickly migrate applications between clusters in order to maintain business continuity during outages. In many cases, out-of-order or missing records are entirely unacceptable. MirrorMaker is a popular tool for replicating topics between clusters, but it has proven inadequate for these enterprise multi-cluster environments. Here we present MirrorMaker 2.0, an upcoming all-new replication engine designed specifically to provide disaster recovery and high availability for Kafka. We describe various replication topologies and recovery strategies using MirrorMaker 2.0 and associated tooling.
Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019confluent
Cloud migration: it's practically a rite of passage for anyone who's built infrastructure on bare metal. When we migrated our 5-year-old Kafka deployment from the datacenter to GCP, we were faced with the task of making our highly mutable server infrastructure more cloud-friendly. This led to a surprising decision: we chose to run our Kafka cluster on Kubernetes. I'll share war stories from our Kafka migration journey, explain why we chose Kubernetes over arguably simpler options like GCP VMs, and present the lessons we learned while making our way toward a stable and self-healing Kubernetes deployment. I'll also go through some improvements in the more recent Kafka releases that make upgrades crucial for any Kafka deployment on immutable and ephemeral infrastructure. You'll learn what happens when you try to run one complex distributed system on top of another, and come away with some handy tricks for automating cloud cluster management, plus some migration pitfalls to avoid. And if you're not sure whether running Kafka on Kubernetes is right for you, our experiences should provide some extra data points that you can use as you make that decision.
2015년 8월 29일 강남역 네이버 D2에서 열린 SOCC 6th Conference 에서 발표한 자료입니다.
첫 번째로는 많은 사람들이 언제 처음 공부를 시작했는지를 중요하게 생각하는데, 어떻게가 더 중요하다는 점을 강조하고자 했습니다.
두 번째로는 내가 어떻게 공부를 했었는지에 대한 방법을 소개했습니다.
마지막으로 신기한 방법으로 두 문제를 풀어봅니다.
https://www.acmicpc.net/problem/9457
https://www.acmicpc.net/problem/2329
2015. 09. 05 도커 서울 밋업 4번째(Open Container Korea 주최).
elasticsearch에 은전한닢 한국어 형태소 분석기를 적용하고 운영한 사례 발표.
- 사용자 사전별로 이미지를 만들기
- nginx를 이용해 http basic auth 적용하기
넥슨 글로벌 플랫폼 구축 이야기 : DB Migration case study (임현수 플랫폼인프라실 Technical Manager, 넥...Amazon Web Services Korea
넥슨 글로벌 플랫폼 구축 이야기 : DB Migration case study
이 세션에서는 넥슨의 Case study를 통하여 글로벌플랫폼 구축을 위해 기존 플랫폼을 AWS로 Migration하는 과정 및 발생가능한 이슈를 공유합니다. 넥슨이 DB서버를 이전하는 과정 속에서 마주한 기술적 고민과 이슈를 통하여 AWS 활용 시 고려해야 할 부분들에 대해 소개하고 함께 이야기 나누고자 합니다.
1. 웹서비스 성능 향상을 위한
오픈소스 Arcus 주요기능 및 활용사례
박준현 (junhyun.park@navercorp.com)
시스템스컴퓨팅G / NAVER LABS
2014-06-27
2. 2
NAVER에서 개발한 Memory Cache Cloud
Memcached 기반의 extended key-value 모델 (collection 지원)
ZooKeeper 기반의 elastic cache cloud 구현
NAVER 서비스들의 여러 요구 사항들 반영
Open Source SW – Apache License 2.0
ARCUS [ɑ́ :rkəs] : 아커스
아치형 구름
http://en.wikipedia.org/wiki/Arcus_cloud
ARCUS
3. 3
ARCUS 개발 배경
Service Applications
Write
Database
Read
Large-scale 웹 서비스
데이터 및 요청 양 증가
성능 이슈
Throughput
Response Time
DB Scalability 이슈
4. 4
ARCUS 개발 배경 – Memory Cache Cloud
Service Applications
1) DB Write
Database
Arcus Cache Cloud
2) DB Read
1) Arcus
Get
3) Arcus
Set
2) Arcus Delete
성능 해결
Low Latency
High Throughput
DB 부하 경감
5. 5
ARCUS 기반 기술 - memcached
http://www.memcached.org/
In-memory key-value store
get, set, add, incr/decr, CAS, delete, …
operation cost - O(1)
High-performance (single cache node)
100K ~ 200K requests / second
Avg. 400 ~ 500 micro second
6. 6
ARCUS 기반 기술 - ZooKeeper
http://zookeeper.apache.org/
Coordination service for distributed applications.
Service discovery
Dynamic configuration management
Distributed locking
For managing Arcus cache clouds
Cache node list management
Cache node failure detection & notification
8. 8
Service Code (or Cloud Name)
Arcus Cloud(or Cluster)를 유일하게 구분하는 식별자
Arcus 클라이언트는 service code로 특정 cloud를 접근
ARCUS Cloud 관리 – Service Code
Arcus
Cloud A
Arcus
Cloud B
Arcus
Cloud C
Servicecode = “ServiceA”
Servicecode = “ServiceB”
9. 9
ARCUS Cloud 관리 – Server의 ZK Node 등록
/cache_server_mapping
/cache_list
/arcus
/10.0.0.1:11211
/10.0.0.2:11211
/blog
/naverme
/blog
/naverme
/10.0.0.1:11211
/10.0.0.3:11211
1. Arcus 서버의 IP:PORT를 사용하여 자
싞이 서비스되어야 하는 코드를 조회한다.
예) “10.0.0.1:11211”
2. 서비스 코드아래 자싞의 IP:PORT를
ephemeral node로 생성
Arcus server
Cloud에 참여하는 서버 정보 관리를 위한 ZooKeeper의 znode구조
ephemeral node : zookeeper 와의 세션이 유효한 동안에만 존재하는 node
10. 10
ARCUS Cloud 관리 – Client의 ZK Node 조회
/cache_list
/arcus
/blog
/10.0.0.1:11211
ZooKeeper 서버에 저장된 znode의 구조
3. 자싞의 서비스 코드의 변경사
항을 감시 한다. 자식 znode에
변경사항이 생기면 변경사항을
통지 받아 서버 목록을 갱싞한다.
2. Arcus server로 접속을 시도한다.
1. 서비스코드에 등록된 Arcus
서버 목록을 조회한다.
/10.0.0.2:11211
Arcus client가 cache cloud에서 사용 가능한 server를 찾는 방법
사용자, 서비스
(Arcus client)
11. 11
ARCUS 데이터 분산
Key-value item을 cloud의 어떤 cache node로 분산할 것인지
Key-to-Node Mapping
Arcus client
key1=AA
key2=BB
key3=CC
12. 12
ARCUS 데이터 분산 – consistent hashing
Hash Ring (2^32)
(Hash결과 값으로 정렬된 구조로
Arcus Client 내부에서 관리된다.)
1000
…
1010
…
1011
…
key hash
1110…
1. Arcus cache node들의 hash값을 모두
구한다.
2. key의 hash값을 구한다.
3. Key hash값을 기준으로 시계 방향으로
가장 처음 만나는 node hash값이 가리
키는 Arcus cache node를 선택한다.
13. 13
ARCUS 데이터 분산 – cache node 분산
Hash 값에 따라서 Arcus cache
node 별로 담당해야 하는
key 개수의 차이가 크다.
요청이 서버에 골고루
분산되지 않는다.
1000
…
1010
…
1011
…
Cache node 분산 이슈 (1 hash point / cache node)
14. 14
ARCUS 데이터 분산 – cache node 분산
Cache node 분산 – 약 160개 hash points / 1 cache node
동일한 Arcus cache node를 가리
키는 hash point를 여러 개 만들어
요청이 골고루 분산될 수 있도록
한다.
요청이 골고루 분산되면 데이터 또
한 고르게 분산되어 저장된다.
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
…
15. 15
Key – key-value item 식별자
Format: <prefix>:<subkey> (max 250 characters)
<prefix> 단위로 key들을 그룹화하여 관리 (ex, delete, stats)
<subkey>로 prefix 내의 특정 item을 식별
Value – key-value item에서 value 부분
Single value (max 1MB)
A collection of values : List, Set, B+Tree
max 50000 elements
max 4KB value in each element
ARCUS Key-Value 모델
16. 16
ARCUS Collections
유형 구조 및 특징
List Double linked list 구조
Set An unordered set of unique data
Membership checking (예, 친구 정보, 구독 정보, …)
B+Tree An ordered data set based on b+tree key
Element 구조: < bkey, [eflag,] value >
bkey(b+tree key): 8 bytes integer or 1~31 bytes array
eflag(element flag): 1~31 bytes array
주요 연산
Bkey 기반 range scan & eflag filter & offset, count
여러 b+tree들에 대한 smget(sort-merge get)
B+tree position 조회 / position 기반 element 조회
17. 17
대표 성능 이슈 – 친구 글 또는 구독 글 모아보기
IN 리스트 커질수록, 질의 응답이 급격히 느림.
SNS 사용자 증가 및 활성화 => DB 성능 튜닝만으롞 해결이 어려움
관계(relationship) 증가
데이터 규모와 조회 요청 량 증가
ARCUS Collection - SNS
SELECT *
FROM posts
WHERE user IN (friend-1, friend-2, … friend-N)
AND create_time < sysdate()
ORDER BY create_time DESC
LIMIT 20;
18. 18
Arcus B+Tree - <bkey: create_time, value: postid>
ARCUS Collection - SNS
Push 방식 Pull 방식
Cache • 사용자 별 inbox cache 유지 • 사용자 별 outbox cache 유지
Post
작성
• 모든 친구의 inbox cache에
post 정보를 push delivery
• 친구 수만큼 write 발생
• 작성자의 outbox cache에만
post 정보를 보관
• 1회 write 발생
Post
모아보기
• 자싞의 inbox cache만 조회
• B+tree get 연산 사용
• 모든 친구의 outbox cache를
조회하여 sort-merge 수행
• B+tree smget 연산 사용
19. 19
B+tree smget(sort-merge get) 연산
ARCUS Collection - SNS
0 7 14 21 28
A
1 5 10 15 20
B
3 6 9 12 16
C
A, B, C로 부터 bkey가 30 ~ 10인 element 조회
[ 28, 21, 20, 16, 15, 14, 12, 10 ]
10 1214 15 1620
20. 20
대표 조회 요청
사용자의 score에 대한 순위 조회하고,
그 score 보다 앞뒤에 있는 N개 score 및 user 조회
B+tree – top game scores 저장
<bkey: score, value: userinfo>
B+tree position 관련 연산들
Find position: <key, bkey, order> => position
Get by position: <key, position_range, order> => elements
Find position with get: <key, bkey, order, count> => <position, elements>
주어진 bkey의 position 조회하고, 그 position의 element 포함하여 양방
향(forward & backward)으로 count개 element 조회 (release 예정 기능)
ARCUS Collection– Game Ranking
23. 23
ARCUS – Eager Item Expiration
기존 Item Expiration
Search 범위
각 LRU 리스트의 tail에 위치한 N개 items만 검사
이슈
LRU 상위에 expired items이 존재하더라도 reclaim 되지 않고, LRU tail에
있는 valid item이 먼저 evict 됨.
Valid items의 memory 사용량 확인이 어려움
Eager Item Expiration
Search 범위
각 LRU 리스트의 tail에 위치한 N개 items 검사 외에도
LRU 리스트를 점진적 traverse하면서 N개 items 검사
효과 - Memory 효율성 증가
24. 24
Dynamic configuration 변경
memlimit, maxconns, …
다양한 Item attributes 제공
Item attributes 조회 및 변경
Delete by prefix
Command pipelining for bulk operations
Sticky Items
stats 정보 확장
…
ARCUS – 그 외 특징들
25. 25
ARCUS Client 초기화
SERVICE_CODE의
Server list 조회
Arcus Java Client
Arcus admin
ZK watcher를 통해서
Server list 변경 감지
Arcus server
Arcus server
Arcus server
Locator, Connector
ZooKeeper Client
ConnectionFactoryBuilder cfb = new ConnectionFactoryBuilder();
ArcusClient client = ArcusClient.createArcusClient(ADMIN_ADDR, SERVICE_CODE, cfb);
26. 26
ARCUS Client Pool 초기화
ArcusClientPool
Arcus admin
Arcus server
Arcus server
Arcus server
Arcus Java Client
ZooKeeper Client
Arcus Java Client
internal pool
ConnectionFactoryBuilder cfb = new ConnectionFactoryBuilder();
ArcusClientPool pool = ArcusClient.createArcusClientPool(ADMIN_ADDR, SERVICE_CODE, cfb, 2);
적당 pool size ?? (응용 서버의 peak 요구 성능(rps) – 1) / single client 처리량(rps) + 1
27. 27
ARCUS Client Pool 사용
Arcus Java Client Pool
Arcus Java Client
Arcus Java Client
ConnectionFactoryBuilder cfb = new ConnectionFactoryBuilder();
ArcusClientPool pool = ArcusClient.createArcusClientPool(ADMIN_ADDR, SERVICE_CODE, cfb, 2);
// pool의 API를 직접 호출함, Pool로부터 client를 얻은 다음 API를 호출하는 방식이 아님.
pool.set(KEY, EXPIRE_TIME, VALUE);
사용자
(서비스)
Set
API
client pool의 API는 pool이 가지고 있는 임의의 client로 명령을 젂달한다.
28. 28
Arcus Client 설정
Connection Factory Builder 설정들
Transcoder, Key-Value 압축, Key-Value Local Cache (ehcache) 사용, …
Key-Value Local Cache 사용 설정
ConnectionFactoryBuilder cfb = new ConnectionFactoryBuilder();
// local cache의 expire time을 1초로 지정
cfb.setFrontCacheExpireTime(1000L);
// local cache에 저장 가능한 key의 개수를 1만개로 지정
cfb.setMaxFrontCacheElements(10000);
// 이제 client를 사용해서 저장/조회되는 key/value는 local cache를 거치게 된다.
ArcusClient client = ArcusClient.createArcusClient(ADMIN_ADDR, SERVICE_CODE, cfb);
// Key/Value local cache는 Arcus client / Arcus client pool별로 생성된다.
29. 29
Arcus Client Local Cache
Request
Arcus get
DB query
Arcus set
(DB query result)
Response
cache hit
cache miss
ehcache Get
ehcache miss or expired
ehcache
set /extend
ehcache hit
optional
skip ehcache set
Key-value item을 local cache(ehcache)에 저장하여 응답속도를 높일 수 있다.
(Local caching은 Key-Value 유형만 지원하며, collection 유형은 불가)
30. 30
Arcus Open – 2014/05/15
http://naver.github.io/arcus/ (Apache License 2.0)
Linux : CentOS 6.x 64bit, Ubuntu 12.04 LTS 64bit
Arcus Code Repositories
arcus : arcus super project including setup tools
arcus-memcached : arcus memory cache server
arcus-java-client : arcus java client
arcus-c-client : arcus c client
arcus-zookeeper : zookeeper with arcus modifications
arcus-hubble : dashboard service to monitor arcus clusters
…
ARCUS Open Source
33. 33
문의/질문 - openarcus 구글 그룹스
https://groups.google.com/d/forum/openarcus
openarcus@googlegroups.com
공지 – openarcus 구글 그룹스 (회원가입 필요)
버그/이슈
개별 code repository issues에 등록
Contribution
Github pull request 방식 선호
아주 갂단하면, patch 젂달 방식도 가능
ARCUS Open Source - Communication
34. 34
Replication
2 Copy & Semi-Synchronous
Others
지원하는 Linux 플랫폼 확장
지원하는 Client 언어 확장
Slow Request Logging => 응용 Request 검증에 활용
Prefix 기능 확장
Arcus Monitoring 기능 확장
…
ARCUS Open Source – Future Dev