This document discusses messaging queues and platforms. It begins with an introduction to messaging queues and their core components. It then provides a table comparing 8 popular open source messaging platforms: Apache Kafka, ActiveMQ, RabbitMQ, NATS, NSQ, Redis, ZeroMQ, and Nanomsg. The document discusses using Apache Kafka for streaming and integration with Google Pub/Sub, Dataflow, and BigQuery. It also covers benchmark testing of these platforms, comparing throughput and latency. Finally, it emphasizes that messaging queues can help applications by allowing producers and consumers to communicate asynchronously.
This document discusses messaging queues and platforms. It begins with an introduction to messaging queues and their core components. It then provides a table comparing 8 popular open source messaging platforms: Apache Kafka, ActiveMQ, RabbitMQ, NATS, NSQ, Redis, ZeroMQ, and Nanomsg. The document discusses using Apache Kafka for streaming and integration with Google Pub/Sub, Dataflow, and BigQuery. It also covers benchmark testing of these platforms, comparing throughput and latency. Finally, it emphasizes that messaging queues can help applications by allowing producers and consumers to communicate asynchronously.
This document discusses exactly once semantics in Apache Kafka 0.11. It provides an overview of how Kafka achieved exactly once delivery between producers and consumers. Key points include:
- Kafka 0.11 introduced exactly once semantics with changes to support transactions and deduplication.
- Producers can write in a transactional fashion and receive acknowledgments of committed writes from brokers.
- Brokers store commit markers to track the progress of transactions and ensure no data loss during failures.
- Consumers can read from brokers in a transactional mode and receive data only from committed transactions, guaranteeing no duplication of records.
- This allows reliable message delivery semantics between producers and consumers with Kafka acting as
The document discusses big data and Hadoop concepts. It covers Hadoop operations like put, get, scan, filter, delete as well as join and group by. It also discusses the different types of data access patterns like random write, sequential read, sequential write and random read. The document focuses on big data, Hadoop operations, and data access patterns.
This document discusses exactly once semantics in Apache Kafka 0.11. It provides an overview of how Kafka achieved exactly once delivery between producers and consumers. Key points include:
- Kafka 0.11 introduced exactly once semantics with changes to support transactions and deduplication.
- Producers can write in a transactional fashion and receive acknowledgments of committed writes from brokers.
- Brokers store commit markers to track the progress of transactions and ensure no data loss during failures.
- Consumers can read from brokers in a transactional mode and receive data only from committed transactions, guaranteeing no duplication of records.
- This allows reliable message delivery semantics between producers and consumers with Kafka acting as
The document discusses big data and Hadoop concepts. It covers Hadoop operations like put, get, scan, filter, delete as well as join and group by. It also discusses the different types of data access patterns like random write, sequential read, sequential write and random read. The document focuses on big data, Hadoop operations, and data access patterns.
This document appears to be test results from running the Yahoo! Cloud Serving Benchmark on a system. It includes performance metrics like request latency distributions and throughput for different request sizes and concurrency levels. Various graphs and tables are presented showing results from multiple benchmark runs. The benchmark was run to test the performance of the system for serving requests in a cloud computing environment.
Tokyo HBase Meetup - Realtime Big Data at Facebook with Hadoop and HBase (ja)tatsuya6502
This is the Japanese translation of the presentation at Tokyo HBase Meetup (July 1, 2011)
Author:
Jonathan Gray
Software Engineer / HBase Commiter at Facebook
活用段階に入ったNoSQLですがまだまだ実際どう使えるのかご存じ無い方も多いのでは無いでしょうか。当セッションでは、MapR-DB(Hbase互換のNoSQL)が企業でどう活用されているのか、インドのマイナンバー事例や国内事例を元に実際の使い方のイメージと技術的な裏付けをご説明します。2015年6月10〜12日に開催されたdb tech showcase Tokyo 2015での講演資料です。
第2回NHNテクノロジーカンファレンスで発表した資料ですー。
References: LINE Storage: Storing billions of rows in Sharded-Redis and HBase per Month (http://tech.naver.jp/blog/?p=1420), I posted this entry in 2012.3.
37. 基本的な構造
BigTable HBase Cassandra Dynamo
CAP CP CP AP AP
データ
分散方法
シャーディング コンシステントハッシング法
データモデル 列志向 KeyValue
MemTable
ストレージ MySQL
CommitLog / SSTable
37
52. 一貫性強度の選択 (複製数3の場合)
B
• 「幾つの複製データに処理を施すか」の選択
Aという値をBに書き換え、読み出す処理の例
B B
A A B
B
Write
B
A A B A B B A B B
Read B
A A B
W:書込数 R:読込数 N:複製数 B B B
W+R>N
の時、「強い一貫性」を得られる B
52