Couchbase introduction - Chinese
•Download as PPTX, PDF•
1 like•353 views
Introduction for Couchbase technical architecture, basic operations, fundimentals- in Chinese
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Couchbase introduction - Chinese
Similar to Couchbase introduction - Chinese (20)
Couchbase introduction - Chinese
- 2. 核心技术 产品概要
- 3. ©2014 Couchbase Inc. Couchbase提供了端到端的数据管理解决方案 3 高可用性的缓存 键值存储 文档数据库 嵌入式的移动数据 库 数据同步功能 通用的平台能力，支持广泛的应用和用户案例 Enterprises often start with cache, then broaden usage to other apps and use cases
- 4. 核心能力 ©2014 Couchbase, Inc. 4 • 开发人员的易用性为中心 支持JSON 数据的索引和查询 增量的Map-Reduce索引数据 更新 • 灵活的水平扩展 同一的节点类型，以及 Shared-nothing 架构 跨数据中心的数据复制 一键式的扩展能力 • 一致性下的高性能 内嵌的基于对象的分布式缓存层 细粒度的锁机制 哈希分区 • 高可用性 无须宕机的管理操作和升级操作 流式数据复制，机架感知能力 强大的集群监控能力（web console， restful API，及命令行）
- 5. ©2014 Couchbase Inc. 不同的行业客户及互联网客户 5 CommunicationsTechnology Travel & Hospitality Media & Entertainment E-Commerce & Digital Advertising Retail & Apparel Games & GamingFinance & Business Services
- 6. ©2014 Couchbase Inc. 用户案例总结 6 360 Degree Customer View Profile managment Catalog Fraud Detection Content Management Internet of Things Digital Communicatio n Real Time Big Data Mobile Application s Personalization
- 8. 键值存储 ©2014 Couchbase, Inc. 8 • Couchbase 是一个键值文档数据库 • 键-支持长度最大为256字节的UTF-8字符串 • 值可以是: - 简单数据类型：字符, 数字, 日期, 布尔型, 以及二进制数据都可以被存储 – 上述数据都是以 Base64编码的字符形式存储 - 复杂数据类型：字典/哈希, 数组/列表, 以JSON格式存储 (简单列表可以表达为由分隔符分割 的字符串) - JSON 可以理解为特殊格式的字符串- 包含自嵌套的简单以及复杂数据类型 - 可以没有明确要求的Schema，动态的更改文档结构，现在更新，同类型的文档可以包含完 全不同的属性
- 9. Couchbase 支持丰富数据类型的存储 Key-Value 存储 文档 存储 2014-06-23-10:15am : 75F 2014-06-23-11:30am : 77F 2014-06-23-02:00pm : 82F 0001: {firstname: “Dipti”, lastname: “Borkar”, language: “English”, time_zone: “PST”, zip: 94403 } 键 - UTF-8字符，最长256字节 值 – 可以使0-20兆字节 (推荐值的大小不超过1 MB) ©2014 Couchbase, Inc. 9
- 10. 可以表达复杂的对象和数据结构 非常简单的表达, 轻量, 结构紧凑, 可读性高 通用API的返回类型 Facebook, Twitter, 等大型SNS都返回JSON 基于Javascript Couchbase支持原生的存储JSON 序列化/反序列化十分快速 JSON数据格式的优势 ©2014 Couchbase, Inc. 10
- 11. 文档的存储和获取 Couchbase 集群 服务器节点 用户/应用 数据 存在于 Data Buckets 文档 读出/ 写入 组成了 客户端 服务器 动态扩展 基于哈希分区 ©2014 Couchbase, Inc. 11
- 12. User Object string uid string firstname string lastname int age array favorite_colors string email u::john@couchbase.com { “uid”: 123456, “firstname”: “John”, “lastname”: “Smith”, “age”: 22, “favorite_colors”: [“blue”, “black”], “email”: “john@couchbase.com” } User Object string uid string firstname string lastname int age array favorite_colors string email u::john@couchbase.com { “uid”: 123456, “firstname”: “John”, “lastname”: “Smith”, “age”: 22, “favorite_colors”: [“blue”, “black”], “email”: “john@couchbase.com” } insert()/rep lace()/upse rt() get() 对象到JSON的序列化和反序列化 ©2014 Couchbase, Inc. 12
- 13. 演示一 数据建模 Data model ©2014 Couchbase, Inc.
- 14. 核心架构 同一的节点类型
- 15. 所有服务器上安装同样节点 ©2014 Couchbase, Inc. 15 Heartbeat Processmonitor Globalsingletonsupervisor Configurationmanager on each node Rebalanceorchestrator Nodehealthmonitor one per clusa vBucketstateandreplicationmanager http RESTmanagementAPI/WebUI HTTP 8091 Erlang port mapper 4369 Distributed Erlang 21100 - 21199 Erlang/OTP storage interface Couchbase EP Engine 11210 Memcapable 2.0 Moxi 11211 Memcapable 1.0 Memcached Persistence Layer 8092 Query API QueryEngine Data Manager Cluster Manager
- 16. 节点- 写入操作 ©2014 Couchbase, Inc. 16 33 2 Managed Cache DiskQueue Disk Replication Queue App Server 节点间-内存到内存的数 据复制 Doc DocDoc
- 17. Managed Cache Disk 节点- 读取操作 ©2014 Couchbase, Inc. 17 Managed Cache Doc 1 Get Doc 1 Doc 1Doc 1 App Server DiskQueue Replication Queue 节点间-内存到内存的数 据复制
- 18. Disk Managed Cache 节点- 缓存删除 ©2014 Couchbase, Inc. 18 Doc 1 Doc 1 Doc 2Doc 3Doc 4Doc 5Doc 6 Doc 2Doc 3Doc 4Doc 5Doc 6 App Server DiskQueue Replication Queue 节点间-内存到内存的数 据复制
- 19. 节点- 缓存失效 ©2014 Couchbase, Inc. 19 33 2 DiskQueue Disk Replication Queue App Server 节点间-内存到内存的数 据复制 Doc 1 Doc 2Doc 3Doc 4Doc 5Doc 6 Doc 2Doc 3Doc 4Doc 5Doc 6 Doc 1 Doc 1Doc 1 Managed Cache Get Doc 1
- 21. 每个bucket中都包含活跃的和备份的数据集 每个数据集（活跃/备份） 都包含1024 个分区(vBuckets) 文档的读取和写入等操作与分区vBuckets进行交互 相同的文档ID，每次都被哈希到相同的分区 分区vBucket可以在节点之间移动(rebalance) vBucket和物理服务器之间的对应关系存储在cluster map中 每个virtual bucket 包含整个数据集的1/1024 自动分区 – Bucket and vBuckets vB Data buckets vB 1 ….. 1024 1024个逻辑分区 virtual bucket ©2014 Couchbase, Inc. 21
- 22. ©2014 Couchbase Inc. Cluster Map
- 23. ©2014 Couchbase Inc. Cluster Map
- 24. ©2014 Couchbase Inc. Cluster Map – 新增两个节点
- 25. read/write/update Active SERVER 1 Active SERVER 2 Active SERVER 3 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Shard 5 Shard 2 Shard 9 Shard Shard Shard Shard 4 Shard 7 Shard 8 Shard Shard Shard Shard 1 Shard 3 Shard 6 Shard Shard Shard Replica Replica Replica Shard 4 Shard 1 Shard 8 Shard Shard Shard Shard 6 Shard 3 Shard 2 Shard Shard Shard Shard 7 Shard 9 Shard 5 Shard Shard Shard 集群层操作 • 数据均匀的分布在每个节点上 • 每个节点即包含活跃数据也包含备 份数据 - 在同一时间，数据只有一个拷贝处于 active状态 • 客户端library 提供给应用与数据库 交互的接口 • Cluster map 记录了1024个vbuket 和节点的映射关系 - 应用无需操心任何与数据分区和数据寻 址相关的工作 • 应用负责读，写，更新数据 • 多个应用可以同时访问相同的数据 ©2014 Couchbase, Inc. 25
- 26. SERVER 4 SERVER 5 Replica Active Replica Active read/write/update APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Active SERVER 1 Shard 9 Shard Replica Shard 4 Shard 1 Shard 8 Shard Shard Shard Active SERVER 2 Shard 8 Shard Replica Shard 6 Shard 3 Shard 2 Shard Shard Shard Active SERVER 3 Shard 6 Shard Replica Shard 7 Shard 9 Shard 5 Shard Shard Shard read/write/update Shard 5 Shard 2 Shard Shard Shard 4 Shard 7 Shard Shard Shard 1 Shard 3 Shard Shard 增加节点 • 一键式的扩展性，在集群中增加两 个服务器 • 随着节点容量的增加，文档数据自 动的在节点间进行重分布 - 平均分布 - 最小量的节点间数据移动 • 数据重分布完成后，更新Cluster map • 应用的请求平均分布在新的处理能 力的集群上 ©2014 Couchbase, Inc. 26
- 27. 故障恢复 SERVER 4 SERVER 5 Replica Active Replica Active App Server 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP App Server 2 Active SERVER 1 Shard 5 Shard 2 Shard 9Shard Shard Shard Replica Shard 4 Shard 1 Shard 8Shard Shard Shard Active SERVER 2 Shard 4 Shard 7 Shard 8 Shard Shard Shard Replica Shard 6 Shard 3 Shard 2 Shard Shard Shard Active SERVER 3 Shard 1 Shard 3 Shard 6Shard Shard Shard Replica Shard 7 Shard 9 Shard 5Shard Shard Shard • 应用访问数据分片 • Server3发生故障，发送到 Server 3的请求失败 • 集群侦测到服务器失效 o 将其他节点的备份数据分 区编程活跃数据分区 o 更新cluster map • 请求被路由到可用的节点 • 自动故障恢复后，需要进行 数据的重分布 Shard 1 Shard 3 Shard ©2014 Couchbase, Inc. 27
- 28. 跨数据中心复制 (XDCR) ©2014 Couchbase, Inc. 28 • 持续的从源端集群向目标端集群进行集群间数据复制 • 支持单向或者双向的数据操作 • 任一集群都支持应用的读写操作 (active – active 复制) • 复制的吞吐量可以线性扩展 • 易于管理：通过console，Rest ，或者命令行CLI
- 29. 跨数据中心复制(XDCR) ©2014 Couchbase, Inc. 29 单向复制 • 热备份/ 灾难恢复 • 开发/测试 环境的拷贝 • 复制到索引集群 • 集成Solr，ElasticSearch分布式文本检索引擎 • 定制化集成
- 30. 跨数据中心复制(XDCR) ©2014 Couchbase, Inc. 30 双向复制 • 多个活跃集群 • 为全球性应用提供Data locality • 灾难恢复
- 31. 33 2 数据写入后的跨数据中心复制 2 Managed Cache DiskQueue Disk Replication Queue App Server Couchbase Server Node Doc 1 Doc 1 XDCR Queue Doc 1Doc 1 (New in 3.0) 内存到内存的跨级群 节点数据复制节点间-内存到内存的数 据复制 ©2014 Couchbase, Inc. 31
- 32. ©2014 Couchbase Inc. 数据视图View ©2014 Couchbase, Inc. 32 • 视图和数据查询 – 分布式的索引和查询 – Secondary indexes – 基于index的灵活查询 • 增量的索引更新 – 分布式的实时数据分析 – 只有更新的数据进行view engine • 全文检索Full Text Search – 和ElasticSearch / Solr 的集成 – 使用XDCR的能力
- 33. ©2014 Couchbase Inc. 33 2 33 Managed Cache DiskQueue Disk Replication Queue 应用服务器 Couchbase Server Node Doc 1 Doc 1 节点间-内存到内存的数据复制 View engine Doc 1Doc 1 数据写入后的索引视图构建
- 34. ©2014 Couchbase Inc. Active SERVER 1 Shard 5 Shard 2 Shard Shard Replica Shard 4 Shard 1 Shard Shard Shard 1 Active SERVER 3 Shard 5 Shard 2 Shard Shard Replica Shard 4 Shard 1 Shard Shard Shard 1 Active SERVER 2 Shard 5 Shard 2 Shard Shard Replica Shard 4 Shard 1 Shard Shard Shard 1 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Couchbase Server 架构 – 视图 ©2014 Couchbase, Inc. 34 • 分布式的索引视图 • 支持海量的数据集 • 并行的索引构建 • 每个节点的数据对应自己的 索引视图 • 查询合并结果，返回客户端
- 35. 演示二 集群操作& XDCR演示&视图查询 ©2014 Couchbase, Inc.
- 36. SQL for Document （开发者评估版） 下一代, NoSQL 查询语言 类SQL : SELECT * FROM WHERE/LIKE/JOIN/GROUP/etc, CREATE INDEX 针对JSON的扩展，用以支持嵌套的和层次型的数据结构 支持索引视图，和新的secondary索引 Query (DQL), Manipulation (DML), Description (DDL) ODBC/JDBC drivers –由simba提供 整合到Couchbase 4.0（目前官网上有开发者评估版下载）: 集成安装 多线程，无状态的查询和索引组件 基于Couchbase 高性能，高可扩展性的
- 37. Couchbase中的数据查询 表达式查询语言 Java script 视图& spatial views 使用增加维护的Map Reduce计算 对于聚合操作的预构建和计算 适合分析，报表以及大数据等用力场景 SQL for Document SQL-Like 构建secondary indexes 极其适合即席查询和数据分析查询
- 38. SQL For Document的拓扑 ©2014 Couchbase, Inc. 38 Client SDK App Index CB Node ODBC / JDBC App Query Manager Data Index CB Node Query Manager Data Index CB Node Query Manager Data Index CB Node Query Manager Data Index CB Node Query Manager Data Index CB Node Query Manager Data Connectivity & ecosystem Node services & independent scaling Scale-out query throughput ODBC / JDBC BI Tool
- 39. 查询过程 ©2014 Couchbase, Inc. 39 Index Nodes Clients Data Nodes (1) Request (6) Response Query Nodes (2) Plan: Query to execution plan (5) Evaluate: Documents to results (3) Scan: Attributes to keys (4) Fetch: Keys to documents
- 40. 在查询服务内部的流程 ©2014 Couchbase, Inc. 40 Index Client Data FetchScanParse Plan Join Filter Pre-Aggregate Offset Limit Project Data-parallel — Query latency benefits from N cores Memory-bound Pluggable architecture — datastore, index, client, … Request Response SortAggregate
- 41. 演示三 Couchbase SQL for Document ©2014 Couchbase, Inc.
Editor's Notes
- KEY POINT: COUCHBASE PROVIDES A SET OF MULTI-PURPOSE, CORE CAPABILITIES THAT SUPPORT A BROAD RANGE OF APPLICATIONS AND USE CASES, ALL IN A SINGLE DATA MANAGEMENT PLATFORM. Couchbase provides a set of technology capabilities to support a broad range of applications and use cases: High Availability Cache: Couchbase provides an integrated managed object cache, so you can start out using Couchbase as a high availability cache on top of your existing relational database. For example, you can use Couchbase as a session store in front of your relational database, if your relational DB is struggling to keep up with the load required for online interactive applications. Key-Value Store: Many customers start with Couchbase as a cache and then broaden their usage to other capabilities, like using Couchbase as a Key-Value Store for things like Profile Management. Document Database: From there, you can grow into using Couchbase as a Document Database, where you can do more with capabilities like indexing and Cross Data Center Replication. Embedded Database: Couchbase also provides an embedded database called Couchbase Lite. It’s a purpose-built database for the device, so you can build applications that are always available and always work, whether offline or online. Sync Management: Finally, as part of our solution for mobile applications, we provide Couchbase Sync Gateway, which automatically synchronizes data on the device with Couchbase Server in the cloud so your developer doesn’t have to write code to manage the complex sync process. Starting with cache and then expanding to other capabilities is often a good way to learn the technology and get comfortable with Couchbase for a wider set of use cases.
- KEY POINT: MAJOR ENTERPRISES ACROSS NUMEROUS INDUSTRIES ARE ADOPTING COUCHBASE NOSQL TECHNOLOGY TO SUPPORT THEIR DATA MANAGEMENT NEEDS. We’ve looked at what’s driving NoSQL and the advantages it delivers. So who are some of the companies that are actually adopting Couchbase? As this slide shows, major enterprises across many different industries are adopting Couchbase NoSQL solution. You see many innovative Internet companies here – like eBay, LinkedIn, Orbitz, and PayPal. They were the early adopters of NoSQL. But it’s clear that NoSQL is now being adopted by a broad range of companies in many other industries: Consumer electronics and technology companies like Apple and Cisco Retail companies like Walmart and Tesco Financial Services companies like VISA and Wells Fargo Telcos like Verizon, AT&T and Vodafone What’s also interesting is that we’re seeing the use of NoSQL expand inside many of these companies. Orbitz, the online travel company, is a great example – they started using Couchbase to store their hotel rate data, and now they use Couchbase in many other ways. So the big takeaway is that an increasing number of companies across industries are seeing the value of NoSQL for a growing number of use cases.
- KEY POINT: ENTERPRISES ARE USING COUCHBASE ACROSS A RANGE OF MISSION CRITICAL USE CASES. As the slide shows, Couchbase supports a wide range of use cases, from Profile Management to Fraud Detection. Each use case has its own set of requirements – some need very high performance, some need very high availability, some need flexibility of the data model. The ability to meet all of these requirements is what has driven adoption of Couchbase.
- All information that you store in Couchbase Server are documents with keys. Keys are unique identifiers for a document, and values are either JSON documents or if you choose the data you want to store can be byte stream, data types, or other forms of serialized objects. Value can be JSON or binary objects, such as integers and strings. Keys are also known as document IDs and serve the same function as a SQL primary key. A key in Couchbase Server can be any string, including strings with separators and identifiers, such as ‘person_93679.’ A key is unique. When Couchbase Server is used as a store for JSON documents, the records can be indexed and queried. Couchbase Server provides a JavaScript-based query engine to find records based on field values.
- The application makes a call for a key called NYC MQ1 We run the key through the crc 32 function and the result of that hash function is that it points to vbucket3 Which in turn points to couchbase server number 1
- We now run a different key through through the has and we now come up with differnet vbucket, vbucket 4 and that points to server 3
- We now run a different key through through the has and we now come up with differnet vbucket, vbucket 4 and that points to server 3
- This slide has an click-by-click animation 1. (click) A set request comes in from the application . 2. Couchbase Server responses back that they key is written 3. (click)Couchbase Server then Replicates the data out to memory in the other nodes At the same time it is put the data into a write queue to be persisted to disk (click)Once it is on disk, the item is processed by the view engine and sent out any configured XDCR link to one or more clusters
- Indexing and querying distributed create indexes on the fields in JSON documents Called Views in Couchabse Server Views are queried to find the objects you are interested in, e.g. range queries to find all players that have black sheep on their farm (if asked: No ad-hoc query language. Index are described via simple Javascript.) Incremental Map Reduce “Normal” map reduce is batch based: i.e. it has to run across all data everytime. So you don’t get updated results often, especially over large data sets. Incremental Map reduce is only considering data that has changed and then calculates the updated result. This happens fast, in near real-time. Distributed across all nodes, so able to cope with large data amounts Does single map and reduce step, so great for simple analytics like leaderboards, counts sums, across data having specific attributes/charcteristics. Full Text Search Integration with separate Elastic search cluster, using XDCR technology Robust, so will efficiently cope with node failures rebalances or interrupted connections to keep the full text index in sync Elastic search is a very fast JSON document based full text indexing open source solution, based on Apache Lucene (the same as used by SOLR that more people will know) Elastic search is also clustered and scales easily and provides very flexible and powerful full text search capabilities
- This slide has an click-by-click animation 1. (click) A set request comes in from the application . 2. Couchbase Server responses back that they key is written 3. (click)Couchbase Server then Replicates the data out to memory in the other nodes At the same time it is put the data into a write queue to be persisted to disk (click)Once it is on disk, the item is processed by the view engine and sent out any configured XDCR link to one or more clusters