Sergey Sverchkov
Project Manager
sergey.sverchkov@altoros.com

© ALTOROS Systems | CONFIDENTIAL
ORDER
Order
ID: 1001
Order Date: 15.9.2012
Customer






Billing Address
Street: Somestreet 10
City: Somewhere
Postal ...
•
•
•
•
•
•
•
•
•

© ALTOROS Systems | CONFIDENTIAL

3
•
•

•
•
•
•

© ALTOROS Systems | CONFIDENTIAL

4
•



• Workload is defined by different distributions



•

Operations of the following types:





© ALTOROS Syst...
•




“user234123”



•





© ALTOROS Systems | CONFIDENTIAL

6
© ALTOROS Systems | CONFIDENTIAL

7
© ALTOROS Systems | CONFIDENTIAL

8
•
 Single availability zone eu-west-1b, Ireland region
 Single security group with all required port opened
 4 m1.xlarg...
•
 partitioner: org.apache.cassandra.dht.Murmur3Partitioner
 key_cache_size_in_mb: 1024
 row_cache_size_in_mb: 6096
 J...
•
 Replica factor 1
 Memory + disk mode

•
 JVM heap size 12GB
 Replica factor 1

 Snappy compressor

© ALTOROS Syste...
Performance of the systems was evaluated under different workloads:







© ALTOROS Systems | CONFIDENTIAL

12
Load phase, 100.000.000 records * 1 KB, [INSERT]
9

Average latency, ms

8
7
6
5

hbase

4

cassandra

3

couchbase
mongod...
Workload A: Update (Update 50%, Read 50%)
120
100

cassandra

80

couchbase
hbase

60

mongodb
40
20
0
0

500

1000

1500
...
Workload A: Read (Update 50%, Read 50%)

80
70
60

50

cassandra
couch

40

hbase
mongo

30
20
10
0
0

500

1000

1500

20...
Workload B: Update (update 5% , read 95%)
120
100
80
cassandra
60

couch
hbase

40

mongo

20
0
0

500

1000

1500

© ALTO...
Workload B: Read (update 5% , read 95%)
90

80
70
60
cassandra

50

couch

40

hbase

30

mongo

20
10
0
0

500

1000

150...
Workload C: 100% Read
80
70
60
50

cassandra

40

couch
hbase

30

mongo
20
10
0
0

500

1000

1500

2000

© ALTOROS Syste...
Workload D: Insert (insert 5% , read 95%)
60
50
40
cassandra
30

couch
hbase

20

mongo

10
0
0

500

1000

1500

2000

© ...
Workload D: Read (insert 5% , read 95%)
90
80
70
60
cassandra

50

couch

40

hbase

30

mongo

20
10
0
0

500

1000

1500...
400

Workload E: Insert (Insert 5%, Scan 95%)

350
300
250
200

cassandra

150

hbase

100
50
0

0

50

100

150

© ALTORO...
Workload F: read (Read-Modify-Write 50%, Read 50%)
80
70

60
50

cassandra

40

couch
hbase

30

mongo
20
10
0
0

500

100...
Workload F: Update (Read-Modify-Write 50%, Read 50%)
140
120

100
cassandra

80

couch
60

hbase
mongo

40
20
0
0

500

10...
Workload F: Read-Modify-Write (Read-Modify-Write 50%, Read 50%)
200
180
160
140
120

cassandra

100

couch

80

hbase

60
...
Workload G: Insert (Insert 90%, Read 10%)
35

30
25
cassandra

20

couch
15

hbase
mongo

10
5
0
0

1000

2000

3000

4000...
Workload G: Read (Insert 90%, Read 10%)
60
50
40
cassandra
30

couch
hbase

20

mongo

10
0
0

1000

2000

3000

4000

500...
•
•

•
•
•
•
•
•

© ALTOROS Systems | CONFIDENTIAL

27
Sergey Sverchkov

Project Manager
sergey.sverchkov@altoros.com
Altoros, 2013

© ALTOROS Systems | CONFIDENTIAL

28
Upcoming SlideShare
Loading in …5
×

Evaluating NoSQL performance: Which database is right for your data? - Sergey Sverchkov (Altoros)

389
-1

Published on

Presented at JAX London 2013

The need to operate terabyte-size databases becomes very common these days. Unless you have implemented architectures that use NoSQL databases and frameworks that support data-intensive distributed applications, then many technology options available are probably a slight enigma. This session focuses on real-world successful attempts to benchmark four of the most popular NoSQL databases side by side. The base tool selected for the purpose of this research is Yahoo Cloud Serving Benchmark and benchmarking is performed on Amazon Elastic Compute Cloud instances.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
389
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
16
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Often referred to as NoSQL, non-relational databases feature elasticity and scalability. In addition, they can store big data and work with cloud computing systems. All of these factors make them extremely popular.
  • Why did NoSQL data stores appear? Mostly because relational databases (RDBMS) have a number of disadvantages, if you have to work with large datasets.For example, RDBMS are hard to scale and their architecture is designed to work on a single machine. - Scaling write operations is either hard, expensive, or impossible.- Vertical scaling (or upgrading equipment) is either limited or very expensive. Unfortunately, this is often the only possible way you can scale.- Horizontal scaling (or adding new nodes to the cluster) is either unavailable or you can only implement it partially. There are some solutions from Oracle and Microsoft that make it possible to have computing instances on several servers. Still, the database itself remains in shared storage.In addition to poor scalability, RDBMS have strict data models. The schema is created together with the database and you will need a lot of time and effort to change this structure. In most cases it is an extremely complex task. Apart from that, RDBMS have difficulties with semistructured data.
  • NoSQL solutions address many of these problems.POINT 1: In 2013, the number of NoSQL products reached 150+ and the figure is still growing. That variety makes it difficult to select the best tool for a particular case.POINT 2: They come in many types--key-value, columnar, document-oriented, and graph.POINT 3: There is one thing in common for all NoSQL databases. They don't use the relational data model. This means they do not use the SQL query language.POINT 4: NoSQL data management systems are inherently schema-free (with no obsessive complexity and a flexible data model) and eventually consistent (complying with BASE rather than ACID)POINT 5: They provide APIs to perform various operations. Some of NoSQL data stores support query language operations, for example, Cassandra and Hbase. However, there is no standard. This is another difference between NoSQL databases and traditional RDBMS.POINT 6: RDBMS usually have strong data consistency. In contrast to that, NoSQL data stores operate with eventual consistency. When you add data to the system, it becomes consistent after some time.POINT 7: NoSQL architectures are designed to run in cluster that consist of several nodes. This makes it possible to scale them horizontally by increasing the number of nodes.In addition, NoSQL data stores serve huge amounts of data and provide high throughput.
  • POINT 1: NoSQL databases differ from RDBMS in their data models. These systems can be divided into 4 groups:A. Key Value StoresKey value stores are similar to maps or dictionaries where data is addressed by a unique key.B. Document StoresDocument Stores encapsulate key value pairs in JSON or JSON like documents. Within documents, keys have to be unique. In contrast to key-value stores, values are not opaque to the system and can be queried as well.C. Column Family StoresColumn Family Stores are also known as column oriented stores, extensible record stores and wide columnar stores.D. Graph databasesKey-value stores, document stores, and column family stores have a common feature. They do store denormalized data in order to gain advantages in distribution.In contrast to relational databases and the already introduced key oriented NoSQL databases, graph databases are specialized on efficient management of heavily linked data.POINT 2: All NoSQL data stores have an API to work with data. Some DBs use certain SQL operations. Others support MapReduce aggregation.POINT 3: Multiversion concurrency control (MVCC) relaxes strict consistency in favor of performance. In order to support transactions without reserving multiple datasets for exclusive access, optimistic locking is provided by many stores. Before changed data is committed, each transaction checks, whether another transactions made any conflicting modifications to the same datasets.POINT 4: NoSQL databases differ in the way they distribute data on multiple machines. Since data models of key-value stores, document stores and column family stores are key oriented, the two common partition strategies are based on keys, too.The first strategy distributes datasets by the range of their keys. A routing server splits the whole keyset into blocks and allocates these blocks to different nodes. Afterwards, one node is responsible for storage and request handling of his specific key ranges. In order to find a certain key, clients have to contact the routing server for getting the partition table.Higher availability and much simpler cluster architecture can be achieved with the second distributionstrategy called consistent hashing. In contrast to range based partitioning, keys are distributed by using hash functions. Since every server is responsible for a certain hash region, addresses of certain keys within the cluster can be calculated very fast.In addition to better read performance through load balancing, replication also brings better availability and durability, because failing nodes can be replaced by other servers. If all replicas of a master server were updated synchronously, the system would not be available until all slaves had committed a write operation. Ifmessages got lost due to network problems, the system would not be available for a longer period of time. This solution is not suitable for platforms that rely on high availability, because even a few milliseconds of latency can have a big influence on user behavior.POINT 5: (PERFORMANCE: TYPICAL WORKLOADS)Obviously, performance is a very important factor. Performance of data storage solutions can be evaluated using typical scenarios. These scenarios simulate the most common operations performed by applications that use the data store, also known as typical workloads. The tests that we performed to compare performance of several NoSQL data stores also used typical workloads.
  • Database vendors usually measure productivity of their products with custom hardware and software settings designed to demonstrate the advantages of their solutions. In our tests we tried to see how NoSQL data stores perform under the same conditions.POINT 1: For benchmarking, we used the Yahoo Cloud Serving Benchmark (YCSB)The kernel of YCSB has a a framework with a workload generator that creates test workload and a set of workload scenarios.POINT 2: Developers need to describe the scenario of the workload by operation type: what operations are performed on what types of records. POINT 3: Supported operations include: insert, update (change one of the fields), read (one random field or all the field of one record), and scan (read the records in the order of the key starting from a randomly selected record).We can define the workload by the data that will be loaded into the database during the loading phase and the operations that will be executed against the data set during the transaction phase. 
  • Each workload was targeted at a table of 100,000,000 records; each record was 1,000 bytes in size and contained 10 fields. A primary key identified each record, which was a string, such as “user234123”. Each field was named field0, field1, and so on. The values in each field were random strings of ASCII characters, 100 bytes each. Database performance was defined by the speed at which a database computed basic operations. A basic operation is an action performed by the workload executor, which drives multiple client threads. Each thread executes a sequential series of operations by making calls to the database interface layer both to load the database (the load phase) and to execute the workload (the transaction phase). The threads throttle the rate at which they generate requests, so that we may directly control the offered load against the database. In addition, the threads measure the latency and achieved throughput of their operations and report these measurements to the statistics module.-- The tests:We defined the following values for the workload executor:the number of threadsthe types of operations in the workload and the desired number of operations per second (target throughput)Then we measured the time it took to perform these transactions (latency).
  • This is a component diagram of the YCSB framework. It consists of several modules.Workload executor applies the workload to the data store. For each session, when the client accesses the DB, a client thread is initiated. Each thread performs a set of operations from the workload. The results in the form of statistics are then sent to the statistics module, which prints the output of the test to console where benchmark is started. These tests are consequently repeated for all the selected solutions.The YCSB framework has connectors for a wide range of DBs. For each database tested with YCSB, a developer needs to determine the type of database, target throughput, the number of concurrent threads on the client side, and how many operations we want to perform. This is necessary to create and start a test.
  • Now let's take a look at the NoSQL data stores that we tested:Cassandra 2.0: This is a column-value data store. We ran it on a virtual machines with Java 1.7.40 installed. The transactions were performed with the non-default configuration. In particular, we used a random partitioner to section data by nodes. The amount of data cash for the keys was 1 GB. The size of row cash was 6 GB. The size of JVM heap was 6 GB. Data was not replicated (there were no copies). This approach was intentional, we wanted to test performance, not failure tolerance of the cluster. MongoDB: This is a document-oriented DB. Here, we didn't do much additional configuration or tuning. As I have mentioned before, for Mongo, we added two VMs that served as routers because according to official documentation, the mongo router process should run on a separate machine. However, if you need to simplify the model, mongo router may run on the same machine, where the YCSB client is. In one of our earlier tests, we discovered that it uses a lot of CPU power. This is why it should be placed on a separate machine. Data sharding for MongoDB was based on document key.
  • We used the following workloads: Workload A: Update-heavily mode. Workload A is an update-heavily scenario that simulates how a database works, when recording typical actions of an e-commerce solution user.Settings for the workload: Read/update ratio: 50/50Zipfian request distributionWorkload BWorkload B is a read-mostly workload that has a 95/5 (ninety five to five percent) read/update ratio. It recaps content tagging, when adding a tag is an update, but most operations include reading tags.Workload CWorkload C is a read-only workload that simulates a data caching layer, for example a user profile cache.Workload D Workload D has 95/5 read/insert ratio. The workload simulates access to the latest data, such as user status updates or working with inbox messages first.Workload EWorkload E is a scan-short-ranges workload with a scan/insert percentile proportion of 95/5. It corresponds to threaded conversations that are clustered by a thread ID. Each scan is performed for the posts of a given thread.Workload F Workload F has read-modify-write/read ops in a proportion of 50/50. It simulates access to user database, where user records are read and modified by the user. User activity is also recorded to this database.Workload G Workload G has a 10/90 read/insert ratio. It simulates data migration process or highly intensive data creation.
  • For this benchmark, we generated a number of workloads based on a target throughputs and then measured the number of operations per second or the actual throughput of the each database. We also measured their latencies, or how long it took to perform each operation.During the first stage of the test, we uploaded 100 mln records of 1 kb each to each data store. YCSB measured average throughput in operations per second and latency in milliseconds.Hbase demonstrated the lowest throughput, probably because we turned on the auto-flash mode. This mode ensures that each operation that creates a record will be sent from the client to the server and then persisted to the database. Hbase also supports an alternative mode that generates additional cash on the client side. When the client is out of storage it sends this cash to the server. In this alternative mode, Hbase saves data to disk in batches.As we expected, Couchbase and Cassandra demonstrated excellent results. Cassandra simultaneously updates data in memory and writes it to the transaction journal on disk. Couchbase writes data to memory and then asynchronously persists it to disc. The result of the transaction returns after everything has been saved to memory. In this particular test, all data was loaded in a single iteration. But insert, update, and read operations that I will describe later were performed in several iterations.
  • The last workload, G, mostly consists of insert operations. It simulates the process of data migration or inserting a lot of data. The results are similar to what we saw on the previous graph. Hbase, Cassandra, and Couchbase demonstrate low latencies and high throughput. MongoDB’s performance starts to decline at about 4000 ops per second with an average latency of up to 5 times greater than in other databases.
  • This is the last diagram that shows the results of read operations, which make up 10% of workload G. Here we can see that latencies vary for different solutions. This might be because the data is in network storage on the cloud. Couchbase and Cassandra show a maximum throughput of up to 6000 operations per second.
  • 1. What you choose depends on your needs. Before making the decision, answer the following questions: determine what your data sets and your data model will be like; the type data model will depend on the data sets and typical operations that your app will performdetermine your requirements to transaction support; decide whether you need transactionschoose whether you need replication; decide on your requirements to data consistencydetermine your performance requirements (how fast your DB should be)If your project is based on an existing solution, you should see if it is possible to migrate your data2. Then, taking into account all these factors, evaluate different solutions and test their performance (that’s what this presentation is all about). It is very useful to build a prototype and perform proof of concept. Then based on this prototype, you can select the best solution for your system. 3. Prototyping makes it possible to see how the solution will work in a real-life project. If it doesn’t work well enough, you need to review the architecture, the components, and build a new prototype. 4. There are no perfect solutions and there are no bad NoSQL or RDBMS data stores. The database and its implementation depends on a particular use case. The tests we have performed show that in different scenarios, different solutions have very different results. Your final choice might be a compromise. The main determinant will be what you want to achieve and what properties you need most. The solution may use several different solutions, including relational and NoSQL databases.
  • Evaluating NoSQL performance: Which database is right for your data? - Sergey Sverchkov (Altoros)

    1. 1. Sergey Sverchkov Project Manager sergey.sverchkov@altoros.com © ALTOROS Systems | CONFIDENTIAL
    2. 2. ORDER Order ID: 1001 Order Date: 15.9.2012 Customer    Billing Address Street: Somestreet 10 City: Somewhere Postal Code: 55901  ADDRESS Line Items Quantity Price Ipod Touch 1 220.95 Monster Beat 2 190.00 Apple Mouse 1 69.90 Name  CUSTOMER First Name: Peter Last Name: Sample ORDER_LINES     © ALTOROS Systems | CONFIDENTIAL 2
    3. 3. • • • • • • • • • © ALTOROS Systems | CONFIDENTIAL 3
    4. 4. • • • • • • © ALTOROS Systems | CONFIDENTIAL 4
    5. 5. •   • Workload is defined by different distributions   • Operations of the following types:     © ALTOROS Systems | CONFIDENTIAL 5
    6. 6. •    “user234123”  •     © ALTOROS Systems | CONFIDENTIAL 6
    7. 7. © ALTOROS Systems | CONFIDENTIAL 7
    8. 8. © ALTOROS Systems | CONFIDENTIAL 8
    9. 9. •  Single availability zone eu-west-1b, Ireland region  Single security group with all required port opened  4 m1.xlarge 64bit instances for cluster nodes: 16GB RAM, 4 vCPU, 8 ECU, highperformance network  1 c1.xlarge 64bit instance for YSCB client: 7GB RAM, 8 vCPU, 20 ECU, highperformance network  2 additional c1.medium 64bit instances for mongo routers: 1.7GB RAM, 2 vCPU, 5 ECU, moderate network •  4 EBS volumes by 25 GB each in RAID0  EBS optimized volumes, no Provisioned IOPS © ALTOROS Systems | CONFIDENTIAL 9
    10. 10. •  partitioner: org.apache.cassandra.dht.Murmur3Partitioner  key_cache_size_in_mb: 1024  row_cache_size_in_mb: 6096  JVM heap size: 6GB  Snappy compressor  Replica factor 1 •  2 c1.medium nodes with mongo router process - mongos  Replica factor 1  Sharding by internal key “_id” © ALTOROS Systems | CONFIDENTIAL 10
    11. 11. •  Replica factor 1  Memory + disk mode •  JVM heap size 12GB  Replica factor 1  Snappy compressor © ALTOROS Systems | CONFIDENTIAL 11
    12. 12. Performance of the systems was evaluated under different workloads:       © ALTOROS Systems | CONFIDENTIAL 12
    13. 13. Load phase, 100.000.000 records * 1 KB, [INSERT] 9 Average latency, ms 8 7 6 5 hbase 4 cassandra 3 couchbase mongodb 2 1 0 0 10000 20000 30000 40000 Throughput, ops/sec © ALTOROS Systems | CONFIDENTIAL 13
    14. 14. Workload A: Update (Update 50%, Read 50%) 120 100 cassandra 80 couchbase hbase 60 mongodb 40 20 0 0 500 1000 1500 2000 © ALTOROS Systems | CONFIDENTIAL 2500 3000 14
    15. 15. Workload A: Read (Update 50%, Read 50%) 80 70 60 50 cassandra couch 40 hbase mongo 30 20 10 0 0 500 1000 1500 2000 © ALTOROS Systems | CONFIDENTIAL 2500 3000 15
    16. 16. Workload B: Update (update 5% , read 95%) 120 100 80 cassandra 60 couch hbase 40 mongo 20 0 0 500 1000 1500 © ALTOROS Systems | CONFIDENTIAL 2000 2500 16
    17. 17. Workload B: Read (update 5% , read 95%) 90 80 70 60 cassandra 50 couch 40 hbase 30 mongo 20 10 0 0 500 1000 1500 © ALTOROS Systems | CONFIDENTIAL 2000 2500 17
    18. 18. Workload C: 100% Read 80 70 60 50 cassandra 40 couch hbase 30 mongo 20 10 0 0 500 1000 1500 2000 © ALTOROS Systems | CONFIDENTIAL 2500 3000 18
    19. 19. Workload D: Insert (insert 5% , read 95%) 60 50 40 cassandra 30 couch hbase 20 mongo 10 0 0 500 1000 1500 2000 © ALTOROS Systems | CONFIDENTIAL 2500 3000 19
    20. 20. Workload D: Read (insert 5% , read 95%) 90 80 70 60 cassandra 50 couch 40 hbase 30 mongo 20 10 0 0 500 1000 1500 2000 © ALTOROS Systems | CONFIDENTIAL 2500 3000 20
    21. 21. 400 Workload E: Insert (Insert 5%, Scan 95%) 350 300 250 200 cassandra 150 hbase 100 50 0 0 50 100 150 © ALTOROS Systems | CONFIDENTIAL 200 250 21
    22. 22. Workload F: read (Read-Modify-Write 50%, Read 50%) 80 70 60 50 cassandra 40 couch hbase 30 mongo 20 10 0 0 500 1000 1500 © ALTOROS Systems | CONFIDENTIAL 2000 2500 22
    23. 23. Workload F: Update (Read-Modify-Write 50%, Read 50%) 140 120 100 cassandra 80 couch 60 hbase mongo 40 20 0 0 500 1000 1500 © ALTOROS Systems | CONFIDENTIAL 2000 2500 23
    24. 24. Workload F: Read-Modify-Write (Read-Modify-Write 50%, Read 50%) 200 180 160 140 120 cassandra 100 couch 80 hbase 60 mongo 40 20 0 0 500 1000 1500 © ALTOROS Systems | CONFIDENTIAL 2000 2500 24
    25. 25. Workload G: Insert (Insert 90%, Read 10%) 35 30 25 cassandra 20 couch 15 hbase mongo 10 5 0 0 1000 2000 3000 4000 5000 © ALTOROS Systems | CONFIDENTIAL 6000 7000 25
    26. 26. Workload G: Read (Insert 90%, Read 10%) 60 50 40 cassandra 30 couch hbase 20 mongo 10 0 0 1000 2000 3000 4000 5000 © ALTOROS Systems | CONFIDENTIAL 6000 7000 26
    27. 27. • • • • • • • • © ALTOROS Systems | CONFIDENTIAL 27
    28. 28. Sergey Sverchkov Project Manager sergey.sverchkov@altoros.com Altoros, 2013 © ALTOROS Systems | CONFIDENTIAL 28
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×