Assignment 3: YCSB, SQL and NoSQLCSCI 599: NewSQL Database Management SystemCSCI 599: NewSQL Database Management SystemsPr...
I. Workload A - read 50% update 50%1. Tier 1 – Performance Benchmarking: For constant hardware, increase Offered Throughpu...
Fig A.1: WorkloadA Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQ...
a. Read Latency MeasurementREAD PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.947 0.4191000 0.812 0.3262500...
2. Tier 2 – Scalability: Scaleup – Increase hardware, data size and workload proportionally. Measurelatency; should be con...
1000 1000 default 1213 824.402 505 0.411 0 6410000 1000 default 1191 839.63 542 0.25 0 6100000 1000 default 1198 834.724 5...
Fig B.1: WorkloadB Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQ...
15000 1.98 0.4417500 2.54 0.5220000 3.292 0.6422500 3.6 0.7425000 4.23 1.05630000 4.95 1.21Fig B.2: WorkloadB Performance ...
Fig B.3: WorkloadB Performance Benchmarking - Read RecordsObservation – PostgreSQL show higher avg latencies for Workload ...
10000 1000 default 1211 825.76 48 0.437 0 5100000 1000 default 1201 832.63 40 1 0 151000000 1000 default 6710 149.03 55 19...
Conclusion – The read and update for the workload B show similar results in terms of throughput. Hereupdate operations sho...
Fig C.1: WorkloadC Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQ...
Fig C.2: WorkloadC Performance Benchmarking - Read RecordsObservation – PostgreSQL show higher avg latencies for Workload ...
Fig D.1: WorkloadA Offered Throughput vs Achieved ThroughputObservation- In the above graph PostgreSQL is not able to matc...
Fig D.2: WorkloadD Performance Benchmarking - Insert Recordsb. ReadREADPostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLat...
Observation – PostgreSQL show higher avg latencies for Workload D Inserts & Reads compared toMongoDB. MongoDB insert and r...
50000 13073.9 903Fig E.1: WorkloadE Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can obser...
Fig E.2: WorkloadE Performance Benchmarking - Scan RecordsObservation – PostgreSQL show very stable and low avg latency th...
Fig F.1: WorkloadF Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQ...
Fig A.2: WorkloadF Performance Benchmarking – Read/Modify Recordsb. ReadREADREAD PostgreSQL MongoDBOfferedThroughputAvgLat...
Observation – Postgre show very low latencies for read/modifies than Mongodb. Mongodb initially ithas high latency but the...
Upcoming SlideShare
Loading in...5
×

Analysis postgre sql-vs_mongodb_report

1,342

Published on

· A comprehensive evaluation of performance of NoSQL DBMS MongoDB with Postgre SQL DBMS using YCSB.
· Measured the Benchmarks for Tier1: Performance and Tier2: Scalability using the YCSB tool.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,342
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Analysis postgre sql-vs_mongodb_report

  1. 1. Assignment 3: YCSB, SQL and NoSQLCSCI 599: NewSQL Database Management SystemCSCI 599: NewSQL Database Management SystemsProblem Statement –Compare the performance of a SQL and NoSQL DBMS using YCSB.Overview: In the first two assignments, we analyzed the performance of a NoSQL DBMS and a SQLDBMS independent of one another. In this assignment, we compare the performance of the twosystems assigned to you by generating graphs similar to those shown in the YCSB paper: Response timeas a function of the throughput.System Configuration –Number of CPU(s) One Physical Processor / 8 Cores / 4 Logical Processors / 64 bitsCPU Name Intel(R) Core(TM) i5-2410M CPU @ 2.30GHzInstalled RAM ChannelA – 2048 Mbytes + ChannelB – 2048 MbytesSpeed (RAM) 1333MHzDisk Space Disk C: 151 GB Available, 227 GB Total, 151 GB FreeDisk Interface IDE SATA-IIController Buffer Size 16 MBytesOperating System Windows 7 64bitPhysical Memory 4011 MB Total, 863 MB FreeVirtual Memory 8019 MB Total, 2034 MB FreeMemory Load 78%PageFile Size 4010 MBIn use 2431 MBMax used 2437 MBAssumptions:Turbo boost and Hyper threading for the processor mentioned above is ON for the whole experimentand may cause some erratic variations in the performance measured values.Benchmark TiersTier1 – Performance- Constant Hardware , Increase offered throughput.- Measure the Achieved throughput and the latency/throughput curve.Tier2 – Scalability- Increase the workload (records) and measure the latency and throughputResults of YCSB –The following are the results calculated by the YCSB for the Postgre(JDBC) client on the systemdescribed above.Scenario :-Record Count: - 10000Operation Count: - 50KOffered Throughput: - X-Axis (Knob) Range 10 to 10000Achieved Throughput: - Y-axis (measured metric) andAverage Latency: - Y-axis (measured metric).
  2. 2. I. Workload A - read 50% update 50%1. Tier 1 – Performance Benchmarking: For constant hardware, increase Offered Throughput (op/sec)until saturation.Throughput Performance MeasurePostgreSQL MongoDBOfferedThroughputAchievedThroughputAchievedThroughput100 96.5 97.71000 982.5 981.22500 2370 23825000 4610 4549.57500 4620 6587.610000 4670 8438.812500 5420 1015015000 5330 1156017500 6330 1300020000 6190 1398022500 6350 1567025000 7260 1666030000 7030 1631050000 7100 1710070000 7150 16900100000 6960 16980
  3. 3. Fig A.1: WorkloadA Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQL is not able to match up with theconsistent increase in achieved throughput as mongoDB does. MongoDB shows higher achievedthroughput values for offered load.Conclusion – MongoDB actually does behave like an RDMS system for workload A. It shows overall goodperformance than PostgreSQL for workload A, as reading and writing is conducted within the usablememory and use of memory mapped files for data storage which hence achieves high-performance.a. Update Latency MeasurementUPDATE PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 1.919 0.861000 1.793 0.4892500 1.683 0.3565000 1.83 0.3767500 1.887 0.38310000 1.91 0.412500 1.87 0.43615000 1.98 0.4417500 2.54 0.5220000 3.292 0.6422500 3.6 0.7425000 4.23 1.05630000 4.95 1.21Fig A.2: WorkloadA Performance Benchmarking - Update Records
  4. 4. a. Read Latency MeasurementREAD PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.947 0.4191000 0.812 0.3262500 0.869 0.2325000 0.88 0.2577500 0.95 0.2810000 0.96 0.29812500 0.972 0.3115000 0.979 0.31417500 1.008 0.31920000 1.26 0.43222500 2.289 0.79525000 2.805 1.0330000 3.466 1.45Fig A.3: WorkloadA Performance Benchmarking - Read RecordsObservation – PostgreSQL show higher avg latencies for Workload A Updates & Reads compared toMongoDB. For updates we can see that Mongodb shows stable avg latencies. The latency starts to climbup a bit at higher offered load. But Postgres latency remains stable for some initial low loads and thenshoots up. Same is observed for Read. Mongodb latency measure is a bit higher for reads than updates.Conclusion – MongoDB actually does behave like an RDMS system for workload A. It shows overall goodperformance than PostgreSQL for workload A, as reading and writing is conducted within the usablememory, and hence high-performance is possible. As MongoDB makes use of memory mapped files fordata storage it shows fast performance and low latencies than SQL counterpart. PostgreSQL cannothandle updates after certain amount of offered load and reached a threshold and shoots up. Mongo isstable for both operations and shows extremely good performance due to mmap read and writes.
  5. 5. 2. Tier 2 – Scalability: Scaleup – Increase hardware, data size and workload proportionally. Measurelatency; should be constant for constant hardware, increase Offered Throughput (op/sec) untilsaturationa. UpdateUpdateRecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencyusMinLatencyMaxLatency1000 1000 default 1517 659.19 514 1660.19 57810602010000 1000 default 1148 871.08 504 1300.34 604 90786100000 1000 default 1504 664.89 475 2006.14 6433537971000000 1000 default 6893 145.07 494 8273.98 909309653RecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencymsMinLatencyMaxLatency1000 1000 default 1213 824.402 495 0.319 0 610000 1000 default 1191 839.63 458 0.364 0 58100000 1000 default 1198 834.724 488 0.247 0 41000000 1000 default 8801 113.623 501 5.92 0 316b. ReadReadRecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencyusMinLatencyMaxLatency1000 1000 default 1517 659.19 486 926.02 196 2817910000 1000 default 1148 871.08 496 518.19 183 25832100000 1000 default 1504 664.89 525463.691 200 182301000000 1000 default 6893 145.07 5064965.13 224 39424RecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencymsMinLatencyMaxLatency
  6. 6. 1000 1000 default 1213 824.402 505 0.411 0 6410000 1000 default 1191 839.63 542 0.25 0 6100000 1000 default 1198 834.724 512 0.347 0 621000000 1000 default 8801 113.623 499 9.18 0 1787Observation- This too behaves the same as the update operation. When we increase the record count,the Achieved Throughput (op/sec) and the Avg Latency (usec) remains stable until 1 million records. Butwhen we do transactions on 1 million records the Achieved Throughput (op/sec) plummets. Even theAvg Latency (usec) has a steep increase.Conclusion – The read and update for the workload A show similar results in terms of throughput. This isbecause the reads and updates have 50% distribution each. Here, if we compare the avg latencies forthe read and update operations, updates perform well than read operations.Compared to MongoDb – In lower record counts, Postgre max achieved throughput is lesser thanMongoDB. Even the Avg Latency (usec) of Postgre is much higher than MongoDB. MongoDB performsbetter for WorkloadA. But when Postgre performs on 1 million records even its throughput decline, itshows higher throughput and good latency measure than MongoDB.II. Workload B - read 95% update 5% (read intensive)1. Tier 1 – Performance Benchmarking: For constant hardware, increase Offered Throughput (op/sec)until saturationThroughput Performance MeasurePostgreSQL MongoDBOfferedThroughputAchievedThroughputAchievedThroughput100 97.8 96.41000 983.9 971.22500 2446 23825000 4659 4549.57500 6710 6587.610000 7540 8438.812500 9810 1015015000 11080 1156017500 11640 1300020000 12640 1398022500 13260 1567025000 14020 1631030000 15020 1776050000 15550 17900
  7. 7. Fig B.1: WorkloadB Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQL throughput is good for loweroffered throughput values. But it is not able to match up with the consistent increase inachieved throughput as mongoDB does. MongoDB shows higher achieved throughput values forhigher offered throughput.Conclusion – MongoDB actually does behave like an RDMS system for workload B. It showsoverall good performance than PostgreSQL for workload B, as reading and writing is conductedwithin the usable memory and use of memory mapped files for data storage which henceachieves high-performance. PostgreSQL show good performance for less offered load but as wegradually increase the offered load, its performance start to decline. On other hand, mongodbshows consistent growth in achieved throughput for WorkloadB (95%Read).a. Update Average LatencyUPDATE PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 1.919 0.861000 1.793 0.4892500 1.683 0.3565000 1.83 0.3767500 1.887 0.38310000 1.91 0.412500 1.87 0.436
  8. 8. 15000 1.98 0.4417500 2.54 0.5220000 3.292 0.6422500 3.6 0.7425000 4.23 1.05630000 4.95 1.21Fig B.2: WorkloadB Performance Benchmarking - Update Recordsb. Read Average LatencyREAD PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.947 0.4191000 0.812 0.3262500 0.869 0.2325000 0.88 0.2577500 0.95 0.2810000 0.96 0.29812500 0.972 0.3115000 0.979 0.31417500 1.008 0.31920000 1.26 0.43222500 2.289 0.79525000 2.805 1.0330000 3.466 1.45
  9. 9. Fig B.3: WorkloadB Performance Benchmarking - Read RecordsObservation – PostgreSQL show higher avg latencies for Workload B Updates & Reads compared toMongoDB. For updates we can see that Mongodb shows comparatively stable avg latencies. The latencystarts to climb up a bit at higher offered load. But Postgres latency remains stable for some initial lowloads during Reads and then shoots up. It shows consistent increase in avg latency for Updates though.Mongodb latency measure for Reads is better than Updates in this case.Conclusion – MongoDB actually does behave like an RDMS system for workload B. It shows overall goodperformance than PostgreSQL for workload B, as reading and writing is conducted within the usablememory, and hence high-performance is possible. As MongoDB makes use of memory mapped files fordata storage it shows fast performance and low latencies than SQL counterpart. Postgres show goodlatencies for Reads than MongoDB. But when the load increases to higher levels the latencies shoots up.In this case the update latencies for Postgre is consistently increased. Mongo is stable for bothoperations and shows extremely good performance due to mmap read and writes. Though for updatesMongodb update latencies are not that good as compared to latencies for Workload B. This may bebecause mongod process uses a modified reader/writer lock with dynamic yielding on page faults andlong operations. Any number of concurrent read operations are allowed, but a write operation can blockall other operations. Write lock acquisition is greedy and will prevent further read lock acquisitions untilfulfilled. Thus yielding by reads can be important. So it gives priority to Reads than Updates.2. Tier 2 – Scalability: Scaleup – Increase hardware, data size and workload proportionally. Measurelatency; should be constant for constant hardware, increase Offered Throughput (op/sec) untilsaturationa. UpdatePostgreRecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencyMinLatencyMaxLatency1000 1000 default 1219 820.34 51 0.803 0 9
  10. 10. 10000 1000 default 1211 825.76 48 0.437 0 5100000 1000 default 1201 832.63 40 1 0 151000000 1000 default 6710 149.03 55 19.2 0 146MongoRecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencyMinLatencyMaxLatency1000 1000 default 1219 820.34 51 0.803 0 910000 1000 default 1211 825.76 48 0.437 0 5100000 1000 default 1201 832.63 40 1 0 151000000 1000 default 6710 149.03 55 19.2 0 146b. ReadPostgreRecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencyMinLatencyMaxLatency1000 1000 default 1219 820.3 949 0.6 0 12610000 1000 default 1211 825.8 952 0.3 0 58100000 1000 default 1201 832.6 960 0.36 0 591000000 1000 default 6710 149 945 5.68 0 461MongoRecordCountOfferedThroughputThreads (DBper client)RuntimeAchievedThroughputOperationsAvgLatencyMinLatencyMaxLatency1000 1000 default 1219 820.3 949 0.6 0 12610000 1000 default 1211 825.8 952 0.3 0 58100000 1000 default 1201 832.6 960 0.36 0 591000000 1000 default 6710 149 945 5.68 0 461Observation- This too behaves the same as the update operation. When we increase the record count,the Achieved Throughput (op/sec) and the Avg Latency (usec) remains stable until 1 million records. Butwhen we do transactions on 1 million records the Achieved Throughput (op/sec) plummets. Even theAvg Latency (usec) has a steep increase.
  11. 11. Conclusion – The read and update for the workload B show similar results in terms of throughput. Hereupdate operations show more latency than reads. Hence postgre performs well in reads for workloadBCompared to MongoDb –Posgre shows slightly better achieved throughput values than Mongodb. TheAvg Latency (usec) of Postgre is higher than MongoDB for updates but Postgre does really well during95% reads. It shows good throughput and even less avg latency. According the above result Posgreperforms well for WorkloadB for reads when it scales up.III. Workload C – Read 100%1. Tier 1 – Performance Benchmarking: For constant hardware, increase Offered Throughput(op/sec) until saturationThroughput Performance MeasurePostgreSQL MongoDBOfferedThroughputAchievedThroughputAchievedThroughput100 98.8 97.31000 996.9 9852500 2467 2409.85000 4780 46807500 6890 890210000 9980 1020317500 11890 1363022500 13560 1570830000 15390 1795050000 16100 17084100000 15930 17401
  12. 12. Fig C.1: WorkloadC Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQL throughput is good for lower offeredthroughput values. But it is not able to match up with the consistent increase in achieved throughput asmongoDB does. MongoDB shows higher achieved throughput values for higher offered throughput.Conclusion – MongoDB actually does behave like an RDMS system for workload C. It shows overallgood performance than PostgreSQL for workload C, as reading and writing is conducted within theusable memory and use of memory mapped files for data storage which hence achieves high-performance. Postgre is better than Mongdb for lower load values. But show slight lower performancethan Mongod for higher offered load. It seems Postgre in this case for Workload C tries to match upwith MongoDB for 100% Reads. And it succeeds in that!a. Average Read LatencyREAD PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.442 0.3181000 0.395 0.362500 0.385 0.335000 0.393 0.337500 0.407 0.3310000 0.457 0.34417500 0.466 0.3522500 0.578 0.4930000 0.87 0.5350000 1.08 0.63
  13. 13. Fig C.2: WorkloadC Performance Benchmarking - Read RecordsObservation – PostgreSQL show higher avg latencies for Workload C Reads compared to MongoDB. Wecan see that Mongodb shows stable avg latencies for low loads. The latency starts to climb up a bit athigher offered load. But Postgres latency remains stable for some initial low loads and then shoots up.Conclusion – MongoDB actually does behave like an RDMS system for workload C. It shows overall goodperformance than PostgreSQL for workload C, as reading and writing is conducted within the usablememory, and hence high-performance is possible. As MongoDB makes use of memory mapped files fordata storage it shows fast performance and low latencies than SQL counterpart. PostgreSQL cannothandle reads after certain amount of offered load and reached a threshold and shoots up. Mongo isstable and the latencies does not show any linear rise. Postgre tries to much with mongodb but fails todo that for higher loads.Tier 2: Scalability : The results pretty similar to WorkloadB with 95% Reads.IV. Workload D – read 95% insert 5% (read intensive)The Output for this Workload is similar to Workload C output. So we can refer the results fromWorkload C for Workload D. For Insert query performance, please refer the figure below –Throughput Performance MeasureOfferedThroughputAchievedThroughputAchievedThroughput100 98.74 99.561000 997.2 995.445000 3785.29 4892.8410000 3761.37 9035.05650000 3752.9 9172.62
  14. 14. Fig D.1: WorkloadA Offered Throughput vs Achieved ThroughputObservation- In the above graph PostgreSQL is not able to match up with the consistent increase inachieved throughput as mongoDB does. MongoDB shows higher achieved throughput values for higheroffered throughput. The achieved throughput increases and then gets stable at some threshold points.The threshold point for postgre is less than mongodb for workload D.Conclusion – MongoDB actually does behave like an RDMS system for workload D. It shows overall goodperformance than PostgreSQL for workload D, as reading and writing is conducted within the usablememory and use of memory mapped files for data storage which hence achieves high-performance.Mongodb performs effeciently for inserts as well than SQL because, MongoDB uses a format calledBSON which is a binary representation of this data. MongoDB is quite fast at a series of singleton insertsas it is a document oriented DBMS.a. InsertINSERTPostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 1.05 0.221000 1.094 0.235000 1.12 0.2410000 1.636 0.4250000 3.96 1.34
  15. 15. Fig D.2: WorkloadD Performance Benchmarking - Insert Recordsb. ReadREADPostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.2 0.081000 0.21 0.0895000 0.215 0.0910000 0.5 0.1250000 1.42 0.46Fig D.3: WorkloadD Performance Benchmarking - Read Records
  16. 16. Observation – PostgreSQL show higher avg latencies for Workload D Inserts & Reads compared toMongoDB. MongoDB insert and reads show similar avg latencies and are stable and then increaseslightly. Postgre shows linear increase in latencies when load is increased.Conclusion – MongoDB actually does behave like an RDMS system for workload D. It shows overall goodperformance than PostgreSQL for workload D, as reading and writing is conducted within the usablememory, and hence high-performance is possible. As MongoDB makes use of memory mapped files fordata storage it shows fast performance and low latencies than SQL counterpart. Mongodb performseffeciently for inserts as well than SQL because, MongoDB uses a format called BSON which is a binaryrepresentation of this data. MongoDB is quite fast at a series of singleton inserts as it is a documentoriented DBMS. Postgres have higher latencies for inserts as indexing is done on a field.Here we can see -a. Only Insert - 1,00,000 recordsPostgrSQL =>RecordCountOfferedThroughput(op/sec)Threads(DB perclient)Runtime(msec)AchievedThroughput(op/sec) OperationsAvgLatency(usec)MinLatency(usec)MaxLatency(usec)100000 1000 default 117366 852.03 100000 1.16148 447 308977Fig 1.d.1: WorkloadD Performance Benchmarking – Insert Bulk Records PostgreSQLMongoDBRecordCountOfferedThroughput (op/sec)Threads(DB perclient)Runtime(msec)AchievedThroughput (op/sec)OperationsAvgLatency(usec)MinLatency(usec)MaxLatency(usec)100000 1000 default 100189 998.113 1000000.056690 591Compared to MongoDb – Posgre shows has a slight less throughput for inserting 1 Lac records. Andeven the latency is higher than mongodb as index is created on a key. Mongodb shows good resultsfor inserting records.V. Workload E – scan 95% insert 5% (scan intensive)1. Tier 1 – Performance Benchmarking: For constant hardware, increase Offered Throughput(op/sec) until saturationThroughput Performance MeasureOfferedThroughputAchievedThroughputAchievedThroughput100 99.71 99.541000 917.29 815.952000 1989.57 853.435000 2287.2 894.7110000 9511.6 809.1730000 11755.3 824.34
  17. 17. 50000 13073.9 903Fig E.1: WorkloadE Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQL throughput is performance veryefficiently than Mongodb. Its indexing helps it a lot to achieve higher throughput. Mongodbperformance cannot even match SQL for scan opeartions.Conclusion – Postgres shows better results as it has indexing on a key which helps to search efficientlythan mongodb. We haven’t created indexes on Mongodb for this experiment. May be mongodb will dowell if indexes are created. The throughput remains stable at around 900 for mongodb. Postgre showlinear increase in the throughput as the load increases.a. Scan 95%SCANREAD PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.227 1.121000 0.349 1.132000 0.391 1.25000 0.736 1.2510000 0.819 1.3430000 0.945 1.5350000 1.02 2.57
  18. 18. Fig E.2: WorkloadE Performance Benchmarking - Scan RecordsObservation – PostgreSQL show very stable and low avg latency than mongodb. Mongodb latencyincreases gradually as load increase.Conclusion – Mongodb shows superb performance for scan intensive operations. Indexing helps Postgreto maintain high achieved throughput and low latencies. Mongodb struggles to keep up with postgre forscan operations.VI. Workload F – Read 50% & ReadModifyWrite 50%1. Tier 1 – Performance Benchmarking: For constant hardware, increase Offered Throughput(op/sec) until saturationThroughput Performance MeasurePostgreSQL MongodbOfferedThroughputAchievedThroughputAchievedThroughput100 99.96 99.541000 971.28 995.485000 1025.97 4892.3610000 1003.81 7392.0750000 1027.96 7237.98
  19. 19. Fig F.1: WorkloadF Offered Throughput vs Achieved ThroughputObservation- In the above graph, we can observe that PostgreSQL throughput show poor performancethan Mogodb. Mongodb throughput increases and becomes stable higher than postgre.Conclusion – MongoDB actually does behave like an RDMS system for workload F. It shows overall goodperformance than PostgreSQL for workload F, as reading and writing is conducted within the usablememory and use of memory mapped files for data storage which hence achieves high-performance.Postgre follows consistency (atomic operations) in read modifies and hence hampers the throughput.a. Read-Modify-WriteREAD/ModifiedREAD PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.0013 1.341000 0.0011 0.215000 0.0015 0.1410000 0.0018 0.1550000 0.0014 0.16
  20. 20. Fig A.2: WorkloadF Performance Benchmarking – Read/Modify Recordsb. ReadREADREAD PostgreSQL MongoDBOfferedThroughputAvgLatencyAvgLatency100 0.951 0.651000 0.517 0.085000 0.514 0.0710000 0.519 0.0750000 0.507 0.08Fig A.3: WorkloadF Performance Benchmarking – Read Records
  21. 21. Observation – Postgre show very low latencies for read/modifies than Mongodb. Mongodb initially ithas high latency but then drops and remains stable. But mongodb has higher latencies than sql. Forreads, mongo shows better latency values than postgre.Conclusion – The mongod process uses a modified reader/writer lock with dynamic yielding on pagefaults and long operations. Any number of concurrent read operations is allowed, but a write operationcan block all other operations. Write lock acquisition is greed and a pending write lock acquisition willprevent further read lock acquisitions until fulfilled. Thus yielding by reads can be important. Hencemongodb show higher latencies than postgre. Reads have higher priority than updates and modify andhence show lower latencies than postgre.Conclusion -In this experiment, we did performance analysis of a NoSQL DBMS with a SQL DBMS. Wecompared PostregreSQL with MongoDB on Tier: Performance and Tier2: Scalability. MongoDB showedbetter performance measure in most cases like workload A , B, C,D and F. PostgreSQL tried to matchwith MongoDB in Workload B and C. Postgre showed way better performance than Mongodb forWorkload E (Scan) due to indexing. Hence we can conclude that MongoDB is in general a better DBMSsystem if consistency and atomicity is not main primary goal. In transactional systems like Banking weneed to care of consistency and it will a huge task to measure and compare the performance andbehavior of NoSQL and SQL DBMS. But on basis of this experiment, in that case SQL will performsuperior than NOSQL it seems. Anways, for a small scale application which can scale up in future,NoSQL’s are very good choice to design the DBMS architecture. We can think of using Documen typeand key value pair stores for such applications than RDMS which involves indexing and joins which tendto show higher average latency value as seen in this experiment.In last three assignments we have even learnt about how to use YCSB tool to observe thetradeoffs between the write and read performance of database systems NoSQL and SQL under differentkind of workloads.

×