0
1JUNE 2014
Performance Tuning
on the Fly at CMP.LY
Michael De Lorenzo
CTO, CMP.LY Inc.
michael@cmp.ly
@mikedelorenzo
2JUNE 2014
Agenda
• CMP.LY and CommandPost
• What is MongoDB Management Service?
• Performance Tuning
• MongoDB Issues we’...
3JUNE 2014
A venture-funded NYC startup that offers proprietary social media, monitoring,
measurement, insight and complia...
4JUNE 2014
Use CommandPost to:
• Track and measure cross-platform in real-time
• Identify and attribute high-value engagem...
5JUNE 2014
What is MongoDB
Management Service?
6JUNE 2014
MongoDB Management Service
• Free MongoDB Monitoring
• MongoDB Backup in the Cloud
• Free Cloud service or Avai...
7JUNE 2014
Who Is MMS for?
• Developers
• Ops Team
• MongoDB Technical Service Team
8JUNE 2014
Performance Tuning
9JUNE 2014
How To Do Performance Tuning?
• Assess the problem and establish acceptable behavior.
• Measure the performance...
10JUNE 2014
What We’ve Faced
11JUNE 2014
Issues We’ve Faced
• Concurrency Issues
• Slow response times and delayed writes
• Querying without indexes
• ...
12JUNE 2014
Concurrency
Slow responses and delayed writes
13JUNE 2014
Concurrency
• What is it?
• How did it affect us?
• How did MMS help identify it?
• How did we diagnose the is...
14JUNE 2014
Concurrency in MongoDB
• MongoDB uses a readers-writer lock
• Many read operations can use a read lock
• If a ...
15JUNE 2014
How Did This Affect Us?
• Slow API response times due to slow database operations
• Delayed writes
• Backed up...
16JUNE 2014
MMS: Identify Concurrency Issues
17JUNE 2014
Lock % Greater than 100%?!?!?
• Global lock percentage is a derived metric:
% of time in global lock (small nu...
18JUNE 2014
Diagnosis
• Identified the write-heavy collections in our applications
• Used application logs to identify slo...
19JUNE 2014
Our Remedies
• Schema changes
• Message queues
• Multiple databases
• Sharding
20JUNE 2014
Schema Changes
• Changed our schema
• Allowed for atomic updates
• Customized documents’ _id attribute
• Lever...
21JUNE 2014
Normalized Schema
// Social Content Collection
{
_id: “12345”,
_type: “tweet”,
text: “Welcome to #MongoDBWorld...
22JUNE 2014
Denormalized Schema
// Social Content Collection
{
"_id": “tweet_123456789”,
“text”: 'Welcome to #MongoDBWorld...
23JUNE 2014
Modeling for Atomic Operations
Document
{
"_id": “tweet_123456789”,
"text": "Welcome to #MongoDBWorld!",
"twit...
24JUNE 2014
Message Queues
• Controlled writes to specific collections using Pub/Sub
• We chose Amazon SQS
• Other options...
25JUNE 2014
Using Multiple Databases
• As of version 2.2, MongoDB implements locks at a per database granularity for
most ...
26JUNE 2014
Using Sharding
• Improves concurrency by distributing databases across multiple mongod
instances
• Locks are p...
27JUNE 2014
Lock %: Today
28JUNE 2014
Queries without Indexes
Slow responses and timeouts
29JUNE 2014
Indexing
• What is it?
• How did it affect us?
• How did MMS help identify it?
• How did we diagnose the issue...
30JUNE 2014
Indexing with MongoDB
• Support for efficient execution of queries
• Without indexes, MongoDB must scan every ...
31JUNE 2014
How Did this Affect Us?
• Our web apps became slow
• Queries began to timeout
• Longer operations mean longer ...
32JUNE 2014
MMS: Identifying Indexing Issues
Page Faults
• The number of times that
MongoDB requires data
not located in p...
33JUNE 2014
Diagnosis
• Log Analysis
• Use mtools
A collection of scripts to parse and visualize MongoDB log files develop...
34JUNE 2014
Diagnosis
• Monitoring application logs
• Enabling ‘notablescan’ option in development and testing versions of...
35JUNE 2014
The MongoDB Profiler
• Collects fine grained data about MongoDB write operations, cursors, database
commands o...
36JUNE 2014
Our Remedies
• Add indexes!
• Make sure queries are covered
• Utilize the projection specification to limit fi...
37JUNE 2014
Adding Indexes
• Improved performance for common queries
• Alleviates the need to go to disk for many operatio...
38JUNE 2014
Projection Specification
Controls the amount of data that needs to be (de-)serialized for use in your app
• We...
39JUNE 2014
Page Faults: Today
40JUNE 2014
Rising Replication Lag +
Falling oplog Window
41JUNE 2014
Replication
• What is it?
• How did it affect us?
• How did MMS help identify it?
• How did we diagnose the is...
42JUNE 2014
What is Replication?
• A replica set is a group of mongod
processes that maintain the same data
set.
• Replica...
43JUNE 2014
What Is the Oplog?
• A special capped collection that keeps a rolling record of all operations that
modify the...
44JUNE 2014
What is Replication Lag?
• A delay between an operation on the primary and the application of that
operation f...
45JUNE 2014
How did this affect us?
• Degraded overall health of our production deployment.
• Distributed reads are no lon...
46JUNE 2014
Identifying Replication Lag Issues
with MMS
The Replication Lag chart displays the lag for your deployment
47JUNE 2014
Diagnosis
• Possible causes of replication lag include network latency, disk throughput,
concurrency and/or ap...
48JUNE 2014
Concurrency/Write Concern
• Our applications apply many updates very quickly
• All operations need to be repli...
49JUNE 2014
Concurrency Wasn’t the Issue
Lock Percentage
50JUNE 2014
Operation Size Was the Issue
Collection A (most active)
Total Updates: 3,373
Total Size of updates: 6.5 GB
Act...
51JUNE 2014
Fast Growing oplog causes issues
Replication oplog Window – approximate hours available in the primary’s oplog
52JUNE 2014
How We Fixed It
• Changed our schema
• Changed the types of updates that were made to documents
• Both allowed...
53JUNE 2014
Replication Lag: Today
54JUNE 2014
oplog Window: Today
55JUNE 2014
Keeping Your Deployment
Healthy
56JUNE 2014
MMS Alerts
57JUNE 2014
Watch for Warnings
• Be warned if you are
• Running outdated versions
• Have startup warnings
• If a mongod is...
58JUNE 2014
MMS Backups
• Engineered by MongoDB
• Continuous backup with point-in-time recovery
• Fully managed backups
59JUNE 2014
Using MMS Backups
• Seeding new secondaries
• Repairing replica set members
• Development and testing database...
60JUNE 2014
Summary
• Know what’s expected and “normal” in your systems
• Know when and what changes in your systems
• Uti...
61JUNE 2014
Questions?
Michael De Lorenzo
CTO, CMP.LY Inc.
michael@cmp.ly
@mikedelorenzo
Upcoming SlideShare
Loading in...5
×

Performance Tuning On the Fly at CMP.LY Using MongoDB Management Service

584

Published on

Using MongoDB, you’re able to develop and deploy applications quickly. But how do you maintain high performance when you have a small team and are developing at a breakneck pace? At CMP.LY, we tune our applications on the fly using key performance metrics from MongoDB Management Service. Identifying bottlenecks before they become production issues let’s us keep our focus on our application. You’ll walk away from this talk with a clear understanding of how to leverage MMS and key performance metrics to keep your application, and team, humming as your MongoDB usage grows.

Published in: Software, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
584
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Free MongoDB Monitoring
    - mongodb specific metrics, visualization of performance, custom alerting
    Backup
    - industrial strength, point-in-time recovery, free usage tier
  • Developers, what we’re focused on today – track bottlenecks
    Ops team :: great for small teams where your developers are also part of your ops team (DevOps) – monitor health of clusters, backup dbs, automate updates and add capacity
    MongoDB technical service team :: helps them help you

    Important for us because we maintain a small tech team
  • PRO-TIP: Know what is “normal” for you system.

    Know what changed when something happens, what do you expect to be normal behavior, what are you normal MMS metrics
  • readers-writer lock allows concurrent read access to the db,
    but exclusive access to a single write
    “Writer-greedy” - When both a read and write are waiting for a
    lock, MongoDB grants the lock to the write.
    The exclusivity of write locks is one of the keys to why getting
    our lock % under control is so important.
  • Lock %
    time spent in write lock state; sum of global lock + hottest database at that time, can make value > 100%

    Our Issue: Primary database maintaining a write lock of 150-175% of the time
  • Global lock percentage has remained about the same
    Primary client-facing database has seen lock % drop
  • Developed by a MongoDB engineer
  • - Purple bar indicates downtime
  • - Alerts for down hosts, down agents and more
  • - According to Technical Services, In many cases, fixing warnings will fix issues
  • Transcript of "Performance Tuning On the Fly at CMP.LY Using MongoDB Management Service"

    1. 1. 1JUNE 2014 Performance Tuning on the Fly at CMP.LY Michael De Lorenzo CTO, CMP.LY Inc. michael@cmp.ly @mikedelorenzo
    2. 2. 2JUNE 2014 Agenda • CMP.LY and CommandPost • What is MongoDB Management Service? • Performance Tuning • MongoDB Issues we’ve faced • Slow response times and delayed writes • Unindexed queries • Rising Replication Lag + Falling oplog Window • Keep your deployment healthy with MMS • Using MMS Alerts • Using MMS Backups
    3. 3. 3JUNE 2014 A venture-funded NYC startup that offers proprietary social media, monitoring, measurement, insight and compliance solutions for Fortune 100 A Monitoring, Measurement & Insights (MMI) tool for managed social communications.
    4. 4. 4JUNE 2014 Use CommandPost to: • Track and measure cross-platform in real-time • Identify and attribute high-value engagement • Analyze and segment engaged audience • Optimize content and engagement strategies • Address compliance needs
    5. 5. 5JUNE 2014 What is MongoDB Management Service?
    6. 6. 6JUNE 2014 MongoDB Management Service • Free MongoDB Monitoring • MongoDB Backup in the Cloud • Free Cloud service or Available to run On-Prem for Standard or Enterprise Subscriptions • Automation coming soon—FTW! Ops Makes MongoDB easier to use and manage
    7. 7. 7JUNE 2014 Who Is MMS for? • Developers • Ops Team • MongoDB Technical Service Team
    8. 8. 8JUNE 2014 Performance Tuning
    9. 9. 9JUNE 2014 How To Do Performance Tuning? • Assess the problem and establish acceptable behavior. • Measure the performance before modification. • Identify the bottleneck. • Remove the bottleneck. • Measure performance after modification to confirm. • Keep it or revert it and repeat. Adapted from [http://en.wikipedia.org/wiki/Performance_tuning]
    10. 10. 10JUNE 2014 What We’ve Faced
    11. 11. 11JUNE 2014 Issues We’ve Faced • Concurrency Issues • Slow response times and delayed writes • Querying without indexes • Slow reads, timeouts • Increasing Replication Lag + Plummeting oplog Window
    12. 12. 12JUNE 2014 Concurrency Slow responses and delayed writes
    13. 13. 13JUNE 2014 Concurrency • What is it? • How did it affect us? • How did MMS help identify it? • How did we diagnose the issue in our app and fix it? • Today
    14. 14. 14JUNE 2014 Concurrency in MongoDB • MongoDB uses a readers-writer lock • Many read operations can use a read lock • If a write lock exists, a single write lock holds the lock exclusively • No other read or write operations can share the lock • Locks are “writer-greedy”
    15. 15. 15JUNE 2014 How Did This Affect Us? • Slow API response times due to slow database operations • Delayed writes • Backed up queues
    16. 16. 16JUNE 2014 MMS: Identify Concurrency Issues
    17. 17. 17JUNE 2014 Lock % Greater than 100%?!?!? • Global lock percentage is a derived metric: % of time in global lock (small number) + % of time locked by hottest (“most locked”) database • Data is sampled and combined, it is possible to see values over 100%.
    18. 18. 18JUNE 2014 Diagnosis • Identified the write-heavy collections in our applications • Used application logs to identify slow API responses • Analyzed MongoDB logs to identify slow database queries
    19. 19. 19JUNE 2014 Our Remedies • Schema changes • Message queues • Multiple databases • Sharding
    20. 20. 20JUNE 2014 Schema Changes • Changed our schema • Allowed for atomic updates • Customized documents’ _id attribute • Leveraged existing index on _id attribute
    21. 21. 21JUNE 2014 Normalized Schema // Social Content Collection { _id: “12345”, _type: “tweet”, text: “Welcome to #MongoDBWorld!” twitter_user: “mongodb” } // Campaign Collection { _id: “mongodbworld_campaign”, name: “MongoDB World” } // Campaign Content Collection (joins content + campaigns) { campaign_id: “#mongodbworld_campaign”, content_id: “12345” }
    22. 22. 22JUNE 2014 Denormalized Schema // Social Content Collection { "_id": “tweet_123456789”, “text”: 'Welcome to #MongoDBWorld!' “twitter_user”: 'mongodb', “campaigns”: [“mongodbworld_campaign”] }
    23. 23. 23JUNE 2014 Modeling for Atomic Operations Document { "_id": “tweet_123456789”, "text": "Welcome to #MongoDBWorld!", "twitter_user": "mongodb", “campaigns": [ ] } Update Operation db.social_content.update( { _id: “tweet_123456789” }, { $addToSet: { campaigns: ”mongodbworld_campaign” } } ); Result WriteResult({ "nMatched": 1,"nUpserted”:0,"nModified": 1 })
    24. 24. 24JUNE 2014 Message Queues • Controlled writes to specific collections using Pub/Sub • We chose Amazon SQS • Other options include Redis, Beanstalkd, IronMQ or any other message queue • Created consistent flow of writes versus bursts • Reduced length and frequency of write locks by controlling flow/speed of writes
    25. 25. 25JUNE 2014 Using Multiple Databases • As of version 2.2, MongoDB implements locks at a per database granularity for most read and write operations • Planned to be at the document level in version 2.8 • Moved write-heavy collections to new (separate) databases
    26. 26. 26JUNE 2014 Using Sharding • Improves concurrency by distributing databases across multiple mongod instances • Locks are per-mongod instance
    27. 27. 27JUNE 2014 Lock %: Today
    28. 28. 28JUNE 2014 Queries without Indexes Slow responses and timeouts
    29. 29. 29JUNE 2014 Indexing • What is it? • How did it affect us? • How did MMS help identify it? • How did we diagnose the issue in our app and fix it? • Today
    30. 30. 30JUNE 2014 Indexing with MongoDB • Support for efficient execution of queries • Without indexes, MongoDB must scan every document • Example Wed Jul 17 13:40:14 [conn28600] query x.y [snip] ntoreturn:16 ntoskip:0 nscanned:16779 scanAndOrder:1 keyUpdates:0 numYields: 906 locks(micros) r:46877422 nreturned:16 reslen:6948 38172ms 38 seconds! Scanned 17k documents, returned 16 • Create indexes to cover all queries, especially support common and user-facing • Collection scans can push entire working set out of RAM
    31. 31. 31JUNE 2014 How Did this Affect Us? • Our web apps became slow • Queries began to timeout • Longer operations mean longer lock times
    32. 32. 32JUNE 2014 MMS: Identifying Indexing Issues Page Faults • The number of times that MongoDB requires data not located in physical memory, and must read from virtual memory.
    33. 33. 33JUNE 2014 Diagnosis • Log Analysis • Use mtools A collection of scripts to parse and visualize MongoDB log files developed by MongoDB Engineer Thomas Rueckstiess. • mlogfilter • filter logs for slow queries, collection scans, etc. • mplotqueries • graph query response times and volumes • https://github.com/rueckstiess/mtools
    34. 34. 34JUNE 2014 Diagnosis • Monitoring application logs • Enabling ‘notablescan’ option in development and testing versions of apps • MongoDB profiling
    35. 35. 35JUNE 2014 The MongoDB Profiler • Collects fine grained data about MongoDB write operations, cursors, database commands on a running mongod instance. • Default slowOpThreshold value is 100ms, can be changed from the Mongo shell • When enabled, profiling has a minor effect on performance
    36. 36. 36JUNE 2014 Our Remedies • Add indexes! • Make sure queries are covered • Utilize the projection specification to limit fields (data) returned
    37. 37. 37JUNE 2014 Adding Indexes • Improved performance for common queries • Alleviates the need to go to disk for many operations
    38. 38. 38JUNE 2014 Projection Specification Controls the amount of data that needs to be (de-)serialized for use in your app • We used it to limit data returned in embedded documents and arrays db.content.find( { tweet_id: ’12345678' }, { text: 1, screen_name: 1 });
    39. 39. 39JUNE 2014 Page Faults: Today
    40. 40. 40JUNE 2014 Rising Replication Lag + Falling oplog Window
    41. 41. 41JUNE 2014 Replication • What is it? • How did it affect us? • How did MMS help identify it? • How did we diagnose the issue in our app? • How did we fix it? • Today
    42. 42. 42JUNE 2014 What is Replication? • A replica set is a group of mongod processes that maintain the same data set. • Replica sets provide redundancy and high availability, and are the basis for all production deployments
    43. 43. 43JUNE 2014 What Is the Oplog? • A special capped collection that keeps a rolling record of all operations that modify the data stored in your databases. • Operations are first applied on the primary and then recorded to its oplog. • Secondary members then copy and apply these operations in an asynchronous process.
    44. 44. 44JUNE 2014 What is Replication Lag? • A delay between an operation on the primary and the application of that operation from the oplog to the secondary. • Effects of excessive lag • “Lagged” members ineligible to quickly become primary • Increases the possibility that distributed read operations will be inconsistent.
    45. 45. 45JUNE 2014 How did this affect us? • Degraded overall health of our production deployment. • Distributed reads are no longer eventually consistent. • Unable to bring new secondary members online. • Caused MMS Backups to do full re-syncs.
    46. 46. 46JUNE 2014 Identifying Replication Lag Issues with MMS The Replication Lag chart displays the lag for your deployment
    47. 47. 47JUNE 2014 Diagnosis • Possible causes of replication lag include network latency, disk throughput, concurrency and/or appropriate write concern • Size of operations to be replicated • Confirmed Non-Issues for us • Network latency • Disk throughput • Possible Issues for us • Concurrency/write concern • Size of op is an issue because entire document is written to oplog
    48. 48. 48JUNE 2014 Concurrency/Write Concern • Our applications apply many updates very quickly • All operations need to be replicated to secondary members • We use the default write concern—Acknowledge (w:1) • The mongod confirms receipt of the write operation • Allows clients to catch network, duplicate key and other errors
    49. 49. 49JUNE 2014 Concurrency Wasn’t the Issue Lock Percentage
    50. 50. 50JUNE 2014 Operation Size Was the Issue Collection A (most active) Total Updates: 3,373 Total Size of updates: 6.5 GB Activity accounted for nearly 87% of total traffic Collection B (next most active) Total Updates: 85,423 Total Size of updates: 740 MB
    51. 51. 51JUNE 2014 Fast Growing oplog causes issues Replication oplog Window – approximate hours available in the primary’s oplog
    52. 52. 52JUNE 2014 How We Fixed It • Changed our schema • Changed the types of updates that were made to documents • Both allowed us to utilize atomic operations • Led to smaller updates • Smaller updates == less oplog space used
    53. 53. 53JUNE 2014 Replication Lag: Today
    54. 54. 54JUNE 2014 oplog Window: Today
    55. 55. 55JUNE 2014 Keeping Your Deployment Healthy
    56. 56. 56JUNE 2014 MMS Alerts
    57. 57. 57JUNE 2014 Watch for Warnings • Be warned if you are • Running outdated versions • Have startup warnings • If a mongod is publicly visible • Pay attention to these warnings
    58. 58. 58JUNE 2014 MMS Backups • Engineered by MongoDB • Continuous backup with point-in-time recovery • Fully managed backups
    59. 59. 59JUNE 2014 Using MMS Backups • Seeding new secondaries • Repairing replica set members • Development and testing databases • Restores are free!
    60. 60. 60JUNE 2014 Summary • Know what’s expected and “normal” in your systems • Know when and what changes in your systems • Utilize MMS alerts, visualizations and warnings to keep things running smoothly
    61. 61. 61JUNE 2014 Questions? Michael De Lorenzo CTO, CMP.LY Inc. michael@cmp.ly @mikedelorenzo
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×