Mongodb improvement by using indexes

1. MongoDB
Performance

2. 1. G3 use case
2. How to detect slow queries
3. How to optimize for speed
4. Discussion
Summary

3. I. G3 Use case
● ~ X M records
● Insert XX K records/day
● Bidding system requirements:
○ Update result near real-time
○ Use updated result for bidding process
○ Many campaigns run at the same time
-> Many queries at the same time, in a short period
-> Many queries have the same result (May be)
● Inactive clients is not used for pushing process

4. I. G3 Use case
● Before:
4 instances x 4 consumers

5. I. G3 Use case
● After:
(1 instance x 4 consumers)

6. How to detect slow queries
● Mongodb logs
○ /var/log/mongodb/mongodb.log
● Use explain function
○ totalDocsExamined
○ totalDocsExamined
● Visualize query explain on Studio 3T

7. explain() function
● How many documents were scanned
● How many documents were returned
● Which index was used
● How long the query took to be executed
● Which alternative execution plans were evaluated

8. Example for explain()
db.users.createIndex({ "amount": 1 })
db.users.find({ amount : { $in : [ 15, 305, 292 ] } }).explain(true)
Result:
"nReturned" : 4.0,
"executionTimeMillis" : 0.0,
"totalKeysExamined" : 7.0,
"totalDocsExamined" : 4.0,

9. Example for explain()
db.users.createIndex({ "amount": 1 })
db.users.find({ amount : { $in : [ 5, 10, 15 ] } }).explain(true)
Result:
"nReturned" : 4.0,
"executionTimeMillis" : 0.0,
"totalKeysExamined" : 5.0,
"totalDocsExamined" : 4.0,

10. Example for explain()
db.users.createIndex({ "amount":"hashed"})
db.users.find({ amount : { $in : [ 5, 10, 15 ] } }).explain(true)
Result:
"nReturned" : 4.0,
"executionTimeMillis" : 0.0,
"totalKeysExamined" : 7.0,
"totalDocsExamined" : 4.0,

11. Index types
● _id
● Single field indexes
● MultiKey indexes
● Compound indexes
● Hash indexes
● Text sparse
● TTL indexes
● Unique
● Partial (filtered)
● GeoIndexes (GeoHaystack, 2d spherical, flat 2d indexes)

12. How does index work?

13. Costs of maintaining indexes
● Each Index adds cost to the writing process
● A lot of indexes will probably slow down the write performance.

14. Default index _id
● _id is the default primary key in MongoDB.
● _id is not a clustered index and the database must perform another operation so as to
read all the values of the document.

15. ● Single field is a simple index and it indexes only one field at a time.
● Creating a single field index will help the query optimizer find the desired
document quickly.
Single field index

16. Compound indexes
● Compound indexes can support queries that match on multiple fields
● If we filter by field A and field B and only the field A is indexed, we may need to open a lot
of documents to read the field B. On the other hand, if field B is already indexed, there is
no need to do so
● Use the higher cardinality field as first field in the btree.

17. Multikey Indexes
● Multikey Indexes are used when the field value is an array
● For the document { likes : ['mongodb', 'mysql'] }, the multikey will generate 2 different
index keys pointing to the same document.

18. ● Support text search queries on string content. text indexes can include any field
whose value is a string or an array of string elements.
● A collection only create a text indexes, but a compound index can include a text
index key.
● Supported Languages and Stop Words, Scoring, Stemming
Text Index

19. ● Hash indexes are commonly used in shards to create random keys for the writes
and increase the write performance
● Cannot find by range
● Document reference https://www.slideshare.net/daumdna/mongodb-scaling-write-performance
Hashed Index

20. Other index
● TTL indexes : is a separate thread that runs periodically (usually every minute) and
scans a collection, that has a TTL index defined, for any expired documents and
removes them in the background.
● Unique indexes : A unique index ensures that the indexed fields do not store
duplicate values
● Partial (filtered) indexes: only indexes the documents that meet a specified filter
expression.
● GeoIndexes (GeoHaystack, 2d spherical, flat 2d indexes)

21. Performance tips for MongoDB
- Should not use queries: $ne / $nin / $or
- Ensure Indexes Fit RAM

22. Performance tips for MongoDB
- Build the index in the background to not affect the performance of MongoDB.
- Regex should not be used with large collection
Example: Link

23. Performance tips for MongoDB
● Slow Aggregates
○ Aggregates in Mongo are super powerful! However unlike other queries they will be
touching most data in the Database (because you probably are trying to generate
aggregate data or some kind of report). So they can easily become the bottleneck of
your service.
○ MongoDB has a Hard limit for the data that is passed in the aggregation pipeline of
100MB. You can turn on Disk usage to go around this but it will really slow down
everything. ↗
● The $match and $sort pipeline operators can take advantage of an index
when they occur at the beginning of the pipeline.
-> Hạn chế sử dụng Aggregation và thay bằng việc tính trước ra một collection khác.
Reference document

24. Selective index
Rules of Compound Index
- Equality fields before range fields
- Order equality fields
from most selective (most unique)
to least selective (least unique)
- Add sorted field to the end of index

25. Selective index - demo
Query
{
“active”: true
“enqueued_time”: {
$lte: new Date(1565756739076)
},
“modified_at”: {
$lte: new Date(1565756739076)
},
“country_code”: “VN”
}
Sort: { “enqueued_time”: 1 }
Data range
active: true/false
country_code: ~150 countries (VN, ID, TH)
enqueued_time: timestamp
modified_at: timestamp
Index
{ “country_code”: 1, “active”: 1, “enqueued_time”: 1 }

26. Selective index - demo
Using Partial Indexes
Only index the documents in a collection that meet a specified filter expression
Index
{ “country_code”: 1, “active”: 1, “enqueued_time”: 1 }
Query
{
“active”: true
…
}
New Index
{ “country_code”: 1, “enqueued_time”: 1 }
{ partialFilterExpression: { active: true } }

27. Selective index - demo
Using Partial Indexes
Only index the documents in a collection that meet a specified filter expression
Index
{ “country_code”: 1, “active”: 1, “enqueued_time”: 1 }
Query
{
“active”: true
…
}
New Index
{ “country_code”: 1, “enqueued_time”: 1 }
{ partialFilterExpression: { active: true } }

28. Keep in mind
● Use MongoDB logs, query explain… to detect slow query
● Indexes support the efficient execution of queries in MongoDB
● Should create index in background
● Ensure indexes fit in RAM
● Create indexes & queries that ensure selectivity

29. References
- Mongo Performance https://github.com/danielabar/mongo-performance
- Tips and Tricks for Avoiding Common Query Pitfalls https://www.slideshare.net/mongodb/mongodblocal-dc-2018-tips-
and-tricks-for-avoiding-common-query-pitfalls
- Performance notes for MongoDB: https://blogs.msdn.microsoft.com/shacorn/2016/01/08/performance-notes-for-
mongodb/
- Index strategies:
https://docs.mongodb.com/manual/applications/indexes/
- Mongodb logging config:
https://docs.mongodb.com/v3.2/reference/configuration-options/
- Scaling write performance:
https://www.slideshare.net/daumdna/mongodb-scaling-write-performance
- MongoDB Indexes and Performance
https://hackernoon.com/mongodb-indexes-and-performance-2e8f94b23c0a