This document discusses Time To Live (TTL) indexes in MongoDB, which allow documents to be automatically removed from a collection after a specified amount of time. It covers how to create and configure TTL indexes, considerations for using them such as impact on storage and performance, and how to determine if a TTL index is working properly. The document also provides code examples for gradually reducing the TTL time over multiple runs to minimize performance impacts.

1. MATH is Hard :
TTL Index Configuration and
Considerations
Kimberly Wilkins
Senior DBA, ObjectRocket
kimberly@objectrocket.com

2. Background
• 16 yrs in databases, operations, replication &
infrastructure
• MongoDB
• Oracle (RAC, Exadata, Golden Gate, Enterprise Mgr)
• Vertica
• MySQL
• Supporting databases of all sizes and types
• Senior DBA @ ObjectRocket

3. Overview
• What and Why
• Design and Creation
• Considerations and Gotchas
• MATH and Queries
• Enabling / Disabling

4. TTL Collections and Indexes
What is a TTL Collection/Index - “Time To Live”
• Starting in 2.2, normal collection - special TTL index type
• TTL index defines how many seconds a document exists
• TTLMonitor - mongod background thread
• Why/When to Use TTL Indexes
• Control collection/database size
• Automatically remove data based on BSON date type
• No MANUAL mass or batch deletes or cron jobs
• Any time data needs to persist only for a specific period of time
• Useful with short term event data, logs, cache, and session info

5. Things to Think About Before Creating a TTL Index
Goal?
• Save space? Improve Performance?
• Data retention policies?
• Business logic or requirements?
• System load? More Reads vs. Writes?
Use Expire AFTER vs. AT ?
Global type expiration for all documents ‘expireAfterSeconds’ (after 3 days)
• { "createdAt": 1 } , { expireAfterSeconds: 259200 } -global policy determines
Per document type expiration for each document at a Certain Clock Time
• {"expiresAt": 1 } , { expireAfterSeconds: 0 } -application sets FUTURE time
Combo - {"expiresAt": 1 } , { expireAfterSeconds: 86400 } –application and global

6. How to Create TTL’s
Create a new index with ensureIndex
Specify expireAfterSeconds option
use "createdAt or "expiresAt” leader
2 real Types : global & application set, then combination
‘expireAfterSeconds’ -global type – all docs expire X seconds after CREATED

7. How to Create TTL’s –cont’d
Expire at Certain Clock Time -application set
- a date set in future by the application code
- expireAfterSeconds set to ZERO
combo type
- set an “expires_at” date in application code
- but apply a global type expiration policy with "expireAfterSeconds”
Messages documents set to expire 24 hours after creation

8. TTL Mods
Sample seconds minutes hours days math
Total seconds / 60 seconds
TTLSeconds / (60 *1minute)
3600 seconds/60 seconds = 60 minutes/60 minutes = 1 hour
1200 seconds/60 seconds = 20 minutes
How to Modify your TTL Time
- Index must be created as a TTL index from the beginning
{"expiresAt" : 1 } alone != a TTL index
- Must use the collMod command to change seconds
Example – increase to a 3 day TTL
db.runCommand({"collMod" : "messages" , "index" : { "keyPattern" :
{"expiresAt" : 1 } , "expireAfterSeconds" : 259200 } })

9. Considerations and Gotchas
Aggressive
deletes
Space Usage
usePowerOf2Sizes
fragmentation
TTL Isolation
drivers
No nulls
Stale timestamps – downstream clients
– supposed missing docs
No _Id fiel
d
timezone
languages
No compounds

10. More Considerations and Gotchas
-Very heavy writes/inserts and turn off TTL…@#$!
-Hard to catch up if app is very busy (7-15 Million inserts per day)
-Version Differences
-TTLMonitor thread runs every 60 seconds
-Background or Foreground Build implications
Look at # and size of documents
Check oplog size – ttl expire time and oplog time at least =
Warning: “The TTL index does not guarantee that expired data
will be deleted immediately.”
The duration of the removal operation depends on the workload of your
mongod instance. Expired data may exist beyond the 60 second period
between runs of the TTL monitor.
- http://docs.mongodb.org/manual/tutorial/expire-data/

11. How to Tell if Your TTL is Working
Do a little pre-checking – gather your information, get prepared,
know what to look for/at

12. How to Tell if Your TTL is Working –cont’d
-Confirm ttl index exists AND is correctly defined
-Define a variable to check with -ttlTime
-Set new Date for checking

13. How to Tell if Your TTL is Working –cont’d
MATH and Associated Queries :
*ISODate – is in MICROSECONDS; TTL is in SECONDS*
db.messages.find({expires_at: {$lt:ISODate("2014-11-
23T03:11:00.013Z") }}).count()

14. Other Associated Queries
If older than exact ttl exists, wait & re-query
get oldest documents in a collection using the _id as the determiner

15. THE MATH- Determine Starting TTL seconds
Good Rule – +20% buffer in TTL seconds = > Math.round(1.2 * …)

16. OK, have Beginning and Ending Seconds– now what?
TTL Reducer snippet javascript example
- Step - # of seconds to shave off at a time, a time interval,
number of seconds to delete for
- Hop – number of iterations to take of your desired step
This is a Process. . . .
- Run in a screen!!!
- Create new ttl index with 20% padded seconds
- Start w/ beginning version of script - high padded ttl time
until deletes start – steps and a low interval
- Monitor impact with mongostat
- Minimize db lock % on Primary to less than 10 seconds
http://goo.gl/W7HAZw

17. ttl_reducer_begin.js – beginning
-Start with a larger step (interval 2000 secs) and fewer hops/iterations (30)
-High initial threshold (+20%) -might take time to 1st start hitting the deletes

18. ttl_reducer_mid.js – mid-way
- Midway - do fewer steps(500 secs) and more hops (2930)
- Caution: deletes may negatively impact performance

19. ttl_reducer_end.js – end
- At end, optimize step (seconds to be deleted) and # hops
to reach desired permanent TTL ‘expireAfterSeconds’

20. Enabling / Disabling TTL Instance Wide
-Single global setting –so no need to drop all TTL indexes
-Version 2.4.x and above: TTLMonitor thread is enabled by default
-Via a command-line parameter and/or server command.
To disable at startup:
setParameter ttlMonitorEnabled=false
#config file of the MONGOD's only
setParameter=ttlMonitorEnabled=false
## To turn it back on (just remove that line)
To disable at runtime:
on ALL replica set members - primary and secondary
on secondaries - -- rs.slaveOk()
db.adminCommand({setParameter:1, ttlMonitorEnabled:false})
To re-enable when ready:
db.adminCommand({setParameter:1, ttlMonitorEnabled:true})

21. A Mass Pre-TTL Delete Loop
Useful MATH, Queries, and Loop for non-TTL based deletes
Run the below query and delete loop for a 30 minute period defined by x below

22. * Questions for me?
More questions? Ask-the-Experts area in the Atrium on first floor
MongoDB Inc. would like feedback if you are happy with the session. You can state
your opinion at the kiosk on the way out the door. Please take a second to give us
yours. Here's a sample of what the kiosks look like: http://www.happy-or-not.
com/en_us/.
Gist of the ttl_reducer.js scriptlets :
http://goo.gl/W7HAZw
Gist of additional queries and examples :
http://goo.gl/C1uYhb
Blogs will be posted in ~2 weeks on
http://engineering.objectrocket.com/