This document summarizes the challenges faced with a large Drupal site handling real estate data. It discusses importing large amounts of node and entity data, performance issues caused by memory leaks, and solutions for optimizing indexing and caching when using Solr search on a site with millions of nodes. Tuning garbage collection, offloading processing to queues and batches, and parallelizing indexing are recommended to improve performance of importing and updating large quantities of data on the site.

1. Big Drupal: Issues We Met
Vadym Abramchuk
Internetdevels

3. Project overview
Realty analytics platform
Used by professional brokers
New data imported each few hours

4. The technologies
PHP, MySQL, Drupal 7

6. The technologies
PHP, MySQL, Drupal 7
MySQL is a primary storage
Solr 5.2 as search engine
PostgreSQL + PostGIS for spatial calculations

7. I wanted to say “One million
nodes”
but I didn’t

8. The facts
958769 nodes
7004555 ECK entities in 7 types
375874 custom entity type rows
MySQL database size: 4G gzipped, 66G disk space usage
Solr indexes: 91G

9. Importing data into Drupal

10. Bulk data import
Know your data
Prepare your data
Avoid direct insertions
Insert directly if you’re brave enough (or you don’t have a choice)
Watch your step

11. Ok, so you don’t batch
Drupal has memory leaks
Entity metadata wrapper has memory leaks
https://www.drupal.org/node/1343196
Everything has memory leaks
Avoid long-running processes
Offload processing with queues and batches

12. Offloading
Put your data into separate table
Add queue
Let it run in background. Queues are made to be indepent
Having small amount of data? Use batches
Need to update really fast? Think about MySQL cursors

13. Tuning garbage collector

15. Tuning garbage collector

17. Playing with garbage collector
gc_disable()
process your batch, then gc_collect_cycles()
gc_enable()
Not a Holy Grail, may become Pandora’s box

18. Living with Solr

19. Drupal and Solr
Solr is a search engine
Solr is fast and scalable
Data is denormalized
Takes lots of space
Needs indexation

20. Data indexing flow

21. Why indexing is slow
Search API indexation is single threaded
Uses entity metadata wrappers
You can run multiple indexation processes, one per each index
Items are not locked, multiple workers at same index will do the same
Indexation via drush does not use batch and is slow due to memory leaks

22. Faster indexing
Index in parallel
One daemon to rule them all
Pool of workers
Daemon maintains list of items to index
Fully utilizes hardware
Needs special Solr configuration due to autocommit

23. Faster indexing
How it looks like

24. Search API Views integration
Make a view like you always do, but for Search API items
Solr returns results set
Views fields handlers loads corresponding entities with entity_load (sic!)
https://www.drupal.org/node/2028337
Database is queried anyway but load is much less
Entity properties are rendered as entity properties
Use Search API processors to avoid loading entity and running getters

25. Caching

26. Cache backends
Database is slow
Memcache is faster
Redis is even faster
Redis caching module locks Redis by running LUA script inside server
local keys = redis.call("KEYS", ARGV[1])
for i, k in ipairs(keys) do
redis.call("DEL", k)
end
return 1

27. So useful useless cache
Field cache litters the cache storage
Use entitycache module with memcached
Large amount of cache entries kills Redis

28. Solr-oriented caching
Solr has an index version
Changed during Solr commit
May be used for cache key

29. That’s all folks!
Thanks for your time everyone
Reach me if you have some questions
Reach me right now if you have some answers ;)