The document discusses how Nutshell, a CRM application, scaled their use of Solr for searching and deduplication. They implemented several techniques including using a separate Solr core per customer, intelligent indexing strategies, and partitioning customers across multiple Solr servers. These changes helped improve performance and allow Nutshell to scale Solr to support thousands of customers.

1. Scaling Solr in the Cloud
By @ablyler & @LindsaySnider
FROM
@NutshellCRM
www.nutshell.com

2. Agenda
About the Speakers
About Nutshell
Nutshell and Solr
Solr Resource Usage
Scaling Methods
Questions and Answers

3. About the Speakers
BitLeap / Barracuda Networks
Developers on the Backup Appliance / Cloud
Scaled databases and storage systems
Used SugarCRM & Salesforce.com

4. About Nutshell
Web and Mobile CRM application
Heavy use of OpenSource technologies:
Gentoo
Nginx / PHP / ZendFramework / jQuery
MySQL / Solr / Gearman
Jenkins / Redmine / Cacti / Nagios

5. Nutshell and Solr
Heavy use of Solr for searching, table views, and
de-duplication
Used for searching / display:
Accounts, Competitors, Contacts, Leads, Products, Sources,
Teams, and Users
Used for de-duplication:
Accounts, Contacts

6. Reads vs Writes
7000
5250
3500
1750
0
Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM
Writes Reads Time
Average Read Query Time:
2.11ms

7. Demo
www.nutshell.com

8. Computer Resources
CPU Disk I/O
Disk
RAM
Storage

9. CPU Disk I/O
Disk
RAM
Storage

10. SOLR Resources
Heavy CPU Disk I/O
Disk
RAM
Storage
Light

11. SOLR Resources
Heavy CPU Disk I/O
Disk
RAM
Storage
Light

12. SOLR Resources
Heavy CPU Disk I/O
Disk
RAM
Storage
Light

13. SOLR Resources
Heavy CPU Disk I/O
Disk
RAM
Storage
Light

14. SOLR Resources
Heavy CPU Disk I/O
Disk
RAM
Storage
Light

15. Dark Ages Age of Enlightenment Modern Era Today
Auto Provisioning
Setup new Solr Jetty app
Create MySQL database
Populate MySQL / Solr with demo data
Send welcome email

16. Dark Ages Age of Enlightenment Modern Era Today
Separate Jetty per Customer
Uses a ton of memory
Separate schema / Solr for each customer
Ran into upper limit morning before launch

17. Dark Ages Age of Enlightenment Modern Era Today
Auto provisioning
Separate Jetty app for each customer

18. Dark Ages Age of Enlightenment Modern Era Today
50
25
0
Auto provisioning
Separate Jetty app for each customer

19. Dark Ages Age of Enlightenment Modern Era Today
Solr Core per Customer
Allows for management of Solr on a per customer
basis: creating / stopping
Contained within a single Jetty app
Shared schema between all cores
Easily managed via simple HTTP API

20. Dark Ages Age of Enlightenment Modern Era Today
Fallback to MySQL
Landing page of application
Allows for graceful handling when Solr is down
Abstracted within the application library

21. Dark Ages Age of Enlightenment Modern Era Today
Sun JVM to IcedTea JVM
IcedTea JVM uses less memory than Sun JVM

22. Dark Ages Age of Enlightenment Modern Era Today
Separate Solr core per customer
Fallback to MySQL for table data
Migrated from Sun JVM to IcedTea JVM

23. Dark Ages Age of Enlightenment Modern Era Today
300
150
0
Separate Solr core per customer
Fallback to MySQL for table data
Migrated from Sun JVM to IcedTea JVM

24. Dark Ages Age of Enlightenment Modern Era Today
Shared Schema Across Cores
Decreases initialization time for each core
Decreases memory usage

25. Dark Ages Age of Enlightenment Modern Era Today
Solr Index Field Selection
More indexed fields = more used memory
Only index fields that are searched
Store other non-indexed fields for display

26. Dark Ages Age of Enlightenment Modern Era Today
Splitting of Reader / Writer
Index building is CPU / disk intensive
Writer = Solr with caching disabled
Reader = Solr slave, that doesn’t build indexes

27. Dark Ages Age of Enlightenment Modern Era Today
Web
Solr
MySQL
Slave
(master-
master)

28. Dark Ages Age of Enlightenment Modern Era Today
Web
Solr
MySQL
Slave
(master-
master)
Solr
Master
Range indexer
Range indexer
Range indexer
Range indexer
Document indexers
Gearman workers

29. Dark Ages Age of Enlightenment Modern Era Today
Web
Solr
MySQL
Slave
(master-
master)
Solr
Master
Range indexer
Range indexer
Range indexer
Range indexer
Document indexers
Gearman workers

30. Dark Ages Age of Enlightenment Modern Era Today
Web
Solr
MySQL
Slave
(master-
master)
Solr
Master
Range indexer
Range indexer
Range indexer
Range indexer
Document indexers
Gearman workers

31. Dark Ages Age of Enlightenment Modern Era Today
Web
Solr
MySQL
Slave
(master-
master)
Solr
Master
Range indexer
Range indexer
Range indexer
Range indexer
Range indexers
Reindex Manager

32. Dark Ages Age of Enlightenment Modern Era Today
Web
Solr
MySQL
Slave
(master-
master)
Solr
Master
Range indexer
Range indexer
Range indexer
Range indexer
Range indexers
Reindex Manager

33. Dark Ages Age of Enlightenment Modern Era Today
Web
Solr
MySQL
Slave
(master-
master)
Solr
Master
Range indexer
Range indexer
Range indexer
Range indexer
Range indexers
Reindex Manager

34. Dark Ages Age of Enlightenment Modern Era Today
Shared schema across cores
Solr index field selection
Splitting of reader / writer roles
Intelligent indexing / reindexing

35. Dark Ages Age of Enlightenment Modern Era Today
1000
500
0
Shared schema across cores
Solr index field selection
Splitting of reader / writer roles
Intelligent indexing / reindexing

36. Dark Ages Age of Enlightenment Modern Era Today
Intelligent Core Management
Least recently used Solr cores are spun down
Solr cores started on login
Reindexing is a database flag, and happens on the
next login

37. Dark Ages Age of Enlightenment Modern Era Today
Partitioning
Pairs of readers and writers
Partitioned based on account id
Left pad the account id w/ zeros to length of two
Reverse the account id
Take last two digits of the account id

38. Dark Ages Age of Enlightenment Modern Era Today
Intelligent core spin down / up
Partitioning of customers to separate Solr servers

39. Dark Ages Age of Enlightenment Modern Era Today
8000
4000
0
Intelligent core spin down / up
Partitioning of customers to separate Solr servers

41. SOLR
Server
8000 User

42. SOLR
SOLR
SOLR
Server
SOLR
Server
SOLR
Server
SOLR
8000 Server
UserSOLR
8000 Server
User
8000 Server
8000User
8000User
Server
8000User
8000User
User

43. SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User

44. SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User

45. SOLR
Server
8000 User
SOLR
Server
8000 User
123456 SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User

46. SOLR
Server
8000 User
SOLR
Server
8000 User
123456 SOLR
Server
8000 User
1 2 3 4 5 6
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User

47. SOLR
Server
8000 User
SOLR
Server
8000 User
123456 SOLR
Server
8000 User
1 2 3 4 5 6
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User

48. SOLR
Server
8000 User
SOLR
Server
8000 User
123456 SOLR
Server
8000 User
1 2 3 4 5 6
SOLR
Server
8000 User
SOLR
1 2 3 4 5 6 Server
8000 User
SOLR
Server
8000 User
SOLR
Server
8000 User

49. SOLR
Server
8000 User
SOLR
Server
8000 User
123456 SOLR
Server
8000 User
1 2 3 4 5 6
SOLR
Server
8000 User
SOLR
1 2 3 4 5 6 Server
8000 User
65 SOLR
Server
8000 User
SOLR
Server
8000 User

50. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
123456 03
SOLR
04 Server
8000 User
1 2 3 4 5 6
65 SOLR
Server
66 8000 User
SOLR
1 2 3 4 5 6 96 Server
8000 User
97
65 SOLR
98 Server
8000 User
99
SOLR
Server
8000 User

51. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
123456 03
SOLR
04 Server
8000 User
1 2 3 4 5 6
65 SOLR
Server
66 8000 User
SOLR
1 2 3 4 5 6 96 Server
8000 User
97
65 SOLR
98 Server
8000 User
99
SOLR
Server
8000 User

52. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
123456 03
SOLR
04 Server
8000 User
1 2 3 4 5 6
65 SOLR
Server
66 8000 User
SOLR
1 2 3 4 5 6 96 Server
8000 User
97
65 SOLR
98 Server
8000 User
99
SOLR
Server
8000 User

53. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User
SOLR
96 Server
8000 User
97
SOLR
98 Server
8000 User
99
SOLR
Server
8000 User

54. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User
SOLR
96 Server
8000 User
97
SOLR
98 Server
8000 User
99
SOLR
Server
8000 User

55. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User SOLR
Server
8000 User
SOLR
96 Server
8000 User
97
SOLR
98 Server
8000 User
99
SOLR
Server
8000 User

56. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User SOLR
Server
8000 User
SOLR
96 Server
8000 User
SOLR
97 Server
SOLR 8000 User
98 Server
8000 User
99
SOLR
Server
8000 User

57. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User SOLR
Server
8000 User
SOLR
96 Server
8000 User
SOLR
97 Server
SOLR 8000 User
98 Server
8000 User
99
SOLR
Server
8000 User

58. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User SOLR
Server
8000 User
SOLR
96 Server
8000 User
SOLR
97 Server
SOLR 8000 User
98 Server
8000 User
99
SOLR
Server
8000 User

59. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User SOLR
Server
8000 User
SOLR
96 Server
8000 User
SOLR
97 Server
SOLR 8000 User
98 Server
8000 User
99
SOLR
Server
8000 User

60. SOLR
Server
8000 User
01
SOLR
02 Server
8000 User
03
SOLR
04 Server
8000 User
65 SOLR
Server
66 8000 User SOLR
Server
8000 User
96
SOLR
97 Server
SOLR 8000 User
98 Server
8000 User
99
SOLR
Server
8000 User

61. Scaling Solr in the Cloud
auto provisioning
50 Users
separate Jetty app for each customer
separate Solr core per customer
fall back to MySQL for table data 300 Users
migrated from Sun JVM to IcedTea JVM
shared schema across cores
solr index field selection 600 Users
splitting reader / writer roles
intelligent core spin down / up
7000 Users
parititioning of customer to separate Solr servers
By @ablyler & @LindsaySnider
FROM
@NutshellCRM