Scalling Solr in the Cloud - OSCON Data 2011

Scaling Solr in the Cloud By @ablyler & @LindsaySnider FROM @NutshellCRM www.nutshell.com

AgendaAbout the SpeakersAbout NutshellNutshell and SolrSolr Resource UsageScaling MethodsQuestions and Answers

About the SpeakersBitLeap / Barracuda Networks Developers on the Backup Appliance / Cloud Scaled databases and storage systems Used SugarCRM & Salesforce.com

About NutshellWeb and Mobile CRM applicationHeavy use of OpenSource technologies: Gentoo Nginx / PHP / ZendFramework / jQuery MySQL / Solr / Gearman Jenkins / Redmine / Cacti / Nagios

Nutshell and SolrHeavy use of Solr for searching, table views, andde-duplicationUsed for searching / display: Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and UsersUsed for de-duplication: Accounts, Contacts

Reads vs Writes 7000 5250 3500 1750 0Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM Writes Reads Time Average Read Query Time: 2.11ms

Demowww.nutshell.com

Computer Resources CPU Disk I/O Disk RAM Storage

CPU Disk I/O DiskRAM Storage

SOLR ResourcesHeavy CPU Disk I/O Disk RAM StorageLight

Dark Ages Age of Enlightenment Modern Era Today Auto Provisioning Setup new Solr Jetty app Create MySQL database Populate MySQL / Solr with demo data Send welcome email

Dark Ages Age of Enlightenment Modern Era TodaySeparate Jetty per Customer Uses a ton of memory Separate schema / Solr for each customer Ran into upper limit morning before launch

Dark Ages Age of Enlightenment Modern Era Today Auto provisioning Separate Jetty app for each customer

Dark Ages Age of Enlightenment Modern Era Today5025 0 Auto provisioning Separate Jetty app for each customer

Dark Ages Age of Enlightenment Modern Era Today Solr Core per Customer Allows for management of Solr on a per customer basis: creating / stopping Contained within a single Jetty app Shared schema between all cores Easily managed via simple HTTP API

Dark Ages Age of Enlightenment Modern Era Today Fallback to MySQL Landing page of application Allows for graceful handling when Solr is down Abstracted within the application library

Dark Ages Age of Enlightenment Modern Era Today Sun JVM to IcedTea JVM IcedTea JVM uses less memory than Sun JVM

Dark Ages Age of Enlightenment Modern Era Today Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM

Dark Ages Age of Enlightenment Modern Era Today300150 0 Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM

Dark Ages Age of Enlightenment Modern Era TodayShared Schema Across Cores Decreases initialization time for each core Decreases memory usage

Dark Ages Age of Enlightenment Modern Era Today Solr Index Field Selection More indexed fields = more used memory Only index fields that are searched Store other non-indexed fields for display

Dark Ages Age of Enlightenment Modern Era Today Splitting of Reader / Writer Index building is CPU / disk intensive Writer = Solr with caching disabled Reader = Solr slave, that doesn’t build indexes

Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master)

Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers

Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager

Dark Ages Age of Enlightenment Modern Era Today Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing

Dark Ages Age of Enlightenment Modern Era Today1000500 0 Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing

Dark Ages Age of Enlightenment Modern Era TodayIntelligent Core Management Least recently used Solr cores are spun down Solr cores started on login Reindexing is a database flag, and happens on the next login

Dark Ages Age of Enlightenment Modern Era Today Partitioning Pairs of readers and writers Partitioned based on account id Left pad the account id w/ zeros to length of two Reverse the account id Take last two digits of the account id

Dark Ages Age of Enlightenment Modern Era Today Intelligent core spin down / up Partitioning of customers to separate Solr servers

Dark Ages Age of Enlightenment Modern Era Today80004000 0 Intelligent core spin down / up Partitioning of customers to separate Solr servers

SOLR Server8000 User

SOLR SOLR SOLR Server SOLR Server SOLR Server SOLR8000 Server UserSOLR 8000 Server User 8000 Server 8000User 8000User Server 8000User 8000User User

SOLR Server8000 User SOLR Server8000 User SOLR Server8000 User SOLR Server8000 User SOLR Server8000 User SOLR Server8000 User SOLR Server8000 User

SOLR Server 8000 User SOLR Server 8000 User123456 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User

SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User

SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User1 2 3 4 5 6 SOLR Server 8000 User SOLR1 2 3 4 5 6 Server 8000 User SOLR Server 8000 User SOLR Server 8000 User

SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User1 2 3 4 5 6 SOLR Server 8000 User SOLR1 2 3 4 5 6 Server 8000 User 65 SOLR Server 8000 User SOLR Server 8000 User

SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User

SOLR Server 8000 User01 SOLR02 Server 8000 User03 SOLR04 Server 8000 User65 SOLR Server66 8000 User SOLR96 Server 8000 User97 SOLR98 Server 8000 User99 SOLR Server 8000 User

SOLR Server 8000 User01 SOLR02 Server 8000 User03 SOLR04 Server 8000 User65 SOLR Server66 8000 User SOLR Server 8000 User SOLR96 Server 8000 User97 SOLR98 Server 8000 User99 SOLR Server 8000 User

SOLR Server 8000 User01 SOLR02 Server 8000 User03 SOLR04 Server 8000 User65 SOLR Server66 8000 User SOLR Server 8000 User SOLR96 Server 8000 User SOLR97 Server SOLR 8000 User98 Server 8000 User99 SOLR Server 8000 User

SOLR Server 8000 User01 SOLR02 Server 8000 User03 SOLR04 Server 8000 User65 SOLR Server66 8000 User SOLR Server 8000 User96 SOLR97 Server SOLR 8000 User98 Server 8000 User99 SOLR Server 8000 User

Scaling Solr in the Cloudauto provisioning 50 Usersseparate Jetty app for each customerseparate Solr core per customerfall back to MySQL for table data 300 Usersmigrated from Sun JVM to IcedTea JVMshared schema across coressolr index ﬁeld selection 600 Userssplitting reader / writer rolesintelligent core spin down / up 7000 Usersparititioning of customer to separate Solr servers By @ablyler & @LindsaySnider FROM @NutshellCRM

http://www.oscon.com	167
http://lanyrd.com	41
http://app-dev.crowdbase.com	7
http://twitter.com	5
http://a0.twimg.com	1
http://crowdbase.com	1
http://app-development.crowdbase.com	1

Scalling Solr in the Cloud - OSCON Data 2011

by OSCON Byrum

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 223

Statistics

Scalling Solr in the Cloud - OSCON Data 2011 Presentation Transcript