SlideShare a Scribd company logo
1 of 61
Download to read offline
Scaling Solr in the Cloud
  By @ablyler & @LindsaySnider
              FROM

         @NutshellCRM



        www.nutshell.com
Agenda
About the Speakers
About Nutshell
Nutshell and Solr
Solr Resource Usage
Scaling Methods
Questions and Answers
About the Speakers


BitLeap / Barracuda Networks
 Developers on the Backup Appliance / Cloud
 Scaled databases and storage systems
 Used SugarCRM & Salesforce.com
About Nutshell

Web and Mobile CRM application
Heavy use of OpenSource technologies:
 Gentoo
 Nginx / PHP / ZendFramework / jQuery
 MySQL / Solr / Gearman
 Jenkins / Redmine / Cacti / Nagios
Nutshell and Solr

Heavy use of Solr for searching, table views, and
de-duplication
Used for searching / display:
 Accounts, Competitors, Contacts, Leads, Products, Sources,
 Teams, and Users

Used for de-duplication:
 Accounts, Contacts
Reads vs Writes
   7000




   5250




   3500




   1750




        0
Jul 20, 2011 3:00 AM        Jul 20, 2011 6:00 AM    Jul 20, 2011 9:00 AM   Jul 20, 2011 12:00 PM   Jul 20, 2011 3:00 PM   Jul 20, 2011 6:00 PM   Jul 20, 2011 9:00 PM   Jul 21, 2011 12:00 AM

                   Writes                   Reads                 Time




                                                            Average Read Query Time:
                                                                     2.11ms
Demo

www.nutshell.com
Computer Resources

   CPU   Disk I/O


           Disk
   RAM
          Storage
CPU   Disk I/O


       Disk
RAM
      Storage
SOLR Resources

Heavy    CPU   Disk I/O


                 Disk
         RAM
                Storage



Light
SOLR Resources

Heavy    CPU   Disk I/O


                 Disk
         RAM
                Storage



Light
SOLR Resources

Heavy    CPU   Disk I/O


                 Disk
         RAM
                Storage



Light
SOLR Resources

Heavy    CPU   Disk I/O


                 Disk
         RAM
                Storage



Light
SOLR Resources

Heavy    CPU   Disk I/O


                 Disk
         RAM
                Storage



Light
Dark Ages   Age of Enlightenment   Modern Era   Today



            Auto Provisioning

 Setup new Solr Jetty app
 Create MySQL database
 Populate MySQL / Solr with demo data
 Send welcome email
Dark Ages   Age of Enlightenment   Modern Era   Today



Separate Jetty per Customer


 Uses a ton of memory
 Separate schema / Solr for each customer
 Ran into upper limit morning before launch
Dark Ages   Age of Enlightenment   Modern Era   Today




 Auto provisioning
 Separate Jetty app for each customer
Dark Ages   Age of Enlightenment   Modern Era   Today
50

25

 0




      Auto provisioning
      Separate Jetty app for each customer
Dark Ages   Age of Enlightenment   Modern Era   Today



      Solr Core per Customer

 Allows for management of Solr on a per customer
 basis: creating / stopping
 Contained within a single Jetty app
 Shared schema between all cores
 Easily managed via simple HTTP API
Dark Ages    Age of Enlightenment   Modern Era   Today



            Fallback to MySQL


 Landing page of application
 Allows for graceful handling when Solr is down
 Abstracted within the application library
Dark Ages   Age of Enlightenment   Modern Era   Today



      Sun JVM to IcedTea JVM



 IcedTea JVM uses less memory than Sun JVM
Dark Ages   Age of Enlightenment   Modern Era   Today




 Separate Solr core per customer
 Fallback to MySQL for table data
 Migrated from Sun JVM to IcedTea JVM
Dark Ages   Age of Enlightenment   Modern Era   Today
300

150

 0




       Separate Solr core per customer
       Fallback to MySQL for table data
       Migrated from Sun JVM to IcedTea JVM
Dark Ages   Age of Enlightenment   Modern Era   Today



Shared Schema Across Cores


 Decreases initialization time for each core
 Decreases memory usage
Dark Ages   Age of Enlightenment   Modern Era   Today



    Solr Index Field Selection


 More indexed fields = more used memory
 Only index fields that are searched
 Store other non-indexed fields for display
Dark Ages   Age of Enlightenment   Modern Era   Today



  Splitting of Reader / Writer


 Index building is CPU / disk intensive
 Writer = Solr with caching disabled
 Reader = Solr slave, that doesn’t build indexes
Dark Ages    Age of Enlightenment   Modern Era           Today




                            Web




                                                 Solr
            MySQL
                                                 Slave
            (master-
            master)
Dark Ages   Age of Enlightenment       Modern Era        Today



                              Web




                                                Solr
                MySQL
                                                Slave
                (master-
                master)




                                                 Solr
                                                Master




                Range indexer
                 Range indexer
                  Range indexer
                    Range indexer
                   Document indexers


              Gearman workers
Dark Ages   Age of Enlightenment       Modern Era        Today



                              Web




                                                Solr
                MySQL
                                                Slave
                (master-
                master)




                                                 Solr
                                                Master




                Range indexer
                 Range indexer
                  Range indexer
                    Range indexer
                   Document indexers


             Gearman workers
Dark Ages   Age of Enlightenment       Modern Era        Today



                              Web




                                                Solr
                MySQL
                                                Slave
                (master-
                master)




                                                 Solr
                                                Master




                Range indexer
                 Range indexer
                  Range indexer
                    Range indexer
                   Document indexers


             Gearman workers
Dark Ages   Age of Enlightenment     Modern Era              Today



                              Web




                                              Solr
                MySQL
                                              Slave
                (master-
                master)




                                               Solr
                                              Master




                Range indexer
                 Range indexer
                  Range indexer
                   Range indexer
                    Range indexers
                                                  Reindex Manager
Dark Ages   Age of Enlightenment     Modern Era            Today



                              Web




                                              Solr
                MySQL
                                              Slave
                (master-
                master)




                                               Solr
                                              Master




                Range indexer
                 Range indexer
                  Range indexer
                   Range indexer
                    Range indexers
                                               Reindex Manager
Dark Ages   Age of Enlightenment     Modern Era              Today



                              Web




                                                  Solr
                 MySQL
                                                  Slave
                 (master-
                 master)




                                               Solr
                                              Master




                Range indexer
                 Range indexer
                  Range indexer
                   Range indexer
                    Range indexers
                                                  Reindex Manager
Dark Ages   Age of Enlightenment   Modern Era   Today




 Shared schema across cores
 Solr index field selection
 Splitting of reader / writer roles
 Intelligent indexing / reindexing
Dark Ages   Age of Enlightenment   Modern Era   Today
1000

500

  0




        Shared schema across cores
        Solr index field selection
        Splitting of reader / writer roles
        Intelligent indexing / reindexing
Dark Ages   Age of Enlightenment   Modern Era   Today



Intelligent Core Management


 Least recently used Solr cores are spun down
 Solr cores started on login
 Reindexing is a database flag, and happens on the
 next login
Dark Ages      Age of Enlightenment    Modern Era       Today



                    Partitioning

 Pairs of readers and writers
 Partitioned based on account id
    Left pad the account id w/ zeros to length of two
    Reverse the account id
    Take last two digits of the account id
Dark Ages   Age of Enlightenment   Modern Era   Today




 Intelligent core spin down / up
 Partitioning of customers to separate Solr servers
Dark Ages   Age of Enlightenment   Modern Era   Today
8000

4000

   0




        Intelligent core spin down / up
        Partitioning of customers to separate Solr servers
SOLR
  Server
8000 User
SOLR
   SOLR
    SOLR
  Server
     SOLR
   Server
      SOLR
    Server
       SOLR
8000 Server
     UserSOLR
 8000 Server
      User
  8000 Server
   8000User
    8000User
        Server
     8000User
      8000User
            User
SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User
SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User

  SOLR
  Server
8000 User
SOLR
           Server
         8000 User

           SOLR
           Server
         8000 User
123456     SOLR
           Server
         8000 User

           SOLR
           Server
         8000 User

           SOLR
           Server
         8000 User

           SOLR
           Server
         8000 User

           SOLR
           Server
         8000 User
SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
  123456        SOLR
                Server
              8000 User
1 2 3 4 5 6
                SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
  123456        SOLR
                Server
              8000 User
1 2 3 4 5 6
                SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
  123456        SOLR
                Server
              8000 User
1 2 3 4 5 6
                SOLR
                Server
              8000 User

                SOLR
1 2 3 4 5 6     Server
              8000 User

                SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
  123456        SOLR
                Server
              8000 User
1 2 3 4 5 6
                SOLR
                Server
              8000 User

                SOLR
1 2 3 4 5 6     Server
              8000 User

     65         SOLR
                Server
              8000 User

                SOLR
                Server
              8000 User
SOLR
                     Server
                   8000 User
              01
                     SOLR
              02     Server
                   8000 User
  123456      03
                     SOLR
              04     Server
                   8000 User
1 2 3 4 5 6
              65     SOLR
                     Server
              66   8000 User

                     SOLR
1 2 3 4 5 6   96     Server
                   8000 User
              97
     65              SOLR
              98     Server
                   8000 User
              99
                     SOLR
                     Server
                   8000 User
SOLR
                     Server
                   8000 User
              01
                     SOLR
              02     Server
                   8000 User
  123456      03
                     SOLR
              04     Server
                   8000 User
1 2 3 4 5 6
              65     SOLR
                     Server
              66   8000 User

                     SOLR
1 2 3 4 5 6   96     Server
                   8000 User
              97
     65              SOLR
              98     Server
                   8000 User
              99
                     SOLR
                     Server
                   8000 User
SOLR
                     Server
                   8000 User
              01
                     SOLR
              02     Server
                   8000 User
  123456      03
                     SOLR
              04     Server
                   8000 User
1 2 3 4 5 6
              65     SOLR
                     Server
              66   8000 User

                     SOLR
1 2 3 4 5 6   96     Server
                   8000 User
              97
     65              SOLR
              98     Server
                   8000 User
              99
                     SOLR
                     Server
                   8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User

       SOLR
96     Server
     8000 User
97
       SOLR
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User

       SOLR
96     Server
     8000 User
97
       SOLR
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User     SOLR
                   Server
                 8000 User
       SOLR
96     Server
     8000 User
97
       SOLR
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User     SOLR
                   Server
                 8000 User
       SOLR
96     Server
     8000 User
                   SOLR
97                 Server
       SOLR      8000 User
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User     SOLR
                   Server
                 8000 User
       SOLR
96     Server
     8000 User
                   SOLR
97                 Server
       SOLR      8000 User
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User     SOLR
                   Server
                 8000 User
       SOLR
96     Server
     8000 User
                   SOLR
97                 Server
       SOLR      8000 User
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User     SOLR
                   Server
                 8000 User
       SOLR
96     Server
     8000 User
                   SOLR
97                 Server
       SOLR      8000 User
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
SOLR
       Server
     8000 User
01
       SOLR
02     Server
     8000 User
03
       SOLR
04     Server
     8000 User

65     SOLR
       Server
66   8000 User     SOLR
                   Server
                 8000 User
96
                   SOLR
97                 Server
       SOLR      8000 User
98     Server
     8000 User
99
       SOLR
       Server
     8000 User
Scaling Solr in the Cloud
auto provisioning
                                                       50 Users
separate Jetty app for each customer
separate Solr core per customer
fall back to MySQL for table data                     300 Users
migrated from Sun JVM to IcedTea JVM
shared schema across cores
solr index field selection                             600 Users
splitting reader / writer roles
intelligent core spin down / up
                                                     7000 Users
parititioning of customer to separate Solr servers

              By @ablyler & @LindsaySnider
                                  FROM

                         @NutshellCRM

More Related Content

Similar to Scalling Solr in the Cloud - OSCON Data 2011

[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
James Chen
 
UKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA'sUKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA's
FromDual GmbH
 
Microsoft Openness Mongo DB
Microsoft Openness Mongo DBMicrosoft Openness Mongo DB
Microsoft Openness Mongo DB
Heriyadi Janwar
 
Stathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IACStathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IAC
Stathy Touloumis
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011
Andy Parsons
 

Similar to Scalling Solr in the Cloud - OSCON Data 2011 (20)

Big Search with Big Data Principles
Big Search with Big Data PrinciplesBig Search with Big Data Principles
Big Search with Big Data Principles
 
ApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big DataApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big Data
 
RADOS for Eucalyptus
RADOS for EucalyptusRADOS for Eucalyptus
RADOS for Eucalyptus
 
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
 
CloudStack technical overview
CloudStack technical overviewCloudStack technical overview
CloudStack technical overview
 
No Sql
No SqlNo Sql
No Sql
 
February 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with DockerFebruary 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with Docker
 
1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day 1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day
 
UKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA'sUKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA's
 
MySQL Consegi
MySQL ConsegiMySQL Consegi
MySQL Consegi
 
Microsoft Openness Mongo DB
Microsoft Openness Mongo DBMicrosoft Openness Mongo DB
Microsoft Openness Mongo DB
 
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear ScalabilityBeyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
 
Stathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IACStathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IAC
 
Geek out 2014-lagergren-final
Geek out 2014-lagergren-finalGeek out 2014-lagergren-final
Geek out 2014-lagergren-final
 
Solr @ eBay Kleinanzeigen
Solr @ eBay KleinanzeigenSolr @ eBay Kleinanzeigen
Solr @ eBay Kleinanzeigen
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011
 
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,..."Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
 
Yokozuna
YokozunaYokozuna
Yokozuna
 
Netflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconNetflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at Gluecon
 
SV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformSV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source Platform
 

More from OSCON Byrum

Big Data for each one of us
Big Data for each one of usBig Data for each one of us
Big Data for each one of us
OSCON Byrum
 
Declarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScriptDeclarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScript
OSCON Byrum
 

More from OSCON Byrum (20)

OSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom FifieldOSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
 
Protecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent LicenseProtecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent License
 
Using Cascalog to build an app with City of Palo Alto Open Data
Using Cascalog to build an app with City of Palo Alto Open DataUsing Cascalog to build an app with City of Palo Alto Open Data
Using Cascalog to build an app with City of Palo Alto Open Data
 
Finite State Machines - Why the fear?
Finite State Machines - Why the fear?Finite State Machines - Why the fear?
Finite State Machines - Why the fear?
 
Open Source Automotive Development
Open Source Automotive DevelopmentOpen Source Automotive Development
Open Source Automotive Development
 
How we built our community using Github - Uri Cohen
How we built our community using Github - Uri CohenHow we built our community using Github - Uri Cohen
How we built our community using Github - Uri Cohen
 
The Vanishing Pattern: from iterators to generators in Python
The Vanishing Pattern: from iterators to generators in PythonThe Vanishing Pattern: from iterators to generators in Python
The Vanishing Pattern: from iterators to generators in Python
 
Distributed Coordination with Python
Distributed Coordination with PythonDistributed Coordination with Python
Distributed Coordination with Python
 
An overview of open source in East Asia (China, Japan, Korea)
An overview of open source in East Asia (China, Japan, Korea)An overview of open source in East Asia (China, Japan, Korea)
An overview of open source in East Asia (China, Japan, Korea)
 
Oscon 2013 Jesse Anderson
Oscon 2013 Jesse AndersonOscon 2013 Jesse Anderson
Oscon 2013 Jesse Anderson
 
US Patriot Act OSCON2012 David Mertz
US Patriot Act OSCON2012 David MertzUS Patriot Act OSCON2012 David Mertz
US Patriot Act OSCON2012 David Mertz
 
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
 
Big Data for each one of us
Big Data for each one of usBig Data for each one of us
Big Data for each one of us
 
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
 
Declarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScriptDeclarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScript
 
Using and Building Open Source in Google Corporate Engineering - Justin McWil...
Using and Building Open Source in Google Corporate Engineering - Justin McWil...Using and Building Open Source in Google Corporate Engineering - Justin McWil...
Using and Building Open Source in Google Corporate Engineering - Justin McWil...
 
A Look at the Network: Searching for Truth in Distributed Applications
A Look at the Network: Searching for Truth in Distributed ApplicationsA Look at the Network: Searching for Truth in Distributed Applications
A Look at the Network: Searching for Truth in Distributed Applications
 
Life After Sharding: Monitoring and Management of a Complex Data Cloud
Life After Sharding: Monitoring and Management of a Complex Data CloudLife After Sharding: Monitoring and Management of a Complex Data Cloud
Life After Sharding: Monitoring and Management of a Complex Data Cloud
 
Faster! Faster! Accelerate your business with blazing prototypes
Faster! Faster! Accelerate your business with blazing prototypesFaster! Faster! Accelerate your business with blazing prototypes
Faster! Faster! Accelerate your business with blazing prototypes
 
Comparing open source private cloud platforms
Comparing open source private cloud platformsComparing open source private cloud platforms
Comparing open source private cloud platforms
 

Recently uploaded

Recently uploaded (20)

Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 

Scalling Solr in the Cloud - OSCON Data 2011

  • 1. Scaling Solr in the Cloud By @ablyler & @LindsaySnider FROM @NutshellCRM www.nutshell.com
  • 2. Agenda About the Speakers About Nutshell Nutshell and Solr Solr Resource Usage Scaling Methods Questions and Answers
  • 3. About the Speakers BitLeap / Barracuda Networks Developers on the Backup Appliance / Cloud Scaled databases and storage systems Used SugarCRM & Salesforce.com
  • 4. About Nutshell Web and Mobile CRM application Heavy use of OpenSource technologies: Gentoo Nginx / PHP / ZendFramework / jQuery MySQL / Solr / Gearman Jenkins / Redmine / Cacti / Nagios
  • 5. Nutshell and Solr Heavy use of Solr for searching, table views, and de-duplication Used for searching / display: Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and Users Used for de-duplication: Accounts, Contacts
  • 6. Reads vs Writes 7000 5250 3500 1750 0 Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM Writes Reads Time Average Read Query Time: 2.11ms
  • 8. Computer Resources CPU Disk I/O Disk RAM Storage
  • 9. CPU Disk I/O Disk RAM Storage
  • 10. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 11. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 12. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 13. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 14. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 15. Dark Ages Age of Enlightenment Modern Era Today Auto Provisioning Setup new Solr Jetty app Create MySQL database Populate MySQL / Solr with demo data Send welcome email
  • 16. Dark Ages Age of Enlightenment Modern Era Today Separate Jetty per Customer Uses a ton of memory Separate schema / Solr for each customer Ran into upper limit morning before launch
  • 17. Dark Ages Age of Enlightenment Modern Era Today Auto provisioning Separate Jetty app for each customer
  • 18. Dark Ages Age of Enlightenment Modern Era Today 50 25 0 Auto provisioning Separate Jetty app for each customer
  • 19. Dark Ages Age of Enlightenment Modern Era Today Solr Core per Customer Allows for management of Solr on a per customer basis: creating / stopping Contained within a single Jetty app Shared schema between all cores Easily managed via simple HTTP API
  • 20. Dark Ages Age of Enlightenment Modern Era Today Fallback to MySQL Landing page of application Allows for graceful handling when Solr is down Abstracted within the application library
  • 21. Dark Ages Age of Enlightenment Modern Era Today Sun JVM to IcedTea JVM IcedTea JVM uses less memory than Sun JVM
  • 22. Dark Ages Age of Enlightenment Modern Era Today Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
  • 23. Dark Ages Age of Enlightenment Modern Era Today 300 150 0 Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
  • 24. Dark Ages Age of Enlightenment Modern Era Today Shared Schema Across Cores Decreases initialization time for each core Decreases memory usage
  • 25. Dark Ages Age of Enlightenment Modern Era Today Solr Index Field Selection More indexed fields = more used memory Only index fields that are searched Store other non-indexed fields for display
  • 26. Dark Ages Age of Enlightenment Modern Era Today Splitting of Reader / Writer Index building is CPU / disk intensive Writer = Solr with caching disabled Reader = Solr slave, that doesn’t build indexes
  • 27. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master)
  • 28. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
  • 29. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
  • 30. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
  • 31. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
  • 32. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
  • 33. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
  • 34. Dark Ages Age of Enlightenment Modern Era Today Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
  • 35. Dark Ages Age of Enlightenment Modern Era Today 1000 500 0 Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
  • 36. Dark Ages Age of Enlightenment Modern Era Today Intelligent Core Management Least recently used Solr cores are spun down Solr cores started on login Reindexing is a database flag, and happens on the next login
  • 37. Dark Ages Age of Enlightenment Modern Era Today Partitioning Pairs of readers and writers Partitioned based on account id Left pad the account id w/ zeros to length of two Reverse the account id Take last two digits of the account id
  • 38. Dark Ages Age of Enlightenment Modern Era Today Intelligent core spin down / up Partitioning of customers to separate Solr servers
  • 39. Dark Ages Age of Enlightenment Modern Era Today 8000 4000 0 Intelligent core spin down / up Partitioning of customers to separate Solr servers
  • 40.
  • 42. SOLR SOLR SOLR Server SOLR Server SOLR Server SOLR 8000 Server UserSOLR 8000 Server User 8000 Server 8000User 8000User Server 8000User 8000User User
  • 43. SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 44. SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 45. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 46. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 47. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 48. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 49. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User 65 SOLR Server 8000 User SOLR Server 8000 User
  • 50. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 51. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 52. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 53. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 54. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 55. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 56. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 57. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 58. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 59. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 60. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User 96 SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 61. Scaling Solr in the Cloud auto provisioning 50 Users separate Jetty app for each customer separate Solr core per customer fall back to MySQL for table data 300 Users migrated from Sun JVM to IcedTea JVM shared schema across cores solr index field selection 600 Users splitting reader / writer roles intelligent core spin down / up 7000 Users parititioning of customer to separate Solr servers By @ablyler & @LindsaySnider FROM @NutshellCRM