This document summarizes a presentation about moving Solr cores to a centralized blob storage system to improve high availability and scalability for Salesforce's search implementation. The current architecture has cores distributed across Solr servers, which limits elasticity. The new design stores core data and metadata in a central blob storage, allowing cores to be loaded and indexed from any Solr server. This improves availability, as cores can quickly be loaded elsewhere if a server fails. Initial testing shows cores can be loaded within seconds on another server using this approach. The system is undergoing further testing and refinement before potential adoption for Salesforce search deployments on public clouds.

1. Who moved my state?
A Blob Storage Solr story
Ilan Ginzburg
Architect, Salesforce
@ilanGinzburg
#Activate18 #ActivateSearch

2. Forward-Looking Statement
This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties
materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or
implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking,
including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements
regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded
services or technology developments and customer contracts or use of our services.
The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality
for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and
rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any litigation, risks associated with
completed and any possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our
ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer
deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further
information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-K for the
most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter. These documents and others containing
important disclosures are available on the SEC Filings section of the Investor Information section of our Web site.
Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available
and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that
are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

3. Agenda
• Salesforce and current Search implementation
• New paradigm
• Solr on Blob Store implementation
• Numbers and conclusion

5. Customers...

6. Search requirements at Salesforce 1/2
• Actually finding what you’re looking for
• There are 2300 “John Smith”...
• Features: suggestions, synonyms, etc.
• Multi-tenant with strict tenant isolation
• 150,000+ customers (tenants)

7. Search requirements at Salesforce 2/2
• Mostly unknown individual customer growth
• High variance access patterns
Time
Access rate
Data volume
Black Friday
Large upload

8. Current architecture
• Not based on SolrCloud
• Cores are for single customer (tenant isolation)
• Automated core split and rebalancing
• Local cores: relatively static cluster size
• 3 replicas per core
• Replica doing indexing
changes in case of server failure!
Solr
Cores
Solr
Cores
Solr
Cores

9. Changing the replica doing indexing
1. replicas are identical
Segments
Core

10. Indexing on multiple replicas
1. replicas are identical
2. indexing on one replica

11. Indexing on multiple replicas
1. replicas are identical
2. indexing on one replica
3. indexing on another replica

12. Indexing on multiple replicas
1. replicas are identical
2. indexing on one replica
3. indexing on another replica
4. Inconsistency!
? ?
?

13. Indexing on multiple replicas
1. replicas are identical
2. indexing on one replica
3. indexing on another replica
4. Inconsistency!
→ Error detection and correction
using indexing replay

14. Agenda
• Salesforce, current Search implementation
• New paradigm
• Solr on Blob Store implementation
• Numbers and conclusion

15. New paradigm: Decorrelate
compute from storage
Blob based cluster Existing cluster
Server elasticity Relatively static size
Core elasticity
(0-n replicas)
Three replicas per core
Public cloud efficiency Sized for max load/data
Simpler high availability Multiple replicas for HA
Store (and share!) core data in a central storage system
Solr
Cores
Blob Store central storage
Solr
Cores
Solr
Cores
Solr
Cores

16. In context (old/new)
Search implementation
cluster management
Search implementation
Cluster management

17. Indexing on a single replica...
Search implementation
cluster management
...simplifies pushing updates to
Blob Store.
But server failures happen, and
indexing for a core may have to
switch to another server.

18. Indexing on a single replica...
Search implementation
cluster management
...simplifies pushing updates to
Blob Store.
But server failures happen, and
indexing for a core may have to
switch to another server.
More on that later.

19. Agenda
• Salesforce, current Search implementation
• New paradigm
• Solr on Blob Store implementation
• Numbers and conclusion

20. /replication (refresher)
Solr
Replica
Solr
Primary
.../indexversion - which generation are you at?
.../filelist - list files in index generation
.../filecontent - get file
• Solr (Lucene) segments are immutable
• Replica resolves conflicts, decides which files to fetch
• Primary does not ship corrupt cores

21. Solr
Cores
Blob Store central storage
Solr
Cores
Solr
Cores
Design principles
13
2 Build upon
Replication
logic
Gracefully handle
concurrent updates
Local “cache”:
no impact on
performance
Gracefully handle
concurrent updates
to Blob Store

22. Replication vs. Blob push/pull
Replication Blob push/pull
Two servers negotiate data
exchange
Single server making decisions
Logic on both sides Blob store does not know search
Asynchronous exchange ● Pushing asynchronously
● Pulling synchronously/asynchronously

23. Anatomy of a core on Blob Store
Solr core files
_0.fdt
_0.fdx
_0.fnm
_0_Lucene41_0.doc
_0_Lucene41_0.pos
_0_Lucene41_0.tim
_0_Lucene41_0.tip
_0_Lucene42_0.dvd
_0_Lucene42_0.dvm
_0.nvd
_0.nvm
_0.si
segments_2
segments.gen
Blob files
a8620b64-781a-402f-9da5-39d4468f86cc
b99536c0-18ea-4e9b-85ac-e8d4e7696bff
d59094e7-2e6a-471d-8300-614935f6ab63
4ebd2367-ae34-4d2b-aa81-90717ec5cb31
5cdccf4e-423d-4661-9bec-b7dcfbb42d5d
ae970e08-21b0-456a-9858-11a753c11a12
755e4030-52b3-44bb-8dd7-ab536f96ced0
7a31af5c-d5bd-484f-a008-afba5f8ae983
0a83c0ca-8630-4279-950e-7bc793c4301d
5c0a6fe7-676d-4805-af0f-783500afdf26
3151ac3e-4c8b-4496-9ea8-6f48dc1475ab
fac7f404-2b33-40ec-bcfc-662dc66cf9de
3fe658b4-cd04-4434-902b-600d52599f25
core.metadata
Blob file names in bucket prefixed by core name

24. core.metadata content
Solr core files
_0.fdt
_0.fdx
_0.fnm
_0_Lucene41_0.doc
_0_Lucene41_0.pos
_0_Lucene41_0.tim
_0_Lucene41_0.tip
_0_Lucene42_0.dvd
_0_Lucene42_0.dvm
_0.nvd
_0.nvm
_0.si
segments_2
segments.gen
Blob files
a8620b64-781a-402f-9da5-39d4468f86cc
b99536c0-18ea-4e9b-85ac-e8d4e7696bff
d59094e7-2e6a-471d-8300-614935f6ab63
4ebd2367-ae34-4d2b-aa81-90717ec5cb31
5cdccf4e-423d-4661-9bec-b7dcfbb42d5d
ae970e08-21b0-456a-9858-11a753c11a12
755e4030-52b3-44bb-8dd7-ab536f96ced0
7a31af5c-d5bd-484f-a008-afba5f8ae983
0a83c0ca-8630-4279-950e-7bc793c4301d
5c0a6fe7-676d-4805-af0f-783500afdf26
3151ac3e-4c8b-4496-9ea8-6f48dc1475ab
fac7f404-2b33-40ec-bcfc-662dc66cf9de
3fe658b4-cd04-4434-902b-600d52599f25
core.metadata{
"coreName": "activateSlidesCore",
"sequenceNumber": 10,
"generation": 2,
"uniqueIdentifier":
"c2d34229-637a-4ca5-a0ff-49513ce5d445",
"isCorrupt": false,
"isDeleted": false,
"blobFiles": [
{
"solrFileName": "_0.fdt",
"blobName":
"a8620b64-781a-402f-9da5-39d4468f86cc",
"checksum": e430606
},
...mo le p s e ...
{
"solrFileName": "_0_Lucene41_0.doc",
"blobName": "4ebd2367-ae34-4d2b-aa81-90717ec5cb31",
"checksum": a9b320e
},
{
"solrFileName": "segments_2",
"blobName": "3fe658b4-cd04-4434-902b-600d52599f25",
"checksum": 0a32c8e
}
],
"blobConfigFiles": [],
"blobFilesToDelete": []
}

25. Pushing to Blob Store
• Local change (indexing, encryption)
captured in (deduplicated) memory log
• Async “pusher” tasks push updates to
Blob store, use unique file names.

26. Pulling from Blob Store
Synchronous:
• When core not present locally,
• When indexing on a core not up to date.
Asynchronous:
• Getting updates pushed by other
servers
Restores original Solr file names!

27. Deleting from Blob Store
1. Files no longer in the current commit point
2. Deleted (no longer referenced) cores
3. Lost files due to blob write races
...Done asynchronously.

28. Why unique file names?
_0.fnm
_0_Lucene41_0.doc
_0_Lucene41_0.pos
d59094e7-2e6a-471d-8300-614935f6ab63
4ebd2367-ae34-4d2b-aa81-90717ec5cb31
5cdccf4e-423d-4661-9bec-b7dcfbb42d5d
On local core update pushed to Blob Store:
• Solr server pushes all new files to Blob
• Once written, server updates core.metadata
In the case of concurrent writes from Solr to Blob Store
(due to indexing switching servers and pushing being
async), unique file names guarantee data consistency!

29. Agenda
• Salesforce, current Search implementation
• New paradigm
• Solr on Blob Store implementation
• Numbers and conclusion

30. Is this enough for HA?
Does having the Blob Store version allow us to
consider we’re Highly Available (HA), even
if cores are otherwise only loaded only
on a single Solr server?
It depends...

31. 65 cores accessed in first 10 seconds
88 cores accessed in first minute
95 cores accessed in first two minutes
cores have to be loaded and accessed on another server
If a server fails...
Load time for 65 cores < 6 seconds!
Starting Wednesday 2:00pm on a specific Solr server:

32. Current status and next steps
• Code being finalized and perf/robustness tested
• Will soon enter shadow production testing
• Improve Salesforce Search deployments
on Public Cloud
• Would it be useful to you?
• Open source?

33. Thank you!
Ilan Ginzburg
Architect, Salesforce
@ilanGinzburg
#Activate18 #ActivateSearch