We've shared our experience of migrating from External Files to Solr Payloads for storage of ranking data. The migration resulted in a huge savings in memory footprint and more stable systems, capable of supporting multiple A/B tests. Presentation by Soubhik Bhattacharya

1. Solr Payloads for Ranking Data
Soubhik
Search Quality, BloomReach, SNAP

2. Outline
● Ranking Data in External File
● Issues
● Payloads
● Benefits
Primary Contributors:
● Renuka Khandelwal, Ricardo Shih, Parag Agrawal

3. Ranking Data

4. Ranking Data
● computed offline
● a score is attached to terms in a product
● stored in an external data file
● a Solr FunctionQuery [2] is used to read the score from the file and apply in the
ranking equation
○ loads the data from the file to a Java HashMap<String, Float>

5. Issues

6. Issues
● collection specific ranking data
○ 500 MB memory for one collection

7. Issues
● collection specific ranking data
○ 500 MB memory for one collection
● 6 data centers, 6 replicas : 18 GB for one collection

8. Issues
● collection specific ranking data
○ 500 MB memory for one collection
● 6 data centers, 6 replicas : 18 GB for one collection
● in a multi-sharded collection: X number of shards

9. Issues
● collection specific ranking data
○ 500 MB memory for one collection
● 6 data centers, 6 replicas : 18 GB for one collection
● in a multi-sharded collection: X number of shards
● X number of merchants (~75)

10. Issues
● collection specific ranking data
○ 500 MB memory for one collection
● 6 data centers, 6 replicas : 18 GB for one collection
● in a multi-sharded collection: X number of shards
● X number of merchants (~75)
● reload: 2X

11. Issues
● collection specific ranking data
○ 500 MB memory for one collection
● 6 data centers, 6 replicas : 18 GB for one collection
● in a multi-sharded collection: X number of shards
● X number of merchants (~75)
● reload: 2X
● what about A/B test?

12. Payloads

13. Payloads
A payload is an arbitrary data that can be attached to an indexed term. It is stored as a
byte array.
● the precomputed ranking score is stored as payloads
● during scoring, the stored payload is used instead of the TF-IDF

14. Benefits

15. ● better memory utilization

16. ● better memory utilization
○ indexed terms are not replicated in memory

17. ● better memory utilization
○ indexed terms are not replicated in memory
○ saved the overhead of Java HashMap: the String and Map.Entry objects
■ for English words, the average overhead of String is 3X

18. ● better memory utilization
○ indexed terms are not replicated in memory
○ saved the overhead of Java HashMap: the String and Map.Entry objects
■ for English words, the average overhead of String is 3X
○ only those terms are stored that are present in index of a given shard

19. ● better memory utilization
○ indexed terms are not replicated in memory
○ saved the overhead of Java HashMap: the String and Map.Entry objects
■ for English words, the average overhead of String is 3X
○ only those terms are stored that are present in index of a given shard
○ 60% reduction in JVM memory utilization

20. ● better memory utilization
○ indexed terms are not replicated in memory
○ saved the overhead of Java HashMap: the String and Map.Entry objects
■ for English words, the average overhead of String is 3X
○ only those terms are stored that are present in index of a given shard
○ 60% reduction in JVM memory utilization

21. ● better memory utilization
○ indexed terms are not replicated in memory
○ saved the overhead of Java HashMap: the String and Map.Entry objects
■ for English words, the average overhead of String is 3X
○ only those terms are stored that are present in index of a given shard
○ 60% reduction in JVM memory utilization
● no additional reloads

22. ● A/B Testability
● use ranking data from field1 or field2 depending on the request parameters
field2
field1 Algo1
field2
field1
Algo2

23. References

24. 1. Working with External Files and Processes
2. FunctionQuery
3. Payloads

25. EFF

26. EFF (External File Field)
● a schema fieldtype that allows the data to be read from a file outside of the index
[1] e.g.
the value of the field is read from a file external_category_rank.txt
<fieldType name="rankingData" keyField="category" defVal="0" stored="false" indexed="false" class="solr.
ExternalFileField" valType="pfloat"/>
<field name="category_rank" type="rankingData" />
fashion=2.2
nike=0.366