The document outlines key lessons learned from the development of searchd by Tomáš Kramár, highlighting the benefits of replication for production data management and testing. It emphasizes the importance of asynchronous processing for error recovery and data integrity, as well as dynamic partitioning in Elasticsearch for efficient querying. Additionally, the document discusses improving site speed by adopting asynchronous loading techniques.

1. Lessons learned from
SearchD development
Tomáš Kramár

2. #1. Replication is awesome
production
staging
localhost
localhost
API
Redis
PUB/SUB

3. #1. Replication is awesome
● Production data on staging!
● Production data on localhost! (on demand)
● Testing new code is easy

4. #2. Asynchronous processing
API Queue
Job
Job

5. #2. Asynchronous processing

6. #2. Asynchronous processing

7. #2. Q
● fatal errors are recoverable & pause
processing
● we are not losing data
● data migrations
● parallel
● in batches
● distributed

8. #3. Dynamic partitioning in ElasticSearch
● Week-based indices
● Elasticsearch templates for mapping
management
● Query only relevant indices

9. #4. Asynchronous UI FTW
1. Site is slow
2. Load everything asynchronously
3. Site is fast! (it seems)