Download as PDF, PPTX
More Related Content
![Jacob Marble [InfluxData] | Observability with InfluxDB IOx and OpenTelemetry...](https://cdn.slidesharecdn.com/ss_thumbnails/influxdbioxandopentelemetry-27-oct-2021jmarble-211022195239-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Algolia - Hosted Search API
Algolia - Hosted Search API
- 1. Instant Search API BuildUnique Search Experiences Sylvain Utard VP of Engineering sylvain@algolia.com @sylvainutard Enterprise Search and Analytics
- 2. @algolia Who am I? 5years @ Exalead, leading the core-engine & NLP teams • C++ • ExaScript (RIP) • Java 2 years @ Algolia, VP of Engineering • C++ • Ruby • Java • and 10+ other languages… @sylvainutard
- 3.
- 4.
- 5.
- 6. @algolia A hosted searchAPI Replies in milliseconds
- 7. @algolia A hosted searchAPI Replies in milliseconds From anywhere
- 8. @algolia A hosted searchAPI Replies in milliseconds From anywhere With intuitive relevance
- 9.
- 10. @algolia 800+customers in 80+countries Algolia Today
- 11. @algolia 800+customers in 80+countries 40B+ Write operationsper month 4B+ User-generated queriesper month Algolia Today
- 12. @algolia Algolia Today 13locations 800+customers in80+ countries 40B+ Write operationsper month 4B+ User-generated queriesper month
- 13.
- 14. @algolia Speed matters Half asecond delay caused 20% drop in traffic Every 100ms of latency costs them 1% in sales
- 15.
- 16. @algolia Unique set ofconstraints High volume of Read & Write operations
- 17. @algolia Unique set ofconstraints High volume of Read & Write operations High-availability
- 18. @algolia Unique set ofconstraints High volume of Read & Write operations High-availability Worldwide data distribution
- 19. @algolia API Software Stack Startedas a mobile offline SDK Written in C++ Search code embedded in Nginx as a module Indexing is done in a separate process Two redis instances
- 20. @algolia API Hardware Fast CPU(Xeon E5 >3.5GHz) In Memory (128GB) Backed by High-end SSD in Raid-0 (800GB) Specific kernel settings
- 21. @algolia Scaling horizontally Several clustersper location A user is assigned to one master cluster A user can be replicated to N replicate clusters
- 22. @algolia What is acluster Master-Master Stream of writes via Consensus At least 3 machines
- 23. @algolia A write inpractice One of the machines accept the write operation via the API (https) /1/indexes/MyFirstIndex/batch
- 24. @algolia A write inpractice The file is saved on the three machines as a temporary file tmp1265 tmp7864 tmp2357
- 25. @algolia A write inpractice Launch the consensus by contacting the RAFT master startConsensus(tmp2357, tmp7864, tmp1265)
- 26. @algolia A write inpractice 1 -Master send the commit order to all nodes 2- Each node returns the next job ID to master 3- If there is a majority the file is committed
- 27. @algolia A write inpractice Same job ID on all hosts Send to slave replicate in parallel Processed in parallel on all hosts job42 job42 job42
- 28. @algolia In case onehost is down Continue to accept writes The two other hosts keep jobs Jobs are sequential, will catch up at restart job42job42
- 29. @algolia Distribution Replicate jobs, notthe result Send to all machines in parallel Consistent with few seconds delay
- 30. @algolia High availability Multi-regions inone location
- 31. @algolia High availability 13 fullyindependent locations
- 32. @algolia Network Optimisations API usagemoving from servers to browser and mobile apps Get close to end users
- 33. @algolia Distributed Search Network- Worldwide Synchronization
- 34. @algolia Distributed Search Network- Worldwide Synchronization
- 35. @algolia • 13 locations= 25 datacenters • No ideal worldwide provider • AWS is not in India, Eastern EU, Africa… • Need to handle several providers • Anticipate long deliveries / customs • Keep as few providers as possible Distributed Search Network - Worldwide Synchronization
- 36. @algolia DNS is key Usedto find the closest location Several DNS providers Good anycast network
- 37. @algolia API Clients DNS healthchecks are not enough Smart retry logic in all our API Clients
- 38. @algolia Analytics • What aremy users searching for? • Top search • Top search without hits • Top refinements • From where do they search for?
- 39.
- 40.
- 41. @algolia Analytics • Billions ofuser-generated queries per month • As-you-type aggregation • ~3 months retentions • Storing all of them in…
- 42. @algolia Analytics • Elasticsearch o/ •… without FTS :) • but with aggregations
- 43. @algolia Analytics• No FTS •No source • Doc values everywhere • SSD only • Custom aggregations (deprecated since ES 1.1.0)
- 44. @algolia Top-k Aggregation • Before •Linear memory consumption • Exhaustivity • After • Constant memory consumption • Approximative but enough
- 45. @algolia Building your worldwideinfra - Is long and difficult quest - Is a real asset & differentiator The Future of APIs is Distributed
- 46. @algolia All the detailsof our architecture are on HighScalability.com Want to know more?
- 47. THANK YOU! sylvain@algolia.com @algolia Build UniqueSearch Experiences W e are hiring in SF, NYC and Paris 😊