Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
ScyllaDB, profile picture
Uploaded byScyllaDB
17 views

The Challenges of Real-Time AI at Scale by ScyllaDB

What matters most as your AI workloads grow – beyond model accuracy Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production – for example, volatile latencies, stale features reaching the model, and vector index degradation. This webinar takes a practical look at why such problems occur and how to address them. We'll cover what matters most for real-time AI, walk through a framework for what’s required, and get into specifics on how ScyllaDB addresses real-time AI challenges. You will learn how to: - Build pipelines that keep features and embeddings fresh for real-time inference - Design feature stores for consistent low-latency access at scale - Maintain predictable vector search performance, even under continuous updates and large datasets

Technology◦
Download to read offline
Challenges of Real-Time AI at Scale Felipe Mendes, Vibe Coder Tim Koopmans, Token Churner
Felipe Mendes, Technical Director at ScyllaDB + Published Author on Linux and Databases + Uses LLMs to help with college homework (no cheating!) + Vibe code all the things, just because… Why not? Tim Koopmans, Senior Director at ScyllaDB + Fought latency and (sometimes) won + Trained ML models a gazillion times and still can't get it right + Built my very own Bloomberg terminal Introductions 2
Powering India's top social media platform Video recommendation management Real-time fraud detection Seamless experiences across content + devices Network security threat detection Content personalization & recommendation platform Mobile Growth & Monetization Platform Inventory hub for retail operations Property listings and updates Cryptocurrency exchange app Real-time auctions advertising platform Predictable performance for on sale surges Online gaming ad targeting Media streaming for 45M+ subscribers Bridging AI to IT Service Management Real-time ML-driven recommendations Real-time endpoint threat detection and security Real-time personalized recommendations World leading beauty platform behind Avon Real-time AI decisioning for digital advertisers AI-centric customer research platform Powering Unreal Engine real-time asset distribution Real-time interactions at massive scale Always-on e-commerce platform for millions of fans 3 ScyllaDB Users
Agenda + Problems with Scaling AI + ScyllaDB for Real-time AI + Takeaways 4
How relevant is AI for your organization today?
6 “Most AI pipelines work at a prototype level but collapse under production scale”
Latency Spikes 7
“You can’t dig yourself out of tail latency” 8
9 Feature Freshness
“Stale features kill accuracy” 10
11 Vector Index Degradation
“Vectors indexes need maintenance” 12
13 Resource Contention
“You gotta keep’em separated” 14
15 Model Retraining Impact
“Retraining is inevitable” 16
Coupled Failure Loops 17
Which of the following challenges do you anticipate within the next year?
Why ScyllaDB for Real-time AI? + Performance-first distributed architecture + Highly concurrent, lock-free, multi-writer design + Just in time elastic scaling App App App App App App Active/Active, replicated, auto-sharded
AI Trilemma 20 Performance Cost Quality Low recall No high availability High memory usage Multiple replicas >100 ms tail latency
Vector Similarity Feature Store Highly Scalable Database within an AI Stack Dissimilar Dissimilar Similar Results (Matches) Vector Space Query Vector Fast, Accurate Matching of Complex Data Raw Data Sources Real-time Event Streams ML Training ML Serving (Interference) Central Feature Store (ScyllaDB) Ingestion Processing & Training Serving & Inference High Throughput & Low Latency Access 21 AI Umbrella
Vector Similarity Functions 22 SELECT commenter, comment, similarity_cosine(comment_vector, [0.12, 0.34, 0.56, 0.78, 0.91]) FROM myapp.comments ORDER BY comment_vector ANN OF [0.12, 0.34, 0.56, 0.78, 0.91] LIMIT 5; + similarity_cosine + General semantic search (incl. caching) + RAG applications + similarity_dot_product + Blending scores (eg: e-commerce product search) + similarity_euclidean + Anomaly detection + Image processing
23 ScyllaDB Vector Search Vector Database Saved in Searched in Embeddings Embeddings Document Image Audio Transformed into User’s Search Query Transformed into 1 2 3 Perform Similarity Search Accurate Search Result To Provide
24 ScyllaDB Vector Search Architecture
Scaling to 1 Billion Vectors 25 Low Latency, Moderate Recall Slightly Higher Latency, High Recall
The following params impact the recall % Tuning recall % @ ScyllaDB 26 Controls the quality of the index during construction, Higher values → better recall but slower and more memory-intensive indexing. efconstruct (Index Quality) efsearch (Search Accuracy vs. Latency) M (Links per Node) Controls how many bi-directional links each node has in the index graph. Higher values → more connectivity → better recall but more memory and slower build time, also impact search time. Controls accuracy vs. latency during search, Higher values → higher recall but slower queries. Have the highest impact on recall% but also on query performance / latency (the higher efsearch the lower QPS/vCPU). Recall % Recommended M 90% 32 95% 48 99% 64 Recall % Recommended efconstruct 90% 128 95% 256 99% 350 Recall % Recommended efsearch 90% 128 95% 256 99% 500
What is a feature? https://sharechat.com/blogs/artificial-intelligence/how-sharechat-built-a-scalable-cost-efficient-ml-feature-system 27
Feature Store Pipeline 28
29 https://www.scylladb.com/tech-talk/inside-tripadvisors-real-time-personalization-scylladb-aws/
Recommendation system + ML models drive 'For you', Daily Digests features + Hot path + Low-latency, high throughput needs 30
Scaling AI with ScyllaDB 31 Trilemma: Freshness, Latency, Recall + Network fan-out gets expensive + Top-k candidates → Separate DB + Thundering Herd problem + Streaming vs Batched updates + Fresher signals vs Delayed personalization + Popularity bias + High-signal content often favored + Leading to Cold Start Image Ref: https://docs.databricks.com/aws/en/machine-learning/train-recommender-models
What actually matters at scale? 32 + Data freshness + Tail latency consistency + Resource isolation + Index health + Upgrade safety + Cost predictability Model accuracy alone is not sufficient.
What do you need to do? 33 + Monitor freshness, backlog, tail latency, index health + Load test beyond steady state + Isolate workloads aggressively + Design for zero-downtime upgrades + Choose storage engines that tolerate write pressure
“Real-time AI is a distributed systems problem under a costume.” 34
“ScyllaDB solved that problem many years ago” https://deathbyclawd.com/?url=scylladb.com 35
What’s your primary AI use case today?
37 Must-Read Books for High- Performance System Design Access 4 PDFs on building and optimizing data-intensive applications lp.scylladb.com/book-library-offer
Thank you for joining us today. @scylladb scylladb/ slack.scylladb.com @scylladb company/scylladb/ scylladb/