SlideShare a Scribd company logo

Distributed Search in Riak - Integrating Search in a NoSQL Database: Presented by Fred Dushin, Basho Technologies

Lucene/Solr Revolution 2015

1 of 32
Download to read offline
O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
Distributed Search in Riak
Integrating search in a NoSQL database
Fred Dushin
Member of Technical Staff
Basho Technologies
3
About Me
CORBA -> Web Services -> MoM
Joined Basho Jan 2015
Reach out!
github://fadushin
lr2015@dushin.net
4
What I want to talk about
How is Query even possible in a distributed
NoSQL database?
What happens when things break?
How does Riak distribute data?
How does Riak repair divergence?
What is Riak? What is Riak Search?
What does Solr bring to Riak?
What does Riak bring to Solr?
5
What is Riak?
A Distributed key-value store
Prioritizes availability over consistency
Provides elasticity without downtime
6
A Riak Glossary
• Key
... any sequence of bytes
• Value
... any opaque blob of data
• Bucket
... an organizing namespace for keys
• Bucket Type
... an organizing namespace for buckets
{{BucketType, Bucket}, Key} -> Value
"BKey"

Recommended

Riak at The NYC Cloud Computing Meetup Group
Riak at The NYC Cloud Computing Meetup GroupRiak at The NYC Cloud Computing Meetup Group
Riak at The NYC Cloud Computing Meetup Groupsiculars
 
Annotation processing and code gen
Annotation processing and code genAnnotation processing and code gen
Annotation processing and code genkoji lin
 
Apache Spark for Library Developers with Erik Erlandson and William Benton
Apache Spark for Library Developers with Erik Erlandson and William BentonApache Spark for Library Developers with Erik Erlandson and William Benton
Apache Spark for Library Developers with Erik Erlandson and William BentonDatabricks
 
PostgreSQL Open SV 2018
PostgreSQL Open SV 2018PostgreSQL Open SV 2018
PostgreSQL Open SV 2018artgillespie
 
OWASP AppSecCali 2015 - Marshalling Pickles
OWASP AppSecCali 2015 - Marshalling PicklesOWASP AppSecCali 2015 - Marshalling Pickles
OWASP AppSecCali 2015 - Marshalling PicklesChristopher Frohoff
 
"ClojureScript journey: from little script, to CLI program, to AWS Lambda fun...
"ClojureScript journey: from little script, to CLI program, to AWS Lambda fun..."ClojureScript journey: from little script, to CLI program, to AWS Lambda fun...
"ClojureScript journey: from little script, to CLI program, to AWS Lambda fun...Julia Cherniak
 

More Related Content

What's hot

Sane Sharding with Akka Cluster
Sane Sharding with Akka ClusterSane Sharding with Akka Cluster
Sane Sharding with Akka Clustermiciek
 
Intravert Server side processing for Cassandra
Intravert Server side processing for CassandraIntravert Server side processing for Cassandra
Intravert Server side processing for CassandraEdward Capriolo
 
Redis for the Everyday Developer
Redis for the Everyday DeveloperRedis for the Everyday Developer
Redis for the Everyday DeveloperRoss Tuck
 
Don't Be Afraid of Abstract Syntax Trees
Don't Be Afraid of Abstract Syntax TreesDon't Be Afraid of Abstract Syntax Trees
Don't Be Afraid of Abstract Syntax TreesJamund Ferguson
 
Automated malware analysis
Automated malware analysisAutomated malware analysis
Automated malware analysisIbrahim Baliç
 
Solr & Lucene @ Etsy by Gregg Donovan
Solr & Lucene @ Etsy by Gregg DonovanSolr & Lucene @ Etsy by Gregg Donovan
Solr & Lucene @ Etsy by Gregg DonovanGregg Donovan
 
Node.js: Continuation-Local-Storage and the Magic of AsyncListener
Node.js: Continuation-Local-Storage and the Magic of AsyncListenerNode.js: Continuation-Local-Storage and the Magic of AsyncListener
Node.js: Continuation-Local-Storage and the Magic of AsyncListenerIslam Sharabash
 
Scala ActiveRecord
Scala ActiveRecordScala ActiveRecord
Scala ActiveRecordscalaconfjp
 
Object Oriented Exploitation: New techniques in Windows mitigation bypass
Object Oriented Exploitation: New techniques in Windows mitigation bypassObject Oriented Exploitation: New techniques in Windows mitigation bypass
Object Oriented Exploitation: New techniques in Windows mitigation bypassSam Thomas
 
Wprowadzenie do technologii Big Data / Intro to Big Data Ecosystem
Wprowadzenie do technologii Big Data / Intro to Big Data EcosystemWprowadzenie do technologii Big Data / Intro to Big Data Ecosystem
Wprowadzenie do technologii Big Data / Intro to Big Data EcosystemSages
 
AST - the only true tool for building JavaScript
AST - the only true tool for building JavaScriptAST - the only true tool for building JavaScript
AST - the only true tool for building JavaScriptIngvar Stepanyan
 
Redis in Practice
Redis in PracticeRedis in Practice
Redis in PracticeNoah Davis
 
Beyond Profilers: Tracing Node.js Transactions
Beyond Profilers: Tracing Node.js TransactionsBeyond Profilers: Tracing Node.js Transactions
Beyond Profilers: Tracing Node.js TransactionsTerral R Jordan
 
Juggling Chainsaws: Perl and MongoDB
Juggling Chainsaws: Perl and MongoDBJuggling Chainsaws: Perl and MongoDB
Juggling Chainsaws: Perl and MongoDBDavid Golden
 
A JIT Smalltalk VM written in itself
A JIT Smalltalk VM written in itselfA JIT Smalltalk VM written in itself
A JIT Smalltalk VM written in itselfESUG
 
Powershell for Log Analysis and Data Crunching
 Powershell for Log Analysis and Data Crunching Powershell for Log Analysis and Data Crunching
Powershell for Log Analysis and Data CrunchingMichelle D'israeli
 
Say Hello To Ecmascript 5
Say Hello To Ecmascript 5Say Hello To Ecmascript 5
Say Hello To Ecmascript 5Juriy Zaytsev
 
Http4s, Doobie and Circe: The Functional Web Stack
Http4s, Doobie and Circe: The Functional Web StackHttp4s, Doobie and Circe: The Functional Web Stack
Http4s, Doobie and Circe: The Functional Web StackGaryCoady
 

What's hot (20)

Sane Sharding with Akka Cluster
Sane Sharding with Akka ClusterSane Sharding with Akka Cluster
Sane Sharding with Akka Cluster
 
JavaScript on the GPU
JavaScript on the GPUJavaScript on the GPU
JavaScript on the GPU
 
Intravert Server side processing for Cassandra
Intravert Server side processing for CassandraIntravert Server side processing for Cassandra
Intravert Server side processing for Cassandra
 
Redis for the Everyday Developer
Redis for the Everyday DeveloperRedis for the Everyday Developer
Redis for the Everyday Developer
 
Don't Be Afraid of Abstract Syntax Trees
Don't Be Afraid of Abstract Syntax TreesDon't Be Afraid of Abstract Syntax Trees
Don't Be Afraid of Abstract Syntax Trees
 
Automated malware analysis
Automated malware analysisAutomated malware analysis
Automated malware analysis
 
Solr & Lucene @ Etsy by Gregg Donovan
Solr & Lucene @ Etsy by Gregg DonovanSolr & Lucene @ Etsy by Gregg Donovan
Solr & Lucene @ Etsy by Gregg Donovan
 
Node.js: Continuation-Local-Storage and the Magic of AsyncListener
Node.js: Continuation-Local-Storage and the Magic of AsyncListenerNode.js: Continuation-Local-Storage and the Magic of AsyncListener
Node.js: Continuation-Local-Storage and the Magic of AsyncListener
 
Scala ActiveRecord
Scala ActiveRecordScala ActiveRecord
Scala ActiveRecord
 
Object Oriented Exploitation: New techniques in Windows mitigation bypass
Object Oriented Exploitation: New techniques in Windows mitigation bypassObject Oriented Exploitation: New techniques in Windows mitigation bypass
Object Oriented Exploitation: New techniques in Windows mitigation bypass
 
Wprowadzenie do technologii Big Data / Intro to Big Data Ecosystem
Wprowadzenie do technologii Big Data / Intro to Big Data EcosystemWprowadzenie do technologii Big Data / Intro to Big Data Ecosystem
Wprowadzenie do technologii Big Data / Intro to Big Data Ecosystem
 
AST - the only true tool for building JavaScript
AST - the only true tool for building JavaScriptAST - the only true tool for building JavaScript
AST - the only true tool for building JavaScript
 
Redis in Practice
Redis in PracticeRedis in Practice
Redis in Practice
 
Beyond Profilers: Tracing Node.js Transactions
Beyond Profilers: Tracing Node.js TransactionsBeyond Profilers: Tracing Node.js Transactions
Beyond Profilers: Tracing Node.js Transactions
 
Solr @ Etsy - Apache Lucene Eurocon
Solr @ Etsy - Apache Lucene EuroconSolr @ Etsy - Apache Lucene Eurocon
Solr @ Etsy - Apache Lucene Eurocon
 
Juggling Chainsaws: Perl and MongoDB
Juggling Chainsaws: Perl and MongoDBJuggling Chainsaws: Perl and MongoDB
Juggling Chainsaws: Perl and MongoDB
 
A JIT Smalltalk VM written in itself
A JIT Smalltalk VM written in itselfA JIT Smalltalk VM written in itself
A JIT Smalltalk VM written in itself
 
Powershell for Log Analysis and Data Crunching
 Powershell for Log Analysis and Data Crunching Powershell for Log Analysis and Data Crunching
Powershell for Log Analysis and Data Crunching
 
Say Hello To Ecmascript 5
Say Hello To Ecmascript 5Say Hello To Ecmascript 5
Say Hello To Ecmascript 5
 
Http4s, Doobie and Circe: The Functional Web Stack
Http4s, Doobie and Circe: The Functional Web StackHttp4s, Doobie and Circe: The Functional Web Stack
Http4s, Doobie and Circe: The Functional Web Stack
 

Viewers also liked

Schema Design for Riak
Schema Design for RiakSchema Design for Riak
Schema Design for RiakSean Cribbs
 
Riak (Øredev nosql day)
Riak (Øredev nosql day)Riak (Øredev nosql day)
Riak (Øredev nosql day)Sean Cribbs
 
Riak Operations
Riak OperationsRiak Operations
Riak Operationsgschofield
 
Riak a successful failure
Riak   a successful failureRiak   a successful failure
Riak a successful failureGiltTech
 
Riak - From Small to Large
Riak - From Small to LargeRiak - From Small to Large
Riak - From Small to LargeRusty Klophaus
 
Distributed Key-Value Stores- Featuring Riak
Distributed Key-Value Stores- Featuring RiakDistributed Key-Value Stores- Featuring Riak
Distributed Key-Value Stores- Featuring Riaksamof76
 
Riak in Ten Minutes
Riak in Ten MinutesRiak in Ten Minutes
Riak in Ten MinutesJon Meredith
 
Introducing Riak
Introducing RiakIntroducing Riak
Introducing RiakKevin Smith
 
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011Riak Training Session — Surge 2011
Riak Training Session — Surge 2011DstroyAllModels
 
Introduction to Riak - Red Dirt Ruby Conf Training
Introduction to Riak - Red Dirt Ruby Conf TrainingIntroduction to Riak - Red Dirt Ruby Conf Training
Introduction to Riak - Red Dirt Ruby Conf TrainingSean Cribbs
 

Viewers also liked (11)

Schema Design for Riak
Schema Design for RiakSchema Design for Riak
Schema Design for Riak
 
Riak (Øredev nosql day)
Riak (Øredev nosql day)Riak (Øredev nosql day)
Riak (Øredev nosql day)
 
Riak Operations
Riak OperationsRiak Operations
Riak Operations
 
Riak a successful failure
Riak   a successful failureRiak   a successful failure
Riak a successful failure
 
Riak - From Small to Large
Riak - From Small to LargeRiak - From Small to Large
Riak - From Small to Large
 
Distributed Key-Value Stores- Featuring Riak
Distributed Key-Value Stores- Featuring RiakDistributed Key-Value Stores- Featuring Riak
Distributed Key-Value Stores- Featuring Riak
 
Riak in Ten Minutes
Riak in Ten MinutesRiak in Ten Minutes
Riak in Ten Minutes
 
Introducing Riak
Introducing RiakIntroducing Riak
Introducing Riak
 
Relational Databases to Riak
Relational Databases to RiakRelational Databases to Riak
Relational Databases to Riak
 
Riak Training Session — Surge 2011
Riak Training Session — Surge 2011Riak Training Session — Surge 2011
Riak Training Session — Surge 2011
 
Introduction to Riak - Red Dirt Ruby Conf Training
Introduction to Riak - Red Dirt Ruby Conf TrainingIntroduction to Riak - Red Dirt Ruby Conf Training
Introduction to Riak - Red Dirt Ruby Conf Training
 

Similar to Distributed Search in Riak - Integrating Search in a NoSQL Database: Presented by Fred Dushin, Basho Technologies

Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteBram Vogelaar
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013Roy Russo
 
Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)Sean Cribbs
 
Adding Riak to your NoSQL Bag of Tricks
Adding Riak to your NoSQL Bag of TricksAdding Riak to your NoSQL Bag of Tricks
Adding Riak to your NoSQL Bag of Trickssiculars
 
Service Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud InternalsService Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud InternalsAleksandr Tarasov
 
Innovation and Security in Ruby on Rails
Innovation and Security in Ruby on RailsInnovation and Security in Ruby on Rails
Innovation and Security in Ruby on Railstielefeld
 
PUT Knowledge BUCKET Brain KEY Riak
PUT Knowledge BUCKET Brain KEY RiakPUT Knowledge BUCKET Brain KEY Riak
PUT Knowledge BUCKET Brain KEY RiakPhilipp Fehre
 
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜Michitoshi Yoshida
 
Coding with Riak (from Velocity 2015)
Coding with Riak (from Velocity 2015)Coding with Riak (from Velocity 2015)
Coding with Riak (from Velocity 2015)Basho Technologies
 
PuppetDB: A Single Source for Storing Your Puppet Data - PUG NY
PuppetDB: A Single Source for Storing Your Puppet Data - PUG NYPuppetDB: A Single Source for Storing Your Puppet Data - PUG NY
PuppetDB: A Single Source for Storing Your Puppet Data - PUG NYPuppet
 
Understanding OpenStack Deployments - PuppetConf 2014
Understanding OpenStack Deployments - PuppetConf 2014Understanding OpenStack Deployments - PuppetConf 2014
Understanding OpenStack Deployments - PuppetConf 2014Puppet
 
Riak from Small to Large
Riak from Small to LargeRiak from Small to Large
Riak from Small to LargeRusty Klophaus
 
KazooCon 2014 - Introduction to Kazoo APIs!
KazooCon 2014 - Introduction to Kazoo APIs!KazooCon 2014 - Introduction to Kazoo APIs!
KazooCon 2014 - Introduction to Kazoo APIs!2600Hz
 
Who is afraid of privileged containers ?
Who is afraid of privileged containers ?Who is afraid of privileged containers ?
Who is afraid of privileged containers ?Marko Bevc
 
IronSmalltalk
IronSmalltalkIronSmalltalk
IronSmalltalkESUG
 
Sending a for ahuh. win32 exploit development old school
Sending a for ahuh. win32 exploit development old schoolSending a for ahuh. win32 exploit development old school
Sending a for ahuh. win32 exploit development old schoolNahidul Kibria
 
About elasticsearch
About elasticsearchAbout elasticsearch
About elasticsearchMinsoo Jun
 
Open Source Search: An Analysis
Open Source Search: An AnalysisOpen Source Search: An Analysis
Open Source Search: An AnalysisJustin Finkelstein
 

Similar to Distributed Search in Riak - Integrating Search in a NoSQL Database: Presented by Fred Dushin, Basho Technologies (20)

Integrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suiteIntegrating icinga2 and the HashiCorp suite
Integrating icinga2 and the HashiCorp suite
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013
 
Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)Introduction to Riak and Ripple (KC.rb)
Introduction to Riak and Ripple (KC.rb)
 
Adding Riak to your NoSQL Bag of Tricks
Adding Riak to your NoSQL Bag of TricksAdding Riak to your NoSQL Bag of Tricks
Adding Riak to your NoSQL Bag of Tricks
 
Service Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud InternalsService Discovery. Spring Cloud Internals
Service Discovery. Spring Cloud Internals
 
Innovation and Security in Ruby on Rails
Innovation and Security in Ruby on RailsInnovation and Security in Ruby on Rails
Innovation and Security in Ruby on Rails
 
PUT Knowledge BUCKET Brain KEY Riak
PUT Knowledge BUCKET Brain KEY RiakPUT Knowledge BUCKET Brain KEY Riak
PUT Knowledge BUCKET Brain KEY Riak
 
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜
DBA だってもっと効率化したい!〜最近の自動化事情とOracle Database〜
 
Yokozuna
YokozunaYokozuna
Yokozuna
 
Coding with Riak (from Velocity 2015)
Coding with Riak (from Velocity 2015)Coding with Riak (from Velocity 2015)
Coding with Riak (from Velocity 2015)
 
PuppetDB: A Single Source for Storing Your Puppet Data - PUG NY
PuppetDB: A Single Source for Storing Your Puppet Data - PUG NYPuppetDB: A Single Source for Storing Your Puppet Data - PUG NY
PuppetDB: A Single Source for Storing Your Puppet Data - PUG NY
 
Understanding OpenStack Deployments - PuppetConf 2014
Understanding OpenStack Deployments - PuppetConf 2014Understanding OpenStack Deployments - PuppetConf 2014
Understanding OpenStack Deployments - PuppetConf 2014
 
Riak from Small to Large
Riak from Small to LargeRiak from Small to Large
Riak from Small to Large
 
KazooCon 2014 - Introduction to Kazoo APIs!
KazooCon 2014 - Introduction to Kazoo APIs!KazooCon 2014 - Introduction to Kazoo APIs!
KazooCon 2014 - Introduction to Kazoo APIs!
 
Who is afraid of privileged containers ?
Who is afraid of privileged containers ?Who is afraid of privileged containers ?
Who is afraid of privileged containers ?
 
IronSmalltalk
IronSmalltalkIronSmalltalk
IronSmalltalk
 
Sending a for ahuh. win32 exploit development old school
Sending a for ahuh. win32 exploit development old schoolSending a for ahuh. win32 exploit development old school
Sending a for ahuh. win32 exploit development old school
 
About elasticsearch
About elasticsearchAbout elasticsearch
About elasticsearch
 
Elastic{ON} 2017 Recap
Elastic{ON} 2017 RecapElastic{ON} 2017 Recap
Elastic{ON} 2017 Recap
 
Open Source Search: An Analysis
Open Source Search: An AnalysisOpen Source Search: An Analysis
Open Source Search: An Analysis
 

More from Lucidworks

Search is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategySearch is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategyLucidworks
 
How Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsHow Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsLucidworks
 
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks
 
Connected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesConnected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesLucidworks
 
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Lucidworks
 
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...Lucidworks
 
Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Lucidworks
 
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Lucidworks
 
AI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and RosetteAI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and RosetteLucidworks
 
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentThe Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentLucidworks
 
Webinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeWebinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeLucidworks
 
Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Lucidworks
 
Applying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchApplying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchLucidworks
 
Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1Lucidworks
 
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyWebinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyLucidworks
 
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Lucidworks
 
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceApply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceLucidworks
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchLucidworks
 
Why Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondWhy Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondLucidworks
 
Webinar: Lucidworks Managed Search
Webinar: Lucidworks Managed SearchWebinar: Lucidworks Managed Search
Webinar: Lucidworks Managed SearchLucidworks
 

More from Lucidworks (20)

Search is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce StrategySearch is the Tip of the Spear for Your B2B eCommerce Strategy
Search is the Tip of the Spear for Your B2B eCommerce Strategy
 
How Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant ProductsHow Crate & Barrel Connects Shoppers with Relevant Products
How Crate & Barrel Connects Shoppers with Relevant Products
 
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product DiscoveryLucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
Lucidworks & IMRG Webinar – Best-In-Class Retail Product Discovery
 
Connected Experiences Are Personalized Experiences
Connected Experiences Are Personalized ExperiencesConnected Experiences Are Personalized Experiences
Connected Experiences Are Personalized Experiences
 
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
Intelligent Insight Driven Policing with MC+A, Toronto Police Service and Luc...
 
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
[Webinar] Intelligent Policing. Leveraging Data to more effectively Serve Com...
 
Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020Preparing for Peak in Ecommerce | eTail Asia 2020
Preparing for Peak in Ecommerce | eTail Asia 2020
 
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
Accelerate The Path To Purchase With Product Discovery at Retail Innovation C...
 
AI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and RosetteAI-Powered Linguistics and Search with Fusion and Rosette
AI-Powered Linguistics and Search with Fusion and Rosette
 
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual MomentThe Service Industry After COVID-19: The Soul of Service in a Virtual Moment
The Service Industry After COVID-19: The Soul of Service in a Virtual Moment
 
Webinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - EuropeWebinar: Smart answers for employee and customer support after covid 19 - Europe
Webinar: Smart answers for employee and customer support after covid 19 - Europe
 
Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19Smart Answers for Employee and Customer Support After COVID-19
Smart Answers for Employee and Customer Support After COVID-19
 
Applying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 ResearchApplying AI & Search in Europe - featuring 451 Research
Applying AI & Search in Europe - featuring 451 Research
 
Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1Webinar: Accelerate Data Science with Fusion 5.1
Webinar: Accelerate Data Science with Fusion 5.1
 
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce StrategyWebinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
Webinar: 5 Must-Have Items You Need for Your 2020 Ecommerce Strategy
 
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
Where Search Meets Science and Style Meets Savings: Nordstrom Rack's Journey ...
 
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision IntelligenceApply Knowledge Graphs and Search for Real-World Decision Intelligence
Apply Knowledge Graphs and Search for Real-World Decision Intelligence
 
Webinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise SearchWebinar: Building a Business Case for Enterprise Search
Webinar: Building a Business Case for Enterprise Search
 
Why Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and BeyondWhy Insight Engines Matter in 2020 and Beyond
Why Insight Engines Matter in 2020 and Beyond
 
Webinar: Lucidworks Managed Search
Webinar: Lucidworks Managed SearchWebinar: Lucidworks Managed Search
Webinar: Lucidworks Managed Search
 

Recently uploaded

"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes", Volodymyr TsapFwdays
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaISPMAIndia
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsEvangelia Mitsopoulou
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1Inbay UK
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Product School
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolProduct School
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotelPhilippines
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor FesenkoFwdays
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...DianaGray10
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfkatalinjordans1
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...UiPathCommunity
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stackSummit
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeJosh Gellers
 
Pragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdfPragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdfinfogdgmi
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17Ana-Maria Mihalceanu
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceSusan Ibach
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceVijayananda Mohire
 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, TripadvisorProduct School
 

Recently uploaded (20)

"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
 
Battle of React State Managers in frontend applications
Battle of React State Managers in frontend applicationsBattle of React State Managers in frontend applications
Battle of React State Managers in frontend applications
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product School
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company Profile
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdf
 
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
Dev Dives: Leverage APIs and Gen AI to power automations for RPA and software...
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stack
 
Artificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human JusticeArtificial Intelligence, Design, and More-than-Human Justice
Artificial Intelligence, Design, and More-than-Human Justice
 
Pragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdfPragmatic UI testing with Compose Semantics.pdf
Pragmatic UI testing with Compose Semantics.pdf
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
 
Confoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data scienceConfoo 2024 Gettings started with OpenAI and data science
Confoo 2024 Gettings started with OpenAI and data science
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial Intelligence
 
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner,  Challenge Like a VC by former CPO, TripadvisorAct Like an Owner,  Challenge Like a VC by former CPO, Tripadvisor
Act Like an Owner, Challenge Like a VC by former CPO, Tripadvisor
 

Distributed Search in Riak - Integrating Search in a NoSQL Database: Presented by Fred Dushin, Basho Technologies

  • 1. O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
  • 2. Distributed Search in Riak Integrating search in a NoSQL database Fred Dushin Member of Technical Staff Basho Technologies
  • 3. 3 About Me CORBA -> Web Services -> MoM Joined Basho Jan 2015 Reach out! github://fadushin lr2015@dushin.net
  • 4. 4 What I want to talk about How is Query even possible in a distributed NoSQL database? What happens when things break? How does Riak distribute data? How does Riak repair divergence? What is Riak? What is Riak Search? What does Solr bring to Riak? What does Riak bring to Solr?
  • 5. 5 What is Riak? A Distributed key-value store Prioritizes availability over consistency Provides elasticity without downtime
  • 6. 6 A Riak Glossary • Key ... any sequence of bytes • Value ... any opaque blob of data • Bucket ... an organizing namespace for keys • Bucket Type ... an organizing namespace for buckets {{BucketType, Bucket}, Key} -> Value "BKey"
  • 7. 7 Riak Partitions 1 2 3 45 6 8 7 ring_size=8 2^160/4 0 BKey_1 BKey_2 BKey_3 BKey_4 BKey_n ... sha1(BKey_i) = 3671 A68E 1098 CDEE 9F4B 2^160 * 3/4 2^160/2 A 20 byte hash, or, A really big number between 0 and 2^160 - 1 2^160 = 1461501637330902918203684832716283019655932542976 {1, 0} {2, 182687704666362864775460604089535377456991567872} {3, 365375409332725729550921208179070754913983135744} {4, 548063113999088594326381812268606132370974703616} {5, 730750818665451459101842416358141509827966271488} {6, 913438523331814323877303020447676887284957839360} {7, 1096126227998177188652763624537212264741949407232} {8, 1278813932664540053428224228626747642198940975104}
  • 8. Node 5 Node 4 Node 3 Node 2 Node 1 8 How Partitions are distributed ring_size=8 num_nodes=5 1 2 3 45 6 8 7 1 6 27 3 8 45 A Riak "cluster"
  • 9. 9 How entries are replicated 1 2 3 45 6 8 7 sha1(BKey) -> 6 "responsible" partition "primary" replicas n_val = 3
  • 10. Node 5 Node 4 Node 3 Node 2 Node 1 10 Riak/KV Put 1 6 2 7 3 8 4 5 #!/usr/bin/env python import riak client = riak.RiakClient( host='node4' ) bucket = client.bucket('agents') key = 'agentp' value = {'name_s': "perry", 'type_s': "reptile"} obj = bucket.new(key, value) obj.store() bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} n_val = 3 w = quorum (⌊n_val/2⌋ + 1) ok sha1({agents, agentp}) -> 6
  • 11. n_val = 3 w = quorum Node 5 Node 4 Node 3 Node 2 Node 1 11 Riak/KV Write Availability 1 6 2 7 3 8 4 5 bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} #!/usr/bin/env python import riak client = riak.RiakClient( host='node4' ) bucket = client.bucket('agents') key = 'agentp' value = {'name_s': "perry", 'type_s': "mammal"} obj = bucket.new(key, value) obj.store() ok bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} Hinted Handoff fallback
  • 12. Node 5 Node 4 Node 3 Node 2 Node 1 12 Riak/KV Read Repair 1 6 2 7 3 8 4 5 #!/usr/bin/env python import riak client = riak.RiakClient( host='node4' ) bucket = client.bucket('agents') obj = bucket.get('agentp') bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} n_val = 3 r = quorum {'name_s': "perry", 'type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"}
  • 13. 13 Riak K/V Active Anti-Entropy {BKey, #} {BKey, #} {BKey, #} #Segment_1 {BKey, #} {BKey, #} {BKey, #} #Segment_2 ... {BKey, #} {BKey, #} #Segment_k ... #Seg_1..k #Seg_j..n... #root {BKey, #} {BKey, #} {BKey, #} #Segment_n... ... {BKey, #} {BKey, #} {BKey, #}
  • 14. Node 5 Node 4 Node 3 Node 2 Node 1 14 Riak/KV AAE 1 6 2 7 3 8 4 5 bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} Riak maintains multiple hashtrees for each partition, one for each "replica set" that can be overlap on the partition. Hashtrees are stored persistently on disk Asynchronously updated on data inserts Periodically exchanged between neighbors Divergence in values triggers read repair
  • 16. 16 What is Yokozuna? An extension of Riak which provides search capability over values stored in Riak Data stored and replicated in Riak is automatically indexed in Solr Solr queries are distributed across the Riak cluster http://github.com/basho/yokozuna
  • 17. Erlang BEAM 17 Yokozuna Riak K/V Yokozuna Admin API Query API Solr Monitor Solr Query extractors YZ AAE http http http pipe index/delete/repair http protobuf http protobuf operations analysis Solr Indexing
  • 18. Node 5 Node 4 Node 3 Node 2 Node 1 18 Indexing 1 2 7 3 8 4 5 #!/usr/bin/env python import riak client = riak.RiakClient(host='node4') client.create_search_index('my_index') my_index my_index my_index my_index my_index bucket = client.bucket('agents') bucket.set_properties( {'search_index': 'my_index'} ) key = 'agentp' value = {'name_s': "perry", 'type_s': "mammal"} obj = bucket.new(key, value) obj.store() bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} 6 name_s: ["perry"] type_s: ["mammal"] name_s: ["perry"] type_s: ["mammal"] name_s: ["perry"] type_s: ["mammal"]
  • 19. 19 <!-- XML --> <schema name="default" version="1.5">
 <fields> ... <dynamicField name="*_s" type="string" indexed="true" stored="true" multiValued="false"/> <dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/> ... <!-- Required fields --> <field name="_yz_id" type="_yz_str" indexed="true" stored="true" multiValued="false" required="true"/>
 <field name="_yz_rt" type="_yz_str" indexed="true" stored="true" multiValued="false"/>
 <field name="_yz_rb" type="_yz_str" indexed="true" stored="true" multiValued="false"/>
 <field name="_yz_rk" type="_yz_str" indexed="true" stored="true" multiValued="false"/>
 <field name="_yz_pn" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_fpn" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_vtag" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_err" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_ed" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 </fields> <uniqueKey>_yz_id</uniqueKey> <types> ... <fieldType name="_yz_str" class="solr.StrField" sortMissingLast="true" /> </types> 
 </schema> Default/Custom Schema https://github.com/basho/yokozuna/blob/develop/priv/default_schema.xml
  • 20. 20 Riak Query All Solr queries are made on the Riak endpoint Riak uses distributed (legacy) Solr to route queries to nodes in the Riak cluster using the shards parameter Solr aggregates results and returns result through Riak Riak supports all query features supported in distributed Solr* * Protobuf interfaces currently have some limitations.
  • 21. Node 5 Node 4 Node 3 Node 2 Node 1 21 Covering Sets bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "reptile"} 6 7 83 4 5 1 2 A Covering Set is a subset of all partitions such that for all BKeys in the keyspace, there is exactly one partition in the covering set in which that BKey can be found. Covering Sets are not unique!
  • 22. Node 5 Node 4 Node 3 Node 2 Node 1 22 Query 1 2 7 3 8 4 5 #!/usr/bin/env python import riak client = riak.RiakClient(host='node4') results = bucket.search( 'type_s:mammal' index='my_index' ) my_index my_index my_index my_index my_index 6 name_s: ["perry"] type_s: ["mammal"] name_s: ["perry"] type_s: ["mammal"] name_s: ["perry"] type_s: ["mammal"] _yz_pn: 8 _yz_pn: 7 _yz_pn: 6
  • 23. 23 Query prompt$ curl 'http://node4:8098/search/query/my_index?wt=json&indent=true&q=type_s:mammal' { "responseHeader":{ "status": 0, "QTime": 88, "params":{ "q" :"type_s:reptile", "shards": "node3:8093/internal_solr/my_index,node5:8093/internal_solr/my_index", "node5:8093": "(_yz_pn:5 AND (_yz_fpn:5 OR _yz_fpn:4))", "node3:8093": "_yz_pn:8 OR _yz_pn:3", "indent": "true", "wt": "json"}}, "response":{"numFound":1,"start":0,"maxScore":0.30685282,"docs":[ { "name_s": "perry", "type_s": "mammal", "_yz_id": "1*default*agents*agentp*8", "_yz_rk": "agentp", "_yz_rt": "default", "_yz_rb": "agents"}] } }
  • 25. Node 5 Node 4 Node 3 Node 2 Node 1 25 YZ AAE 1 2 7 3 8 4 5 my_index my_index my_index my_index my_index 6 name_s: ["perry"] type_s: ["mammal"] name_s: ["perry"] type_s: ["mammal"] name_s: ["perry"] type_s: ["mammal"] bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"}
  • 26. Node 2 26 YZ AAE 2 my_index name_s: ["perry"] type_s: ["mammal"] bucket: agents key: agentp value: {"name_s": "perry", "type_s': "mammal"} 7 Yokozuna maintains its own set of AAE tress for data stored in Solr. Hashtrees are stored persistently on disk Updated on indexing operations Periodically exchanged between the K/V AAE tree on the same node If a value is missing in Solr, it is reindexed; if a value is indexed when it shouldn't be, it is deleted. Riak K/V is canonical.
  • 27. 27 Entropy Data Field <!-- XML --> <schema name="default" version="1.5">
 <fields> ... <!-- Required fields --> <field name="_yz_id" type="_yz_str" indexed="true" stored="true" multiValued="false" required="true"/>
 <field name="_yz_rt" type="_yz_str" indexed="true" stored="true" multiValued="false"/>
 <field name="_yz_rb" type="_yz_str" indexed="true" stored="true" multiValued="false"/>
 <field name="_yz_rk" type="_yz_str" indexed="true" stored="true" multiValued="false"/>
 <field name="_yz_pn" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_fpn" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_vtag" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_err" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 <field name="_yz_ed" type="_yz_str" indexed="true" stored="false" multiValued="false"/>
 </fields> <uniqueKey>_yz_id</uniqueKey> <types> ... <fieldType name="_yz_str" class="solr.StrField" sortMissingLast="true" /> </types> 
 </schema>
  • 28. 28 _yz_ed field 2 default my_bucket agentp 8 g2IHDNr2 version bucket type bucket name key object hash partition
  • 29. 29 Entropy Data Query prompt$ curl 'http://node3:8093/internal_solr/my_index/entropy_data?partition=8&limit=1000&wt=json&indent=true' { "responseHeader":{ "status":0, "QTime":1}, "response":{"numFound":135,"start":0,"docs":[ ... { "vsn":"2", "riak_bucket_type":"default", "riak_bucket_name":"my_bucket", "riak_key":"agentp", "base64_hash":"g2IHDNr2" }, ... ]}, "more":false}
  • 31. 31 What does Solr bring to Riak? What does Riak bring to Solr?