SlideShare a Scribd company logo
1 of 42
Download to read offline
Stop Worrying
Love the SQL! (the Quepid story)
OpenSource Connections
Me/Us
Doug Turnbull
@softwaredoug
Likes: Solr, Elasticsearch, Cassandra,
Postgres
OpenSource Connections
@o19s
Search, Discovery and Analytics
Let us introduce you to freelancing!
OpenSource Connections
Most Importantly we do...
Make my search results more relevant!
“Search Relevancy”
What database works best for problem X?
“(No)SQL Architect/Trusted Advisor”
OpenSource Connections
How products actually get built
Rena: Doug, John can you come by this afternoon?
One of our Solr-based products needs some urgent relevancy
work
Its Friday, it needs to get done today!
Us: Sure!
The Client
(Rena!)
smart
cookie!
OpenSource Connections
A few hours later
Us: we’ve made a bit of progress!
image frustration-1081 by jseliger2
Rena: but everytime we fix something, we break an
existing search!
Us: yeah! we’re stuck in a whack-a-mole-game
other image: whack a mole by jencu
OpenSource Connections
Whack-a-Mole
What search relevancy
work actually looks like
OpenSource Connections
I HAVE AN IDEA
● Middle of the afternoon, I stop doing search
work and start throwing together some
python from flask import Flask
app = Flask(__name__)
Everyone: Doug, stop that, you have important search work to do!
Me: We’re not making any progress!
WE NEED A WAY TO REGRESSION TEST OUR RELEVANCY AS WE TUNE!
Everyone: You’re nuts!
OpenSource Connections
What did I make?
Focus on gathering stakeholder (ie Rena)
feedback on search, coupled w/ workbench
tuning against that feedback
Today we have customers...
… forget that, tell me about your failures!
OpenSource Connections
Our war story
My mistakes:
● Building a product
● Selling a product
● As a user experience engineer
● As an Angular developer
● At choosing databases
OpenSource Connections
Quepid 0.0.0.0.0.0.1
Track multiple user searches
for this query (hdmi cables) Rena rates this document
as a good/bad search result
need to store:
<search> -> <id for search result> -> <rating 1-10>
“hdmi cables” -> “doc1234” -> “10”
*Actual UI may have been much uglier
OpenSource Connections
Data structure selection under duress
● What’s simple, easy, and will persist our
data?
● What plays well with python?
● What can I get working now in Rena’s office?
OpenSource Connections
Redis
● In memory “Data Structure Server”
○ hashes, lists, simple key-> value storage
● Persistent -- write to disk every X minutes
OpenSource Connections
Redis
from redis import Redis
redis = Redis()
redis.set("foo", "bar")
redis.get("foo") # gets ‘bar’
$ pip install redis
Easy to install and go! Specific to our problem:
from redis import Redis
redis = Redis()
ratings = {“doc1234”: “10”,
“doc532”: “5”}
searchQuery = “hdmi cables”
redis.hsetall(searchQuery, ratings)
Store a hash table
at “hdmi cables”
with:
“doc1234” -> “10”
“doc532” -> “5”
OpenSource Connections
Success!
● My insanity paid off that afternoon
● Now we’re left with a pile of hacked together
(terrible) code -- now what?
OpenSource Connections
Adding some features
● Would like to add multiple “cases”
(different search projects that solve different problems)
● Would like to add user accounts
● Still a one-off for Silverchair
OpenSource Connections
Cases
Tuning a cable shopping site... … vs state laws
OpenSource Connections
Cases in Redis?
from redis import Redis
redis = Redis()
ratings = {“doc1234”: “10”,
“doc532”: “5”}
searchQuery = “hdmi cables”
redis.hset(searchQuery, ratings)
Recall our existing implementation
“data model”
Out of the box, redis can deal with 2 levels deep:
{
“hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“ethernet cables”
...
}
Can’t add extra layer (redis hash only one layer)
{“cable site”: {
“hdmi cables”: {...}
“ethernet cables”: {...}
}
“laws site: {...}}
OpenSource Connections
Time to give up Redis?
“All problems in computer science can be solved by another level of indirection” -- David Wheeler
Crazy Idea: Add dynamic prefix to query keys to indicate case, ie:
{
“case_cablestore_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“case_cablestore_ethernet cables”: {
… },
“case_statelaws_car tax”: {
…}
}
Queries for “Cable Store” case
Query for “State Laws” case
redis.keys(“case_cablestore*”)
To Fetch:
OpenSource Connections
Store other info about cases?
New problem: we need to store some information about cases, case name, et
{
“case_cablestore_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“case_cablestore_ethernet cables”: {
… },
“case_statelaws_car tax”: {
…}
}
Where would it go here?
{
“case_cablestore” {
“name”: “cablestore”,
“created” “20140101”
},
“case_cablestore_query_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“case_cablestore_query_ethernet cables”:
{
… },
“case_statelaws_query_car tax”: {
…}
}
OpenSource Connections
Oh but let’s add users
Extrapolating on past patterns {
“user_doug” {
“name”: “Doug”,
“created_date”: “20140101”
},
“user_doug_case_cablestore” {
“name”: “cablestore”,
“created_date” “20140101”
},
“user_doug_case_cablestore_query_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“user_doug_case_cablestore_query_ethernet cables”:
{
… },
“user_tom_case_statelaws_query_car tax”: {
…}
}image: Rage Wallpaper from Flickr user Thoth God of Knowledge
You right now!
OpenSource Connections
Step Back
We ask ourselves: Is this tool a
product? Is it useful outside of this
customer?
What level of software engineering helps us move forward?
● Migrate to RDMS?
● “NoSQL” options?
● Clean up use of Redis somehow?
OpenSource Connections
SubRedis
Operationalizes hierarchy inside of redis
https://github.com/softwaredoug/subredis
from redis import Redis
from subredis import SubRedis
redis = Redis()
sr = SubRedis(“case_%s” % caseId , redis)
ratings = {“doc1234”: “10”,
“doc532”: “5”}
searchQuery = “hdmi cables”
sr.hsetall(searchQuery, ratings)
Create a redis sandbox for this case
Interact with this case’s queries with redis
sandbox specific to that case
Behind the scenes, subredis
queries/appends the case_1 prefix to
everything
OpenSource Connections
SubRedis == composable
userSr = SubRedis(“user_%s” % userId , redis)
caseSr = SubRedis(“case_%s” % caseId , userSr)
# Sandbox redis for queries about user
ratings = {“doc1234”: “10”,
“doc532”: “5”}
searchQuery = “hdmi cables”
caseSr.hsetall(searchQuery, ratings)
SubRedis takes any Redis like
thing, and works safely in that
sandbox
Now working on sandbox, within a sandbox
OpenSource Connections
Does something reasonable under the hood
{
“user_1_name”: “Doug”,
“user_1_created_date”: “Doug”,
“user_1_case_1_name”: “name”: “cablestore”
“user_1_case_1_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“user_2_name”, “Rena”,
...
}
All
Redis
user_1
subred.
case_1
subred.
OpenSource Connections
We reflect again
● Ok we tried this out as a product. Launched.
● Paid off *some* tech debt, but wtf are we
doing
● Works well enough, we’ve got a bunch of
new features, forge ahead
OpenSource Connections
We reflect again
● We have real customers
● Our backend is evolving away from simple
key-value storage
○ user accounts? users that share cases? stored
search snapshots? etc etc
OpenSource Connections
Attack of the relational
Given our current set of tools, how would we solve the problem
“case X can be shared between multiple users”?
{
“user_1_name”: “Doug”,
“user_1_created_date”: “Doug”,
“user_1_case_1_name”: “name”: “cablestore”
“user_1_case_1_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“user_2_name”, “Rena”,
“user_2_case_1_name”: “name”: “cablestore”
“user_2_case_1_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
}
Could duplicate the data?
This stinks!
● Updates require visiting many (every?)
user, looking for this case
● Bloated database
Duplicate the data?
OpenSource Connections
Attack of the relational
Given our current set of tools, how would we solve the problem
“case X can be shared between multiple users”?
{
“user_1_name”: “Doug”,
“user_1_created_date”: “Doug”,
“user_1_cases”: [1, ...]
“case_1_name”: “name”: “cablestore”
“case_1_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“user_2_name”, “Rena”,
“user_2_cases”: [1, ...]
...
}
User 1
Case 1
User 2
Store list of
owned cases
Break out cases to a top-level record?
OpenSource Connections
SudRedisRelational?
{
“user_1_name”: “Doug”,
“user_1_created_date”: “Doug”,
“user_1_cases”: [1, ...]
“case_1_name”: “name”: “cablestore”
“case_1_hdmi cables”: {
“doc1234”: “10”,
“doc532”: “5”
},
“user_2_name”, “Rena”,
“user_2_cases”: [1, ...]
...
}
We’ve actually just normalized our data.
Why was this good?
● We want to update case 1 in isolation
without anomalies
● We don’t want to visit every user to
update case 1!
● We want to avoid duplication
We just made our “NoSQL” database a bit relational
OpenSource Connections
Other Problems
● Simple CRUD tasks like “delete a case”
need to be coded up
● We’re managing our own record ids
● Is any of this atomic? does it occur in
isolation?
OpenSource Connections
What’s our next DB?
● These problems are hard, we need a new
DB
● We also need better tooling!
OpenSource Connections
Irony
● This is the exact situation we warn clients
about in our (No)SQL Architect Roles.
○ Relational == General Purpose
○ Many-many, many-one, one-many, etc
○ Relational == consistent tooling
○ NoSQL == solve specific problems well
OpenSource Connections
So we went relational!
● Took advantage of great tooling: MySQL,
Sqlalchemy (ORM), Alembic (migrations)
● Modeled our data relationships exactly like
we needed them to be modeled
OpenSource Connections
Map db Python classes
class SearchQuery(Base):
__tablename__ = 'query'
id = Column(Integer, primary_key=True)
search_string = Column(String)
ratings = relationship("QueryRating")
class QueryRating(Base):
__tablename__ = 'rating'
id = Column(Integer, primary_key=True)
doc_id = Column(String)
rating = Column(Integer)
Can model my domain in coder-friendly
classes class SearchQuery(Base):
__tablename__ = 'query'
id = Column(Integer, primary_key=True)
search_string = Column(String)
ratings = relationship("QueryRating")
class QueryRating(Base):
__tablename__ = 'rating'
id = Column(Integer, primary_key=True)
doc_id = Column(String)
rating = Column(Integer)
OpenSource Connections
Easy CRUD
q = SearchQuery(search_string=”hdmi cable”)
db.session.add(q)
db.session.commit()
del q.ratings[0]
db.session.add(q)
db.session.commit()
q = SearchQuery.query.filter(id=1).one()
q.search_string=”foo”
db.session.add(q)
db.session.commit()
Create!
Delete!
Update!
OpenSource Connections
Migrations are good
alembic revision --autogenerate -m "name for tries"
alembic upgrade head
alembic downgrade 0ab51c25c
How do you upgrade your database to add/move/reorganize data?
● Redis this was always done manually/scripted
● Migrations with RDMS are a very robust/well-understood way to
handle this
SQLAlchemy has “alembic” to help:
OpenSource Connections
Modeling Users ←→ Cases
association_table = Table(case2users, Base.metadata,
Column('case_id', Integer, ForeignKey('case.id')),
Column('user_id', Integer, ForeignKey('user.id'))
)
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
cases = relationship("Case",
secondary=association_table)
class Case(Base):
__tablename__ = 'case'
id = Column(Integer, primary_key=True)
Can model many-many relationships
OpenSource Connections
Ultimate Query Flexibility
for user in User.query.all():
for case in user.cases:
print case.caseName
for user in User.query.filter(User.isPaying==True):
for case in user.cases:
print case.caseName
Print all cases:
Cases from paying members:
OpenSource Connections
Lots of things easier
● backups
● robust hosting services (RDS)
● industrial strength ACID with flexible
querying
● 3rd-party tooling (ie VividCortex for MySQL)
OpenSource Connections
When NoSQL?
● Solve specific problems well
○ Optimize for specific query patterns
○ Full-Text Search (Elasticsearch, Solr)
○ Caching, shared data structure (Redis)
● Optimize for specific scaling problems
○ Provide a denormalized “view” of your data for
specific task
OpenSource Connections
Final Thoughts
Sometimes RDMS’s have harder initial hurdle
for setup, figuring out migrations; data
modeling; etc
Why isn’t the easy path the wise path?
OpenSource Connections
In conclusion

More Related Content

What's hot

Incrementalism: An Industrial Strategy For Adopting Modern Automation
Incrementalism: An Industrial Strategy For Adopting Modern AutomationIncrementalism: An Industrial Strategy For Adopting Modern Automation
Incrementalism: An Industrial Strategy For Adopting Modern AutomationSean Chittenden
 
Creating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at ScaleCreating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at ScaleSean Chittenden
 
HTTP For the Good or the Bad
HTTP For the Good or the BadHTTP For the Good or the Bad
HTTP For the Good or the BadXavier Mertens
 
Deploying E.L.K stack w Puppet
Deploying E.L.K stack w PuppetDeploying E.L.K stack w Puppet
Deploying E.L.K stack w PuppetColin Brown
 
Leveraging Open Source for Database Development: Database Version Control wit...
Leveraging Open Source for Database Development: Database Version Control wit...Leveraging Open Source for Database Development: Database Version Control wit...
Leveraging Open Source for Database Development: Database Version Control wit...All Things Open
 
Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.Prajal Kulkarni
 
PostgreSQL Open SV 2018
PostgreSQL Open SV 2018PostgreSQL Open SV 2018
PostgreSQL Open SV 2018artgillespie
 
Terracotta's OffHeap Explained
Terracotta's OffHeap ExplainedTerracotta's OffHeap Explained
Terracotta's OffHeap ExplainedChris Dennis
 
REST with Eve and Python
REST with Eve and PythonREST with Eve and Python
REST with Eve and PythonPiXeL16
 
#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015
#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015
#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015Matt Raible
 
Introduction to rest.li
Introduction to rest.liIntroduction to rest.li
Introduction to rest.liJoe Betz
 
[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화NAVER D2
 
A million connections and beyond - Node.js at scale
A million connections and beyond - Node.js at scaleA million connections and beyond - Node.js at scale
A million connections and beyond - Node.js at scaleTom Croucher
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4琛琳 饶
 
Building servers with Node.js
Building servers with Node.jsBuilding servers with Node.js
Building servers with Node.jsConFoo
 

What's hot (20)

Incrementalism: An Industrial Strategy For Adopting Modern Automation
Incrementalism: An Industrial Strategy For Adopting Modern AutomationIncrementalism: An Industrial Strategy For Adopting Modern Automation
Incrementalism: An Industrial Strategy For Adopting Modern Automation
 
Creating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at ScaleCreating PostgreSQL-as-a-Service at Scale
Creating PostgreSQL-as-a-Service at Scale
 
HTTP For the Good or the Bad
HTTP For the Good or the BadHTTP For the Good or the Bad
HTTP For the Good or the Bad
 
Deploying E.L.K stack w Puppet
Deploying E.L.K stack w PuppetDeploying E.L.K stack w Puppet
Deploying E.L.K stack w Puppet
 
Leveraging Open Source for Database Development: Database Version Control wit...
Leveraging Open Source for Database Development: Database Version Control wit...Leveraging Open Source for Database Development: Database Version Control wit...
Leveraging Open Source for Database Development: Database Version Control wit...
 
Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.Null Bachaav - May 07 Attack Monitoring workshop.
Null Bachaav - May 07 Attack Monitoring workshop.
 
The tale of 100 cve's
The tale of 100 cve'sThe tale of 100 cve's
The tale of 100 cve's
 
PostgreSQL Open SV 2018
PostgreSQL Open SV 2018PostgreSQL Open SV 2018
PostgreSQL Open SV 2018
 
FreeBSD: Dev to Prod
FreeBSD: Dev to ProdFreeBSD: Dev to Prod
FreeBSD: Dev to Prod
 
Terracotta's OffHeap Explained
Terracotta's OffHeap ExplainedTerracotta's OffHeap Explained
Terracotta's OffHeap Explained
 
REST with Eve and Python
REST with Eve and PythonREST with Eve and Python
REST with Eve and Python
 
#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015
#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015
#NoXML: Eliminating XML in Spring Projects - SpringOne 2GX 2015
 
Introduction to rest.li
Introduction to rest.liIntroduction to rest.li
Introduction to rest.li
 
[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화
 
A million connections and beyond - Node.js at scale
A million connections and beyond - Node.js at scaleA million connections and beyond - Node.js at scale
A million connections and beyond - Node.js at scale
 
{{more}} Kibana4
{{more}} Kibana4{{more}} Kibana4
{{more}} Kibana4
 
Move Over, Rsync
Move Over, RsyncMove Over, Rsync
Move Over, Rsync
 
Building servers with Node.js
Building servers with Node.jsBuilding servers with Node.js
Building servers with Node.js
 
Node.js in production
Node.js in productionNode.js in production
Node.js in production
 
Node.js - A Quick Tour
Node.js - A Quick TourNode.js - A Quick Tour
Node.js - A Quick Tour
 

Similar to Stop Worrying & Love the SQL - A Case Study

Discover the Power of the NoSQL + SQL with MySQL
Discover the Power of the NoSQL + SQL with MySQLDiscover the Power of the NoSQL + SQL with MySQL
Discover the Power of the NoSQL + SQL with MySQLDave Stokes
 
Discover The Power of NoSQL + MySQL with MySQL
Discover The Power of NoSQL + MySQL with MySQLDiscover The Power of NoSQL + MySQL with MySQL
Discover The Power of NoSQL + MySQL with MySQLDave Stokes
 
Open Source World June '21 -- JSON Within a Relational Database
Open Source World June '21 -- JSON Within a Relational DatabaseOpen Source World June '21 -- JSON Within a Relational Database
Open Source World June '21 -- JSON Within a Relational DatabaseDave Stokes
 
Buildingsocialanalyticstoolwithmongodb
BuildingsocialanalyticstoolwithmongodbBuildingsocialanalyticstoolwithmongodb
BuildingsocialanalyticstoolwithmongodbMongoDB APAC
 
MVP Cloud OS Week Track 1 9 Sept: Data liberty
MVP Cloud OS Week Track 1 9 Sept: Data libertyMVP Cloud OS Week Track 1 9 Sept: Data liberty
MVP Cloud OS Week Track 1 9 Sept: Data libertycsmyth501
 
MVP Cloud OS Week: 9 Sept, Track 1 Data Liberty
MVP Cloud OS Week: 9 Sept, Track 1 Data LibertyMVP Cloud OS Week: 9 Sept, Track 1 Data Liberty
MVP Cloud OS Week: 9 Sept, Track 1 Data Libertycsmyth501
 
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...Dave Stokes
 
RedisConf18 - Writing modular & encapsulated Redis code
RedisConf18 - Writing modular & encapsulated Redis codeRedisConf18 - Writing modular & encapsulated Redis code
RedisConf18 - Writing modular & encapsulated Redis codeRedis Labs
 
Nomad Multi-Cloud
Nomad Multi-CloudNomad Multi-Cloud
Nomad Multi-CloudNic Jackson
 
Living the Nomadic life - Nic Jackson
Living the Nomadic life - Nic JacksonLiving the Nomadic life - Nic Jackson
Living the Nomadic life - Nic JacksonParis Container Day
 
Json within a relational database
Json within a relational databaseJson within a relational database
Json within a relational databaseDave Stokes
 
MySQL Without the SQL -- Oh My!
MySQL Without the SQL -- Oh My!MySQL Without the SQL -- Oh My!
MySQL Without the SQL -- Oh My!Data Con LA
 
Datacon LA - MySQL without the SQL - Oh my!
Datacon LA - MySQL without the SQL - Oh my! Datacon LA - MySQL without the SQL - Oh my!
Datacon LA - MySQL without the SQL - Oh my! Dave Stokes
 
MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019
MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019
MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019Dave Stokes
 
Kerberizing spark. Spark Summit east
Kerberizing spark. Spark Summit eastKerberizing spark. Spark Summit east
Kerberizing spark. Spark Summit eastJorge Lopez-Malla
 
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)Johannes Hoppe
 
Redis for Security Data : SecurityScorecard JVM Redis Usage
Redis for Security Data : SecurityScorecard JVM Redis UsageRedis for Security Data : SecurityScorecard JVM Redis Usage
Redis for Security Data : SecurityScorecard JVM Redis UsageTimothy Spann
 
MySQL Without the SQL -- Oh My! Longhorn PHP Conference
MySQL Without the SQL -- Oh My!  Longhorn PHP ConferenceMySQL Without the SQL -- Oh My!  Longhorn PHP Conference
MySQL Without the SQL -- Oh My! Longhorn PHP ConferenceDave Stokes
 
Introduction to Azure DocumentDB
Introduction to Azure DocumentDBIntroduction to Azure DocumentDB
Introduction to Azure DocumentDBDenny Lee
 
Ruby new wheels_condensed
Ruby new wheels_condensedRuby new wheels_condensed
Ruby new wheels_condensedosake
 

Similar to Stop Worrying & Love the SQL - A Case Study (20)

Discover the Power of the NoSQL + SQL with MySQL
Discover the Power of the NoSQL + SQL with MySQLDiscover the Power of the NoSQL + SQL with MySQL
Discover the Power of the NoSQL + SQL with MySQL
 
Discover The Power of NoSQL + MySQL with MySQL
Discover The Power of NoSQL + MySQL with MySQLDiscover The Power of NoSQL + MySQL with MySQL
Discover The Power of NoSQL + MySQL with MySQL
 
Open Source World June '21 -- JSON Within a Relational Database
Open Source World June '21 -- JSON Within a Relational DatabaseOpen Source World June '21 -- JSON Within a Relational Database
Open Source World June '21 -- JSON Within a Relational Database
 
Buildingsocialanalyticstoolwithmongodb
BuildingsocialanalyticstoolwithmongodbBuildingsocialanalyticstoolwithmongodb
Buildingsocialanalyticstoolwithmongodb
 
MVP Cloud OS Week Track 1 9 Sept: Data liberty
MVP Cloud OS Week Track 1 9 Sept: Data libertyMVP Cloud OS Week Track 1 9 Sept: Data liberty
MVP Cloud OS Week Track 1 9 Sept: Data liberty
 
MVP Cloud OS Week: 9 Sept, Track 1 Data Liberty
MVP Cloud OS Week: 9 Sept, Track 1 Data LibertyMVP Cloud OS Week: 9 Sept, Track 1 Data Liberty
MVP Cloud OS Week: 9 Sept, Track 1 Data Liberty
 
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...
MySQL Without the SQL - Oh My! August 2nd presentation at Mid Atlantic Develo...
 
RedisConf18 - Writing modular & encapsulated Redis code
RedisConf18 - Writing modular & encapsulated Redis codeRedisConf18 - Writing modular & encapsulated Redis code
RedisConf18 - Writing modular & encapsulated Redis code
 
Nomad Multi-Cloud
Nomad Multi-CloudNomad Multi-Cloud
Nomad Multi-Cloud
 
Living the Nomadic life - Nic Jackson
Living the Nomadic life - Nic JacksonLiving the Nomadic life - Nic Jackson
Living the Nomadic life - Nic Jackson
 
Json within a relational database
Json within a relational databaseJson within a relational database
Json within a relational database
 
MySQL Without the SQL -- Oh My!
MySQL Without the SQL -- Oh My!MySQL Without the SQL -- Oh My!
MySQL Without the SQL -- Oh My!
 
Datacon LA - MySQL without the SQL - Oh my!
Datacon LA - MySQL without the SQL - Oh my! Datacon LA - MySQL without the SQL - Oh my!
Datacon LA - MySQL without the SQL - Oh my!
 
MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019
MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019
MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019
 
Kerberizing spark. Spark Summit east
Kerberizing spark. Spark Summit eastKerberizing spark. Spark Summit east
Kerberizing spark. Spark Summit east
 
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
2012-08-29 - NoSQL Bootcamp (Redis, RavenDB & MongoDB für .NET Entwickler)
 
Redis for Security Data : SecurityScorecard JVM Redis Usage
Redis for Security Data : SecurityScorecard JVM Redis UsageRedis for Security Data : SecurityScorecard JVM Redis Usage
Redis for Security Data : SecurityScorecard JVM Redis Usage
 
MySQL Without the SQL -- Oh My! Longhorn PHP Conference
MySQL Without the SQL -- Oh My!  Longhorn PHP ConferenceMySQL Without the SQL -- Oh My!  Longhorn PHP Conference
MySQL Without the SQL -- Oh My! Longhorn PHP Conference
 
Introduction to Azure DocumentDB
Introduction to Azure DocumentDBIntroduction to Azure DocumentDB
Introduction to Azure DocumentDB
 
Ruby new wheels_condensed
Ruby new wheels_condensedRuby new wheels_condensed
Ruby new wheels_condensed
 

More from All Things Open

Building Reliability - The Realities of Observability
Building Reliability - The Realities of ObservabilityBuilding Reliability - The Realities of Observability
Building Reliability - The Realities of ObservabilityAll Things Open
 
Modern Database Best Practices
Modern Database Best PracticesModern Database Best Practices
Modern Database Best PracticesAll Things Open
 
Open Source and Public Policy
Open Source and Public PolicyOpen Source and Public Policy
Open Source and Public PolicyAll Things Open
 
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...All Things Open
 
The State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil NashThe State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil NashAll Things Open
 
Total ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScriptTotal ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScriptAll Things Open
 
What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?All Things Open
 
How to Write & Deploy a Smart Contract
How to Write & Deploy a Smart ContractHow to Write & Deploy a Smart Contract
How to Write & Deploy a Smart ContractAll Things Open
 
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
 Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlowAll Things Open
 
DEI Challenges and Success
DEI Challenges and SuccessDEI Challenges and Success
DEI Challenges and SuccessAll Things Open
 
Supercharging tutorials with WebAssembly
Supercharging tutorials with WebAssemblySupercharging tutorials with WebAssembly
Supercharging tutorials with WebAssemblyAll Things Open
 
Using SQL to Find Needles in Haystacks
Using SQL to Find Needles in HaystacksUsing SQL to Find Needles in Haystacks
Using SQL to Find Needles in HaystacksAll Things Open
 
Configuration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit InterceptConfiguration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit InterceptAll Things Open
 
Scaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship ProgramScaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship ProgramAll Things Open
 
Build Developer Experience Teams for Open Source
Build Developer Experience Teams for Open SourceBuild Developer Experience Teams for Open Source
Build Developer Experience Teams for Open SourceAll Things Open
 
Deploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache BeamDeploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache BeamAll Things Open
 
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML ApplicationsFortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML ApplicationsAll Things Open
 
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...All Things Open
 
Building AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources codeBuilding AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources codeAll Things Open
 
Open Source evaluation: A comprehensive guide on what you are using
Open Source evaluation: A comprehensive guide on what you are usingOpen Source evaluation: A comprehensive guide on what you are using
Open Source evaluation: A comprehensive guide on what you are usingAll Things Open
 

More from All Things Open (20)

Building Reliability - The Realities of Observability
Building Reliability - The Realities of ObservabilityBuilding Reliability - The Realities of Observability
Building Reliability - The Realities of Observability
 
Modern Database Best Practices
Modern Database Best PracticesModern Database Best Practices
Modern Database Best Practices
 
Open Source and Public Policy
Open Source and Public PolicyOpen Source and Public Policy
Open Source and Public Policy
 
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
 
The State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil NashThe State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil Nash
 
Total ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScriptTotal ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScript
 
What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?
 
How to Write & Deploy a Smart Contract
How to Write & Deploy a Smart ContractHow to Write & Deploy a Smart Contract
How to Write & Deploy a Smart Contract
 
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
 Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
 
DEI Challenges and Success
DEI Challenges and SuccessDEI Challenges and Success
DEI Challenges and Success
 
Supercharging tutorials with WebAssembly
Supercharging tutorials with WebAssemblySupercharging tutorials with WebAssembly
Supercharging tutorials with WebAssembly
 
Using SQL to Find Needles in Haystacks
Using SQL to Find Needles in HaystacksUsing SQL to Find Needles in Haystacks
Using SQL to Find Needles in Haystacks
 
Configuration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit InterceptConfiguration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit Intercept
 
Scaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship ProgramScaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship Program
 
Build Developer Experience Teams for Open Source
Build Developer Experience Teams for Open SourceBuild Developer Experience Teams for Open Source
Build Developer Experience Teams for Open Source
 
Deploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache BeamDeploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache Beam
 
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML ApplicationsFortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
 
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
 
Building AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources codeBuilding AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources code
 
Open Source evaluation: A comprehensive guide on what you are using
Open Source evaluation: A comprehensive guide on what you are usingOpen Source evaluation: A comprehensive guide on what you are using
Open Source evaluation: A comprehensive guide on what you are using
 

Recently uploaded

Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 

Recently uploaded (20)

Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 

Stop Worrying & Love the SQL - A Case Study

  • 1. Stop Worrying Love the SQL! (the Quepid story)
  • 2. OpenSource Connections Me/Us Doug Turnbull @softwaredoug Likes: Solr, Elasticsearch, Cassandra, Postgres OpenSource Connections @o19s Search, Discovery and Analytics Let us introduce you to freelancing!
  • 3. OpenSource Connections Most Importantly we do... Make my search results more relevant! “Search Relevancy” What database works best for problem X? “(No)SQL Architect/Trusted Advisor”
  • 4. OpenSource Connections How products actually get built Rena: Doug, John can you come by this afternoon? One of our Solr-based products needs some urgent relevancy work Its Friday, it needs to get done today! Us: Sure! The Client (Rena!) smart cookie!
  • 5. OpenSource Connections A few hours later Us: we’ve made a bit of progress! image frustration-1081 by jseliger2 Rena: but everytime we fix something, we break an existing search! Us: yeah! we’re stuck in a whack-a-mole-game other image: whack a mole by jencu
  • 6. OpenSource Connections Whack-a-Mole What search relevancy work actually looks like
  • 7. OpenSource Connections I HAVE AN IDEA ● Middle of the afternoon, I stop doing search work and start throwing together some python from flask import Flask app = Flask(__name__) Everyone: Doug, stop that, you have important search work to do! Me: We’re not making any progress! WE NEED A WAY TO REGRESSION TEST OUR RELEVANCY AS WE TUNE! Everyone: You’re nuts!
  • 8. OpenSource Connections What did I make? Focus on gathering stakeholder (ie Rena) feedback on search, coupled w/ workbench tuning against that feedback Today we have customers... … forget that, tell me about your failures!
  • 9. OpenSource Connections Our war story My mistakes: ● Building a product ● Selling a product ● As a user experience engineer ● As an Angular developer ● At choosing databases
  • 10. OpenSource Connections Quepid 0.0.0.0.0.0.1 Track multiple user searches for this query (hdmi cables) Rena rates this document as a good/bad search result need to store: <search> -> <id for search result> -> <rating 1-10> “hdmi cables” -> “doc1234” -> “10” *Actual UI may have been much uglier
  • 11. OpenSource Connections Data structure selection under duress ● What’s simple, easy, and will persist our data? ● What plays well with python? ● What can I get working now in Rena’s office?
  • 12. OpenSource Connections Redis ● In memory “Data Structure Server” ○ hashes, lists, simple key-> value storage ● Persistent -- write to disk every X minutes
  • 13. OpenSource Connections Redis from redis import Redis redis = Redis() redis.set("foo", "bar") redis.get("foo") # gets ‘bar’ $ pip install redis Easy to install and go! Specific to our problem: from redis import Redis redis = Redis() ratings = {“doc1234”: “10”, “doc532”: “5”} searchQuery = “hdmi cables” redis.hsetall(searchQuery, ratings) Store a hash table at “hdmi cables” with: “doc1234” -> “10” “doc532” -> “5”
  • 14. OpenSource Connections Success! ● My insanity paid off that afternoon ● Now we’re left with a pile of hacked together (terrible) code -- now what?
  • 15. OpenSource Connections Adding some features ● Would like to add multiple “cases” (different search projects that solve different problems) ● Would like to add user accounts ● Still a one-off for Silverchair
  • 16. OpenSource Connections Cases Tuning a cable shopping site... … vs state laws
  • 17. OpenSource Connections Cases in Redis? from redis import Redis redis = Redis() ratings = {“doc1234”: “10”, “doc532”: “5”} searchQuery = “hdmi cables” redis.hset(searchQuery, ratings) Recall our existing implementation “data model” Out of the box, redis can deal with 2 levels deep: { “hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “ethernet cables” ... } Can’t add extra layer (redis hash only one layer) {“cable site”: { “hdmi cables”: {...} “ethernet cables”: {...} } “laws site: {...}}
  • 18. OpenSource Connections Time to give up Redis? “All problems in computer science can be solved by another level of indirection” -- David Wheeler Crazy Idea: Add dynamic prefix to query keys to indicate case, ie: { “case_cablestore_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “case_cablestore_ethernet cables”: { … }, “case_statelaws_car tax”: { …} } Queries for “Cable Store” case Query for “State Laws” case redis.keys(“case_cablestore*”) To Fetch:
  • 19. OpenSource Connections Store other info about cases? New problem: we need to store some information about cases, case name, et { “case_cablestore_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “case_cablestore_ethernet cables”: { … }, “case_statelaws_car tax”: { …} } Where would it go here? { “case_cablestore” { “name”: “cablestore”, “created” “20140101” }, “case_cablestore_query_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “case_cablestore_query_ethernet cables”: { … }, “case_statelaws_query_car tax”: { …} }
  • 20. OpenSource Connections Oh but let’s add users Extrapolating on past patterns { “user_doug” { “name”: “Doug”, “created_date”: “20140101” }, “user_doug_case_cablestore” { “name”: “cablestore”, “created_date” “20140101” }, “user_doug_case_cablestore_query_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “user_doug_case_cablestore_query_ethernet cables”: { … }, “user_tom_case_statelaws_query_car tax”: { …} }image: Rage Wallpaper from Flickr user Thoth God of Knowledge You right now!
  • 21. OpenSource Connections Step Back We ask ourselves: Is this tool a product? Is it useful outside of this customer? What level of software engineering helps us move forward? ● Migrate to RDMS? ● “NoSQL” options? ● Clean up use of Redis somehow?
  • 22. OpenSource Connections SubRedis Operationalizes hierarchy inside of redis https://github.com/softwaredoug/subredis from redis import Redis from subredis import SubRedis redis = Redis() sr = SubRedis(“case_%s” % caseId , redis) ratings = {“doc1234”: “10”, “doc532”: “5”} searchQuery = “hdmi cables” sr.hsetall(searchQuery, ratings) Create a redis sandbox for this case Interact with this case’s queries with redis sandbox specific to that case Behind the scenes, subredis queries/appends the case_1 prefix to everything
  • 23. OpenSource Connections SubRedis == composable userSr = SubRedis(“user_%s” % userId , redis) caseSr = SubRedis(“case_%s” % caseId , userSr) # Sandbox redis for queries about user ratings = {“doc1234”: “10”, “doc532”: “5”} searchQuery = “hdmi cables” caseSr.hsetall(searchQuery, ratings) SubRedis takes any Redis like thing, and works safely in that sandbox Now working on sandbox, within a sandbox
  • 24. OpenSource Connections Does something reasonable under the hood { “user_1_name”: “Doug”, “user_1_created_date”: “Doug”, “user_1_case_1_name”: “name”: “cablestore” “user_1_case_1_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “user_2_name”, “Rena”, ... } All Redis user_1 subred. case_1 subred.
  • 25. OpenSource Connections We reflect again ● Ok we tried this out as a product. Launched. ● Paid off *some* tech debt, but wtf are we doing ● Works well enough, we’ve got a bunch of new features, forge ahead
  • 26. OpenSource Connections We reflect again ● We have real customers ● Our backend is evolving away from simple key-value storage ○ user accounts? users that share cases? stored search snapshots? etc etc
  • 27. OpenSource Connections Attack of the relational Given our current set of tools, how would we solve the problem “case X can be shared between multiple users”? { “user_1_name”: “Doug”, “user_1_created_date”: “Doug”, “user_1_case_1_name”: “name”: “cablestore” “user_1_case_1_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “user_2_name”, “Rena”, “user_2_case_1_name”: “name”: “cablestore” “user_2_case_1_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, } Could duplicate the data? This stinks! ● Updates require visiting many (every?) user, looking for this case ● Bloated database Duplicate the data?
  • 28. OpenSource Connections Attack of the relational Given our current set of tools, how would we solve the problem “case X can be shared between multiple users”? { “user_1_name”: “Doug”, “user_1_created_date”: “Doug”, “user_1_cases”: [1, ...] “case_1_name”: “name”: “cablestore” “case_1_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “user_2_name”, “Rena”, “user_2_cases”: [1, ...] ... } User 1 Case 1 User 2 Store list of owned cases Break out cases to a top-level record?
  • 29. OpenSource Connections SudRedisRelational? { “user_1_name”: “Doug”, “user_1_created_date”: “Doug”, “user_1_cases”: [1, ...] “case_1_name”: “name”: “cablestore” “case_1_hdmi cables”: { “doc1234”: “10”, “doc532”: “5” }, “user_2_name”, “Rena”, “user_2_cases”: [1, ...] ... } We’ve actually just normalized our data. Why was this good? ● We want to update case 1 in isolation without anomalies ● We don’t want to visit every user to update case 1! ● We want to avoid duplication We just made our “NoSQL” database a bit relational
  • 30. OpenSource Connections Other Problems ● Simple CRUD tasks like “delete a case” need to be coded up ● We’re managing our own record ids ● Is any of this atomic? does it occur in isolation?
  • 31. OpenSource Connections What’s our next DB? ● These problems are hard, we need a new DB ● We also need better tooling!
  • 32. OpenSource Connections Irony ● This is the exact situation we warn clients about in our (No)SQL Architect Roles. ○ Relational == General Purpose ○ Many-many, many-one, one-many, etc ○ Relational == consistent tooling ○ NoSQL == solve specific problems well
  • 33. OpenSource Connections So we went relational! ● Took advantage of great tooling: MySQL, Sqlalchemy (ORM), Alembic (migrations) ● Modeled our data relationships exactly like we needed them to be modeled
  • 34. OpenSource Connections Map db Python classes class SearchQuery(Base): __tablename__ = 'query' id = Column(Integer, primary_key=True) search_string = Column(String) ratings = relationship("QueryRating") class QueryRating(Base): __tablename__ = 'rating' id = Column(Integer, primary_key=True) doc_id = Column(String) rating = Column(Integer) Can model my domain in coder-friendly classes class SearchQuery(Base): __tablename__ = 'query' id = Column(Integer, primary_key=True) search_string = Column(String) ratings = relationship("QueryRating") class QueryRating(Base): __tablename__ = 'rating' id = Column(Integer, primary_key=True) doc_id = Column(String) rating = Column(Integer)
  • 35. OpenSource Connections Easy CRUD q = SearchQuery(search_string=”hdmi cable”) db.session.add(q) db.session.commit() del q.ratings[0] db.session.add(q) db.session.commit() q = SearchQuery.query.filter(id=1).one() q.search_string=”foo” db.session.add(q) db.session.commit() Create! Delete! Update!
  • 36. OpenSource Connections Migrations are good alembic revision --autogenerate -m "name for tries" alembic upgrade head alembic downgrade 0ab51c25c How do you upgrade your database to add/move/reorganize data? ● Redis this was always done manually/scripted ● Migrations with RDMS are a very robust/well-understood way to handle this SQLAlchemy has “alembic” to help:
  • 37. OpenSource Connections Modeling Users ←→ Cases association_table = Table(case2users, Base.metadata, Column('case_id', Integer, ForeignKey('case.id')), Column('user_id', Integer, ForeignKey('user.id')) ) class User(Base): __tablename__ = 'user' id = Column(Integer, primary_key=True) cases = relationship("Case", secondary=association_table) class Case(Base): __tablename__ = 'case' id = Column(Integer, primary_key=True) Can model many-many relationships
  • 38. OpenSource Connections Ultimate Query Flexibility for user in User.query.all(): for case in user.cases: print case.caseName for user in User.query.filter(User.isPaying==True): for case in user.cases: print case.caseName Print all cases: Cases from paying members:
  • 39. OpenSource Connections Lots of things easier ● backups ● robust hosting services (RDS) ● industrial strength ACID with flexible querying ● 3rd-party tooling (ie VividCortex for MySQL)
  • 40. OpenSource Connections When NoSQL? ● Solve specific problems well ○ Optimize for specific query patterns ○ Full-Text Search (Elasticsearch, Solr) ○ Caching, shared data structure (Redis) ● Optimize for specific scaling problems ○ Provide a denormalized “view” of your data for specific task
  • 41. OpenSource Connections Final Thoughts Sometimes RDMS’s have harder initial hurdle for setup, figuring out migrations; data modeling; etc Why isn’t the easy path the wise path?