NoSQL for SQL Server Developers using Couchbase

NoSQL for SQL
Server developers
using Couchbase
TriNUG Data SIG
3/7/2018

Who is this guy?
• Brant Burnett - @btburnett3
• Systems Architect at CenterEdge Software
• .NET since 1.0, SQL Server since 7.0
• MCSD, MCDBA
• Experience from desktop apps to large
scale cloud services

NoSQL Credentials
• Couchbase user since 2012 (v1.8)
• Couchbase Expert
• Open source contributions:
• Couchbase .NET SDK
• Couchbase.Extensions for .NET Core
• Couchbase LINQ provider (Linq2Couchbase)
• CouchbaseFakeIt
• couchbase-index-manager

Content
Attributions
• Matthew Groves
Developer Advocate at Couchbase
@mgroves on Twitter
• Raju Suravarjjala
Couchbase R&D
• Keshav Murthy
Couchbase R&D

Agenda
What’s NoSQL?
Why NoSQL?
Some popular NoSQL options
SQL Server to Couchbase Mindmap
Live Demo: ASP.NET Core Microservice
Querying and Indexing Couchbase

Let’s get some
questions…
first???
This Photo by Unknown Author is licensed under CC BY-NC-ND

What’s
NoSQL?
It’s just the opposite of SQL,
right?
Document
• Couchbase
• MongoDB
• DynamoDB
• CosmosDB
Graph
• OrientDB
• Neo4J
• GraphBase
• CosmosDB
Key-Value
• Couchbase
• Riak
• BerkeleyDB
• Redis
• CosmosDB
Wide Column
• Hbase
• Cassandra
• Hypertable
• CosmosDB

NoSQL Document
Databases
• Get and set documents by key
• Imagine a giant folder full of JSON
files
• If you know the filename, you can
get or update the content
• Often offer map/reduce or query
functionality

Why NoSQL?
SQL Server can do it all, right?

Why NoSQL?
Scalability
• Horizontal scaling
• Scaling SQL horizontally is difficult
• NoSQL is designed for horizontal scaling
• Couchbase shards horizontally automatically
and transparently
• Scaling vertically has limits
• Scaling SQL usually has downtime
• Big Data!
This Photo by Unknown Author is licensed under CC BY

Why NoSQL?
Availability
• Multi-node design protects against
failure
• COUCH = Cluster of Unreliable
Commodity Hardware
• Zero down time upgrades
• Zero down time scaling

Why NoSQL?
Performance
• Designed for very high throughput
• Multi-node architecture shares the load
• Can be more cost effective than read-
replicas
• Couchbase has a memory-first architecture
• No more separate memory cache!

Why NoSQL? Agility
• Most NoSQL systems are “schema-less”
• Schema changes can slow the team
• Schema changes can cause downtime for SQL
• Updating large tables can lock the whole
table until complete
• Many modern architectures don’t really need
database-enforced schema anymore
This Photo by Unknown Author is licensed under CC BY-SA

Okay, so why SQL?
• Small scale
• On-premise, single server
• ACID transactions are required
• a.k.a. two-phase commit
• Transaction logs
• High-consistency DB backups
• Keeping it old school

Which NoSQL system
is right for me?
I prefer Couchbase, but
here’s some (hopefully)
unbiased info

Popular NoSQL
Choices
Selection criteria:
• Document database
• Runs on Microsoft / Microsoft-friendly
• Popular – db-engines.com

Disclaimer
• I am primarily a Couchbase user
• Please let me know if I’ve made any mistakes
• Most of the information here comes from
Matthew Groves (@mgroves)
This Guy

Evaluation
Criteria
Querying
•Quality of query support
•Ease of query language
Scaling
•Ease of scaling
•Cost effectiveness of scaling
Usability
•Ease of installation
•Ease of maintenance
•Ease of development
Speed
•How fast is it?
Deployment
•OS choices
•Cloud provider choices
Support
•Support options
•Licensing
•Community
•Longevity

Querying
• Proprietary JSON-based query
• Text search
• .NET and .NET Core SDK
• LINQ provider in .NET

Scaling
• Multiple node types
• Sharding options
• Master-slave / Primary-secondary

Usability
• MongoDB Compass
• Scaling / Replication decisions

Speed
• Indexing for queries
• In-memory options

Deployment
• Windows
• MacOS
• Linux
• Azure
• AWS
• Docker

Support
• MongoDB, Inc.
• Open-source (AGPL & Apache)
• Huge community / popularity

Querying
• N1QL (SQL for JSON)
• MapReduce
• Full Text Search (FTS)
• Linq2Couchbase

Scaling
• Single node type
• Multi-master / "masterless"
• Auto-sharding
• Replication
• XDCR

Usability
• Built in web console
• .NET SDK idioms
• N1QL is just SQL
• Scaling
• mongoose and ottoman

Speed
• Memory-first architecture
• Indexing for N1QL
• YCSB benchmarks
• Memory-optimized indexes

Support
• Couchbase, Inc.
• Open-source (Apache 2)
• Active community

Querying
• REST focused
• MapReduce views
• Mango query language (2.x)

Scaling
• Single node type
• Multi-master
• HAProxy
• Sharding configuration
• Replication

Usability
• Built in Futon / Fauxton
• Scaling / Replication work
• Cluster Setup Wizard (2.x)
• There is no "official" .NET SDK
• .NET SDKs are REST wrappers

Speed
• Indexing for Mango queries
• MapReduce
• Caching is external

Support
• Open-source (Apache)
• Cloudant (IBM)

Querying
• SQL (limited)
• Stored Procedures (JS)
• Triggers (JS)
• UDFs (JS)

Scaling
• Handled by Azure
• Geographic distribution / affinity
• Consistency options
• Guaranteed zero data loss during
failovers

Usability
• Azure doing work for you
• Local emulator
• Transactions (with sprocs)
• Mongo compatible API

Speed
• Guaranteed ~10ms latency reads
• Guaranteed ~15ms latency writes

Deployment
• Windows (emulator)
• Azure

Support
• Microsoft
• Popularity is increasing, so more
community growth should be
expected

So which do I
choose?

I don't have
your answer. I
just have more
questions.
• Mobile?
• Querying?
• Cost /
licensing?
• Ops / DevOps?
• Hobby / Side?
• Resume?
• Speed?
• Transactions?
• Security?
• Integrations?

SQL Server to
Couchbase Mindmap

SQL is English for
Relational Database
SQL Invented by Don
Chamberlin & Raymond
Boyce at IBM
N1QL is English for JSON
N1QL was invented by
Gerald Sangudi at
Couchbase
SQL
Instance
Database
Table
Row
Column
Index
Datatypes
N1QL
Cluster
Bucket
Bucket, Keyspace
Document
Attribute
Index
JSON Datatypes
SQL
Input and Output: Set(s) of
Tuples
N1QL
Input and Output:
Set(s) of JSON
N1QL STMT
CREATE BUCKET
None
CREATE INDEX
None
SELECT
INSERT
UPDATE
DELETE
MERGE
Subqueries
JOIN
GROUP BY
ORDER BY
OFFSET, LIMIT
EXPLAIN
PREPARE
EXECUTE
GRANT ROLE
REVOKE ROLE
FLUSH
Tuples
SQL Model
Set of
JSON
N1QL Model
Set of
Tuples
Set of
JSON
N1QL Tooling
Web Console
Monitoring
Profiling
Dev workbench
SDK
Simba, CdataSQL Tooling
ODBC, JDBC, .NET
Hibernate, Entity Framework
N1QLResources
http://query.couchbase.com
SQL Indexes
Primary Key
Secondary Key
Composite
Range Partitioned
Expression
(Functional)
Spatial
Search
N1QL Indexes
Primary
Secondary
Composite
Partial
Expression (Functional)
Array Index
Replica(HA)
Adaptive
Spatial
SQL Logic
3 valued logic
TRUE, FALSE, NULL
N1QL Logic
4 valued logic
TRUE, FALSE, NULL,
MISSING
SQL Transactions
ACID
Multi-Statement
Savepoints
Commit/Rollback
N1QL Transactions
Single Document
atomicity
SQL Datatypes
Numeric
Decimal
Character
Date/Time
BLOB
Spatial
JSON
N1QL Datatypes
Numeric
Boolean
Character
Array
Object
Null
Conversion Functions
JSON
SQL Optimizer
Rule Based
Cost Based
Index Selection
Query Rewrites
NL, Hash, Merge join
N1QL Optimizer
Rule based
Index Selection
NL Join
SQL ACID
Atomic
Consistent
Isolated
Durable
N1QL BASE
Single doc Atomic
Consistent Data*
Optimistic Concurrency
Tunable Durability
N1QL Index Scan Consistency*
NOT_BOUNDED
AT_PLUS
REQUEST_PLUS
SQL Engine
(SMP
Scale UP)
N1QL
Engine
(MPP
Cluste Scale
OUT)
Additional SQL Features
Triggers
Stored Procedures
XML
Constraints
RAC
SQL STMT
CREATE DATABASE
CREATE TABLE
CREATE INDEX
ALTER TABLE
SELECT
INSERT
UPDATE
DELETE
MERGE
Subqueries
JOIN
GROUP BY
ORDER BY
OFFSET, FETCH, TOP
EXPLAIN
sp_prepare
sp_execute
GRANT
REVOKE
TRUNCATE

Couchbase Architecture
STORAGE
Couchbase Server 1
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Managed Cache
Storage
Data
Service STORAGE
Couchbase Server 2
Managed Cache
Cluster
ManagerCluster
Manager
Query
Service STORAGE
Couchbase Server 3
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Index
Service STORAGE
Couchbase Server 4
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Search
Service STORAGE
Couchbase Server 5
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Analytics
Service*
Managed Cache
Storage
SDK SDK

Architecture
Product SQL Server Couchbase
System Architecture SMP
Always-On readable secondary
replicas
MPP
MDS: Multi Dimensional Scaling
Data Service: MPP, Hash Partitioning
Indexing, FTS: MDS Scale-out
Query: MDS Scale-out
Query SQL, search, JSON extensions N1QL, Key-value, Full text search
Analytics (preview)
High Availability Always-On,
Log Shipping,
DB mirroring (deprecated)
Built-in intranode replication (up to 3 copies)
Built-in XDCR (cross data center replication)
Transactions ACID
Multi-statement
Single document atomicity
Data Service consistency, Index - eventual consistency
optimistic locking (CAS)
additional confirmation for durability
Drivers JDBC, ODBC, .NET, LINQ Couchbase SDK (Java, .NET, LINQ, PHP, Python, Go), Simba
JDBC/ODBC
Data Model Normalized, Denormalized Denormalized JSON model

Database Objects
Data Feature/Type SQL Server Couchbase
Database Database Bucket
Table Table Bucket, Keyspace
Row Row Document
Column Column Field/Attribute
Partition Partition (manual) Partition (hash automatic)

Data Types
Data Type SQL Server Couchbase JSON
Numbers int, bigint, smallint, tinyint, float, real,
decimal, numeric, money, smallmoney
JSON Number { "id": 5, "balance":2942.59 }
String char, varchar, nchar, nvarchar, text, ntext JSON String { "name": "Joe", "city": "Morrisville" }
Boolean bit JSON Boolean { "premium": true, ”pending": false}
Date/Time datetime, smalldatetime, datetime2,
date, time, datetimeoffset
JSON ISO 8601 string with extract,
convert and arithmetic functions
{ “soldat”: "2017-10-12T13:47:41.068-07:00" }
spatial data geometry, geography Supports nearest neighbor and spatial
distance.
"geometry": {"type": "Point", "coordinates": [-
104.99404, 39.75621]}
MISSING Not applicable, fixed schema MISSING { }
NULL NULL JSON Null { "last_address": null }
Objects Flexible JSON Objects { "address": {"street": "1 Main Street",
"city": Morrisville, "zip":"94824“} }
Arrays Flexible JSON Arrays { "hobbies": ["tennis", "skiing", "lego"] }

Understanding MISSING
Couchbase
MISSING
Value of a field absent in the JSON document or literal.
{“name”: ”joe”} Everything but the field “name” is missing from the document.
IS MISSING
Returns true if the document does not have status field
FROM CUSTOMER WHERE status IS MISSING;
IS NOT MISSING Returns true if the document has status field (even if null)
FROM CUSTOMER WHERE status IS NOT MISSING;
MISSING vs NULL MISSING is a known missing quantity
NULL is a known UNKNOWN.
Valid JSON: {“status”: null}
MISSING value Simply make the field of any type disappear by setting it to MISSING
UPDATE CUSTOMER SET status = MISSING WHERE cxid = “xyz232”

Couchbase 4-valued Boolean Logic
A B A OR B A AND B
TRUE NULL TRUE NULL
FALSE NULL FALSE NULL
TRUE MISSING TRUE MISSING
FALSE MISSING TRUE MISSING
NULL MISSING NULL MISSING
NULL NULL NULL NULL
MISSING MISSING MISSING MISSING

Statements
Feature SQL Server Couchbase
CREATE TABLE CREATE TABLE couchbase-cli bucket-create
ALTER TABLE ALTER TABLE UPDATE customer SET, UNSET
CREATE INDEX i1 on t(a, b, c DESC); CREATE INDEX i1 on t(a, b, c DESC); CREATE INDEX i1 on t(a, b, c DESC);
INSERT INTO INSERT INTO INSERT INTO
SELECT SELECT SELECT
JOINS JOINS JOIN – INNER JOIN, LEFT OUTER JOIN
GROUP BY, HAVING GROUP BY, HAVING GROUP BY, HAVING
ORDER BY a ASC, b DESC ORDER BY a ASC, b DESC ORDER BY a ASC, b DESC
OFFSET, LIMIT OFFSET, FETCH, TOP OFFSET, LIMIT
Subqueries Subqueries Subqueries

Data Modeling
Relationship SQL Server Couchbase
1:1
 Foreign Key
 Denormalize
 Embedded Object (implicit)
 Document Key reference
1:N  Foreign Key
 Embedded Array of Objects (implicit)
 Document Key reference
N:M  Foreign Key
 Embedded Array of Objects
 Arrays of objects with references

Joins
JOIN Type SQL Server Couchbase
INNER JOIN Full ANSI join ON clause requires document key
reference. Equi-join only
LEFT OUTER JOIN Full ANSI join ON clause requires document key
reference. Equi-join only
RIGHT OUTER JOIN Full ANSI join Unsupported
FULL OUTER JOIN Full ANSI join Unsupported

Transactions
Feature SQL Server Couchbase
Index updates Index is synchronously maintained Index is asynchronously maintained
Multi Statement Transaction
Yes
BEGIN, COMMIT, ROLLBACK, SAVE
No
Atomicity Multi update, Multi statement Single Document
Consistency
Consistent
Includes Dirty read support
Data access is always consistent
Index has multiple consistency levels
(NOT_BOUNDED, AT_PLUS, REQUEST_PLUS)
Isolation
Pessimistic locking with distributed lock
manager
Optimistic locking with CAS checking
Durability Durable Durable with confirmation after replication

Do you really need
transactions?
• Oftentimes, you don’t
• Good design can usually reduce the need
• If you’re using a microservice architecture,
distributed transactions have major downsides
• Bye-bye high availability
• Difficult
• Consider eventual consistency using saga patterns

Basic POCO
public class Airline
{
public string Callsign { get; set; }
public string Country { get; set; }
[JsonProperty("iata")]
public string IATA { get; set; }
[JsonProperty("iaco")]
public string IACO { get; set; }
public int Id { get; set; }
public string Name { get; set; }
public string Type { get; set; }
}
Key: airline_10
{
"callsign": "MILE-AIR",
"country": "United States",
"iata": "Q5",
"icao": "MLA",
"id": 10,
"name": "40-Mile Air",
"type": "airline"
}

Nesting Subdocuments
Key: airport_1254
{
"airportname": "Calais Dunkerque",
"city": "Calais",
"country": "France",
"faa": "CQF",
"geo": {
"alt": 12,
"lat": 50.962097,
"lon": 1.954764
},
"icao": "LFAC",
"id": 1254,
"type": "airport",
"tz": "Europe/Paris"
}
// JSON decorators not shown for simplicity
public class Airport
{
public string AirportName { get; set; }
public string City { get; set; }
public string FAA { get; set; }
public Coordinate Geo { get; set; }
public string ICAO { get; set; }
public string Timezone { get; set; }
}
public class Coordinate
{
public int Altitude { get; set; }
public double Latitude { get; set; }
public double Longitude { get; set; }
}

Nesting Arrays as Lists
Key: route_10000
{
"airline": "AF",
"airlineid": "airline_137",
"destinationairport": "MRS",
"distance": 2881.617376098415,
"equipment": "320",
"id": 10000,
"schedule": [
{
"day": 0,
"flight": "AF198",
"utc": "10:13:00"
},
{
"day": 0,
"flight": "AF547",
"utc": "19:14:00"
}
],
"sourceairport": "TLV",
"stops": 0,
"type": "route"
}
// JSON decorators not shown for simplicity
public class Route
{
public string Airline { get; set; }
public string AirlineId { get; set; }
public string DestinationAirport { get; set; }
public double Distance { get; set; }
public string Equipment { get; set; }
public List<Schedule> Schedule { get; set; }
public string SourceAirport { get; set; }
public int Stops { get; set; }
}
public class Schedule
{
public int Day { get; set; }
public string Flight { get; set; }
public TimeSpan UTC { get; set; }
}

Modeling for Joins
Key: route_10000
{
"airline": "AF",
"distance": 2881.617376098415,
"equipment": "320",
"id": 10000,
"schedule": [
{
"day": 0,
"flight": "AF198",
"utc": "10:13:00"
},
{
"day": 0,
"flight": "AF547",
"utc": "19:14:00"
}
],
"stops": 0,
"type": "route"
}
Key: airline_137
{
"callsign": "AIRFRANS",
"country": "France",
"iata": "AF",
"icao": "AFR",
"id": 137,
"name": "Air France",
"type": "airline"
}
Note: Instead of the full key, you may
just store the dynamic parts, such as
“137”, and rebuild the key as needed

Good Practices
[DocumentTypeFilter(TypeString)]
public class Airline
{
public const string TypeString = "airline";
public string Callsign { get; set; }
[JsonProperty("iata")]
public string IATA { get; set; }
[JsonProperty("iaco")]
public string IACO { get; set; }
public string Name { get; set; }
// Type is now read only to maintain consistency
public string Type => TypeString;
}
Key: airline_10
{
"callsign": "MILE-AIR",
"country": "United States",
"iata": "Q5",
"icao": "MLA",
"id": 10,
"name": "40-Mile Air",
"type": "airline"
}

Live Demo!
This should be interesting…
https://github.com/brantburnett/Couchbase.Net.StepByStep
This Photo by Unknown Author is licensed under CC BY-NC-SA

Querying Couchbase Using N1QL
(Pronounced “nickel”)

Start off
Simple
SELECT *
FROM `travel-sample` as airline
WHERE airline.type = 'airline'
ORDER BY airline.name
LIMIT 10

O backtick, backtick!
Wherefore art thou a
backtick?
• ANSI SQL delimits identifiers with double quotes
• SELECT * FROM "table-name"
• T-SQL also delimits identifiers with square
brackets
• SELECT * FROM [table-name]
• Both of these are used in JSON!
• {"array": ["string1", "string2"]}
• So, N1QL uses the backtick instead
• SELECT * FROM `bucket-name`

JOIN
SELECT airline.name, route.stops, route.schedule
FROM `travel-sample` AS route
INNER JOIN `travel-sample` AS airline
ON KEYS route.airlineid
WHERE route.type = 'route'
AND route.sourceairport = 'ATL'
AND route.destinationairport = 'ABE'
ORDER BY airline.name, route.stops

JOIN
with
LINQ
from route in db.Query<Route>()
join airline in db.Query<Airline>()
on route.AirlineId equals N1QlFunctions.Key(airline)
where route.SourceAirport == "ATL"
&& route.DestinationAirport == "ABE"
orderby airline.Name, route.Stops
select new
{
AirlineName = airline.Name,
route.Stops,
route.Schedule
};

What about
nested arrays?
Key: route_10000
{
"airline": "AF",
"distance": 2881.617376098415,
"equipment": "320",
"id": 10000,
"schedule": [
{
"day": 0,
"flight": "AF198",
"utc": "10:13:00"
},
{
"day": 0,
"flight": "AF547",
"utc": "19:14:00"
}
],
"stops": 0,
"type": "route"
}

UNNEST
SELECT route.airline, schedule.*
FROM `travel-sample` as route
UNNEST route.schedule as schedule
WHERE route.type = 'route’
AND route.sourceairport = 'MRS’
AND route.destinationairport = 'TLV'
LIMIT 10

UNNEST
with
LINQ
from route in context.Query<Route>()
from schedule in route.Schedules
where route.SourceAirport == "MRS"
&& route.DestinationAirport == "TLV"
select new
{
route.Airline,
schedule
};

Other N1QL
Specific
Features
• Turn an array of document keys into an array of
documents (NEST)
• Subqueries on nested arrays (ARRAY..END)
• Test nested arrays for matching values
(ANY..END)
• Most are implemented transparently by LINQ

SQL Server vs Couchbase Indexes
Airline
SQL Table
Airport
SQL Table
travel-sample
Bucket
Airline Indexes
Airport Indexes
Bucket Indexes

Index
Predicates
• Predicates only include certain documents in the index
• Reduces index size
• Queries must include the predicate to use the index
CREATE INDEX airlinesByName
ON `travel-sample` (name)
WHERE type = 'airline'

Indexing
Expressions
• Any deterministic function or operation may be used
• Queries must use the same expression to use the index
CREATE INDEX airlinesByNameInsensitive
ON `travel-sample` (LOWER(name))
WHERE type = 'airline'

Indexing
Date/Times
• LINQ assumes DateTime props are stored as ISO8601
• LINQ wraps DateTimes in STR_TO_MILLIS()
• Be sure any indexes also use STR_TO_MILLIS()
CREATE INDEX beersByUpdated
ON `beer-sample` (
STR_TO_MILLIS(updated))
WHERE type = 'beer'

Indexing
Arrays
• A nested array may be indexed
• Limit one array per index
• Queries must use array subqueries to use the index
CREATE INDEX routesByFlight
ON `travel-sample` (airline,
DISTINCT ARRAY p.flight
FOR p IN schedule END)
WHERE type = 'route'

SQL Server
Index
Consistency
SQL Server updates
indexes as part of the
transaction
Follow-on queries
immediately include
any mutations from the
previous query
Locks are used to
manage isolation
between simultaneous
transactions

Couchbase
Index
Consistency
Couchbase indexes
are updated
asynchronously
Follow-on queries
may or may not
include the
preceding mutations
Promotes high
mutation
performance

Querying
With
Consistency –
not_bounded
Returns results based on what has
been indexed at the time of the
query
Fastest, and the default
Most likely to return out-of-date
results

Querying
With
Consistency –
at_plus
Supply a mutation token, which is returned
from previous mutations
Ensures that the mutations have been
indexed
Slower than not_bounded
More up-to-date results
Also known as Read Your Own Write
(RYOW)

Querying
With
Consistency –
request_plus
Ensures that indexes are fully
updated with all mutations prior to
time when the query is submitted
Slowest, only use if necessary
Most up-to-date results

Resources
https://developer.couchbase.com/
https://forums.couchbase.com/
https://github.com/couchbaselabs/Linq2Couchbase
http://centeredgesoftware.com/
@btburnett3 on Twitter

Questions,
Take Two

NoSQL for SQL Server Developers using Couchbase

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to NoSQL for SQL Server Developers using Couchbase

Similar to NoSQL for SQL Server Developers using Couchbase (20)

Recently uploaded

Recently uploaded (20)

NoSQL for SQL Server Developers using Couchbase

Editor's Notes