The document discusses different types of databases and when each may be suitable. It begins with an introduction to the author and their background. It then discusses key differences between SQL and NoSQL databases, such as how SQL databases are better for large amounts of structured data while NoSQL databases can be useful for unstructured or distributed data. The document emphasizes that the type of database chosen depends on factors like the size and shape of the data as well as how the data will be used. Specific databases are described like MongoDB, Elasticsearch, Neo4j and others.

1. C H O O S I N G T H E
R I G H T DATA B AS E
D a v i d S i m o n s
@ S w a m W i t h Tu r t l e s

2. C H O O S I N G T H E
R I G H T DATA B AS E
D a v i d S i m o n s
@ S w a m W i t h Tu r t l e s

3. W H O A M I ?
• Tech Lead/Consultant at
Softwire
• Specification
• Architecture
• Design
• Implementation
• Support
• … and more!

4. W H O A M I ?
Hack with
Databases
Been on TV!
Political Nerd
@SwamWithTurtles

5. A ( P O L I T I C A L ) A S I D E

6. T H E AV R E F E R E N D U M

7. “ N O T O AV ”

8. T H AT ’ S T H E W H O L E
P O I N T
W E L L D U H …

9. I M A G I N I N G A
N E W S Y S T E M
C O U L D G E T U S
T O T H E S A M E
P L A C E B E T T E R
T H E ‘ Y E S ’ C A M PA I G N
WA S N O T…

10. R E - E X A M I N I N G
W H AT M A K E
T H E S Y S T E M
G O O D
T H E ‘ Y E S ’ C A M PA I G N WA S …

12. W H AT D O E S
T H I S H AV E T O
D O W I T H
D ATA B A S E S ?
E R R R …

13. SQL
Clear Market
Standard
• Denormalisation
• Relational Data
• Foreign Keys
• Data Integrity Checks
• Amazing Indexing
• Maturity and Robustness
• SQL as a query language
• Large Community Support
• All the plug-in/library
integration
• Available Support
Contracts

14. W H Y A R E
P E O P L E
A D V O C AT I N G
N O S Q L
D ATA B A S E S ?
S O …

15. T H E Y T H I N K T H E
G O A L P O S T S A R E W R O N G

16. N O S Q L
S O L U T I O N S
C A N ’ T [ V E R B ]
B U T …

17. I T ’ S N O T
M E A N T T O
S O LV E A
P R O B L E M T H AT
N E E D S
[ V E R B ] I N G
B E C A U S E …

18. T O D AY…
• What should we be asking
when we look at
databases?
• What do the results of
those questions mean for
your database?

19. How big is your data?
What shape is your data?
Are you happy to pay?
What uses your data?

20. S Q L I S P R E T T Y
G O O D F O R
L A R G E A M O U N T S
O F D ATA
T R U T H F U L LY

21. W I T H E N O U G H
D ATA , Y O U
H AV E T O
D I S T R I B U T E
T H E H A R D T R U T H

22. W H AT H A P P E N S
W H E N O N E
I N S TA N C E G O E S
D O W N ?
B U T …

23. C A P T H E O R E M
• Choose Two:
• Consistency
• Availability
• Partition Tolerance

24. P R O V I S O
There is a lot of thought in this area,
I am giving a simplified description
that would make many database people
pull their hair out.
https://martin.kleppmann.com/2015/05/11/
please-stop-calling-databases-cp-or-ap.html

25. C A P T H E O R E M
CP AP
Consistent
& Partition Tolerant
Available
& Partition Tolerant

26. C A P T H E O R E M
A
BC
Data = “Cat”
Data = “Cat”
Data = “Cat”

27. C A P T H E O R E M
A
BC
Data = “Cat”
Data = “Dog”
Data = “Cat”

28. C A P T H E O R E M
A
BC
Data = “Dog”
Data = “Dog”
Data = “Dog”

29. C A P T H E O R E M
A
BC
Data = “Dog”
Data = “Dog” Data = “Dog”

30. AVA I L A B L E ( “ A P ” ) S Y S T E M S
A
BC
Data = “Wolf”
Data = “Dog” Data = “Dog”

31. AVA I L A B L E ( “ A P ” ) S Y S T E M S
A
BC
Data = “Wolf”
Data = “Dog” Data = “Wolf”

32. C O N S I S T E N T ( “ C P ” ) S Y S T E M
A
BC
Data = “Dog”
Data = “Dog” Data = “Dog”

33. C O N S I S T E N T ( “ C P ” ) S Y S T E M
A
BC
Data = “Dog”
Data = “Dog” Data = “Dog”

34. C O N S I S T E N T ( “ C P ” ) S Y S T E M
A
BC
Data = “Wolf”
Data = “Dog” Data = “Wolf”

35. T H E R E ’ S A L O T
O F VA L U E I N
C O N S I S T E N C Y…

36. – D Y N A M O : A M A Z O N ’ S H I G H LY AVA I L A B L E K E Y- VA L U E
S T O R E
“Reliability at massive scale is one of the biggest
challenges we face at Amazon.com. Even the
slightest outage has significant financial
consequences and impacts customer trust.”

37. – D Y N A M O : A M A Z O N ’ S H I G H LY AVA I L A B L E K E Y- VA L U E
S T O R E
“Dynamo targets applications that operate with
weaker consistency if this results in high
availability.”

38. D Y N A M O I M P L E M E N TAT I O N S

39. N O T
G U A R A N T E E D
C O N S I S T E N C Y
T H E C O S T ?

46. “open source software project that enables distributed processing of large
data sets across clusters of commodity servers”

47. N O T A LWAY S
AVA I L A B L E
T H E C O S T ?

48. W H AT I F W E
D O N ’ T N E E D
T O D I S T R I B U T E ?
B U T …

49. How big is your data?
What shape is your data?
Are you happy to pay?
What uses your data?

50. R D B M S
( R E L AT I O N A L D ATA B A S E
M A N A G E M E N T S Y S T E M )

51. E V E RY R O W I S A “ T H I N G ”
Name Species
1 Puss
2 Dinah
3 Einstein
4 Jess

52. S E T- B A S E D
O P E R AT I O N
R E A D D A TA O U T W I T H

53. “ W H E R E ” ( I N T E R S E C T I O N )
Name Species
1 Puss
2 Dinah
3 Einstein
4 Jess

54. U N I O N S
Name Species
1 Puss
2 Dinah
3 Einstein
4 Jess
5 Nemo
6 Moby Dick
7 Wanda

55. J O I N S
Name Species
Species Coolness
Rating
1 Puss 0
2 Dinah 0
3 Einstein 10
4 Jess 0

56. C A R T E S I A N P R O D U C T S
0 10
0 10
0 10

57. C A R T E S I A N P R O D U C T S
0 10
0 10
0 10

58. W H AT S Q L
D O E S W E L L
• Modelling objects:
• With a fixed structure
and shape
• With a limited number of
relations
• With no opinion or
opinion of any deeper
underlying domain
R D B M S
( R E L AT I O N A L D ATA B A S E
M A N A G E M E N T S Y S T E M )

59. T H E R E A R E
P R O B L E M S T H I S
I S B A D F O R
B U T …

60. K E V I N B A C O N
S I X D E G R E E S O F …

65. T H E R E I S N O O P E N
E L E C T I O N D ATA
T H E P R O B L E M

66. E L E C T I O N D ATA

67. E L E C T I O N D ATA

68. E L E C T I O N D ATA
E =
(e.g.) member of, held in,
stood in…
V =
elections, constituencies,
years, politicians and parties

69. W O R L D ’ S L E A D I N G G R A P H D B :

70. "embedded, disk-based, fully transactional Java
persistence engine that stores data structured in
graphs rather than in tables"

71. D ATA S T O R A G E

72. D ATA S T O R A G E

73. D ATA
S T O R A G E
• Nodes and edges are all:
• Stored as first-class
objects on the file system
• “typed”
• Key-value stores

74. D ATA I N T H E
R E L AT I O N S
• “Joins” are first class
objects in the database
that can be queried at no
additional cost
• Certain queries become
trivial (e.g. Joins)

75. P R O T O T Y P I N G
• Easy to see and work with
data
• Schemaless
• Active community with a
lot of libraries

76. N E O 4 J U S E R S

77. T H E R E A R E
P R O B L E M S T H I S
I S
O V E R E N G I N E E R E D
F O R
B U T …

78. B A S I C L O C K
M E C H A N I S M S
F O R E X A M P L E

79. C A C H I N G
F O R E X A M P L E

80. K E Y / VA L U E
S T O R E S
P O S S I B L E S O L U T I O N

81. K E Y / VA L U E
S T O R E S
P O S S I B L E S O L U T I O N
DynamoDB
Used by BBC for managing scaled
scheduled processes
Used by Twitter for caching
your timeline

82. T I M E S E R I E S
D ATA B A S E

84. Timestamp Value
2014-06-10T12:00:00+0100 17
2014-06-10T12:15:00+0100 17
2014-06-10T12:30:00+0100 20
2014-06-10T12:45:00+0100 22
2014-06-10T13:00:00+0100 24
2014-06-10T13:15:00+0100 28
2014-06-10T13:30:00+0100 32

85. T H E R E A R E
P R O B L E M S T H I S
I S T O O S T R I C T
F O R
B U T …

87. S C H E M A L E S S

89. E V E RY R O W I S A “ T H I N G ”
N A M E = P U S S
C O O L N E S S = 0
!
N A M E = J E S S
C O O L N E S S = 0
!
N A M E = D I N A H
C O O L N E S S = 0
!
N A M E = E I N S T E I N
C O O L N E S S = 1 0
!
D O C U M E N T

90. S O C I A L M E D I A S I T E

92. Y O U C A N N O T
D E N O R M A L I S E
D ATA
WA R N I N G

93. T H E R E A R E
P R O B L E M S T H AT
S Q L I S A C T U A L LY
G O O D F O R …
B U T …

94. How big is your data?
What shape is your data?
Are you happy to pay?
What uses your data?

95. T H I S M AY S E E M
L I K E A T R I V I A L
P O I N T…

96. C O S T S …
• Oracle: $50,000+
• SQL Server: $10,000+

97. ( B A S I C A L LY )
F R E E
• MySQL
• PostgreSQL
• Riak
• Voldemort
• MariaDB
• Cassandra
• MongoDB

98. How big is your data?
What shape is your data?
Are you happy to pay?
What uses your data?

99. T H I N G S H AV E
T O U S E O U R
D ATA …
R E M E M B E R T H A T

100. A P P L I C AT I O N A R C H I T E C T U R E
D ATA B A S E C O D E Data

101. A P I A R C H I T E C T U R E
D ATA B A S E C O D E Data E N D U S E R

102. A P I A R C H I T E C T U R E
E N D U S E R What is the average age of …?

103. A P I A R C H I T E C T U R E
E N D U S E R
Er….
I think it was something like “Campbell”?

104. O U R C H O I C E I S
I N F O R M E D B Y
O U R P L A N S F O R
T H E A P P L I C AT I O N
R E M E M B E R T H A T

106. G E O S PAT I A L
I N D E X E S

108. (Another rambly aside)

109. L I N K E D
M E D I A
F R A M E W O R K

110. A PA C H E
M A R M O T TA
O U T O F T H E B O X …

112. D O C U M E N T
S T O R E

113. D O C U M E N T
S T O R E
ElasticSearch

114. E V E RY R O W I S A “ T H I N G ”
N A M E = P U S S
C O O L N E S S = 0
!
N A M E = J E S S
C O O L N E S S = 0
!
N A M E = D I N A H
C O O L N E S S = 0
!
N A M E = E I N S T E I N
C O O L N E S S = 1 0
!
D O C U M E N T

115. A PA C H E
L U C E N E

116. “Apache Lucene is a high-performance, full-
featured text search engine library … It is a
technology suitable for nearly any application that
requires full-text search”

117. F O C U S E D
A R O U N D
T E X T
S E A R C H I N G
Q U E R I E S

118. {
"query": {
"match": {"hobbies": "skateboard"}
}
}

119. {
"query": {
{"fuzzy": {"hobbies": “skateboarig"}}
}
}

120. {
"query": {
{"match": {"hobbies": {"query": "writing
reddit comments", "type": "phrase"}}}
}
}

121. I N S U M M A R Y …

122. D ATA B A S E S W E ’ V E TA L K E D A B O U T
T O D AY…
• SQL (Industry standard RDBMS coming on many flavours of different cost. used by many)
• Cassandra (Eventually consistent Dynamo implementation)
• Riak (Eventually consistent Dynamo implementation)
• Hadoop (Large ecosystem focused around scalability - focused on consistency and utilising many nodes. Used by
Facebook for their messenging)
• Neo4j (Graph Database, with poor scalability but high-fidelity data model. Used by many companies for highly
relational data)
• Redis (In-memory key-value store, used by twitter for their caching as a lightweight solution.)
• DynamoDB (AWS managed DBaaS, used by the BBC among others for light-weight key-value store needs such as
locking)
• MongoDB (Document store database, schedules with some interesting indexes. Used well by New York Times and
Foursquar.e Used purely by Diaspora v1)
• Apache Marmotta (Bleeding Edge DB used by Red Bull Media House to comply with Linked Data framework,
allowing easy integration)
• ElasticSearch (Document store database, providing easy searching out the box. Used by github and StackOverflow
among others)

123. How big is your data?
What shape is your
data?
Are you happy to pay?
What uses your
data?
If it gets big enough,
you have to distribute
AP (available)
vs.
CP (consistent)

124. How big is your data?
What shape is your
data?
Are you happy to pay?
What uses your
data?
If it’s not big, you can
use
high-fidelity data
models
Time Series
Graph DBs
Denormalised Rows
.. and more!

125. How big is your data?
What shape is your
data?
Are you happy to pay?
What uses your
data?
You may have to make
sacrifices…

126. How big is your data?
What shape is your
data?
Are you happy to pay?
What uses your
data?
Think about the
queries you’ll be
running up-front…
… this can prevent
costly rearchitecting
down the line

127. How big is your data?
What shape is your
data?
Are you happy to pay?
What uses your
data?
Your system is
unique
NoSQL is not
one thing -
there’s a range
of solutions
Consider them
on their own
merits!

128. A N Y Q U E S T I O N S >
D a v i d S i m o n s
@ S w a m W i t h Tu r t l e s