In 2018 we've seen a huge uptick in applications using Kubernetes for their deployment method. Many times your persistent data layer is a difficult decision. What will you store data in, how long will you need to access this data and who will manage the lifecycle of this data? These are important questions many developers and ops teams have taken to heart. In this talk we'll review how the data layer is managed for high availability and reliability in modern application deployment. The attendee should leave having a better understanding of the options in front of them and their ability to build applications in any hosting environment.

1. Solving The Database
Problem
A tale of yesterday, today and
tomorrow.
Jay Gordon - @jaydestro

2. 2
Prior to MongoDB -
BuzzFeed,
DigitalOcean and
Datapipe
Disclaimer
NYC
Loves: My wife,
Hardcore Punk,
Metal Music, NY
Yankees and
Teaching You!
Jay Gordon
Cloud Developer Advocate
@jaydestro
MongoDB

3. 3

4. 4
WHAT COLOR IS THIS
DRESS?

5. 5
Social Media goes wild…

6. 6
It went viral

7. 7
x

8. 8

9. 9
x

10. 10
The Database Problem - Defined by Me.
1. The common issue that tends to keeps Ops people awake at
night:
"Sorry, my pager went off again - looks like a database
problem! Go back to sleep."
2. A undefinable and mostly generic issue with "the database" that
causes you to throw more resources at it rather than determine
the root cause.
"I have no clue why the site is down, must be a database
problem. But I upgraded the node to larger size and it's up."

11. 11
My personal
introduction to
the database
problem.
Disclaimer
Yeah I did some Windows
System Administration

12. 12
The Database
Problem 2003
Disclaimer
Error in your app then...

13. 13
The Database
Problem 2018
Disclaimer
Fifteen years later...

14. 14
Our Agenda
● A brief history lesson on databases.
● A series of problems.
● A number of potential solutions.
● Where do we go next?

15. A Quick History Lesson
Databases are about 40 years old.

16. 16
Disclaimer
Where we are starting, where we are going...
A brief look at how we got here...
Ted Codd authors his
influential paper "A
Relational Model of Data
for Large Shared Data
Banks"
SQL becomes a standard
query language for
databases.
SQL Server 1.0 is
released - available on
OS/2
MongoDB releases Atlas
cloud service.
RDS is launched by
Amazon WebServices
Johan Oskarsson of
Last.fm introduces
NoSQL.
2016
1970
1980
1989
2009

17. 17
Today - Cloud/Modern Infrastructure
● Distributed
● APIs and Drivers for many languages
● Cloud and Hybrid Solutions
● db-engines.com ranked 342 different databases
● More than just tabular/relational databases

18. PROBLEM ONE
The costs and complexity are different.

19. Computing when SQL was invented

20. Price Per GB of Data Over Time

21. Misconfiguration can cost you a bit...

22. People were cheaper… devalued by businesses

23. Hardware expensive

24. The paradigm has changed.

25. Developers rule in the economy of software.

26. 26
Costs and Planning
● Hardware costs down
● People costs up
● Misconfiguration can be costly
● Software defined economy

27. PROBLEM TWO
Data failure and recovery

28. 28

29. 29
Cool idea, but...

30. 30
Ouch, my repo...

31. 31
The horror...

32. 32
Recovery Point Objective - Recovery Time Objective

33. 33
Backups
● No backups… no app
● We pay for restores
● Data sets are huge
● Different restore methods
● Restores can take long

34. PROBLEM THREE
Don't leave your wallet on the ground in Times
Square, don't leave your database open to world.

35. 35
This is Times Square

36. 36
26 Million Visitors per year

37. 37
Would you leave this laying around here?

38. 38
Improperly configuring security for your database

39. 39
Could lead to this...

40. 40

41. 41
Security
● Security is a priority
● More attack vectors
● OS Patching
● Reputation
● More credentials to manage

42. PROBLEM FOUR
Performance, scaling up and down can be a lot of
work

43. What happens in 60 seconds

44. 44
Are queries taking longer than expected?

45. 45
Are your app processes taking longer than expected?

46. 46
When do we hit a peak? When do we hit a valley?

47. 47
Tests… do you run tests for performance in relation to database?

48. Virtualization, clouds and Containers

49. Containers… Many solutions to pick.

50. Virtualization, clouds and Containers

52. 52
Performance
● Understand Optimization
● Monitor for performance!
● Monitoring for potential failures!
● Need to scale quickly.
● Many storage types
● Code, testing via CI
● A Frankenstein's Monster.

53. These are not all the
problems, just a
subsect.
Let's talk about solutions...

54. 54
Modern answers to long term problems...
● Keeping our costs down
● Ensuring our data is backed up
● Securing our data
● Performance of application

55. 55
Solution One - Costs
● Flexible data… JSON!
● Open Source? Save $
● Understand what you're
provisioning.
● Flexible pricing models (pay-
as-you-go)

56. 56

57. 57
Remember that the cloud keeps your initial costs down...
● Reservations (services like AWS)
● Licensing built into costs +
infrastructure used to serve data.
● No commodity equipment to
consider "what to do with" in future.

58. 58
Services can abstract your licensing and hardware needs...

59. 59
Solution Two - Backups and Data Retention
● Have RTO/RPO
plans
● Perform regular tests
● Look for even faster
solutions
● Data retention and
purging plan

60. 60
TEST YOUR
DATABASE
BACKUPS.
Disclaimer
I quoted myself!
No one pays for backups,
we pay for restores.
- Me.

61. 61

62. 62

63. 63
Keep your results in a database, wiki with info on success...

64. 64
Backup management ● Retention policy
● Backups should be considered a
"mission critical"
● Keep stats on your backup restores
if possible
● Validate your data integrity by
testing the backup restores on a
regular cadence
● Look for services that include
queryable snapshots
● An online replica is not a
replacement for a backup

65. 65
Solution Three - Security
● Security is paramount!
● Use firewalls.
● Encrypt secrets for your
apps
● Don't guess, test

66. 66
Firewall your database...

67. 67
Make use of cloud security tools!

68. 68
Open source tools for security scanning

69. 69
Get help when you need it!

70. 70
Solution Four - Performance and Scaling
● Use modern distributed
systems that can leverage
loud native scaling
● Index, optimize and review
● Use monitoring tools
● Make note of trends
● Write tests

71. 71
Use the metrics available to you for your database.

72. 72
Look for services that provide assistance on optimization.

73. 73
Monitor all the things!

74. 74
Write tests! Review Errors!

75. 75
Look for ease to when you need to scale up or down

76. 76
Avoid over-provisioning - run tests on capacity. Plan ahead!

77. 77
Consider a DBaaS for your database.

78. 78
Distribute your data to improve reliability.

79. 79
Key
Takeaways
Remember this stuff...
• Plan ahead for the future
• Security is paramount for reliability for
your users
• Testing your restore process, having
RTO/RPO policies
• Avoiding lock-in and using flexible
data formats (I like JSON, personally)
Good!
• Don't make assumptions about the
future, the worst is possible if you
don't consider it
• Over provisioning and under
provisioning
• Leaving your wallet on the ground in
Times Square (lack of security)
• "Set and forget" your backups
Problematic!

80. 80
Where do we go from here?
● Improved Security
● Dev first options
● NoOps, GitOps, DevOps,
Serverless
● Businesses will make use
of new services
● Quicker deployment
● More places for more
applications

81. 81
Thanks for
your time, if
you have any
questions…
Disclaimer
Like my buddy Rico, I am all
ears.