This document discusses the history and evolution of data processing, from early accounting and financial transactions to modern databases, data warehouses, and analytics. It describes how organizations initially separated transactional databases from data warehouses for performance and governance reasons. Over time, new technologies like NoSQL, Hadoop, in-memory databases, and distributed systems have enabled merging these systems for improved performance while retaining SQL capabilities and reliability. The document advocates for multi-model solutions that combine the benefits of different data storage approaches.

1. A Brief History
of Data Processing
@garyorenstein
Deckset Theme - Next White
(c) Gary Orenstein 1

2. In The Beginning
(c) Gary Orenstein 2

3. Computers and Data
(c) Gary Orenstein 3

4. Accounting Transactions
(c) Gary Orenstein 4

5. Financial Transactions
(c) Gary Orenstein 5

6. Inventory Management
(c) Gary Orenstein 6

7. Human Resources
(c) Gary Orenstein 7

8. Enter the Database
(c) Gary Orenstein 8

9. Enter the Database
Put stuff in. Take stuff out.
Reliably. Quickly.
(c) Gary Orenstein 9

10. Now Let Me Ask A
Question
(c) Gary Orenstein 10

11. Just A Moment Please
(c) Gary Orenstein 11

12. Let's Build A Bigger,
Faster Database
(c) Gary Orenstein 12

13. maybe this is not as easy
as we thought
(c) Gary Orenstein 13

14. We Need A Data
Warehouse
(c) Gary Orenstein 14

15. Database, Meet Data Warehouse
(c) Gary Orenstein 15

16. Welcome to the ETL Gap
(c) Gary Orenstein 16

17. And Never The Two Shall Meet
(c) Gary Orenstein 17

18. Four Ways Your DBMS is Holding
You Back1
• ETL (Extract, Transform, Load)
• Analytic Latency
• Synchronization
• Copies of data
1
Source: Gartner Hybrid/Transactional/Analytical Processing Will Foster Opportunities for Dramatic Business Innovation,
Published: 28 January 2014
(c) Gary Orenstein 18

19. Why Did We Separate
• Performance
• Performance
• Performance
• Governance
(c) Gary Orenstein 19

20. Primary Performance
Impediment
Disk Drives
(c) Gary Orenstein 20

21. Scale-up for
Databases and Data
Warehouses
(c) Gary Orenstein 21

22. Complex and Costly
(c) Gary Orenstein 22

23. Quest for Scale-out
(c) Gary Orenstein 23

24. Paths Across Databases and Data Warehouses
(c) Gary Orenstein 24

25. NoSQL Wave And Hadoop Ecosystem
(c) Gary Orenstein 25

26. NoSQL Theory
• Scale
• Performance
• Eventual Consistency
• No need for SQL
(c) Gary Orenstein 26

27. NoSQL Reality
• Scale and performance?
• Stick to one thing at a time
• Consistency?
• Just wait
• Analytics?
• Thank goodness for SQL on
NoSQL
(c) Gary Orenstein 27

28. Hadoop Theory
• Just store it
• Who needs a schema
• Let's learn MapReduce
• Compute on disk, no problem
(c) Gary Orenstein 28

29. Hadoop Reality
• Data lakes are deep and dark
• Unclear what is going on
• Hard to fill shoes
• MapReduce
• Hadoop ecosystem engineering
• Occasionally feels like the data
strategy is upside down
(c) Gary Orenstein 29

30. What is the one
thing never
intended for
NoSQL and
Hadoop?
SQL.
(c) Gary Orenstein 30

31. Hadoop (HDFS) is a
filesystem, not a
database
(c) Gary Orenstein 31

32. NoSQL is, well...
Just part of a complete solution
(c) Gary Orenstein 32

33. Why did we pursue a split data
warehouse, NoSQL, HDFS?
Performance, performance, performance, governance
(c) Gary Orenstein 33

34. Idea
(c) Gary Orenstein 34

35. Let's Use Memory
(c) Gary Orenstein 35

36. Let's Use Memory
And understandably
architect for
persistence
(c) Gary Orenstein 36

37. What About Flash
(c) Gary Orenstein 37

38. The Right Solution Spans
Memory, Flash, and Disk
(c) Gary Orenstein 38

39. New Tech: Distributed Systems
(c) Gary Orenstein 39

40. Old Tech: Relational
Databases
Proudly serving SQL since 1970
(c) Gary Orenstein 40

41. Do we really
have to split
databases and
data
warehouses?
(c) Gary Orenstein 41

42. Merge
with in-memory solutions
(c) Gary Orenstein 42

43. Do I need to
worry about
high costs?
(c) Gary Orenstein 43

44. Distribute
scale across low cost machines or
cloud instances
(c) Gary Orenstein 44

45. Do I need to give up SQL?
(c) Gary Orenstein 45

46. Orchestrate
A Multi-Model Solution
• Full transactional SQL
• Inserts, updates, deletes
• JSON
• Geospatial
• Spark
(c) Gary Orenstein 46

47. But not all of my data
needs to be in-memory
Exactly
• Combine with a disk/flash based
columnstore
• Keep real-time data in memory
• Keep historical data on disk
• Query both datastores through a
single interface
(c) Gary Orenstein 47

48. What happens if a
node goes down?
Replicate for availability
(c) Gary Orenstein 48

49. What happens
if I need to
recover?
Persist logs to disk,
take snapshots,
make backups
(c) Gary Orenstein 49

50. Explore the possibilities
• In-memory, distributed database
• Relational and multi-model
• Software for your data center or the cloud
• Real-time data pipelines and analytics
• New world of modern applications
(c) Gary Orenstein 50

51. Find Your Inner SQL
for more
@garyorenstein
(c) Gary Orenstein 51