You have a beautifully modelled warehouse and a lake with all your business data but you’re still looking at the past and making decisions from what happened yesterday. In this talk we’ll look at how you can really get value from your data and make decisions as events happen and even before they do.

1. @mlfowler_
@Claranet
From Warehouses to Lakes: The Value of Streams
Mike Fowler - Principal Data Engineer
PLACE CUSTOMER LOGO HERE
May 14/15 2020

2. @mlfowler_
@Claranet
Data is the New Oil
mattbuck (category) / CC BY-SA (https://creativecommons.org/licenses/by-sa/2.0)Source: https://twitter.com/TheEconomist/status/860135249552003073?s=20

3. @mlfowler_
@Claranet
Toilet Paper is the New Oil
https://wkfr.com/worlds-tallest-toilet-paper-pyramid-created-in-michigan/

4. @mlfowler_
@Claranet
The Data Warehouse
mattbuck (category) / CC BY-SA (https://creativecommons.org/licenses/by-sa/2.0)

5. @mlfowler_
@Claranet
Data Sources

6. @mlfowler_
@Claranet
Data Sources - Extract

7. @mlfowler_
@Claranet
Data Sources - Extract - Transform

8. @mlfowler_
@Claranet
Data Sources - Extract - Transform - Load

9. @mlfowler_
@Claranet
Dashboards! Reports!
Stiegenaufgang [MIT (http://opensource.org/licenses/mit-license.php) or BSD (http://opensource.org/licenses/bsd-license.php)]

10. @mlfowler_
@Claranet
Dashboards! Reports!
Stiegenaufgang [MIT (http://opensource.org/licenses/mit-license.php) or BSD (http://opensource.org/licenses/bsd-license.php)]

11. @mlfowler_
@Claranet
Problem: Pulling too hard on the source

12. @mlfowler_
@Claranet
Problem: Lag / Timeliness
Source: https://www.mentalfloss.com/article/62725/15-things-you-didnt-know-about-persistence-memory

13. @mlfowler_
@Claranet
Problem: Flooding the Warehouse
Source: https://www.shetnews.co.uk/2011/12/08/little-weather-damage-so-far/

14. @mlfowler_
@Claranet
Problem: Model First
Source: https://www.tamr.com/blog/stop-putting-the-ai-cart-before-the-data-horse/

15. @mlfowler_
@Claranet
Problem: Change is Expensive
Source: The Dark Knight, 2008

16. @mlfowler_
@Claranet
The Data Lake
Source:https://trimmtravels.com/best-time-to-visit-lake-louise/

17. @mlfowler_
@Claranet
Data Sources

18. @mlfowler_
@Claranet
Data Sources - Extract

19. @mlfowler_
@Claranet
Data Sources - Extract - Load

20. @mlfowler_
@Claranet
Data Sources - Extract - Load - Transform

21. @mlfowler_
@Claranet
Dashboards! Reports!
Stiegenaufgang [MIT (http://opensource.org/licenses/mit-license.php) or BSD (http://opensource.org/licenses/bsd-license.php)]

22. @mlfowler_
@Claranet
Benefit: Storage Independent of Compute
Source: https://scifi.stackexchange.com/questions/178793/which-way-is-up-in-a-borg-cube

23. @mlfowler_
@Claranet
Benefit: Model on Demand
Source: https://mythcreants.com/blog/implications-of-replicator-technology/

24. @mlfowler_
@Claranet
Problem: Pulling too hard on the source

25. @mlfowler_
@Claranet
Problem: Lag / Timeliness
Source: https://www.mentalfloss.com/article/62725/15-things-you-didnt-know-about-persistence-memory

26. @mlfowler_
@Claranet
Problem: The Data Swamp
Source:https://trimmtravels.com/best-time-to-visit-lake-louise/Source: https://techcrunch.com/2017/02/04/drain-the-swamp/

27. @mlfowler_
@Claranet
Data Streams
Source:https://trimmtravels.com/best-time-to-visit-lake-louise/Source: https://techcrunch.com/2017/02/04/drain-the-swamp/Source: By Bjørn Christian Tørrissen - Own work by uploader, http://bjornfree.com/galleries.html, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=17476156

28. @mlfowler_
@Claranet
Data Sources

29. @mlfowler_
@Claranet
Data Sources - Ingest

30. @mlfowler_
@Claranet
Data Sources - Ingest - Store

31. @mlfowler_
@Claranet
Data Sources - Ingest - Store - Process

32. @mlfowler_
@Claranet
Seriously?

33. @mlfowler_
@Claranet
Data is the New Water

34. @mlfowler_
@Claranet
Sources - Ingest - Store - Process - Repeat

35. @mlfowler_
@Claranet
Ingest
SQS
Messaging

36. @mlfowler_
@Claranet
Ingest
SQS
Streaming
Kinesis Data
Streams
Messaging

37. @mlfowler_
@Claranet
Ingest
SQS Glue
Streaming Integration
Kinesis Data
Streams
Messaging

38. @mlfowler_
@Claranet
Store
Redshift
Structured

39. @mlfowler_
@Claranet
Store
Redshift
Semi-Structured Structured
DynamoDB

40. @mlfowler_
@Claranet
Store
S3 Redshift
Semi-Structured Structured
DynamoDB
Unstructured

41. @mlfowler_
@Claranet
Process & Analyse
Batch
EMR

42. @mlfowler_
@Claranet
Process & Analyse
Kinesis Data
Analytics
Batch Streaming
EMR

43. @mlfowler_
@Claranet
Process & Analyse
SageMaker Kinesis Data
Analytics
Batch Streaming
EMR
Modelling

44. @mlfowler_
@Claranet
Putting it Together
Source: https://nerdist.com/article/star-trek-picard-data-where-he-is-now/

45. @mlfowler_
@Claranet
Scenario: Being On Call
Source: https://www.silicon.co.uk/wp-content/uploads/2017/02/Pager.jpg

46. @mlfowler_
@Claranet
Our engineer rests peacefully
Source: https://i.pinimg.com/originals/cb/32/5f/cb325f9c268bf2135125f512d957f8e6.jpg

47. @mlfowler_
@Claranet
04:03 - Prod is Down!!!!
Source: https://vignette.wikia.nocookie.net/memoryalpha/images/6/6b/RedAlert.jpg/revision/latest?cb=20100117050244&path-prefix=en

48. @mlfowler_
@Claranet
04:04 - All Clear!
Source: https://www.lakelouiseinn.com/wp-content/uploads/2019/01/LakeLouise2-1.jpg

49. @mlfowler_
@Claranet

50. @mlfowler_
@Claranet
The Problem
Many PagerDuty incidents resolve before I respond disrupting my
sleep needlessly

51. @mlfowler_
@Claranet
Introducing Mr Data

52. @mlfowler_
@Claranet
Introducing Mr Data

53. @mlfowler_
@Claranet
Introducing Mr Data

54. @mlfowler_
@Claranet
A Streaming Solution

55. @mlfowler_
@Claranet
A Streaming Solution

56. @mlfowler_
@Claranet
A Streaming Solution

57. @mlfowler_
@Claranet
A Streaming Solution

58. @mlfowler_
@Claranet
A Streaming Solution

59. @mlfowler_
@Claranet
A Streaming Solution

60. @mlfowler_
@Claranet
A Streaming Solution

61. @mlfowler_
@Claranet
A Streaming Solution

62. @mlfowler_
@Claranet
A Streaming Solution

63. @mlfowler_
@Claranet
A Streaming Solution

64. @mlfowler_
@Claranet
An AWS Solution

65. @mlfowler_
@Claranet
An AWS Solution

66. @mlfowler_
@Claranet
An AWS Solution

67. @mlfowler_
@Claranet
An AWS Solution

68. @mlfowler_
@Claranet
An AWS Solution

69. @mlfowler_
@Claranet
An AWS Solution

70. @mlfowler_
@Claranet
An AWS Solution

71. @mlfowler_
@Claranet
An AWS Solution

72. @mlfowler_
@Claranet
An AWS Solution

73. @mlfowler_
@Claranet
An AWS Solution

74. @mlfowler_
@Claranet
Solving the Problems

75. @mlfowler_
@Claranet
Solving the Problems

76. @mlfowler_
@Claranet
Solving the Problems

77. @mlfowler_
@Claranet
Solving the Problems

78. @mlfowler_
@Claranet
Fin

79. Questions?
Mike Fowler mlfowler
gh-mlfowler @mlfowler_