The document discusses the use of containerized databases in machine data environments, highlighting the importance of handling large volumes of diverse data from IoT devices, sensors, and logs. It presents various applications, such as predictive maintenance in fleet management and performance monitoring in smart apartments, emphasizing the need for horizontal scalability and real-time data analysis. Additionally, it introduces CrateDB as a solution for managing structured and unstructured data with features like automatic sharding and a distributed query execution engine.

1. Containerized DBs
In a Machine Data Environment
Docker NYC, 9th March 2017
@claus__m

2. Claus (me)
1.5yrs at Crate.io
DevRel/Field Engineering/Support/
Integrations/…
Talk to me about
Rust, Raspberry Pis, Tech!
@claus__m

3. Crate.io
Founded in 2013
~25 people and growing
Offices
San Francisco, Berlin,
Dornbirn (AT)
Dockerized
1M+ downloads on Docker hub
@claus__m

4. Machine Data
@claus__m

5. Source: HPE Jun 2016
http://www.slideshare.net/penumuru/harness-the-power-of-big-data-with-oracle-63438438/9
@claus__m

6. Machine Data
Characteristics
Millions of data points/second
Streaming in from sensors, devices, logs, etc.
Data diversity
Structured & unstructured JSON, Blobs
Real-time query performance
Monitoring & alerting
Complex queries of big data volumes
With Terabytes of historic data
Growth
Adding sources often means exponential
growth @claus__m

7. Machine Data
Internet of Things
Sensors, cameras, ...
Wearables, Gadgets
Location data, interaction data, ...
Logs & Monitoring data
Component health monitoring, access logs, ...
Industry 4.0, Digitization
Production line insights, automation, ...
Vehicles
Location data, health data, ...
@claus__m

8. Clickdrive.io
Fleet management & vehicle tracking
Vehicle health and tracking data
2,000 data points per car, per second
In-depth & real-time analysis
● Predictive maintenance
● Fix cars before catastrophe strikes
● Route & driver efficiency
● Accident reconstruction
@claus__m

9. Roomonitor
Smart apartments for property managers
● Climate monitoring & control
● Occupancy
● Noise level
● Access monitoring & control
Alerts
● AC is on with window open!
● Make rental properties more efficient,
safe, profitable
@claus__m

10. Skyhigh Networks
Cloud access security broker (CASB)
● Logging access data from cloud services
● Billions of accesses per day from 600+
customers, 10s of thousands of
concurrent TCP connections
Machine data is the fingerprint of fraud
● Run AI algorithms to define “normal”
● Flag deviations/anomalies in real-time
@claus__m

11. Architecture
For Machine Data
@claus__m

12. Microservices
Containers
Isolation by default
Flexibility
Building blocks
Horizontally scalable
Mostly
Stateful containers
Databases?
@claus__m

13. An Open Stack
Example
@claus__m

14. Consumer
Visualizer
An Example
Sensor
Sensor
Produces data
Consumer
Receives and enriches data
Visualizer
Draws stuff
@claus__m

15. LOAD BALANCER
V
C
Deploy!
S
U
S
S
C
V
C
V
Load balancer
For TLS, reverse proxying, load
balancing
High availability
3 instances
A few sensors
One user to actually use it
@claus__m

16. Go Live
More users!
More sensors and users
Data storage
Slow and fast
Monitoring & Analytics
Two different subsystems
LOAD BALANCER
V
C
S
S
U
S S
U
NoSQL DBMessage
Queue
SQL DB
U
S
S
C
V
C
MONITORING
V
S
ANALYTICS
@claus__m

17. But ...
Even more users?
Horizontal scaling?
Maintenance & bug hunting?
Mostly via scheduled downtimes
Reporting?
Kafka? Elasticsearch?
Security?
Access control?
Expertise?
Knowledge transfer?
LOAD BALANCER
V
C
S
U
S S
U
NoSQL DBMessage
Queue
SQL DB
U
S
S
C
V
C
MONITORING
V
S
ANALYTICS
S
@claus__m

18. A Database for
Microservices
Shared nothing
Replication
Sane defaults
Resilient
Cross-functional
@claus__m

19. Another DB?
Yay!
@claus__m

20. Yay!
…
which one
though?
@claus__m

21. CrateDB
github.com/crate/crate
hub.docker.com/r/_/crate
@claus__m

22. Yay!
@claus__m

23. CrateDB
Shared nothing
Partitioning & auto-sharding
Replication
(Almost) Zero config
Multi model: Structured &
unstructured
SQL
@claus__m

24. CrateDB Fundamentals
Disk-based index with
in-memory caching
Fast and efficient OS caching
Shards: Units of data
Concurrency by distributing
shards
Distributed query execution
engine
“Push down” queries
@claus__m

25. Postgres Wire Protocol
Presto Parser
Distributed Query Planner
Query Execution Engine
Elasticsearch
Lucene
CLIENT
@claus__m

26. A better
setup!
Horizontal scalability
Scale out everything
Reduced tech stack
Get to know it quicker
Live reporting
Use ad-hoc
queries on
production data
Flexibility
Schema
Evolution not
required @claus__m
LOAD BALANCER
V
C
S
S
U
S S
U
U
S
S
C
V
C
MONITORING
V
S
ANALYTICS

27. A better
setup!
No single point of failure
As highly available as your service
Reduced network traffic
Better reliability
No queue
Work with
real data
DB isolation
Accessible only
from the host
@claus__m
LOAD BALANCER
V
C
S
S
U
S S
U
U
S
S
C
V
C
MONITORING
V
S
ANALYTICS

28. Demo Time!
@claus__m

29. An Open Stack
for Machine Data
Ad-hoc analysis with SQL
Instant reporting on production
data
Integrates well
Legacy SQL applications included
Horizontally scalable
Container native, highly
availability
@claus__m

30. Thanks!
Links
https://github.com/celaus
https://github.com/crate
https://hub.docker.com/r/_/crate
https://crate.io
@crateio
@claus__m