This document discusses the relevance of time series databases for real-time solutions. It begins with introductions and discusses how business focus is shifting towards real-time opportunities and use cases that require processing data immediately. It then discusses challenges with using traditional databases for real-time solutions and outlines alternatives like time series databases. Specific examples of Druid.io are provided, including its features, the author's experience building a demo using it, and a sample reference architecture.

1. RELEVANCE OF TIMESERIES DATABASES for
REAL-TIME SOLUTIONS
BY :
MUNIRAJU
VENKATESHA REDDY
 MUNIRAJU_V@HOTMAIL.COM

2. 1
2
3
4
5
6
About Me
Now()
Challenges & Opportunities
Solution
Time-Series Databases
Druid.io
Quick introduction about me & my work
Real-Time scenarios and importance of NOW()
Shifting business focus, Opportunities and Use cases
Technology challenges and alternatives
Quick view on various time-series databases
Why druid.io is most fit for real-time analytics ,
my experience, demo design patterns &
reference architecture

3. Muniraju Venkatesha Reddy (Muni)
is a lead solutions architect and having overall 18 years experience in
following areas.
Big Data Analytics & AI | DW& BI | Solution Strategy and advisory|
Cloud Data lake & Insights Solutions (IaaS/PaaS/SaaS) | Enterprise
Architecture | Consulting | Product & Processes Engineering | Delivery
planning & execution | Customer Relations | Grow the business |
People Management.
In my current role with HCL America, working with customers across
industry verticals on designing next generation solutions by leveraging
Big data, Analytics & AI.
Reach me @ : Muniraju_v@hotmail.com

4. NOW ( )

5. Your flight is at 9:00 AM for
important customer
meeting at later part of
the day in near by city
There is a traffic incident at 7:30
hrs and major disruptions to
airport roadway.
You want to know now()?
You want to know Later?
NOW ( )

6. You went out for a Family
& Friends Dinner Event at
6 PM
There is a burglary incident in
your house
NOW ( )
You want to know now()?
You want to know Later?

7. You want to know now()?
You want to know Later?
You are a warehouse
manager for large supply
chain business for
retailers. You are
expecting large stock of
goods to be delivered
tonight for next week
festival season
There will be sever weather
condition declared. Due to
transportation disruptions
arrival of goods will be delayed
for more than 4 days
NOW ( )

8. Important things are happening NOW ( )
NOW( ) is the only time life really happens
whether individual or business

9. Shifting business focus,
Opportunities and Use cases

10. “ The collection and storage
of data, for processing at a
scheduled time when a
sufficient amount of data
has been accumulated..”
“ The immediate processing of data
after the transaction occurs, with
the database being updated at the
time of the event..”
2.5K
Week 1 Week 2 Week 3 Week 4 Month 3 Qtr. 3
2.6K 2.4K2.4K 1.1M 2.9M
Batch() vs. NOW()…

11. Shift in Business Focus…

12. Cutting
Preventable
Losses
Routine
Operations
Missing
Opportunities
New
Opportunities
1
2
3
4
• Closed Loop control systems (Manufacturing)
• Systems & Network Monitoring (IT)
• Field Asset Monitoring (Multiple)
o Vending Machines, Oil rigs, Fleet
management, Telecom
• Transaction Processing (Finance)
o Fraud, Validations, Authentications
• Complex ICU Analytics (Healthcare)
• Disaster Warning Systems (Environment)
• Fraudulent Trades (Finance)
• Preventive Maintenance (Manufacturing)
• Customer Churn (Marketing)
• Brand Reputation (Marketing)
• Social Media (Sentiments)
• Missed Opportunities – Revenue
o Customer Services – Social, Products,
Advertisements,
Up Sell / Cross Cell
• Missed Opportunities – Efficiency
o Supply chain, Quality of Service, Insurance,
Capacity Management
• Autonomous Diagnostics and Connected
automotives
• Tractors are becoming soil sensors
• Community WIFI (Global Projections – 94M (2016) to
541M (2021))
• Cyber Security, Smarter Surveillance,….
1
2
3
4
Business Opportunities & Use Cases

13. Solution

14. Solution…
• Primarily Updates
• Performance
• Memory Swapping
• Scaling
• I/O Intensive
• Distributed Keys
• Memory Intensive
• Complex Index Structure
• Poor Secondary Index
Support
• Primarily Inserts
• Recent Time Interval
• Key association with timestamp
• Time/Space Chunking
o Fixed Duration Intervals
o Fixed Size chunks
o Adaptive Intervals
RDBMS
NOSQL
TIMESEARIES

15. Variants

16. GraphiteDB
Time Series Databases at a Glance…

17. Druid.io

18. Evolving Architecture Standards… Kappa (K)

19. Evolving Architecture Standards… Lambda (λ)
IMMUTABLE DATA (HDFS) PRECOMPUTE VIEWS
STREAM
PROCESS
STREAM
INCREMENT
VIEWS
BATCH
RECOMPUTE
ENTERPRISE
DATA
VIEW 1 VIEW 2 VIEW …n
VIEW
REAL-TIME
INCREMENT
BATCH VIEWS
REAL-TIME VIEWS
MERGED
VIEWS
RESEARCH
SCENARIOS
(<= 30to90 Sec)
OFFLINE
SCENARIOS
(<= 5 Secs)
ONLINE
SCENARIOS
(<= 5 Secs)
BATCH LAYER
SPEEDY LAYER
SERVING LAYER
SPLIT –> TRANSFORM -> SINK
SPLIT –> TRANSFORM -> SINK
Enrich -> Split
Enrich -> Split
Steams
Streams
Streams
Streams
MEDIATION
LAYER

20. Druid.io (supports both Kappa & Lambda)
Druid is an open-source data store designed for sub-second
queries on real-time and historical data. It is primarily used for
business intelligence (OLAP) queries on event data. Druid
provides low latency (real-time) data ingestion, flexible data
exploration, and fast data aggregation.
More Details… Druid.io
Sub-second OLAP Queries: Druid’s unique architecture
enables rapid multi-dimensional filtering, ad-hoc attribute
groupings, and extremely fast aggregations.
Real-time Streaming Ingestion : Druid employs lock-
free ingestion to allow for simultaneous ingestion and
querying of high dimensional, high volume data sets.
Explore events immediately after they occur.
Power Analytic Applications: Druid has numerous
features built for multi-tenancy. Power user-facing analytic
applications designed to be used by thousands of
concurrent users.
Cost Effective: Druid is extremely cost effective at scale
and has numerous features built in for cost reduction. Trade
off cost and performance with simple configuration knobs.
Highly Available: Druid is used to back SaaS
implementations that need to be up all the time. Druid
supports rolling updates so your data is still available and
queryable during software updates. Scale up or down
without data loss.
Scalable: Existing Druid deployments handle trillions of
events, petabytes of data, and thousands of queries every
second.
Key Features
My Experience
When I listening the customer challenge, I mentioned about the druid (I had
only theoretical knowledge) and will best fit for the scenario and this turned
out to be a request for showcasing short demo.
It took me 5 days to build out a simple end to end demo by using simulator
which generates real-time events, capture and ingesting real-time events via
kafka, creating ingestion spec (dimensions and measures) for druid and
developing a dashboard using superset without a line of programming code
on Hortonworks Sandbox.

21. Demo - Cell towers Capacity Monitoring in real time on Geo Map
Design Pattern Used for Demo

22. Real-time
Stream
Topic Ingestion
Spec
Real-time Exploration / BI
Offline Analysis & Analytics
Real-time
Stream
Topic Ingestion
Spec
Real-time Exploration / BI
Offline Analysis & Analytics
Data
Enrichment
Batch
Data
Tranquillity
Batch
Data
Real-time
Stream
Topic Ingestion
Spec
Real-time Exploration / BI
Offline Analysis & Analytics
Data
Enrichment
Batch
Data
Real-time
Stream
Topic Ingestion
Spec
Real-time Exploration / BI
Offline Analysis & Analytics
Data
Enrichment
Batch
Data
Pipeline
Development
1
2 4
3
Druid.io – Design Patterns

23. Druid.io Data Flow and Sample Reference Architecture
Druid.io Data Flow
Reference Architecture
Reference Architecture (Logical)