Javantura v3 - Real-time BigData ingestion and querying of aggregat...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

www.infobip.com
REAL-TIME BIG DATA INGESTION
AND QUERYING OF
AGGREGATED DATA
Davor Poldrugo
software engineer

Davor Poldrugo @ Infobip
Software engineer with interest in backend development,
high availability and distributed systems...

●
MOBILE SERVICES: Professional SMS, number validation, voice, USSD,
mobile payments; deeply integrated into the telecoms ...

Presentation overview
●
Dictionary
●
The real-time use case and the challenges
(because there are no problems ;)
●
The pla...

Dictionary
REAL-TIME noun
“the actual time during which something takes place <the computer may
partly analyze the data in...

Dictionary
INGEST verb
“to take (something, such as food) into your body : to swallow
(something)
— sometimes used figurat...

The real-time use case and the challenges
●
Our new web requirement: provide real-time data and graphs of
traffic
●
SMS Ca...

The platform and how we got here
●
There was only one node – a monolith
●
One transactional database (OLTP)
●
Traffic incr...

The platform and how we got here
●
After that querying has become complex:
– when one or more databases down for maintenan...

The platform and how we got here
●
Our Database Team introduced GREEN – our ODS/DWH
– named after the color of the pencil ...

Our path towards real-time data
●
GREEN ODS/DWH provided an abstract solution for all our traffic
data but was not in REAL...

Our path towards real-time data
1. All data entering the system is dispatched
to both the batch layer and the speed
layer ...

Our path towards real-time data
Know thyself! Adapt lambda architecture to fit your needs!
IpCore
(Core Message
Processing...

Architecture and component overview
Messaging Cloud
App
Message
Event
App
Message
Event
App
Message
Event
REAL-TIME
LAYER
...

Architecture and component overview
REAL-TIME
LAYER
Data Ingestion Service
Kafka cluster
Druid cluster
{
"sendDateTime":"2...

Architecture and component overview
REAL-TIME LAYER
Kafka cluster
Druid cluster
Data Ingestion Service
GREEN DB
ODS
DWH
BA...

Architecture and component overview
REAL-TIME LAYER
Druid cluster
QUERY LAYER
Data
Query
Service
POST /druid/v2 HTTP/1.1
H...

Architecture and component overview
REAL-TIME LAYER
Druid cluster
QUERY LAYER
Data
Query
Service
Response from Druid
[
{
"...

Architecture and component overview
KAFKA - https://kafka.apache.org/
●
Kafka maintains feeds of messages in categories ca...

Architecture and component overview
DRUID - http://druid.io/
Druid is a fast column-oriented distributed data store.
Real-...

Numbers and conlusion
Data pipeline
Max. throughput
(msg/s)
Ingest points → Data Ingestion Service 7700
Billing ingest poi...

Numbers and conclusion
PROBLEMS / CHALLENGES ;)
●
Added complexity to the flow
– Maintenance of “ingest point” code
– Main...

www.infobip.com
Q/A
Davor Poldrugo
software engineer
davor.poldrugo@infobip.com
dpoldrugo@gmail.com
REAL-TIME BIG DATA ING...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Javantura v3 - The Internet of (Leg... by HUJAK - Hrvatska ... 3281 views
Javantura v3 - ES6 – Future Is Now ... by HUJAK - Hrvatska ... 2935 views
Javantura v3 - Just say it – using ... by HUJAK - Hrvatska ... 3174 views
Javantura v3 - Spring Boot under th... by HUJAK - Hrvatska ... 3237 views
Javantura v4 - Security architectur... by HUJAK - Hrvatska ... 1077 views
Javantura v4 - DMN – supplement you... by HUJAK - Hrvatska ... 1257 views

View on Slideshare

1 of 25 Ad

View on Slideshare

1 of 25 Ad