SlideShare a Scribd company logo
www.bottega52.it
Cloud, IoT and Big Data
Systems Engineering
Bottega52 SRL - P.IVA: 08848340967 | Piazza della Vittoria 47, 26900 Lodi (LO), Italy | www.bottega52.it | info@bottega52.it | Phone: +39 02 4003 0539
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Bottega52 SRL - P.IVA: 08848340967 | Piazza della Vittoria 47, 26900 Lodi (LO), Italy | www.bottega52.it | info@bottega52.it | Phone: +39 02 4003 0539
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Scan and go with the flow
How I met Ka+a
Ma0eo Ferroni
ma#eo@bo#ega52.it
1
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
About: Matteo Ferroni
Me
Passionate coder, software architect, (Web) surfer and
musician, local organizer of this meetup
Education
Ph.D., Politecnico di Milano
Visiting Researcher, University of California, Berkeley (UCB)
Work
Teaching, Politecnico di Milano & LIUC
CTO & Co-Founder, Bottega52 Srl
matteo@bottega52.it
@mattferroni
2
Internet Of Things
Cloud Platforms
Big-Data
OUR CORE
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Industry 4.0
33
Who we are
Bo#ega52 SRL is a company
providing so;ware for Cloud,
IoT and Big Data Systems.
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
About: Bottega52
Who
~12 nerds, mostly Ph.D. and M.S. @ Politecnico di Milano
What
Connect stuff, collect data, build real-time systems, create value
When
Founded in Nov’2014
Where
PoliHub, Milan (Italy)
Why
High quality systems by high quality people
4
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
How we met Kafka
“A digital watermark is a kind of marker covertly embedded in a noise-
tolerant signal such as an audio, video or image data.” (Wikipedia)
Photo credits: https://artlawjournal.com/invisible-watermark/
Case study: digital watermarks scan & analytics
5
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Case study: digital watermarks scan & analytics
Consumer
• Updated contents and promotions
• Full and updated product information
• Original Product Certification
Retail
• In-store engagement
• In-store Instant Promotion
• Faster Checkout (the code is repeated multiple
times over package)
Production
• Production control
• Quality control
• Logistics & tracking
6
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
It’s just a PoC…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Scan&Collect	
data
REST API
(Java)
DB
(MySQL)
7
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
It’s just a PoC…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Scan&Collect	
data
Watermarks	
CRUD
Basic	
Statistics
REST API
(Java)
DB
(MySQL)
8
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
It’s just a PoC…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Scan&Collect	
data
Watermarks	
CRUD
Enrich	Data	
(GMaps,	ext.)
Basic	
Statistics
REST API
(Java)
DB
(MySQL)
9
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
It’s just a PoC…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
DB
(MySQL)
Scan&Collect	
data
Watermarks	
CRUD
Multi-Users	
(administration)
Enrich	Data	
(GMaps,	ext.)
Basic	
Statistics
REST API
(Java)
10
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
It’s just a PoC…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
DB
(MySQL)
Scan&Collect	
data
Watermarks	
CRUD
Multi-Users	
(administration)
Multi-Customers	
(administration)
Enrich	Data	
(GMaps,	ext.)
Basic	
Statistics
Real-Time	
Analytics
REST API
(Java)
11
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
REST API
(Java)
REST API
(Java)
REST API
(Java)
It’s just a PoC…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
REST API
(Java)
DB
(MySQL)
Scan&Collect	
data
Watermarks	
CRUD
Multi-Users	
(administration)
Multi-Customers	
(administration)
Custom	Batch	
Reports
Enrich	Data	
(GMaps,	ext.)
Basic	
Statistics
Real-Time	
AnalyticsFault-Tolerancy
Scalability
Real-Time	
Alerts
12
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Evolvability, Reliability, Scalability
Goals:
• do not lose any scan
• meet new requirements, evolve my code fast
• design to scale
bug prone
custom logic,
bad design
…microservices,
anyone?
(Martin Fowler)
“A Microservices architecture as a service-oriented architecture composed of loosely coupled
elements that have bounded contexts.” – Adrian Cockcroft, Cloud Architect at Netflix
“[...] a suite of small services, each running in its own process and communicating with lightweight
mechanisms, often an HTTP resource API.” – Martin Fowler, Chief Scientist at ThoughtWorks
13
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break the PoC into pieces…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
REST API
(Java)
DB
(MySQL)
Scan&Collect	
data
Watermarks	
CRUD
Multi-Users	
(administration)
Multi-Customers	
(administration)
Custom	Batch	
Reports
Enrich	Data	
(GMaps,	ext.)
Basic	
Statistics
Real-Time	
Analytics
Real-Time	
Alerts
14
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break the PoC into pieces…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
DB
(MySQL)
Real-Time	
Alerts
Custom	Batch	
Reports
Basic	
StatisticsEnrich	Data	
(GMaps,	ext.)
Multi-Users	
(administration)
Multi-Customers	
(administration)
Real-Time	
Analytics
Watermarks	
CRUD
Scan&Collect	
data
15
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break data into pieces…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Real-Time	
Alerts
Custom	Batch	
Reports
Basic	
StatisticsEnrich	Data	
(GMaps,	ext.)
Multi-Users	
(administration)
Multi-Customers	
(administration)
Real-Time	
Analytics
Watermarks	
CRUD
Scan&Collect	
data
Fault-Tolerancy
Service	
Discovery
Deployment
Data	consistency,		
modeling	
and	evolution
[complexity]
Scalability
16
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break flow into pieces…
Domain-driven considerations:
• scan logic should be reliable and scalable
• backoffice logic can be slower, under maintenance
but eventually consistent
…CQRS and
Event Collaboration,
anyone? (Martin Fowler)
“CQRS stands for Command Query Responsibility Segregation. […] At its heart is the notion that you can
use a different model to update information than the model you use to read information. […]
CQRS fits well with event-based programming models. […] Having separate models raises questions about
how hard to keep those models consistent, which raises the likelihood of using eventual consistency. […]
CQRS allows you to separate the load from reads and writes allowing you to scale each independently.”
write
read
– Martin Fowler, Chief Scientist at ThoughtWorks
17
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break flow into pieces…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Scan&Collect	
data
Enrich	Data	
(GMaps,	ext.)
Real-Time	
Alerts
Real-Time	
Analytics
Custom	Batch	
Reports
Watermarks	
CRUD
Multi-Users	
(administration)
Multi-Customers	
(administration)
18
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break flow into pieces…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Scan&Collect	
data
Real-Time	
Alerts
Enrich	Data	
(GMaps,	ext.)
Watermarks	
CRUD
Multi-Users	
(administration)
Multi-Customers	
(administration)
…events streams…
Real-Time	
Analytics
Custom	Batch	
Reports
19
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break flow into pieces: how I met Kafka
Enrich	Data	
(GMaps,	ext.)
Real-Time	
Alerts
Scan&Collect	
data
Real-Time	
Analytics
Custom	Batch	
Reports
20
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break flow into pieces: how I met Kafka
Scan&Collect	
data
Enrich	Data	
(GMaps,	ext.)
Custom	Batch	
Reports
Real-Time	
Analytics
Real-Time	
Alerts
21
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break flow into pieces: how I met Kafka
Enrich	Data	
(GMaps,	ext.)
Custom	Batch	
Reports
Real-Time	
Analytics
Real-Time	
Alerts
Scan&Collect	
data
…
elasticsearch
Amazon	RDS
22
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Kafka Connect
Benefits over ‘do-it-yourself’ Producers and Consumers:
• Off-the-shelf, tested Connectors for common data sources are available
• Features fault tolerance and automatic load balancing when running in distributed
mode
• No coding required, just write configuration files for Kafka Connect
• Pluggable/extendable by developers
23
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
JDBC
configuration
example
(standalone)
CENSORED	CENSORED	CENSORED	CENSORED	CENSORED	CENSORED
Mode for detecting DB changes:	
incrementing,	timestamp,	
timestamp+incrementing	
24
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
• Exactly Once Delivery: the connector relies on Elasticsearch’s idempotent write semantics
to ensure exactly once delivery to Elasticsearch. When the keys are not included, or are
explicitly ignored, the connector will use topic+partition+offset as the key
• Mapping Inference: The connector can infer mappings from the Kafka Connect schemas.
If more customizations are needed (e.g. geo_point), we highly recommend to manually
create mappings.
Elasticsearch configuration example (standalone)
25
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Break flow into pieces: how I met Kafka
Enrich	Data	
(GMaps,	ext.)
Custom	Batch	
Reports
Real-Time	
Analytics
Real-Time	
Alerts
Scan&Collect	
data
…
Amazon	RDS
elasticsearch
26
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Kafka Streams
Kafka Streams API is a lightweight Java library for building distributed stream processing
applications in Kafka clusters
• Easy to embed in your own applications
• Supports windowing operations, and stateful processing including distributed joins
and aggregation
• Has fault-tolerance and supports distributed processing
• Includes both a Domain-Specific Language (DSL) and a low-level API
27
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Elasticsearch	
Production
Kafka	
Connect	ES	
Production
Break flow for this demo
Enriched	
production	
topic
Amazon	RDS
Kafka	
Connect	
JDBC
KafkaKafka	
Streams
Enrich	Data	
(GMaps,	ext.)
Scan	
topic
Watermarks	
topic
28
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Elasticsearch	
Demo
Kafka	
Connect	
ES	Demo
Elasticsearch	
Production
Kafka	
Connect	ES	
Production
Break flow for this demo
Enriched	
demo	topic
Enriched	
production	
topic
Amazon	RDS
Kafka	
Connect	
JDBC
Custom	Meetup	
Processor
KafkaKafka	
Streams
Enrich	Data	
(GMaps,	ext.)
Scan	
topic
Watermarks	
topic
28
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Kafka Streams example (custom demo process)
29
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Elasticsearch	
Demo
Kafka	
Connect	
ES	Demo
Elasticsearch	
Production
Kafka	
Connect	ES	
Production
Break flow for this demo
Enriched	
demo	topic
Enriched	
production	
topic
Amazon	RDS
Kafka	
Connect	
JDBC
Custom	Meetup	
Processor
KafkaKafka	
Streams
Enrich	Data	
(GMaps,	ext.)
Scan	
topic
Users	
topic
30
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Livedemo
31
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
REST API
(Java)
REST API
(Java)
REST API
(Java)
Conclusion: from this…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
REST API
(Java)
DB
(MySQL)
Scan&Collect	
data
Watermarks	
CRUD
Multi-Users	
(administration)
Multi-Customers	
(administration)
Custom	Batch	
Reports
Enrich	Data	
(GMaps,	ext.)
Basic	
Statistics
Real-Time	
AnalyticsFault-Tolerancy
Scalability
Real-Time	
Alerts
32
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
…to this…
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Real-Time	
Alerts
Custom	Batch	
Reports
Basic	
StatisticsEnrich	Data	
(GMaps,	ext.)
Multi-Users	
(administration)
Multi-Customers	
(administration)
Real-Time	
Analytics
Watermarks	
CRUD
Scan&Collect	
data
Fault-Tolerancy
Service	
Discovery
Deployment
Data	modeling	
and	evolution
[complexity]
Scalability
33
MOBILE
APP
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
…and finally, this!
Enrich	Data	
(GMaps,	ext.)
Custom	Batch	
Reports
Real-Time	
Analytics
Real-Time	
Alerts
Scan&Collect	
data
…
Amazon	RDS
elasticsearch
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
34
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Scan&Collect	
data
…
Amazon	RDS
Working on…
Enrich	Data	
(GMaps,	ext.)
Custom	Batch	
Reports
Real-Time	
Analytics
Real-Time	
Alerts
elasticsearch
…KSQL?
…Kafka	
Streams?
“It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.)
Ka=a REST
Proxy
35
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Do I really need Kafka?
…Big data? Not yet… [thousands of scan/day]
…Fast data? Not really… [eventual consistency, seconds]
…Do I really need the power of Kafka in this project?
Probably no, but it allowed me to have:
• Evolvability (change requests are coming from customers)
• Reliability (thanks to Connect, Streams and KSQL in the future)
• Scalable (thanks to Kafka)
• Elegant data pipelines (of course, I can do almost everything as a LAMP stack)
36
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Event-streams, data pipelines, everywhere!
37
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
Event-streams, data pipelines, everywhere!
The Log
38
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
My personal takeaways
In my opinion:
• Kafka is NOT ONLY for Big/Fast Data
• Kafka is NOT ONLY for Stream Computation
• Kafka enables new architectural patterns
• Everyone can talk at this meetup, no matter how big (data) he is :)
In the next episodes…
39
LOCKS
Contacts
Matteo Ferroni
Chief Technology Officer
matteo@bottega52.it
Bottega52 SRL
Tel: +39 02 4003 0539
Via Durando 38/A
20158, Milano (MI)
40
Copyright 2017 @ Bottega52 SRL - www.bottega52.it
www.bottega52.it
Cloud, IoT and Big Data
Systems Engineering
Bottega52 SRL - P.IVA: 08848340967 | Piazza della Vittoria 47, 26900 Lodi (LO), Italy | www.bottega52.it | info@bottega52.it | Phone: +39 02 4003 0539
Copyright 2017 @ Bottega52 SRL - www.bottega52.it

More Related Content

Recently uploaded

OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024
OpenMetadata
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
Google
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
kalichargn70th171
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
mz5nrf0n
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
Google
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
Codeigniter VS Cakephp Which is Better for Web Development.pdf
Codeigniter VS Cakephp Which is Better for Web Development.pdfCodeigniter VS Cakephp Which is Better for Web Development.pdf
Codeigniter VS Cakephp Which is Better for Web Development.pdf
Semiosis Software Private Limited
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
Shane Coughlan
 
Enterprise Resource Planning System in Telangana
Enterprise Resource Planning System in TelanganaEnterprise Resource Planning System in Telangana
Enterprise Resource Planning System in Telangana
NYGGS Automation Suite
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
TheSMSPoint
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
Alina Yurenko
 
E-commerce Application Development Company.pdf
E-commerce Application Development Company.pdfE-commerce Application Development Company.pdf
E-commerce Application Development Company.pdf
Hornet Dynamics
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Neo4j
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
Adele Miller
 
APIs for Browser Automation (MoT Meetup 2024)
APIs for Browser Automation (MoT Meetup 2024)APIs for Browser Automation (MoT Meetup 2024)
APIs for Browser Automation (MoT Meetup 2024)
Boni García
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
rickgrimesss22
 

Recently uploaded (20)

OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024OpenMetadata Community Meeting - 5th June 2024
OpenMetadata Community Meeting - 5th June 2024
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
Codeigniter VS Cakephp Which is Better for Web Development.pdf
Codeigniter VS Cakephp Which is Better for Web Development.pdfCodeigniter VS Cakephp Which is Better for Web Development.pdf
Codeigniter VS Cakephp Which is Better for Web Development.pdf
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
 
Enterprise Resource Planning System in Telangana
Enterprise Resource Planning System in TelanganaEnterprise Resource Planning System in Telangana
Enterprise Resource Planning System in Telangana
 
Transform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR SolutionsTransform Your Communication with Cloud-Based IVR Solutions
Transform Your Communication with Cloud-Based IVR Solutions
 
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)GOING AOT WITH GRAALVM FOR  SPRING BOOT (SPRING IO)
GOING AOT WITH GRAALVM FOR SPRING BOOT (SPRING IO)
 
E-commerce Application Development Company.pdf
E-commerce Application Development Company.pdfE-commerce Application Development Company.pdf
E-commerce Application Development Company.pdf
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
 
APIs for Browser Automation (MoT Meetup 2024)
APIs for Browser Automation (MoT Meetup 2024)APIs for Browser Automation (MoT Meetup 2024)
APIs for Browser Automation (MoT Meetup 2024)
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
 

Featured

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
Marius Sescu
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
Expeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
Pixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
ThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
marketingartwork
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
Skeleton Technologies
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
SpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Lily Ray
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
Rajiv Jayarajah, MAppComm, ACC
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
Christy Abraham Joy
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
Vit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
MindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
RachelPearson36
 

Featured (20)

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 

[1st Italian Kafka Meetup 2017] Scan and go with the flow: how I met Kafka

  • 1. www.bottega52.it Cloud, IoT and Big Data Systems Engineering Bottega52 SRL - P.IVA: 08848340967 | Piazza della Vittoria 47, 26900 Lodi (LO), Italy | www.bottega52.it | info@bottega52.it | Phone: +39 02 4003 0539 Copyright 2017 @ Bottega52 SRL - www.bottega52.it Bottega52 SRL - P.IVA: 08848340967 | Piazza della Vittoria 47, 26900 Lodi (LO), Italy | www.bottega52.it | info@bottega52.it | Phone: +39 02 4003 0539 Copyright 2017 @ Bottega52 SRL - www.bottega52.it Scan and go with the flow How I met Ka+a Ma0eo Ferroni ma#eo@bo#ega52.it 1
  • 2. Copyright 2017 @ Bottega52 SRL - www.bottega52.it About: Matteo Ferroni Me Passionate coder, software architect, (Web) surfer and musician, local organizer of this meetup Education Ph.D., Politecnico di Milano Visiting Researcher, University of California, Berkeley (UCB) Work Teaching, Politecnico di Milano & LIUC CTO & Co-Founder, Bottega52 Srl matteo@bottega52.it @mattferroni 2
  • 3. Internet Of Things Cloud Platforms Big-Data OUR CORE Copyright 2017 @ Bottega52 SRL - www.bottega52.it Industry 4.0 33 Who we are Bo#ega52 SRL is a company providing so;ware for Cloud, IoT and Big Data Systems.
  • 4. Copyright 2017 @ Bottega52 SRL - www.bottega52.it About: Bottega52 Who ~12 nerds, mostly Ph.D. and M.S. @ Politecnico di Milano What Connect stuff, collect data, build real-time systems, create value When Founded in Nov’2014 Where PoliHub, Milan (Italy) Why High quality systems by high quality people 4
  • 5. Copyright 2017 @ Bottega52 SRL - www.bottega52.it How we met Kafka “A digital watermark is a kind of marker covertly embedded in a noise- tolerant signal such as an audio, video or image data.” (Wikipedia) Photo credits: https://artlawjournal.com/invisible-watermark/ Case study: digital watermarks scan & analytics 5
  • 6. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Case study: digital watermarks scan & analytics Consumer • Updated contents and promotions • Full and updated product information • Original Product Certification Retail • In-store engagement • In-store Instant Promotion • Faster Checkout (the code is repeated multiple times over package) Production • Production control • Quality control • Logistics & tracking 6 MOBILE APP
  • 7. Copyright 2017 @ Bottega52 SRL - www.bottega52.it It’s just a PoC… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Scan&Collect data REST API (Java) DB (MySQL) 7 MOBILE APP
  • 8. Copyright 2017 @ Bottega52 SRL - www.bottega52.it It’s just a PoC… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Scan&Collect data Watermarks CRUD Basic Statistics REST API (Java) DB (MySQL) 8 MOBILE APP
  • 9. Copyright 2017 @ Bottega52 SRL - www.bottega52.it It’s just a PoC… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Scan&Collect data Watermarks CRUD Enrich Data (GMaps, ext.) Basic Statistics REST API (Java) DB (MySQL) 9 MOBILE APP
  • 10. Copyright 2017 @ Bottega52 SRL - www.bottega52.it It’s just a PoC… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) DB (MySQL) Scan&Collect data Watermarks CRUD Multi-Users (administration) Enrich Data (GMaps, ext.) Basic Statistics REST API (Java) 10 MOBILE APP
  • 11. Copyright 2017 @ Bottega52 SRL - www.bottega52.it It’s just a PoC… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) DB (MySQL) Scan&Collect data Watermarks CRUD Multi-Users (administration) Multi-Customers (administration) Enrich Data (GMaps, ext.) Basic Statistics Real-Time Analytics REST API (Java) 11 MOBILE APP
  • 12. Copyright 2017 @ Bottega52 SRL - www.bottega52.it REST API (Java) REST API (Java) REST API (Java) It’s just a PoC… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) REST API (Java) DB (MySQL) Scan&Collect data Watermarks CRUD Multi-Users (administration) Multi-Customers (administration) Custom Batch Reports Enrich Data (GMaps, ext.) Basic Statistics Real-Time AnalyticsFault-Tolerancy Scalability Real-Time Alerts 12 MOBILE APP
  • 13. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Evolvability, Reliability, Scalability Goals: • do not lose any scan • meet new requirements, evolve my code fast • design to scale bug prone custom logic, bad design …microservices, anyone? (Martin Fowler) “A Microservices architecture as a service-oriented architecture composed of loosely coupled elements that have bounded contexts.” – Adrian Cockcroft, Cloud Architect at Netflix “[...] a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.” – Martin Fowler, Chief Scientist at ThoughtWorks 13
  • 14. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break the PoC into pieces… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) REST API (Java) DB (MySQL) Scan&Collect data Watermarks CRUD Multi-Users (administration) Multi-Customers (administration) Custom Batch Reports Enrich Data (GMaps, ext.) Basic Statistics Real-Time Analytics Real-Time Alerts 14 MOBILE APP
  • 15. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break the PoC into pieces… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) DB (MySQL) Real-Time Alerts Custom Batch Reports Basic StatisticsEnrich Data (GMaps, ext.) Multi-Users (administration) Multi-Customers (administration) Real-Time Analytics Watermarks CRUD Scan&Collect data 15 MOBILE APP
  • 16. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break data into pieces… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Real-Time Alerts Custom Batch Reports Basic StatisticsEnrich Data (GMaps, ext.) Multi-Users (administration) Multi-Customers (administration) Real-Time Analytics Watermarks CRUD Scan&Collect data Fault-Tolerancy Service Discovery Deployment Data consistency, modeling and evolution [complexity] Scalability 16 MOBILE APP
  • 17. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break flow into pieces… Domain-driven considerations: • scan logic should be reliable and scalable • backoffice logic can be slower, under maintenance but eventually consistent …CQRS and Event Collaboration, anyone? (Martin Fowler) “CQRS stands for Command Query Responsibility Segregation. […] At its heart is the notion that you can use a different model to update information than the model you use to read information. […] CQRS fits well with event-based programming models. […] Having separate models raises questions about how hard to keep those models consistent, which raises the likelihood of using eventual consistency. […] CQRS allows you to separate the load from reads and writes allowing you to scale each independently.” write read – Martin Fowler, Chief Scientist at ThoughtWorks 17
  • 18. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break flow into pieces… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Scan&Collect data Enrich Data (GMaps, ext.) Real-Time Alerts Real-Time Analytics Custom Batch Reports Watermarks CRUD Multi-Users (administration) Multi-Customers (administration) 18 MOBILE APP
  • 19. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break flow into pieces… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Scan&Collect data Real-Time Alerts Enrich Data (GMaps, ext.) Watermarks CRUD Multi-Users (administration) Multi-Customers (administration) …events streams… Real-Time Analytics Custom Batch Reports 19 MOBILE APP
  • 20. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break flow into pieces: how I met Kafka Enrich Data (GMaps, ext.) Real-Time Alerts Scan&Collect data Real-Time Analytics Custom Batch Reports 20
  • 21. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break flow into pieces: how I met Kafka Scan&Collect data Enrich Data (GMaps, ext.) Custom Batch Reports Real-Time Analytics Real-Time Alerts 21
  • 22. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break flow into pieces: how I met Kafka Enrich Data (GMaps, ext.) Custom Batch Reports Real-Time Analytics Real-Time Alerts Scan&Collect data … elasticsearch Amazon RDS 22
  • 23. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Kafka Connect Benefits over ‘do-it-yourself’ Producers and Consumers: • Off-the-shelf, tested Connectors for common data sources are available • Features fault tolerance and automatic load balancing when running in distributed mode • No coding required, just write configuration files for Kafka Connect • Pluggable/extendable by developers 23
  • 24. Copyright 2017 @ Bottega52 SRL - www.bottega52.it JDBC configuration example (standalone) CENSORED CENSORED CENSORED CENSORED CENSORED CENSORED Mode for detecting DB changes: incrementing, timestamp, timestamp+incrementing 24
  • 25. Copyright 2017 @ Bottega52 SRL - www.bottega52.it • Exactly Once Delivery: the connector relies on Elasticsearch’s idempotent write semantics to ensure exactly once delivery to Elasticsearch. When the keys are not included, or are explicitly ignored, the connector will use topic+partition+offset as the key • Mapping Inference: The connector can infer mappings from the Kafka Connect schemas. If more customizations are needed (e.g. geo_point), we highly recommend to manually create mappings. Elasticsearch configuration example (standalone) 25
  • 26. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Break flow into pieces: how I met Kafka Enrich Data (GMaps, ext.) Custom Batch Reports Real-Time Analytics Real-Time Alerts Scan&Collect data … Amazon RDS elasticsearch 26
  • 27. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Kafka Streams Kafka Streams API is a lightweight Java library for building distributed stream processing applications in Kafka clusters • Easy to embed in your own applications • Supports windowing operations, and stateful processing including distributed joins and aggregation • Has fault-tolerance and supports distributed processing • Includes both a Domain-Specific Language (DSL) and a low-level API 27
  • 28. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Elasticsearch Production Kafka Connect ES Production Break flow for this demo Enriched production topic Amazon RDS Kafka Connect JDBC KafkaKafka Streams Enrich Data (GMaps, ext.) Scan topic Watermarks topic 28
  • 29. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Elasticsearch Demo Kafka Connect ES Demo Elasticsearch Production Kafka Connect ES Production Break flow for this demo Enriched demo topic Enriched production topic Amazon RDS Kafka Connect JDBC Custom Meetup Processor KafkaKafka Streams Enrich Data (GMaps, ext.) Scan topic Watermarks topic 28
  • 30. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Kafka Streams example (custom demo process) 29
  • 31. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Elasticsearch Demo Kafka Connect ES Demo Elasticsearch Production Kafka Connect ES Production Break flow for this demo Enriched demo topic Enriched production topic Amazon RDS Kafka Connect JDBC Custom Meetup Processor KafkaKafka Streams Enrich Data (GMaps, ext.) Scan topic Users topic 30
  • 32. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Livedemo 31
  • 33. Copyright 2017 @ Bottega52 SRL - www.bottega52.it REST API (Java) REST API (Java) REST API (Java) Conclusion: from this… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) REST API (Java) DB (MySQL) Scan&Collect data Watermarks CRUD Multi-Users (administration) Multi-Customers (administration) Custom Batch Reports Enrich Data (GMaps, ext.) Basic Statistics Real-Time AnalyticsFault-Tolerancy Scalability Real-Time Alerts 32 MOBILE APP
  • 34. Copyright 2017 @ Bottega52 SRL - www.bottega52.it …to this… “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Real-Time Alerts Custom Batch Reports Basic StatisticsEnrich Data (GMaps, ext.) Multi-Users (administration) Multi-Customers (administration) Real-Time Analytics Watermarks CRUD Scan&Collect data Fault-Tolerancy Service Discovery Deployment Data modeling and evolution [complexity] Scalability 33 MOBILE APP
  • 35. Copyright 2017 @ Bottega52 SRL - www.bottega52.it …and finally, this! Enrich Data (GMaps, ext.) Custom Batch Reports Real-Time Analytics Real-Time Alerts Scan&Collect data … Amazon RDS elasticsearch “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) 34
  • 36. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Scan&Collect data … Amazon RDS Working on… Enrich Data (GMaps, ext.) Custom Batch Reports Real-Time Analytics Real-Time Alerts elasticsearch …KSQL? …Kafka Streams? “It’s just a Proof-of-Concept (PoC), we’ll have time for develop production code” (cit.) Ka=a REST Proxy 35
  • 37. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Do I really need Kafka? …Big data? Not yet… [thousands of scan/day] …Fast data? Not really… [eventual consistency, seconds] …Do I really need the power of Kafka in this project? Probably no, but it allowed me to have: • Evolvability (change requests are coming from customers) • Reliability (thanks to Connect, Streams and KSQL in the future) • Scalable (thanks to Kafka) • Elegant data pipelines (of course, I can do almost everything as a LAMP stack) 36
  • 38. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Event-streams, data pipelines, everywhere! 37
  • 39. Copyright 2017 @ Bottega52 SRL - www.bottega52.it Event-streams, data pipelines, everywhere! The Log 38
  • 40. Copyright 2017 @ Bottega52 SRL - www.bottega52.it My personal takeaways In my opinion: • Kafka is NOT ONLY for Big/Fast Data • Kafka is NOT ONLY for Stream Computation • Kafka enables new architectural patterns • Everyone can talk at this meetup, no matter how big (data) he is :) In the next episodes… 39 LOCKS
  • 41. Contacts Matteo Ferroni Chief Technology Officer matteo@bottega52.it Bottega52 SRL Tel: +39 02 4003 0539 Via Durando 38/A 20158, Milano (MI) 40 Copyright 2017 @ Bottega52 SRL - www.bottega52.it
  • 42. www.bottega52.it Cloud, IoT and Big Data Systems Engineering Bottega52 SRL - P.IVA: 08848340967 | Piazza della Vittoria 47, 26900 Lodi (LO), Italy | www.bottega52.it | info@bottega52.it | Phone: +39 02 4003 0539 Copyright 2017 @ Bottega52 SRL - www.bottega52.it