Agile Lab is an Italian company that specializes in leveraging innovative technologies like machine learning, big data, and artificial intelligence to satisfy customers' objectives. They have over 50 specialists with deep experience in production environments. The company believes in investing in its team through conferences, R&D projects, and welfare benefits. They also release open source frameworks on GitHub and share knowledge through meetups in Milan and Turin.

1. Ideas go software
www.agilelab.it

2. Machine Learning
BigData
ArtificialIntelligence
DeepLearning
Training
www.agilelab.it

3. Agile is our philosophy to transform
business ideas in real use cases.
We understand rapidly the key elements
of the Customer’s challenge, without fear
for any challenge.
We are an efficient organization totally focused on
leveraging innovative technologies to satisfy the
customer’s objectives and time-to-market.
Our reference handbook for organization is public:
https://publicagilefactory.gitlab.io/handbook/
The secret of getting ahead is getting started - Mark Twain
www.agilelab.it

4. More than 50 specialists, who have a deep
understanding of the real production
environments.
We constantly invest time and effort to increase
our skills on new technologies and innovative
analysis methodologies.
International Conferences, R&D Projects,
Smart Working, Welfare, Work for Equity…
We believe in our team and we invest for it.
Our reference handbook for organization is
public:
https://publicagilefactory.gitlab.io/handbook
/
www.agilelab.it
OurTeam
Think about the future, think about what you could do, don’t fear anything - Rita Levi Montalcini

5. www.agilelab.it
In memory and Machine learning
Hadoop Frameworks and
storage format
Change Data Capture
NoSQL Data Ingestion and Hi-scalable
applications
Deployment and resource
management
Cloud PaaS deployment
Deep Learning Development and Data Visualization
Certified

6. CHECKOUTOUR
FRAMEWORKONGITHUB
www.agilelab.it
Check ourframeworksreleasedfor“native”
DataQualityinBig Dataenvironmentsorour
SparkSearchlibrary.
SHARING KNOWLEDGE
IN MEETUPS
Don’tmissourMeetUps
appointmentsbothin Milanoand
Torino.
Darwin
Schemaregistry
WASP isourframeworkforstreaming
analyticsatscale.Ithasbeenreleased
OpenSourcesince thebeginningandis
goingtobe evolvedwithnew
developments.

7. Kafka

8. www.agilelab.it
• General purpose Message/Event Bus
• Real-Time Stream Processing in collaboration with other
tools
• Collecting User Activity Data
• Collecting Operational Metrics from applications, server or
devices
• Log Aggregation
• Change Data Capture
• Commit Log for distributed systems

9. www.agilelab.it
• As distributed system and services increasingly become part of modern architecture, this makes
for a fragile system
• As integration complexity increases, evolvability decreases
Slave
System
ETL
ETL
Master
System
Reporting
System
Service 1
Service 3
Service 2
ETL

10. www.agilelab.it
Decoupling integration patterns increase evolvability !!!
Slave
System
Reporting
System
Service 1 Service 3Service 2
Kafka
Kafka is not a «master» system, but a
«central» system for service and data
integration: real time and event based

11. www.agilelab.it
Focuson:
• Simplicity
• Throughput
• ExtremeScalability
Not for:
• Large batch exports
• Low Latency

12. • Message ordering is only guaranteed within a
partition for a topic
• Scaling is achieved through partitions
• At least once delivery
• For a topic with replication factor N, Kafka can
tolerate up to N-1 server failures without “losing”
any messages committed to the log
• Messages sent by a producer to a particular topic
partition will be appended in the order they are
sent
• A consumer instance sees messages in the order
they are store in the log

13. • Topics are broken up into ordered commit logs called partitions.
• To each message in a partition a sequential id (called offset) is assigned
• Data is retained for a configurable period of time

14. Producer

15. • Producers publish to a topic of their choosing (push)
• Load can be distributed
• Typically by “round-robin”
• Can also do “semantic partitioning” based on a key in the message
• Brokers load balance by partition
• Can support async (less durable) sending
• All nodes can answer metadata request about
• Which server is alive
• Where leaders are for the partition of a topic

16. Consumer

17. • Multiple Consumers can read from the same topic
• Each Consumer is responsible for managing it’s own offset
• Messages stay on Kafka… they are not removed after they are
consumed
• Consumers can go away
• And then came back

19. Why is Kafka so
scalable?

20. @Linkedin:
100+ clusters
4K brokers
100K topics
7M partitions
81M msg/sec ???!!!
On Prem scenario:
6 x Server:
• Xeon 2.5 GHz – 6 cores
• 6 x HDD 7200 rpm
• 32 GB RAM
• 1Gb Ethernet
1 topic – 6 partitions – 3x replica – 1 producer:
• 800K msg/sec
• 133K msg/sec x node
1 consumer from a 3x replicated topic:
• 950K msg/sec
Latency end to end:
3ms 99th
14ms 99.9th
Reference:
KV DB with persistence  10K write/sec x node
KV DB in memory  50K write/sec x node
Kafka is persistent...so what ??

21. Where is the trick?

22. Incredible
complex
architecture
from ‘90?
(do you know this system?)

23. Why is so fast?

24. It is simple !!!

25. • Focus on sequential I/O  no Random I/O
• How is this possible with multiple writers and readers ?
• Page cache
• No Indexes
• A partition it’s just a file in append mode
• ZeroCopy: directly copy data from pagecache to NIC buffer through OS sendfile system call,
skipping userspace and socket buffer copies
Writes are sequential by definition
Page Cache
–
no disk access

26. And hide complexity

27. • Everything in Kafka is batched and compressed !!!
• Communication is done via a high performance binary API over TCP protocol
Fake !!!!
• Batching of data to reduce
network calls, and also converting
a lot of random writes into
sequential ones.
• Compression of batches (and not
individual messages) using LZ4,
SNAPPY or GZIP codecs.

28. Questions ?

29. We are
hiring
www.agilelab.it info@agilelab.it www.linkedin.com/company/agile-lab
Thank you!