Kafka short

•

0 likes•877 views

This document discusses using Kafka for processing big data and check-in locations. It proposes using Kafka to build a "Gogobot Checkins Heat-Map" service that collects check-in addresses from various microservices at high volumes, geocodes the addresses, and displays the locations as a heat map. Kafka is presented as a good fit because it can reliably handle very high volumes of data ingestion and can scale to support many producers and consumers. The document also briefly outlines LinkedIn's original architecture challenges at high data volumes that led them to adopt Kafka as a better paradigm for messaging that is stateless, fault-tolerant, and can scale without limiting consumer throughput.

Software

1
Kafka forKafka for
BigDataBigData
ProcessingProcessing
Yanai Franchi , TikalYanai Franchi , Tikal

4
gogobot checkin
Heat Map Service
Lets' Develop
“Gogobot Checkins Heat-Map”

5
Key Notes
● Collector Service - Collects checkins as text addresses
– We need to use GeoLocation ServiceWe need to use GeoLocation Service
● Upon elapsed interval, the last locations list will be
displayed as Heat-Map in GUI.
● Web Scale service – 10Ks checkins/seconds all over the
world (imaginary, but lets do it for the exercise).

6
Heat-Map Context
Text-Address
Checkins Heat-Map
Service
Gogobot System
Gogobot
Micro Service
Gogobot
Micro Service
Gogobot
Micro Service
Geo Location
Service
Get-GeoCode(Address)
Heat-Map
Last Interval Locations

7
Tons of Addresses
Arriving Every Second

9
Checkin HTTP
Reactor Checkins
Topic
Storm Heat-Map
Topology
Hotzones
Topic
Web App
Push via WebSocket
Publish
Checkins
HDFS
Checkin HTTP
Firehose

11
They all are Good
But not for all use-cases

18
Looks Familiar :
Use Messaging
(i.e. JMS, RabbitMQ)

24
Paradigm Change :
Do NOT track message
consumption

28
Stateless Broker &
Doesn't Fear the File System

29
Topics
● Logical collections of partitions (the physical fi les).
● A broker contains some of the partitions for a topic

30
A partition is Consumed by
Exactly One Group's Consumer

32
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

33
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

34
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

35
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

36
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

37
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

38
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

39
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

40
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

41
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

42
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1
Producer 1 Producer 2

43
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1
Producer 1 Producer 2

44
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1
Producer 1 Producer 2

45
Performance Benchmark
1 Broker
1 Producer
1 Consumer

48
LinkedIn Kafka Performance (2012)
● 8 nodes per datacenter
– ~20 GB RAM available for Kafka~20 GB RAM available for Kafka
– 6TB storage, RAID 10, basic SATA drives6TB storage, RAID 10, basic SATA drives
● 10 billion messages/day
● Sustained peak:
– 172,000 messages/second written172,000 messages/second written
– 950,000 messages/second read950,000 messages/second read
● 367 topics
● 40 real-time consumers
● Many ad hoc consumers
● 9.5TB log retained (~ 6 days)
● End-to-end delivery time: A few seconds

Video games are written as a main loop: process player input, update the state of the game, render a new frame to the screen, repeat. They do this 60 times a second, with millisecond timing. Most monitoring tools are also written as loops: send a probe, wait for the response, update a data store, sleep. Often this is done pretty slowly, maybe once a second! In video games if you can’t update fast enough, you skip the rendering step and the frame rate drops. With monitoring tools if your loop takes to long you also stop logging data as often, and instead of choppy gameplay you get gaps in your graphs, often when you need that data the most! Let’s use ping as an example and see how we can rewrite its main loop to function more like a video game, keeping a high frame rate.

Exploring Garbage Collection

ESUG

The Power of Both Choices: Practical Load Balancing for Distributed Stream Pr...

Anis Nasir

Jeff Fischer - Python and IoT: From Chips and Bits to Data Science

PyData

Deploying applications on Kubernetes is getting easier every day. From a minimal deployment to distributed service mesh enabled applications with planning and a little bit of YAML resilient cloud-native applications are the norm. In this session, Christopher Bradford and Ty Morton will help answer the following questions: - What about your data behind these apps? - Are you running those in a multi-cluster environment or sending everything back to a common location? - How do you modernize to a distributed peer-to-peer data architecture? - How do you plan for this change? - Are there pitfalls on the road to enlightened data? Join this session to explore the key concepts needed when investigating multi-cluster deployments for data. This includes: - Cluster planning - Network design - Security - Failure handling

Fun421 stephens

Tess Ferrandez

Super COMPUTING JournalPandey_G

Smallworld Data Check-Out to Microstation

Safe Software

Ray Chien from Fujitsu Consulting, has created a Smallworld plugin capable to "check-out" Smallworld data into Microstation format, where it may be edited and "checked-in" back to Smallworld using FME 2014 desktop transformers. Written for the 4.0 Smallworld Translator, it does not support fme_db_operation and 'true' updates, but can accurately detect changed features and only replace those as needed. While Microstation does not support feature attributes, data integrity is preserved by combining with Microsoft Access Linkage support. In addition, the plugin is written with full Dynamic reader and writer support, making it scalable and adaptive to data model changes.

On heap cache vs off-heap cache

rgrebski

Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...

Igor Sfiligoi

Presented at PEARC20. This talk presents expanding the IceCube’s production HTCondor pool using cost-effective GPU instances in preemptible mode gathered from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and the Google Cloud Platform. Using this setup, we sustained for a whole workday about 15k GPUs, corresponding to around 170 PFLOP32s, integrating over one EFLOP32 hour worth of science output for a price tag of about $60k. In this paper, we provide the reasoning behind Cloud instance selection, a description of the setup and an analysis of the provisioned resources, as well as a short description of the actual science output of the exercise.

Smallworld to GreGG - FME Server Automation

Safe Software

SkyhookDM - Towards an Arrow-Native Storage System

JayjeetChakraborty

Computer Performance Microscopy with SHIMhiyangxi

Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...

Igor Sfiligoi

The San Diego Supercomputer Center (SDSC) and the Wisconsin IceCube Particle Astrophysics Center (WIPAC) at the University of Wisconsin–Madison successfully completed a computational experiment as part of a multi-institution collaboration that marshalled all globally available for sale GPUs (graphics processing units) across Amazon Web Services, Microsoft Azure, and the Google Cloud Platform. In all, some 51,500 GPU processors were used during the approximately 2-hour experiment conducted on November 16 and funded under a National Science Foundation EAGER grant. The experiment – completed just prior to the opening of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19) in Denver, CO – was coordinated by Frank Würthwein, SDSC Lead for High-Throughput Computing, and Benedikt Riedel, Computing Manager for the IceCube Neutrino Observatory and Global Computing Coordinator at WIPAC. Igor Sfiligoi, SDSC’s lead scientific software developer for high-throughput computing, and David Schultz, a production software manager with IceCube, conducted the actual run. This presentation was given at several booths during SC19 by Frank Würthwein.

Alternative cryptocurrencies

vpnmentor

Pig TPC-H Benchmark and Performance Tuning

Jie Li

NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...

Igor Sfiligoi

Burst data retrieval after 50k GPU Cloud run

Igor Sfiligoi

하이퍼커넥트 데이터 팀이 데이터 증가에 대처해온 기록

Jaehyeuk Oh

Pharo Status Fosdem 2015

Marcus Denker

Data-intensive IceCube Cloud Burst

Igor Sfiligoi

Progress_190315

Hyo jeong Lee

CERN OpenStack Cloud Control Plane - From VMs to K8s

Belmiro Moreira

CERN is the home of the Large Hadron Collider (LHC), a 27km circular proton accelerator that generates petabytes of physics data every year. To process all this data, CERN runs an OpenStack Cloud (>300K cores) that helps scientists all around the world to unveil the mysteries of the Universe. The Infrastructure is also used to run all the IT services of the Organization. Delivering these services, with high performance and reliable service levels has been one of the major challenges for the CERN Cloud engineering team. We have been constantly iterating the architecture and deployment model of the Cloud control plane. In this presentation we will describe the different control plane architecture models that we relied over the years. Finally, we will describe all the work done to move the OpenStack Cloud control plane from VMs into a kubernetes cluster. We will report about our experience running this architecture at scale, its advantages and challenges.

Introduction to Apache Kafka

Shiao-An Yuan

OVHcloud Tech Talks S01E09 - OVHcloud Data Processing : Le nouveau service po...

OVHcloud

Nous vivons une époque où tout est connecté, de nos ampoules à notre éditeur de texte, les objets et services qui nous entourent devienne de plus en plus intelligents. Pour ce faire ils génèrent des données, elles sont nécessaires au bon fonctionnement du service ou de l'objet, mais elles sont également utiles pour faire évoluer les produits. Ces données peuvent rapidement représenter de gros volumes, plusieurs dizaines voir centaines de gigaoctets. Une question se pose alors, comment traiter de tels volumes ? Comment en tirer du sens et de la valeur à cette échelle ? Avec OVHcloud Data Processing, une solution basée sur le framework Apache Spark, nous répondons à ce besoin. Venez découvrir comment vous aussi, en quelques cliques, pouvez exécuter votre code sur une infrastructure taillée pour vos besoins. Au travers de différents exemples, comme une analyse du traffic des taxis New-Yorkais, nous verrons comment Data Processing a été pensé, comment il fonctionne et comment il peut être utilisé pour valoriser vos données.

What's hot

証明駆動開発のたのしみ@名古屋reject会議Hiroki Mizuno

Taking Your Database Beyond the Border of a Single Kubernetes Cluster

Christopher Bradford

Fun421 stephens

Tess Ferrandez

Super COMPUTING JournalPandey_G

Smallworld Data Check-Out to Microstation

Safe Software

On heap cache vs off-heap cache

rgrebski

Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...

Igor Sfiligoi

Smallworld to GreGG - FME Server Automation

Safe Software

SkyhookDM - Towards an Arrow-Native Storage System

JayjeetChakraborty

Computer Performance Microscopy with SHIMhiyangxi

Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...

Igor Sfiligoi

Alternative cryptocurrencies

vpnmentor

Pig TPC-H Benchmark and Performance Tuning

Jie Li

NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...

Igor Sfiligoi

Burst data retrieval after 50k GPU Cloud run

Igor Sfiligoi

하이퍼커넥트 데이터 팀이 데이터 증가에 대처해온 기록

Jaehyeuk Oh

Pharo Status Fosdem 2015

Marcus Denker

Data-intensive IceCube Cloud Burst

Igor Sfiligoi

Progress_190315

Hyo jeong Lee

CERN OpenStack Cloud Control Plane - From VMs to K8s

Belmiro Moreira

What's hot (20)

証明駆動開発のたのしみ@名古屋reject会議

Taking Your Database Beyond the Border of a Single Kubernetes Cluster

Fun421 stephens

Super COMPUTING Journal

Smallworld Data Check-Out to Microstation

On heap cache vs off-heap cache

Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...

Smallworld to GreGG - FME Server Automation

SkyhookDM - Towards an Arrow-Native Storage System

Computer Performance Microscopy with SHIM

Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...

Alternative cryptocurrencies

Pig TPC-H Benchmark and Performance Tuning

NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...

Burst data retrieval after 50k GPU Cloud run

하이퍼커넥트 데이터 팀이 데이터 증가에 대처해온 기록

Pharo Status Fosdem 2015

Data-intensive IceCube Cloud Burst

Progress_190315

CERN OpenStack Cloud Control Plane - From VMs to K8s

Similar to Kafka short

Introduction to Apache Kafka

Shiao-An Yuan

OVHcloud Tech Talks S01E09 - OVHcloud Data Processing : Le nouveau service po...

OVHcloud

Apache Pulsar at Yahoo! Japan

StreamNative

More Efficient Object Replication in OpenStack Summit Juno

Kota Tsuyuzaki

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

SaltStack

Google is making the power of its datacenter, network, and technology innovations available to the world through its Cloud services. This presentation will provide an overview of the Google Cloud Platform and a deeper dive on Google Compute Engine. Google recently made an open source contribution to SaltStack and now you can now use Salt Cloud to manage your Compute Engine resources (IaaS virtual machine services). Come find out more about Google's Cloud Platform and how you can leverage Google scale with SaltStack.

Netflix Keystone—Cloud scale event processing pipeline

Monal Daxini

The state of Spark in the cloud

Nicolas Poggi

Originally presented at Strata EU 2017: https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/57631 Cloud providers currently offer convenient on-demand managed big data clusters (PaaS) with a pay-as-you-go model. In PaaS, analytical engines such as Spark and Hive come ready to use, with a general-purpose configuration and upgrade management. Over the last year, the Spark framework and APIs have been evolving very rapidly, with major improvements on performance and the release of v2, making it challenging to keep up-to-date production services both on-premises and in the cloud for compatibility and stability. Nicolas Poggi evaluates the out-of-the-box support for Spark and compares the offerings, reliability, scalability, and price-performance from major PaaS providers, including Azure HDinsight, Amazon Web Services EMR, Google Dataproc, and Rackspace Cloud Big Data, with an on-premises commodity cluster as baseline. Nicolas uses BigBench, the brand new standard (TPCx-BB) for big data systems, with both Spark and Hive implementations for benchmarking the systems. BigBench combines SQL queries, MapReduce, user code (UDF), and machine learning, which makes it ideal to stress Spark libraries (SparkSQL, DataFrames, MLlib, etc.). The work is framed within the ALOJA research project, which features an open source benchmarking and analysis platform that has been recently extended to support SQL-on-Hadoop engines and BigBench. The ALOJA project aims to lower the total cost of ownership (TCO) of big data deployments and study their performance characteristics for optimization. Nicolas highlights how to easily repeat the benchmarks through ALOJA and benefit from BigBench to optimize your Spark cluster for advanced users. The work is a continuation of a paper to be published at the IEEE Big Data 16 conference. (A preprint copy can be obtained here.)

Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)

confluent

Presenter: Tim Berglund, Senior Director of Developer Experience, Confluent It has become a truism in the past decade that building systems at scale, using non-relational databases, requires giving up on the transactional guarantees afforded by the relational databases of yore. ACID transactional semantics are fine, but we all know you can’t have them all in a distributed system. Or can we? In this talk, I will argue that by designing our systems around a distributed log like Apache Kafka®, we can in fact achieve ACID semantics at scale. We can ensure that distributed write operations can be applied atomically, consistently, in isolation between services, and of course with durability. What seems to be a counterintuitive conclusion ends up being straightforwardly achievable using existing technologies, as an elusive set of properties becomes relatively easy to achieve with the right architectural paradigm underlying the application.

Drinking from the Firehose - Real-time Metrics

Samantha Quiñones

To understand an application’s performance, first you have to know what to measure. That’s the easy part. How do you take those measurements? Store them? Analyze them? Get them to the people who need them? Well, that’s where things get complicated, especially in the high-traffic distributed systems of the modern web! Like careful scientists, we must observe our subjects without altering them, and we must report our findings quickly so that we have the data necessary to make smart choices about the health and growth of the system. Let’s explore the lessons learned by engineers at one of the world’s top web companies in their quest to find meaning at 5 MB/s. We’ll discuss the tools and techniques that enable the collection, indexing, and analysis of billions or more datapoints each hour, and learn how these same approaches can empower your applications and your business, no matter the scale.

Moving Toward Deep Learning Algorithms on HPCC Systems

HPCC Systems

Clojure News Feed Performance Testing

Dynamical Software, Inc.

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

Natan Silnitsky

At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker. This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters. In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner. You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant. You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC. When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.

2014 11-05 hpcac-kniep_christian_dockermpi

QNIB Solutions

Docker is going to change the way services are deployed by encapsulating them, thus provide a robust and complete continous development/deployment workflow. But how is docker impacting the compute part of HPC? Containerization uses kernel functions (Namespaces/CGroups) to encapsulate the processes, while allowing a fain grained customization of the compute stack; sitting on top of a stripped down bare-metal OS that provides basic services. This sessions aims to explore if docker already is the 'virtualization' technique the HPC community is waiting for. Could a distributed MPI job across multiple containers placed on different physical nodes beat a natively started job?

Microservices interaction at scale using Apache Kafka

Ivan Ursul

Apache Beam (incubating)

Apache Apex

Presenter: Kenn Knowles, Software Engineer, Google & Apache Beam (incubating) PPMC member Apache Beam (incubating) is a programming model and library for unified batch & streaming big data processing. This talk will cover the Beam programming model broadly, including its origin story and vision for the future. We will dig into how Beam separates concerns for authors of streaming data processing pipelines, isolating what you want to compute from where your data is distributed in time and when you want to produce output. Time permitting, we might dive deeper into what goes into building a Beam runner, for example atop Apache Apex.

Log everything!

ICANS GmbH

Datafying Bitcoins

Tariq Ahmad

Go GC: Prioritizing Low Latency and Simplicity

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1IPjAIS. Rick Hudson discusses the motivation, performance, and technical challenges of Go's low latency concurrent GC and why the approach fits Go well. Filmed at qconsf.com. Rick Hudson is a member of Google’s Go team. Rick has published papers on language runtimes, memory management, concurrency, synchronization, memory models, and transactional memory.

Streaming architecture patterns

hadooparchbook

An Introduction to time series with Team Apache

Patrick McFadin

We as an industry are collecting more data every year. IoT, web, and mobile applications send torrents of bits to our data centers that have to be processed and stored, even as users expect an always-on experience—leaving little room for error. Patrick McFadin explores how successful companies do this every day using the powerful Team Apache: Apache Kafka, Spark, and Cassandra. Patrick walks you through organizing a stream of data into an efficient queue using Apache Kafka, processing the data in flight using Apache Spark Streaming, storing the data in a highly scaling and fault-tolerant database using Apache Cassandra, and transforming and finding insights in volumes of stored data using Apache Spark. Topics include: - Understanding the right use case - Considerations when deploying Apache Kafka - Processing streams with Apache Spark Streaming - A deep dive into how Apache Cassandra stores data - Integration between Cassandra and Spark - Data models for time series - Postprocessing without ETL using Apache Spark on Cassandra

Similar to Kafka short (20)

Introduction to Apache Kafka

OVHcloud Tech Talks S01E09 - OVHcloud Data Processing : Le nouveau service po...

Apache Pulsar at Yahoo! Japan

More Efficient Object Replication in OpenStack Summit Juno

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

Netflix Keystone—Cloud scale event processing pipeline

The state of Spark in the cloud

Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)

Drinking from the Firehose - Real-time Metrics

Moving Toward Deep Learning Algorithms on HPCC Systems

Clojure News Feed Performance Testing

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

2014 11-05 hpcac-kniep_christian_dockermpi

Microservices interaction at scale using Apache Kafka

Apache Beam (incubating)

Log everything!

Datafying Bitcoins

Go GC: Prioritizing Low Latency and Simplicity

Streaming architecture patterns

An Introduction to time series with Team Apache

More from Tikal Knowledge

Clojure - LISP on the JVM

Tikal Knowledge

Clojure presentationTikal Knowledge

Tabtale story: Building a publishing and monitoring mobile games architecture...

Tikal Knowledge

At Tabtale we are setting up an entire server side for the all the publishing services. These services include dynamic game configurations, error collection, analytics, social services and more.Tabtale is among the world’s top app publishers with millions of downloads so we are putting a great deal of effort in creating an extremely highly scalable and fault tolerant architecture. In this talk I will go over the architecture decisions taken to support the scalability and diversity that is required from the server side services while keeping the management of this infrastructure sane. ~30min By Assaf Gannon

Docking your services_with_dockerTikal Knowledge

Who moved my_box

Tikal Knowledge

Writing a Fullstack Application with Javascript - Remote media player

Tikal Knowledge

Writing apps should be fun. This is a statement I try to keep in mind when approaching any application development. When I first realized that a full stack of Javascript can be used to create an application I was very happy, since for me coding in Javascript is a lot of fun. I was soon quite eager to neglect the traditional Java server-side + Javascript client, giving up the Java server side "bliss" and the constant context switching, in favor of a full stack of Javascript whenever I could. For the past 2+ years I have been developing a variety of applications using a pure JS stack: Games, Forex trading middleware, CRM, CMS, sophisticated proxies, and various utilities. I must admit my productivity has increased dramatically and the time-to-market of each and every project was a fraction of what it would have been using traditional Java based server side infra-structure. On October 29th 2013 I gave a talk at the "Javascript.everywhere" meetup event, in which I presented the development route of Youtube Remote Control Application with a "slave" player using Javascript stack all over (NodeJS, Socket.IO, ExpressJS, AngularJS, MongoDB). I'm pleased to share the presentation - See more at: http://www.tikalk.com/js/writing-fullstack-application-javascript#sthash.IjFNbQJ2.dpuf

.Net OSS Ci & CD with Jenkins - JUC ISRAEL 2013

Tikal Knowledge

Tikal's Backbone_js introduction workshop

Tikal Knowledge

TCE Automation

Tikal Knowledge

Tce automation-d4Tikal Knowledge

Cloud computing - an insight into "how does it really work ?"

Tikal Knowledge

Introduction to Cloud Computing

Tikal Knowledge

Tikal Fuse Day Access Layer Implementation (C#) Based On Mongo DbTikal Knowledge

Ship early ship often with Django

Tikal Knowledge

Google App EngineTikal Knowledge

AWS Case StudyTikal Knowledge

Introduction To Cloud ComputingTikal Knowledge

Building Components In Flex3

Tikal Knowledge

Osgi Democamp

Tikal Knowledge

JBUG 11 - Django-The Web Framework For Perfectionists With Deadlines

Tikal Knowledge

More from Tikal Knowledge (20)

Clojure - LISP on the JVM

Clojure presentation

Tabtale story: Building a publishing and monitoring mobile games architecture...

Docking your services_with_docker

Who moved my_box

Writing a Fullstack Application with Javascript - Remote media player

.Net OSS Ci & CD with Jenkins - JUC ISRAEL 2013

Tikal's Backbone_js introduction workshop

TCE Automation

Tce automation-d4

Cloud computing - an insight into "how does it really work ?"

Introduction to Cloud Computing

Tikal Fuse Day Access Layer Implementation (C#) Based On Mongo Db

Ship early ship often with Django

Google App Engine

AWS Case Study

Introduction To Cloud Computing

Building Components In Flex3

Osgi Democamp

JBUG 11 - Django-The Web Framework For Perfectionists With Deadlines

Recently uploaded

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

Google

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-pilot-review/ AI Pilot Review: Key Features ✅Deploy AI expert bots in Any Niche With Just A Click ✅With one keyword, generate complete funnels, websites, landing pages, and more. ✅More than 85 AI features are included in the AI pilot. ✅No setup or configuration; use your voice (like Siri) to do whatever you want. ✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It… ✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again. ✅ZERO Limits On Features Or Usages ✅Use Our AI-powered Traffic To Get Hundreds Of Customers ✅No Complicated Setup: Get Up And Running In 2 Minutes ✅99.99% Up-Time Guaranteed ✅30 Days Money-Back Guarantee ✅ZERO Upfront Cost See My Other Reviews Article: (1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review

Mobile App Development Company In Noida | Drona Infotech

Drona Infotech

Artificia Intellicence and XPath Extension Functions

Octavian Nadolu

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

Crescat

Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry. Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events. With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use. Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements. If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Google

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-fusion-buddy-review AI Fusion Buddy Review: Key Features ✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini ✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique! ✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs! ✅Fully automated AI articles bulk generation! ✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more. ✅With one keyword or URL, generate complete websites, landing pages, and more… ✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7. ✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches. ✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all! ✅Save over $5000 per year and kick out dependency on third parties completely! ✅Brand New App: Not available anywhere else! ✅ Beginner-friendly! ✅ZERO upfront cost or any extra expenses ✅Risk-Free: 30-Day Money-Back Guarantee! ✅Commercial License included! See My Other Reviews Article: (1) AI Genie Review: https://sumonreview.com/ai-genie-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review #AIFusionBuddyReview, #AIFusionBuddyFeatures, #AIFusionBuddyPricing, #AIFusionBuddyProsandCons, #AIFusionBuddyTutorial, #AIFusionBuddyUserExperience #AIFusionBuddyforBeginners, #AIFusionBuddyBenefits, #AIFusionBuddyComparison, #AIFusionBuddyInstallation, #AIFusionBuddyRefundPolicy, #AIFusionBuddyDemo, #AIFusionBuddyMaintenanceFees, #AIFusionBuddyNewbieFriendly, #WhatIsAIFusionBuddy?, #HowDoesAIFusionBuddyWorks

Enterprise Resource Planning System in Telangana

NYGGS Automation Suite

Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics. To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

timtebeek1

Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management

Utilocate

Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly. The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Using Xen Hypervisor for Functional Safety

Ayan Halder

Empowering Growth with Best Software Development Company in Noida - Deuglo

Deuglo Infosystem Pvt Ltd

Do you want Software for your Business? Visit Deuglo Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions. Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC). Requirement — Collecting the Requirements is the first Phase in the SSLC process. Feasibility Study — after completing the requirement process they move to the design phase. Design — in this phase, they start designing the software. Coding — when designing is completed, the developers start coding for the software. Testing — in this phase when the coding of the software is done the testing team will start testing. Installation — after completion of testing, the application opens to the live server and launches! Maintenance — after completing the software development, customers start using the software.

A Sighting of filterA in Typelevel Rite of Passage

Philip Schwarz

openEuler Case Study - The Journey to Supply Chain Security

Shane Coughlan

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Neo4j

Atelier - Innover avec l’IA Générative et les graphes de connaissances Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement. Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.

2024 eCommerceDays Toulouse - Sylius 2.0.pdf

Łukasz Chruściel

AI Genie Review: World’s First Open AI WordPress Website Creator

Google

AI Genie Review: World’s First Open AI WordPress Website Creator 👉👉 Click Here To Get More Info 👇👇 https://sumonreview.com/ai-genie-review AI Genie Review: Key Features ✅Creates Limitless Real-Time Unique Content, auto-publishing Posts, Pages & Images directly from Chat GPT & Open AI on WordPress in any Niche ✅First & Only Google Bard Approved Software That Publishes 100% Original, SEO Friendly Content using Open AI ✅Publish Automated Posts and Pages using AI Genie directly on Your website ✅50 DFY Websites Included Without Adding Any Images, Content Or Doing Anything Yourself ✅Integrated Chat GPT Bot gives Instant Answers on Your Website to Visitors ✅Just Enter the title, and your Content for Pages and Posts will be ready on your website ✅Automatically insert visually appealing images into posts based on keywords and titles. ✅Choose the temperature of the content and control its randomness. ✅Control the length of the content to be generated. ✅Never Worry About Paying Huge Money Monthly To Top Content Creation Platforms ✅100% Easy-to-Use, Newbie-Friendly Technology ✅30-Days Money-Back Guarantee See My Other Reviews Article: (1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review (2) SocioWave Review: https://sumonreview.com/sociowave-review (3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review (4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review #AIGenieApp #AIGenieBonus #AIGenieBonuses #AIGenieDemo #AIGenieDownload #AIGenieLegit #AIGenieLiveDemo #AIGenieOTO #AIGeniePreview #AIGenieReview #AIGenieReviewandBonus #AIGenieScamorLegit #AIGenieSoftware #AIGenieUpgrades #AIGenieUpsells #HowDoesAlGenie #HowtoBuyAIGenie #HowtoMakeMoneywithAIGenie #MakeMoneyOnline #MakeMoneywithAIGenie

Fundamentals of Programming and Language Processors

Rakesh Kumar R

E-commerce Application Development Company.pdf

Hornet Dynamics

Orion Context Broker introduction 20240604

Fermin Galan

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Mind IT Systems

Recently uploaded (20)

AI Pilot Review: The World’s First Virtual Assistant Marketing Suite

Mobile App Development Company In Noida | Drona Infotech

Artificia Intellicence and XPath Extension Functions

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App

Enterprise Resource Planning System in Telangana

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management

Essentials of Automations: The Art of Triggers and Actions in FME

Using Xen Hypervisor for Functional Safety

Empowering Growth with Best Software Development Company in Noida - Deuglo

A Sighting of filterA in Typelevel Rite of Passage

openEuler Case Study - The Journey to Supply Chain Security

Atelier - Innover avec l’IA Générative et les graphes de connaissances

2024 eCommerceDays Toulouse - Sylius 2.0.pdf

AI Genie Review: World’s First Open AI WordPress Website Creator

Fundamentals of Programming and Language Processors

E-commerce Application Development Company.pdf

Orion Context Broker introduction 20240604

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Kafka short

1. 1 Kafka forKafka for BigDataBigData ProcessingProcessing Yanai Franchi , TikalYanai Franchi , Tikal

2. 2 Find “Hot” Places

3. 3

4. 4 gogobot checkin Heat Map Service Lets' Develop “Gogobot Checkins Heat-Map”

5. 5 Key Notes ● Collector Service - Collects checkins as text addresses – We need to use GeoLocation ServiceWe need to use GeoLocation Service ● Upon elapsed interval, the last locations list will be displayed as Heat-Map in GUI. ● Web Scale service – 10Ks checkins/seconds all over the world (imaginary, but lets do it for the exercise).

6. 6 Heat-Map Context Text-Address Checkins Heat-Map Service Gogobot System Gogobot Micro Service Gogobot Micro Service Gogobot Micro Service Geo Location Service Get-GeoCode(Address) Heat-Map Last Interval Locations

7. 7 Tons of Addresses Arriving Every Second

8. 8 First Reaction...

9. 9 Checkin HTTP Reactor Checkins Topic Storm Heat-Map Topology Hotzones Topic Web App Push via WebSocket Publish Checkins HDFS Checkin HTTP Firehose

10. 10

11. 11 They all are Good But not for all use-cases

12. 12 Kafka A little introduction

13. 13

14. 14 Why ?

15. 15 LinkedIn Original Architecture

16. 16

17. 17 What LinkedIn Want...

18. 18 Looks Familiar : Use Messaging (i.e. JMS, RabbitMQ)

19. 19

20. 20

21. 21

22. 22

23. 23 It Didn't Scale...

24. 24 Paradigm Change : Do NOT track message consumption

25. 25

26. 26

27. 27

28. 28 Stateless Broker & Doesn't Fear the File System

29. 29 Topics ● Logical collections of partitions (the physical fi les). ● A broker contains some of the partitions for a topic

30. 30 A partition is Consumed by Exactly One Group's Consumer

31. 31 Distributed & Fault-Tolerant

32. 32 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

33. 33 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

34. 34 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

35. 35 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

36. 36 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

37. 37 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

38. 38 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

39. 39 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

40. 40 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

41. 41 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

42. 42 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Producer 1 Producer 2

43. 43 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Producer 1 Producer 2

44. 44 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Producer 1 Producer 2

45. 45 Performance Benchmark 1 Broker 1 Producer 1 Consumer

46. 46

47. 47

48. 48 LinkedIn Kafka Performance (2012) ● 8 nodes per datacenter – ~20 GB RAM available for Kafka~20 GB RAM available for Kafka – 6TB storage, RAID 10, basic SATA drives6TB storage, RAID 10, basic SATA drives ● 10 billion messages/day ● Sustained peak: – 172,000 messages/second written172,000 messages/second written – 950,000 messages/second read950,000 messages/second read ● 367 topics ● 40 real-time consumers ● Many ad hoc consumers ● 9.5TB log retained (~ 6 days) ● End-to-end delivery time: A few seconds

49. 49 Thanks

Kafka short

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kafka short

Similar to Kafka short (20)

More from Tikal Knowledge

More from Tikal Knowledge (20)

Recently uploaded

Recently uploaded (20)

Kafka short