Kafka short

•

0 likes•877 views

This document discusses using Kafka for processing big data and check-in locations. It proposes using Kafka to build a "Gogobot Checkins Heat-Map" service that collects check-in addresses from various microservices at high volumes, geocodes the addresses, and displays the locations as a heat map. Kafka is presented as a good fit because it can reliably handle very high volumes of data ingestion and can scale to support many producers and consumers. The document also briefly outlines LinkedIn's original architecture challenges at high data volumes that led them to adopt Kafka as a better paradigm for messaging that is stateless, fault-tolerant, and can scale without limiting consumer throughput.

Software

1
Kafka forKafka for
BigDataBigData
ProcessingProcessing
Yanai Franchi , TikalYanai Franchi , Tikal

4
gogobot checkin
Heat Map Service
Lets' Develop
“Gogobot Checkins Heat-Map”

5
Key Notes
● Collector Service - Collects checkins as text addresses
– We need to use GeoLocation ServiceWe need to use GeoLocation Service
● Upon elapsed interval, the last locations list will be
displayed as Heat-Map in GUI.
● Web Scale service – 10Ks checkins/seconds all over the
world (imaginary, but lets do it for the exercise).

6
Heat-Map Context
Text-Address
Checkins Heat-Map
Service
Gogobot System
Gogobot
Micro Service
Gogobot
Micro Service
Gogobot
Micro Service
Geo Location
Service
Get-GeoCode(Address)
Heat-Map
Last Interval Locations

7
Tons of Addresses
Arriving Every Second

9
Checkin HTTP
Reactor Checkins
Topic
Storm Heat-Map
Topology
Hotzones
Topic
Web App
Push via WebSocket
Publish
Checkins
HDFS
Checkin HTTP
Firehose

11
They all are Good
But not for all use-cases

18
Looks Familiar :
Use Messaging
(i.e. JMS, RabbitMQ)

24
Paradigm Change :
Do NOT track message
consumption

28
Stateless Broker &
Doesn't Fear the File System

29
Topics
● Logical collections of partitions (the physical fi les).
● A broker contains some of the partitions for a topic

30
A partition is Consumed by
Exactly One Group's Consumer

32
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

33
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

34
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

35
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

36
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

37
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

38
Broker 1 Broker 4Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

39
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

40
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

41
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1 Consumer 2
Producer 1 Producer 2

42
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1
Producer 1 Producer 2

43
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1
Producer 1 Producer 2

44
Broker 1 Broker 3Broker 2
Zoo Keeper
Consumer 1
Producer 1 Producer 2

45
Performance Benchmark
1 Broker
1 Producer
1 Consumer

48
LinkedIn Kafka Performance (2012)
● 8 nodes per datacenter
– ~20 GB RAM available for Kafka~20 GB RAM available for Kafka
– 6TB storage, RAID 10, basic SATA drives6TB storage, RAID 10, basic SATA drives
● 10 billion messages/day
● Sustained peak:
– 172,000 messages/second written172,000 messages/second written
– 950,000 messages/second read950,000 messages/second read
● 367 topics
● 40 real-time consumers
● Many ad hoc consumers
● 9.5TB log retained (~ 6 days)
● End-to-end delivery time: A few seconds

What's hot

証明駆動開発のたのしみ@名古屋reject会議

Hiroki Mizuno

Deploying applications on Kubernetes is getting easier every day. From a minimal deployment to distributed service mesh enabled applications with planning and a little bit of YAML resilient cloud-native applications are the norm. In this session, Christopher Bradford and Ty Morton will help answer the following questions: - What about your data behind these apps? - Are you running those in a multi-cluster environment or sending everything back to a common location? - How do you modernize to a distributed peer-to-peer data architecture? - How do you plan for this change? - Are there pitfalls on the road to enlightened data? Join this session to explore the key concepts needed when investigating multi-cluster deployments for data. This includes: - Cluster planning - Network design - Security - Failure handling

Taking Your Database Beyond the Border of a Single Kubernetes Cluster

Christopher Bradford

Fun421 stephens

Tess Ferrandez

Super COMPUTING Journal

Pandey_G

Ray Chien from Fujitsu Consulting, has created a Smallworld plugin capable to "check-out" Smallworld data into Microstation format, where it may be edited and "checked-in" back to Smallworld using FME 2014 desktop transformers. Written for the 4.0 Smallworld Translator, it does not support fme_db_operation and 'true' updates, but can accurately detect changed features and only replace those as needed. While Microstation does not support feature attributes, data integrity is preserved by combining with Microsoft Access Linkage support. In addition, the plugin is written with full Dynamic reader and writer support, making it scalable and adaptive to data model changes.

Smallworld Data Check-Out to Microstation

Safe Software

On heap cache vs off-heap cache

rgrebski

Presented at PEARC20. This talk presents expanding the IceCube’s production HTCondor pool using cost-effective GPU instances in preemptible mode gathered from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and the Google Cloud Platform. Using this setup, we sustained for a whole workday about 15k GPUs, corresponding to around 170 PFLOP32s, integrating over one EFLOP32 hour worth of science output for a price tag of about $60k. In this paper, we provide the reasoning behind Cloud instance selection, a description of the setup and an analysis of the provisioned resources, as well as a short description of the actual science output of the exercise.

Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...

Igor Sfiligoi

Smallworld to GreGG - FME Server Automation

Safe Software

SkyhookDM - Towards an Arrow-Native Storage System

JayjeetChakraborty

Computer Performance Microscopy with SHIM

hiyangxi

The San Diego Supercomputer Center (SDSC) and the Wisconsin IceCube Particle Astrophysics Center (WIPAC) at the University of Wisconsin–Madison successfully completed a computational experiment as part of a multi-institution collaboration that marshalled all globally available for sale GPUs (graphics processing units) across Amazon Web Services, Microsoft Azure, and the Google Cloud Platform. In all, some 51,500 GPU processors were used during the approximately 2-hour experiment conducted on November 16 and funded under a National Science Foundation EAGER grant. The experiment – completed just prior to the opening of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19) in Denver, CO – was coordinated by Frank Würthwein, SDSC Lead for High-Throughput Computing, and Benedikt Riedel, Computing Manager for the IceCube Neutrino Observatory and Global Computing Coordinator at WIPAC. Igor Sfiligoi, SDSC’s lead scientific software developer for high-throughput computing, and David Schultz, a production software manager with IceCube, conducted the actual run. This presentation was given at several booths during SC19 by Frank Würthwein.

Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...

Igor Sfiligoi

Alternative cryptocurrencies

vpnmentor

Pig TPC-H Benchmark and Performance Tuning

Jie Li

NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...

Igor Sfiligoi

Burst data retrieval after 50k GPU Cloud run

Igor Sfiligoi

하이퍼커넥트 데이터 팀이 데이터 증가에 대처해온 기록

Jaehyeuk Oh

Pharo Status Fosdem 2015

Marcus Denker

Data-intensive IceCube Cloud Burst

Igor Sfiligoi

Progress_190315

Hyo jeong Lee

CERN is the home of the Large Hadron Collider (LHC), a 27km circular proton accelerator that generates petabytes of physics data every year. To process all this data, CERN runs an OpenStack Cloud (>300K cores) that helps scientists all around the world to unveil the mysteries of the Universe. The Infrastructure is also used to run all the IT services of the Organization. Delivering these services, with high performance and reliable service levels has been one of the major challenges for the CERN Cloud engineering team. We have been constantly iterating the architecture and deployment model of the Cloud control plane. In this presentation we will describe the different control plane architecture models that we relied over the years. Finally, we will describe all the work done to move the OpenStack Cloud control plane from VMs into a kubernetes cluster. We will report about our experience running this architecture at scale, its advantages and challenges.

CERN OpenStack Cloud Control Plane - From VMs to K8s

Belmiro Moreira

What's hot (20)

証明駆動開発のたのしみ@名古屋reject会議

Taking Your Database Beyond the Border of a Single Kubernetes Cluster

Fun421 stephens

Super COMPUTING Journal

Smallworld Data Check-Out to Microstation

On heap cache vs off-heap cache

Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...

Smallworld to GreGG - FME Server Automation

SkyhookDM - Towards an Arrow-Native Storage System

Computer Performance Microscopy with SHIM

Running a GPU burst for Multi-Messenger Astrophysics with IceCube across all ...

Alternative cryptocurrencies

Pig TPC-H Benchmark and Performance Tuning

NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...

Burst data retrieval after 50k GPU Cloud run

하이퍼커넥트 데이터 팀이 데이터 증가에 대처해온 기록

Pharo Status Fosdem 2015

Data-intensive IceCube Cloud Burst

Progress_190315

CERN OpenStack Cloud Control Plane - From VMs to K8s

Similar to Kafka short

Introduction to Apache Kafka

Shiao-An Yuan

Nous vivons une époque où tout est connecté, de nos ampoules à notre éditeur de texte, les objets et services qui nous entourent devienne de plus en plus intelligents. Pour ce faire ils génèrent des données, elles sont nécessaires au bon fonctionnement du service ou de l'objet, mais elles sont également utiles pour faire évoluer les produits. Ces données peuvent rapidement représenter de gros volumes, plusieurs dizaines voir centaines de gigaoctets. Une question se pose alors, comment traiter de tels volumes ? Comment en tirer du sens et de la valeur à cette échelle ? Avec OVHcloud Data Processing, une solution basée sur le framework Apache Spark, nous répondons à ce besoin. Venez découvrir comment vous aussi, en quelques cliques, pouvez exécuter votre code sur une infrastructure taillée pour vos besoins. Au travers de différents exemples, comme une analyse du traffic des taxis New-Yorkais, nous verrons comment Data Processing a été pensé, comment il fonctionne et comment il peut être utilisé pour valoriser vos données.

OVHcloud Tech Talks S01E09 - OVHcloud Data Processing : Le nouveau service po...

OVHcloud

Apache Pulsar at Yahoo! Japan

StreamNative

More Efficient Object Replication in OpenStack Summit Juno

Kota Tsuyuzaki

Google is making the power of its datacenter, network, and technology innovations available to the world through its Cloud services. This presentation will provide an overview of the Google Cloud Platform and a deeper dive on Google Compute Engine. Google recently made an open source contribution to SaltStack and now you can now use Salt Cloud to manage your Compute Engine resources (IaaS virtual machine services). Come find out more about Google's Cloud Platform and how you can leverage Google scale with SaltStack.

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

SaltStack

Netflix Keystone—Cloud scale event processing pipeline

Monal Daxini

Originally presented at Strata EU 2017: https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/57631 Cloud providers currently offer convenient on-demand managed big data clusters (PaaS) with a pay-as-you-go model. In PaaS, analytical engines such as Spark and Hive come ready to use, with a general-purpose configuration and upgrade management. Over the last year, the Spark framework and APIs have been evolving very rapidly, with major improvements on performance and the release of v2, making it challenging to keep up-to-date production services both on-premises and in the cloud for compatibility and stability. Nicolas Poggi evaluates the out-of-the-box support for Spark and compares the offerings, reliability, scalability, and price-performance from major PaaS providers, including Azure HDinsight, Amazon Web Services EMR, Google Dataproc, and Rackspace Cloud Big Data, with an on-premises commodity cluster as baseline. Nicolas uses BigBench, the brand new standard (TPCx-BB) for big data systems, with both Spark and Hive implementations for benchmarking the systems. BigBench combines SQL queries, MapReduce, user code (UDF), and machine learning, which makes it ideal to stress Spark libraries (SparkSQL, DataFrames, MLlib, etc.). The work is framed within the ALOJA research project, which features an open source benchmarking and analysis platform that has been recently extended to support SQL-on-Hadoop engines and BigBench. The ALOJA project aims to lower the total cost of ownership (TCO) of big data deployments and study their performance characteristics for optimization. Nicolas highlights how to easily repeat the benchmarks through ALOJA and benefit from BigBench to optimize your Spark cluster for advanced users. The work is a continuation of a paper to be published at the IEEE Big Data 16 conference. (A preprint copy can be obtained here.)

The state of Spark in the cloud

Nicolas Poggi

Presenter: Tim Berglund, Senior Director of Developer Experience, Confluent It has become a truism in the past decade that building systems at scale, using non-relational databases, requires giving up on the transactional guarantees afforded by the relational databases of yore. ACID transactional semantics are fine, but we all know you can’t have them all in a distributed system. Or can we? In this talk, I will argue that by designing our systems around a distributed log like Apache Kafka®, we can in fact achieve ACID semantics at scale. We can ensure that distributed write operations can be applied atomically, consistently, in isolation between services, and of course with durability. What seems to be a counterintuitive conclusion ends up being straightforwardly achievable using existing technologies, as an elusive set of properties becomes relatively easy to achieve with the right architectural paradigm underlying the application.

Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)

confluent

To understand an application’s performance, first you have to know what to measure. That’s the easy part. How do you take those measurements? Store them? Analyze them? Get them to the people who need them? Well, that’s where things get complicated, especially in the high-traffic distributed systems of the modern web! Like careful scientists, we must observe our subjects without altering them, and we must report our findings quickly so that we have the data necessary to make smart choices about the health and growth of the system. Let’s explore the lessons learned by engineers at one of the world’s top web companies in their quest to find meaning at 5 MB/s. We’ll discuss the tools and techniques that enable the collection, indexing, and analysis of billions or more datapoints each hour, and learn how these same approaches can empower your applications and your business, no matter the scale.

Drinking from the Firehose - Real-time Metrics

Samantha Quiñones

Moving Toward Deep Learning Algorithms on HPCC Systems

HPCC Systems

Clojure News Feed Performance Testing

Dynamical Software, Inc.

At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker. This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters. In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner. You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant. You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC. When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

Natan Silnitsky

Docker is going to change the way services are deployed by encapsulating them, thus provide a robust and complete continous development/deployment workflow. But how is docker impacting the compute part of HPC? Containerization uses kernel functions (Namespaces/CGroups) to encapsulate the processes, while allowing a fain grained customization of the compute stack; sitting on top of a stripped down bare-metal OS that provides basic services. This sessions aims to explore if docker already is the 'virtualization' technique the HPC community is waiting for. Could a distributed MPI job across multiple containers placed on different physical nodes beat a natively started job?

2014 11-05 hpcac-kniep_christian_dockermpi

QNIB Solutions

Microservices interaction at scale using Apache Kafka

Ivan Ursul

Presenter: Kenn Knowles, Software Engineer, Google & Apache Beam (incubating) PPMC member Apache Beam (incubating) is a programming model and library for unified batch & streaming big data processing. This talk will cover the Beam programming model broadly, including its origin story and vision for the future. We will dig into how Beam separates concerns for authors of streaming data processing pipelines, isolating what you want to compute from where your data is distributed in time and when you want to produce output. Time permitting, we might dive deeper into what goes into building a Beam runner, for example atop Apache Apex.

Apache Beam (incubating)

Apache Apex

Log everything!

ICANS GmbH

Datafying Bitcoins

Tariq Ahmad

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1IPjAIS. Rick Hudson discusses the motivation, performance, and technical challenges of Go's low latency concurrent GC and why the approach fits Go well. Filmed at qconsf.com. Rick Hudson is a member of Google’s Go team. Rick has published papers on language runtimes, memory management, concurrency, synchronization, memory models, and transactional memory.

Go GC: Prioritizing Low Latency and Simplicity

C4Media

Streaming architecture patterns

hadooparchbook

We as an industry are collecting more data every year. IoT, web, and mobile applications send torrents of bits to our data centers that have to be processed and stored, even as users expect an always-on experience—leaving little room for error. Patrick McFadin explores how successful companies do this every day using the powerful Team Apache: Apache Kafka, Spark, and Cassandra. Patrick walks you through organizing a stream of data into an efficient queue using Apache Kafka, processing the data in flight using Apache Spark Streaming, storing the data in a highly scaling and fault-tolerant database using Apache Cassandra, and transforming and finding insights in volumes of stored data using Apache Spark. Topics include: - Understanding the right use case - Considerations when deploying Apache Kafka - Processing streams with Apache Spark Streaming - A deep dive into how Apache Cassandra stores data - Integration between Cassandra and Spark - Data models for time series - Postprocessing without ETL using Apache Spark on Cassandra

An Introduction to time series with Team Apache

Patrick McFadin

Similar to Kafka short (20)

Introduction to Apache Kafka

OVHcloud Tech Talks S01E09 - OVHcloud Data Processing : Le nouveau service po...

Apache Pulsar at Yahoo! Japan

More Efficient Object Replication in OpenStack Summit Juno

SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...

Netflix Keystone—Cloud scale event processing pipeline

The state of Spark in the cloud

Dissolving the Problem (Making an ACID-Compliant Database Out of Apache Kafka®)

Drinking from the Firehose - Real-time Metrics

Moving Toward Deep Learning Algorithms on HPCC Systems

Clojure News Feed Performance Testing

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

2014 11-05 hpcac-kniep_christian_dockermpi

Microservices interaction at scale using Apache Kafka

Apache Beam (incubating)

Log everything!

Datafying Bitcoins

Go GC: Prioritizing Low Latency and Simplicity

Streaming architecture patterns

An Introduction to time series with Team Apache

More from Tikal Knowledge

Clojure - LISP on the JVM

Tikal Knowledge

Clojure presentation

Tikal Knowledge

At Tabtale we are setting up an entire server side for the all the publishing services. These services include dynamic game configurations, error collection, analytics, social services and more.Tabtale is among the world’s top app publishers with millions of downloads so we are putting a great deal of effort in creating an extremely highly scalable and fault tolerant architecture. In this talk I will go over the architecture decisions taken to support the scalability and diversity that is required from the server side services while keeping the management of this infrastructure sane. ~30min By Assaf Gannon

Tabtale story: Building a publishing and monitoring mobile games architecture...

Tikal Knowledge

Docking your services_with_docker

Tikal Knowledge

Who moved my_box

Tikal Knowledge

Writing apps should be fun. This is a statement I try to keep in mind when approaching any application development. When I first realized that a full stack of Javascript can be used to create an application I was very happy, since for me coding in Javascript is a lot of fun. I was soon quite eager to neglect the traditional Java server-side + Javascript client, giving up the Java server side "bliss" and the constant context switching, in favor of a full stack of Javascript whenever I could. For the past 2+ years I have been developing a variety of applications using a pure JS stack: Games, Forex trading middleware, CRM, CMS, sophisticated proxies, and various utilities. I must admit my productivity has increased dramatically and the time-to-market of each and every project was a fraction of what it would have been using traditional Java based server side infra-structure. On October 29th 2013 I gave a talk at the "Javascript.everywhere" meetup event, in which I presented the development route of Youtube Remote Control Application with a "slave" player using Javascript stack all over (NodeJS, Socket.IO, ExpressJS, AngularJS, MongoDB). I'm pleased to share the presentation - See more at: http://www.tikalk.com/js/writing-fullstack-application-javascript#sthash.IjFNbQJ2.dpuf

Writing a Fullstack Application with Javascript - Remote media player

Tikal Knowledge

.Net OSS Ci & CD with Jenkins - JUC ISRAEL 2013

Tikal Knowledge

Tikal's Backbone_js introduction workshop

Tikal Knowledge

TCE Automation

Tikal Knowledge

Tce automation-d4

Tikal Knowledge

Cloud computing - an insight into "how does it really work ?"

Tikal Knowledge

Introduction to Cloud Computing

Tikal Knowledge

Tikal Fuse Day Access Layer Implementation (C#) Based On Mongo Db

Tikal Knowledge

Ship early ship often with Django

Tikal Knowledge

Google App Engine

Tikal Knowledge

AWS Case Study

Tikal Knowledge

Introduction To Cloud Computing

Tikal Knowledge

Building Components In Flex3

Tikal Knowledge

Osgi Democamp

Tikal Knowledge

JBUG 11 - Django-The Web Framework For Perfectionists With Deadlines

Tikal Knowledge

More from Tikal Knowledge (20)

Clojure - LISP on the JVM

Clojure presentation

Tabtale story: Building a publishing and monitoring mobile games architecture...

Docking your services_with_docker

Who moved my_box

Writing a Fullstack Application with Javascript - Remote media player

.Net OSS Ci & CD with Jenkins - JUC ISRAEL 2013

Tikal's Backbone_js introduction workshop

TCE Automation

Tce automation-d4

Cloud computing - an insight into "how does it really work ?"

Introduction to Cloud Computing

Tikal Fuse Day Access Layer Implementation (C#) Based On Mongo Db

Ship early ship often with Django

Google App Engine

AWS Case Study

Introduction To Cloud Computing

Building Components In Flex3

Osgi Democamp

JBUG 11 - Django-The Web Framework For Perfectionists With Deadlines

Recently uploaded

%in Harare+277-882-255-28 abortion pills for sale in Harare

masabamasaba

ADR, or Architecture Decision Record, is a valuable tool in software development for several reasons. It provides a centralized location for documenting and tracking architectural decisions, aiding both current and future team members. ADRs enhance communication among team members by documenting the rationale behind architectural decisions, especially beneficial during onboarding of new team members or when revisiting decisions. They serve as a knowledge base, enabling teams to learn from past decisions and refine their decision-making process. Additionally, ADRs contribute to transparency by helping stakeholders understand the reasons behind specific architectural choices. As with any other tool or process, introducing them into an organization can face several obstacles, and overcoming these challenges is crucial for successful implementation. In this talk I go through some common problems and our way of solving them.

Architecture decision records - How not to get lost in the past

Papp Krisztián

Artyushina_Guest lecture_YorkU CS May 2024.pptx

AnnaArtyushina1

Conference: Engage2024 in Antwerp Type: Workshop Speakers: Florian Vogler, Henning Kunz, Christoph Adler Title: Navigating the Future with The Hitchhiker's Guide to Notes and Domino 14 Abstract: Embark on an exhilarating journey with industry trailblazers Florian Vogler, Henning Kunz, and Christoph Adler in this not-to-be-missed workshop at the forefront of the tech universe. Get ready for a thrilling kick-off as we navigate the current state of the HCL universe, setting the stage for an exploration of the groundbreaking Notes and Domino 14. Discover the latest enhancements and revolutionary features that will redefine your experience. In this interactive session, unlock a treasure trove of tips and tricks to elevate your utilization of version 14, both with and without the game-changing panagenda MarvelClient. Brace yourself for also diving into Nomad, Nomad Web, and VoltMX, expanding your horizons in the expansive HCL landscape. Be a part of this exclusive opportunity to stay ahead in the ever-evolving world of HCL technologies. Your journey to mastering Notes and Domino 14 begins here. And remember, in the spirit of intergalactic exploration, don't forget to bring your towel!

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

panagenda

Modeling languages are generally applied for developing systems and software – both internally, with domain-specific languages, and with standardized languages targeting a general purpose and a wide audience. Way too often these languages are weakly created and defined leading to poor quality: Language definitions tend to contain errors and inconsistencies; notations do not recognize the communication and problem-solving needs of humans; standardization organizations push exchange formats that do not fully work and offer certificates that do not measure mastery of the language. We describe common problems in language development and point them out with examples from known cases. To overcome these problems, we suggest several solutions to improve language development, including using modeling languages specifically designed to define modeling languages, continuous testing and prototyping, and keeping language users in the loop.

What Goes Wrong with Language Definitions and How to Improve the Situation

Juha-Pekka Tolvanen

Craft an AI & Machine Learning Pitch with our Editable Professional PowerPoint Template. Ignite your AI & Machine Learning pitch with our cutting-edge PowerPoint template tailored for the industry. Perfect for AI conferences, investor presentations, sales pitches to tech-focused companies, training sessions, and educational programs. - 20+ editable slides: Get a variety of options to choose from for your presentation. - Time-saving solution: Download, replace text/images with a few clicks. - User-friendly customization: Easy to use and personalize. - Modern and attractive design: Captivating visuals, sleek layout. - Tailored to your requirements: Fully alterable for customization. - Well-organized slides: Complete control over content. - Thematic specificity: Reflects healthcare industry with relevant graphics. - Showcase your business idea: Communicate value proposition effectively.

AI & Machine Learning Presentation Template

Presentation.STUDIO

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

WSO2

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

masabamasaba

WSO2CON 2024 - Does Open Source Still Matter?

WSO2

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

Bert Jan Schrijver

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

%in Soweto+277-882-255-28 abortion pills for sale in soweto

masabamasaba

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

masabamasaba

MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...

Jittipong Loespradit

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

WSO2

Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic near me by a powerful astrologer and spell caster in Atlanta. Bring back your lover with Asaf's voodoo-love witchcraft. Psychic Reading | Astrologer | Spell Caster | Obsession Spells | Black Magic | Witchcraft | Protection spell | wiccan spells. Black magic expert and voodoo love spells that work overnight to retrieve that love | lost love spell caster with powerful love psychic reading. Best astrologer & psychic in Sandy Springs, GA to renew your relationship & make your relationship stronger. love spells to bring back the feelings of love for ex-lovers. Increase the intimacy, affection & love between you and your lover using voodoo relationship obsession spells in USA. Money spells, easy love spells with just words, think of me spell, powerful love spell, spells of love, spells that work, love potion to attract a man, easy love spells with just words, pink candle prayer, white magic spells, call me spell, manifestation spell, gay love spells, Commitment spells, business spells and, how to bring back lost love in a relationship, Witchcraft love spells that work immediately to increase love & intimacy in your relationship. Attraction love spells to attract someone, stop a divorce, prevent a breakup & get your ex back. When using love binding spells, there are several things you should know, particularly how to protect yourself from negative energies and how to use the powers of incantations for the good of all people involved. When the focus is love, the results are truly magical. It’s important to remember love is not manipulative, it is not forceful, and it does not bend another to its will. Love is free-flowing, accepting, kind, and generous. For your love spell to work as intended, you must have good intentions in your heart. Below, we share the top five love spells you can use today to shift the energies in your life and create a future filled with love and fulfilment. Obsession spells to Get Your Ex-Lover Back. All women want one thing the most in life to be love and love in return – not much to ask. A lady wants a good man who loves you unconditionally and loyally. You want a man who is honest, hardworking, and loyal – not a complainer or a weakling or a self-centred man. You are a strong, independent, sensual, caring, loving woman. And you don’t think it’s asking too much to want to be with a loving, intelligent man with a good sense of humour and daring, an upright and confident man – who knows who he is. No one wants a chauvinistic and macho man, but you do want someone who will be willing to protect and care for you. Who loves you for you and not some kind of imaginary. You have no doubt that when the right guy appears on your doorstep, you’ll never let him go. But sometimes a man doesn’t realize he has that good woman.

Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...

chiefasafspells

This presentation covers the following topics: What is logging? The purpose of logging: Debugging The purpose of logging: Security The purpose of logging: Stats & analytics Traditional logging Traditional logging: Advantages Traditional logging: Disadvantages The solution: Large-scale logging Large-scale logging: Core principles Large-scale logging: Solution types Large-scale logging: Cloud vs on-prem Large-scale logging: Operational complexity Large-scale logging: Security Large-scale logging: Costs Large-scale logging: On-prem comparison - Elasticsearch - Grafana Loki - VictoriaLogs On-prem comparison: Setup and operation On-prem comparison: Costs On-prem comparison: Full-text search support On-prem comparison: How to efficiently query 100TB of logs? On-prem comparison: Integration with CLI tools VictoriaLogs for large-scale logging VictoriaLogs demo instance - Ingestion rate: 3600 messages / minute - The number of log messages: 1.1 billion - Uncompressed log messages’ size: 1.5TB - Compressed log messages’ size: 23GB - Compression ratio: 47x - Memory usage: 150MB VictoriaLogs CLI integration demo - Which errors have occurred in all the apps during the last hour? - How many errors have occurred during the last hour? - Which apps generated the most of errors during the last hour? - The number of per-minute errors for the last 10 minutes - Status codes for the last hour - Non-200 status codes for the last week - Top client IPs for the last 4 weeks with 404 and 500 response status codes - Per-month stats for the given IP across all the logs Large-scale logging solution MUST provide excellent CLI integration VictoriaLogs: (temporary) drawbacks VictoriaLogs: Recap - Easy to setup and operate - The lowest RAM usage and disk space usage (up to 30x less than Elasticsearch and Grafana Loki) - Fast full-text search - Excellent integration with traditional command-line tools for log analysis - Accepts logs from popular log shippers (Filebeat, Fluentbit, Logstash, Vector, Promtail, Grafana Agent) - Open source and free to use!

Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024

VictoriaMetrics

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

masabamasaba

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

Recently uploaded (20)

%in Harare+277-882-255-28 abortion pills for sale in Harare

Architecture decision records - How not to get lost in the past

Artyushina_Guest lecture_YorkU CS May 2024.pptx

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

What Goes Wrong with Language Definitions and How to Improve the Situation

AI & Machine Learning Presentation Template

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

WSO2CON 2024 - Does Open Source Still Matter?

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

VTU technical seminar 8Th Sem on Scikit-learn

%in Soweto+277-882-255-28 abortion pills for sale in soweto

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...

Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

Kafka short

1. 1 Kafka forKafka for BigDataBigData ProcessingProcessing Yanai Franchi , TikalYanai Franchi , Tikal

2. 2 Find “Hot” Places

3. 3

4. 4 gogobot checkin Heat Map Service Lets' Develop “Gogobot Checkins Heat-Map”

5. 5 Key Notes ● Collector Service - Collects checkins as text addresses – We need to use GeoLocation ServiceWe need to use GeoLocation Service ● Upon elapsed interval, the last locations list will be displayed as Heat-Map in GUI. ● Web Scale service – 10Ks checkins/seconds all over the world (imaginary, but lets do it for the exercise).

6. 6 Heat-Map Context Text-Address Checkins Heat-Map Service Gogobot System Gogobot Micro Service Gogobot Micro Service Gogobot Micro Service Geo Location Service Get-GeoCode(Address) Heat-Map Last Interval Locations

7. 7 Tons of Addresses Arriving Every Second

8. 8 First Reaction...

9. 9 Checkin HTTP Reactor Checkins Topic Storm Heat-Map Topology Hotzones Topic Web App Push via WebSocket Publish Checkins HDFS Checkin HTTP Firehose

10. 10

11. 11 They all are Good But not for all use-cases

12. 12 Kafka A little introduction

13. 13

14. 14 Why ?

15. 15 LinkedIn Original Architecture

16. 16

17. 17 What LinkedIn Want...

18. 18 Looks Familiar : Use Messaging (i.e. JMS, RabbitMQ)

19. 19

20. 20

21. 21

22. 22

23. 23 It Didn't Scale...

24. 24 Paradigm Change : Do NOT track message consumption

25. 25

26. 26

27. 27

28. 28 Stateless Broker & Doesn't Fear the File System

29. 29 Topics ● Logical collections of partitions (the physical fi les). ● A broker contains some of the partitions for a topic

30. 30 A partition is Consumed by Exactly One Group's Consumer

31. 31 Distributed & Fault-Tolerant

32. 32 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

33. 33 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

34. 34 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

35. 35 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

36. 36 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

37. 37 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

38. 38 Broker 1 Broker 4Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

39. 39 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

40. 40 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

41. 41 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Consumer 2 Producer 1 Producer 2

42. 42 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Producer 1 Producer 2

43. 43 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Producer 1 Producer 2

44. 44 Broker 1 Broker 3Broker 2 Zoo Keeper Consumer 1 Producer 1 Producer 2

45. 45 Performance Benchmark 1 Broker 1 Producer 1 Consumer

46. 46

47. 47

48. 48 LinkedIn Kafka Performance (2012) ● 8 nodes per datacenter – ~20 GB RAM available for Kafka~20 GB RAM available for Kafka – 6TB storage, RAID 10, basic SATA drives6TB storage, RAID 10, basic SATA drives ● 10 billion messages/day ● Sustained peak: – 172,000 messages/second written172,000 messages/second written – 950,000 messages/second read950,000 messages/second read ● 367 topics ● 40 real-time consumers ● Many ad hoc consumers ● 9.5TB log retained (~ 6 days) ● End-to-end delivery time: A few seconds

49. 49 Thanks

Kafka short

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kafka short

Similar to Kafka short (20)

More from Tikal Knowledge

More from Tikal Knowledge (20)

Recently uploaded

Recently uploaded (20)

Kafka short