Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

•

1 like•1,385 views

Hunter Kelly presents an approach for using Apache Kafka Streams to perform real-time domain ranking based on a modified HITS algorithm. The system discovers relevant fashion domains from web links. It represents domains as a graph and runs HITS iterations to identify hub and authority domains. By using Kafka Streams, the rankings can be updated continuously in real-time from a stream of new links. The system decomposes the HITS algorithm into separate Kafka streams processes for link extraction, domain reduction, and scoring domains.

Engineering

Hunter Kelly
@retnuh
Real-Time Domain Rankings
with Kafka Streams

What are we doing?
● Discovering relevant domains in the Fashion Web
● Use modified HITS - a.k.a. Hubs & Authorities
Why use Kafka Streams?
● Why not Spark, Hadoop, Whatevs?
● Advantages of Kafka Streams
Show me the nitty gritty!
● Okay!

Curated list of Fashion related sites
● Initial lists from domain experts
● Support multiple languages
● What are the next “best” domains?

cosmopolitan
stylecaster
instyle
racked
harpersbazaar
elle
vogue
hypebeast

http://www.cs.cornell.edu/home/kleinber/auth.pdf
http://en.wikipedia.org/wiki/HITS_algorithm
* Hyperlink-Induced Topic Search, but no-one seems to call it that
Need Data-driven method
● Kleinberg’s HITS* algorithm
● Allows us to discover Hubs and Authorities
● Flattened to domain level

HITS in a nutshell
Adjacency Matrix
Image courtesy of
http://faculty.ycp.edu/~dbabcock/PastCourses/cs360/lectures/lecture15.html

HITS in a nutshell
● Why not just counts?
● If you point to good sites, you’re a better
Hub
● If good sites point to you, you’re a better
Authority
● Each iteration, the weights change, until
they reach convergence

Common Questions
● What about non-fashion domains?
● Why not PageRank?

Please tell me you aren’t using Kafka
to do Matrix multiplication!

Please tell me you aren’t using Kafka
to do Matrix multiplication!
No, I am not crazy enough* to use Kafka to do
matrix multiplication!

Please tell me you aren’t using Kafka
to do Matrix multiplication!
No, I am not crazy enough* to use Kafka to do
matrix multiplication!
*Although, I probably did spend too much
time thinking about it on the bus!

Why not use Map-Reduce?
Basically what it was invented for,
right?

You could use Spark…
● High infrastructure overhead if not using it
for anything else
● Bad initial experience w/ Spark Streaming
snapshotting and recovery
● Already using Kafka!

Why use Kafka Streams?
● Has primitives necessary for Map-Reduce
○ Map step groupBy groupByKey
○ Reduce step reduce aggregate
● Focus is on your data
not distributed computing machinery
● Streaming allows us to have
(near)
real-time, up-to-date data

Kafka Streams 101
● No explicit consumers/producers -
plumbing handled for you
● Topics still a fundamental communication
piece
● Think of individual datum flowing through
KStreams & KTables

Kafka Streams 101 - KSTreams
● Focus is on specific functional
transformations - map, filter, flatMap
● Also supports various flavours of joins with
other KStreams
● Usually created from one or more topics or
a transformation on another KStream

Kafka Streams 101 - KTable
● Still offers functional transforms, but on a
primary-keyed table
● Offers persistent storage
● Created from aggregations on a KStream
or transforms on other KTable

● The bridge between KStream and KTable
● Created by doing a groupBy or
groupByKey on a KStream
● Create KTables by doing reduce,
aggregate, count
Kafka Streams 101 - KGroupedStream

Output Topic
(log compacted)
KTable ops:
groupByKey,
reduce,
toStream
KStream
Input Topic
KStream ops:
flatMapDomain Link Extractor
Domain Reducer
HITS Calculator & API
Data Flow

Current
Domain Link Extractor
Input Topic
Fetch Content
External Links
Resolve Links
Extract Domains
KStream
KStream ops:
flatMap

Extracted Link
Resolver
Domain Link Extractor
Input Topic
Fetch Content
External Links
Resolve Links
Extract Domains
Output
Topic
KStream ops:
flatMap
KStream ops:
flatMap
KStream ops:
flatMap

Domain Link Extractor
Input Topic
Fetch Content
External Links
Resolve Links
Extract Domains
KStream
Other Function(s)
KStream
KStream ops:
flatMap
KStream Topic Whatev’s
With Fork

TL;DR
● Kafka Streams can help solve problems in
your application domain
● Focus on your data!
● Naturally decompose the problem into
flexible microservices

Hunter Kelly
@retnuh
https://github.com/retnuh

This document discusses WePay's use of Kafka and Debezium for real-time data warehousing. Debezium is used to stream database changes from MySQL to Kafka. The Kafka Connect BigQuery connector then loads data from Kafka into BigQuery. This provides lower latency compared to WePay's previous ETL system. Key benefits include handling schema changes, retries on errors, and view deduplication in BigQuery. Future work includes integrating more of WePay's monolithic database and addressing issues like metrics and compatibility checking as the system scales.

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...

confluent

Interactive Queries in Apache Kafka's Streams API allows users to query the local state of a Kafka Streams application without accessing an external database. It treats the Kafka Streams application as an embedded, lightweight database. The local state is fault-tolerant and can be sharded across tasks to scale horizontally. Users can discover other application instances and their state to perform queries on remote state if needed. Interactive Queries simplifies stateful stream processing by reducing moving parts compared to using an external database.

Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day

Ankur Bansal

Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020

confluent

Apache Kafka sits at the center of a technology ecosystem that can be a bit overwhelming to someone just getting started. Fortunately, Apache Kafka is also at the heart of an amazing community that is able and eager to help! So, if you are new, or relatively new, to Apache Kafka, welcome! I’d like to introduce you to the Kafka ecosystem, and present you with a plan for how to learn and be productive with it. I’d also like to introduce you to one of the most helpful and welcoming software communities I’ve ever encountered. I’ll take you through the basics of Kafka—the brokers, the partitions, the topics—and then on and up into the different APIs and tools that are available to work with it. Consider it a Kafka 101, if you will. We’ll stay at a high level, but we’ll cover a lot of ground, with an emphasis on where and how you can dig in deeper. I am still learning myself, so I will share with you what and who have helped me in my journey, and then I’ll invite you to continue that journey with me. It’s going to be a great adventure!

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

HostedbyConfluent

Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security. A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka. In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.

Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example

confluent

This document introduces Kafka Streams and provides an example of using it to process streaming data from Apache Kafka. It summarizes some key limitations of using Apache Spark for streaming use cases with Kafka before demonstrating how to build a simple text processing pipeline with Kafka Streams. The document also discusses parallelism, state stores, aggregations, joins and deployment considerations when using Kafka Streams. It provides an example of how Kafka Streams was used to aggregate metrics from multiple instances of an application into a single stream.

ksqlDB: A Stream-Relational Database System

confluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...

HostedbyConfluent

Having started with classic monolith applications in the late 90s and adopting a new microservice architecture in 2015, our organization needed a convenient, reliable, and low-cost way to push changes back and forth between them. One that preferably utilized technology already on hand and could exchange information between multiple data stores. In this session we will explore how Kafka Connect and its various connectors satisfied this need. We will review the two disparate tech stacks we needed to integrate, and the strategies and connectors we used to exchange information. Finally, we will cover some enhancements we made to our own processes including integrating Kafka Connect and its connectors into our CI/CD pipeline and writing tools to monitor connectors in our production environment.

Follow the (Kafka) Streams

confluent

Mario Molina, Datio, Software Engineer Kafka Streams is an open source JVM library for building event streaming applications on top of Apache Kafka. Its goal is to allow programmers to create efficient, real-time, streaming applications and perform analysis and operations on the incoming data. In this presentation we’ll cover the main features of Kafka Streams and do a live demo! This demo will be partially on Confluent Cloud, if you haven’t already signed up, you can try Confluent Cloud for free. Get $200 every month for your first three months ($600 free usage in total) get more information and claim it here: https://cnfl.io/cloud-meetup-free https://www.meetup.com/Mexico-Kafka/events/271972045/

A Tour of Apache Kafka

confluent

Performance Tuning RocksDB for Kafka Streams’ State Stores

confluent

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...

HostedbyConfluent

When choosing an event streaming platform, Kafka shouldn’t be the only technology you look at. There are a plethora of others in the messaging space today, including open source and proprietary software as well as a range of cloud services. So how do you know you are choosing the right one? A great way to deepen our understanding of event streaming and Kafka is exploring the trade-offs in distributed system design and learning about the choices made by the Kafka project. We’ll look at how Kafka stacks up against other technologies in the space, including traditional messaging systems like Apache ActiveMQ and RabbitMQ as well as more contemporary ones, such as BookKeeper derivatives like Apache Pulsar or Pravega. This talk focuses on the technical details such as difference in messaging models, how data is stored locally as well as across machines in a cluster, when (not) to add tiers to your system, and more. By the end of the talk, you should have a good high-level understanding of how these systems compare and which you should choose for different types of use cases.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

This document summarizes Activision Data's transition from a batch data pipeline to a real-time streaming data pipeline using Apache Kafka and Kafka Streams. Some key points: - The new pipeline ingests, processes, and stores game telemetry data from over 200k messages per second and over 5PB of data across 9 years of games. - Kafka Streams is used to transform the raw streaming data through multiple microservices with low 10-second end-to-end latency, compared to 6-24 hours previously. - Kafka Connect integrates the streaming data with data stores like AWS S3, Cassandra, and Elasticsearch. - The new pipeline provides real-time and historical access to structured

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...

confluent

RocksDB is the default state store for Kafka Streams. In this talk, we will discuss how to improve single node performance of the state store by tuning RocksDB and how to efficiently identify issues in the setup. We start with a short description of the RocksDB architecture. We discuss how Kafka Streams restores the state stores from Kafka by leveraging RocksDB features for bulk loading of data. We give examples of hand-tuning the RocksDB state stores based on Kafka Streams metrics and RocksDB’s metrics. At the end, we dive into a few RocksDB command line utilities that allow you to debug your setup and dump data from a state store. We illustrate the usage of the utilities with a few real-life use cases. The key takeaway from the session is the ability to understand the internal details of the default state store in Kafka Streams so that engineers can fine-tune their performance for different varieties of workloads and operate the state stores in a more robust manner.

Tips & Tricks for Apache Kafka®

confluent

Kat Grigg, Confluent, Senior Customer Success Architect + Jen Snipes, Confluent, Senior Customer Success Architect This presentation will cover tips and best practices for Apache Kafka. In this talk, we will be covering the basic internals of Kafka and how these components integrate together including brokers, topics, partitions, consumers and producers, replication, and Zookeeper. We will be talking about the major categories of operations you need to be setting up and monitoring including configuration, deployment, maintenance, monitoring and then debugging. https://www.meetup.com/KafkaBayArea/events/270915296/

Exactly-once Data Processing with Kafka Streams - July 27, 2017

confluent

This document discusses exactly-once processing in stream processing systems. It begins by defining exactly-once processing and describing some of the challenges in achieving it. It then outlines three options for achieving exactly-once processing with Kafka: at-least-once processing with deduplication, using Kafka's idempotent producer and transactions, and using Kafka Streams. The document focuses on Kafka Streams, describing how it provides exactly-once guarantees through transactional processing of data in batches across the processing topology.

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

confluent

This document discusses Kafka deployment strategies at Goldman Sachs. It describes how Kafka is deployed across multiple datacenters for high availability. It also discusses various failure scenarios like host, network, or entire datacenter failures and how the deployment is designed to minimize impact. The document also summarizes the monitoring, alerting, and management tooling built around Kafka clusters to provide health checks, metrics collection, and topic management capabilities.

Everything you ever needed to know about Kafka on Kubernetes but were afraid ...

HostedbyConfluent

Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.

Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...

confluent

The last 5 years, Kafka and Flink have become mature technologies that have allowed us to embrace the streaming paradigm. You can bet on them to build reliable and efficient applications. They are active projects backed by companies using them in production. They have a good community contributing, and sharing experience and knowledge. Kafka and Flink are solid choices if you want to build a data platform that your data scientists or developers can use to collect, process, and distribute data. You can put together Kafka Connect, Kafka, Schema Registry, and Flink. First, you will take care of their deployment. Then, for each case, you will setup each part, and of course develop the Flink job so it can integrate easily with the rest. Looks like a challenging but exciting project, isn't it? In this session, you will learn how you can build such data platform, what are the nitty-gritty of each part, how you can plug them together, in particular how to plug Flink in the Kafka ecosystem, what are the common pitfalls to avoid, and what it requires to be deployed on kubernetes. Even if you are not familiar with all the technologies, there will be enough introduction so you can follow. Come and learn how we can actually cross the streams!

Real-world Streaming Architectures

confluent

This document discusses four streaming data architectures used by companies like Yelp, LINE, and Altspace VR. It begins by providing context on streaming data and the evolution from batch processing with Hadoop/MapReduce to stream processing with Apache Kafka and frameworks like Storm. The main case studies then describe how each company uses streaming to build real-time data pipelines and services. Key aspects discussed include using Kafka for messaging between services, Kafka Streams for stream processing, and storing streaming data in databases. The document concludes by promoting Apache Kafka as a platform for building scalable streaming applications.

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!

confluent

This document summarizes Mic Hussey's presentation on building streaming data pipelines using Apache Kafka and KSQL. The presentation introduced Kafka and KSQL, demonstrated how to integrate PostgreSQL with Kafka using Kafka Connect, and showed examples of using KSQL to perform streaming ETL operations like joining, filtering, and aggregating event streams in real-time. It also provided several useful links for learning more about change data capture from PostgreSQL to Kafka and building streaming data pipelines with Kafka and KSQL.

Deploying Confluent Platform for Production

confluent

Deploying Kafka Streams Applications with Docker and Kubernetes

confluent

(Gwen Shapira + Matthias J. Sax, Confluent) Kafka Summit SF 2018 Kafka Streams, Apache Kafka’s stream processing library, allows developers to build sophisticated stateful stream processing applications which you can deploy in an environment of your choice. Kafka Streams is not only scalable, but fully elastic allowing for dynamic scale-in and scale-out as the library handles state migration transparently in the background. By running Kafka Streams applications on Kubernetes, you will be able to use Kubernetes powerful control plane to standardize and simplify the application management—from deployment to dynamic scaling. In this technical deep dive, we’ll explain the internals of dynamic scaling and state migration in Kafka Streams. We’ll then show, with a live demo, how a Kafka Streams application can run in a Docker container on Kubernetes and the dynamic scaling of an application running in Kubernetes.

Inside Kafka Streams—Monitoring Comcast’s Outside Plant

confluent

(Mike Graham + Dan Carroll, Comcast) Kafka Summit SF 2018 Comcast manages over 2 million miles of fiber and coax, and over 40 million in home devices. This “outside plant” is subject to adverse conditions from severe weather to power grid outages to construction-related disruptions. Maintaining the health of this large and important infrastructure requires a distributed, scalable, reliable and fast information system capable of real-time processing and rapid analysis and response. Using Apache Kafka and the Kafka Streams Processor API, Comcast built an innovative new system for monitoring, problem analysis, metrics reporting and action response for the outside plant. In this talk, you’ll learn how topic partitions, state stores, key mapping, source and sink topics and processors from the Kafka Streams Processor API work together to build a powerful dynamic system. We will dive into the details about the inner workings of the state store—how it is backed by a Kafka “changelog” topic, how it is scaled horizontally by partition and how the instances are rebuilt on startup or on processor failure. We will discuss how these state stores essentially become like materialized views in a SQL database but are updated incrementally as data flows through the system, and how this allows the developers to maintain the data in the optimal structures for performing the processing. The best part is that the data is readily available when needed by the processors. You will see how a REST API using Kafka Streams “interactive queries” can be used to retrieve the data in the state stores. We will explore the deployment and monitoring mechanisms used to deliver this system as a set of independently deployed components.

Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim va...

confluent

You know the fundamentals of Apache Kafka. You are a Spring Boot developer and working with Apache Kafka. You have chosen Spring Kafka to integrate with Apache Kafka. You implemented your first producer, consumer and maybe some Kafka streams, it's working... Hurray! You are ready to deploy to production what can possibly go wrong? In this talk, Tim will take you on a journey beyond the basics of Spring Kafka and will share his knowledge, pitfalls and lessons learned based on real-life Kafka projects that are running in production for many years at ING Bank in the Netherlands. This deep-dive talk is a balanced combination of code examples, fully working demos and theory behind the lessons learned like: * Survive consuming a poison pill from a Kafka topic without writing gigabytes of log lines and blow up your server's hard drive? * Understand how to deal with exceptions in different cases * Validate incoming data consumed from a Kafka topic. After joining this talk, you will be inspired to bring your Spring Kafka projects to the next level and share some practical tips and tricks with your fellow developer colleagues!

Architecture of a Kafka camus infrastructure

mattlieber

This document summarizes the results of a performance evaluation of Kafka and Camus to ingest streaming data into Hadoop. It finds that Kafka can ingest data at rates from 15,000-50,000 messages per second depending on data format (Avro is fastest). Camus can move the data to HDFS at rates from 54,000-662,000 records per second. Once in HDFS, queries on Avro-formatted data are fastest, with count and max aggregation queries completing in under 100 seconds for 20 million records. The customer's goal of 5000 events per second can be easily achieved with this architecture.

Learn from HomeAway Hadoop Development and Operations Best Practices

Driven Inc.

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

Monal Daxini

Over 100 million subscribers from over 190 countries enjoy the Netflix service. This leads to over a trillion events, amounting to 3 PB, flowing through the Keystone infrastructure to help improve customer experience and glean business insights. The self-serve Keystone stream processing service processes these messages in near real-time with at-least once semantics in the cloud. This enables the users to focus on extracting insights, and not worry about building out scalable infrastructure. I’ll share the details about this platform, and our experience building it.

What's hot

Introduction to Kafka Streams

Guozhang Wang

Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...

HostedbyConfluent

Follow the (Kafka) Streams

confluent

A Tour of Apache Kafka

confluent

Performance Tuning RocksDB for Kafka Streams’ State Stores

confluent

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...

HostedbyConfluent

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...

confluent

Tips & Tricks for Apache Kafka®

confluent

Exactly-once Data Processing with Kafka Streams - July 27, 2017

confluent

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

confluent

Everything you ever needed to know about Kafka on Kubernetes but were afraid ...

HostedbyConfluent

Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...

confluent

Real-world Streaming Architectures

confluent

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!

confluent

Deploying Confluent Platform for Production

confluent

Deploying Kafka Streams Applications with Docker and Kubernetes

confluent

Inside Kafka Streams—Monitoring Comcast’s Outside Plant

confluent

Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim va...

confluent

Architecture of a Kafka camus infrastructure

mattlieber

What's hot (20)

Introduction to Kafka Streams

Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...

Follow the (Kafka) Streams

A Tour of Apache Kafka

Performance Tuning RocksDB for Kafka Streams’ State Stores

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...

Tips & Tricks for Apache Kafka®

Exactly-once Data Processing with Kafka Streams - July 27, 2017

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

Everything you ever needed to know about Kafka on Kubernetes but were afraid ...

Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...

Real-world Streaming Architectures

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!

Deploying Confluent Platform for Production

Deploying Kafka Streams Applications with Docker and Kubernetes

Inside Kafka Streams—Monitoring Comcast’s Outside Plant

Spring Kafka beyond the basics - Lessons learned on our Kafka journey (Tim va...

Architecture of a Kafka camus infrastructure

Similar to Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

Learn from HomeAway Hadoop Development and Operations Best Practices

Driven Inc.

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

Monal Daxini

SQL Engines for Hadoop - The case for Impala

markgrover

Go Big or Go Home: Approaching Kafka Replication at Scale

HostedbyConfluent

"Processing a lot of data with Kafka means knowing how and when to scale horizontally and vertically. When you’ve exhausted the boundaries of scaling inside a single cluster, replication becomes critical but sometimes standard replication is not enough. New Relic once earned the dubious title of “World’s Largest Kafka Cluster”, and in our journey to break this cluster into dozens of smaller clusters, we needed to route events between clusters and topics based on headers. At the time, this meant we had to do it ourselves. Starting out, our goal was fan out (one-to-many) replication. Since then our needs have expanded to include many-to-one and many-to-many replication. In this talk we'll discuss what bottlenecks we have hit as we scaled out, and what measures we took to remove them, such as: - Replicating data based on Kafka Headers - Connecting to many source and destination Kafka clusters - Managing the replication of Kafka topics of varying traffic - The use of an intermediary Kafka cluster At the end of this talk you will understand how we have scaled replication and routing to support New Relic's ever growing data ingestion, and all the mitigations it took to get us there."

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

confluent

1. A streaming platform like Kafka can provide the benefits of Hadoop for batch processing but in a faster, real-time way by processing data as it arrives rather than storing all data. 2. Virtual reality applications require stream processing to power features like VR mirroring and capture in real-time. Kafka's stream processing capabilities address challenges like this for VR. 3. The document discusses how AltspaceVR uses Kafka stream processing for applications like VR mirroring and capture, presence tracking, scheduled tasks, and more to power their real-time VR experiences.

Real time Messages at Scale with Apache Kafka and Couchbase

Will Gardella

Kafka is a scalable, distributed publish subscribe messaging system that's used as a data transmission backbone in many data intensive digital businesses. Couchbase Server is a scalable, flexible document database that's fast, agile, and elastic. Because they both appeal to the same type of customers, Couchbase and Kafka are often used together. This presentation from a meetup in Mountain View describes Kafka's design and why people use it, Couchbase Server and its uses, and the use cases for both together. Also covered is a description and demo of Couchbase Server writing documents to a Kafka topic and consuming messages from a Kafka topic. using the Couchbase Kafka Connector.

Capital One Delivers Risk Insights in Real Time with Stream Processing

confluent

Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways. Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space. Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka. -Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers. -Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions. -Understand how Capital One manages Kafka Docker containers using Kubernetes. Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.

Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp

José Román Martín Gil

Apache Kafka is the most used data streaming broker by companies. It could manage millions of messages easily and it is the base of many architectures based in events, micro-services, orchestration, ... and now cloud environments. OpenShift is the most extended Platform as a Service (PaaS). It is based in Kubernetes and it helps the companies to deploy easily any kind of workload in a cloud environment. Thanks many of its features it is the base for many architectures based in stateless applications to build new Cloud Native Applications. Strimzi is an open source community that implements a set of Kubernetes Operators to help you to manage and deploy Apache Kafka brokers in OpenShift environments. These slides will introduce you Strimzi as a new component on OpenShift to manage your Apache Kafka clusters. Slides used at OpenShift Meetup Spain: - https://www.meetup.com/es-ES/openshift_spain/events/261284764/

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Athens Big Data

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

ssuserd3a367

1) StumbleUpon uses open source tools like Kafka, HBase, Hive and Pig to build a scalable big data infrastructure to process large amounts of data from its services in real-time and batch. 2) Data is collected from various services using Kafka and stored in HBase for real-time analytics. Batch processing is done using Pig and data is loaded into Hive for ad-hoc querying. 3) The infrastructure powers various applications like recommendations, ads and business intelligence dashboards.

Flink Forward Berlin 2017: Fabian Hueske - Using Stream and Batch Processing ...

Flink Forward

Apache Flink's DataStream API is very expressive and gives users precise control over time and state. However, many applications do not require this level of expressiveness and can be implemented more concisely and easily with a domain-specific API. SQL is undoubtedly the most widely used language for data processing but usually applied in the domain of batch processing. Apache Flink features two relational APIs for unified stream and batch processing, the Table API, a language-integrated relational query API for Scala and Java, and SQL. A Table API or SQL query computes the same result regardless whether it is evaluated on a static file or on a Kafka topic. While Flink evaluates queries on batch input like a conventional query engine, queries on streaming input are continuously processed and their results constantly updated and refined. In this talk we present Flink’s unified relational APIs, show how streaming SQL queries are processed, and discuss exciting new use-cases.

Unify Enterprise Data Processing System Platform Level Integration of Flink a...

Flink Forward

In this talk, I will present how Flink enables enterprise customers to unify their data processing systems by using Flink to query Hive data. Unification of streaming and batch is a main theme for Flink. Since 1.9.0, we have integrated Flink with Hive in a platform level. I will talk about: - what features we have released so far, and what they enable our customers to do - best practices to use Flink with Hive - what is the latest development status of Flink-Hive integration at the time of Flink Forward Berlin (Oct 2019), and what to look for in the next release (probably 1.11)

Flink and Hive integration - unifying enterprise data processing systems

Bowen Li

Flink and Hive Integration aims to unify Flink's streaming and batch processing capabilities by integrating with Hive. Flink 1.9 introduced initial integration with Hive by developing new Catalog APIs to integrate Flink with Hive's metadata and metastore. Flink 1.10 will enhance this integration by supporting more Hive versions, improving Hive source and sink, and introducing pluggable function and table modules. The integration strengthens Flink's metadata management and SQL capabilities while promoting its adoption for both streaming and batch processing.

Fast federated SQL with Apache Calcite

Chris Baynes

Real time data pipline with kafka streams

Yoni Farin

Real time fraud detection at 1+M scale on hadoop stack

DataWorks Summit/Hadoop Summit

Rocketfuel processes over 120 billion ad auctions per day and needs to detect fraud in real time to prevent losses. They developed Helios, which ingests event data from Kafka and HDFS into Storm in real time, joins the streams in HBase, then runs MapReduce jobs hourly to populate an OLAP cube for analyzing feature vectors and detecting fraud patterns. This architecture on Hadoop allows them to easily scale real-time processing and experiment with different configurations to quickly react to fraud.

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Xu Jiang

Kylin is an open source Distributed Analytics Engine from eBay Inc. that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets. If you want to do multi-dimension analysis on large data sets (billion+ rows) with low query latency (sub-seconds), Kylin is a good option. Kylin also provides seamless integration with existing BI tools (e.g Tableau).

Integration for real-time Kafka SQL

Amit Nijhawan

The document discusses a new Hive storage handler for Kafka that allows users to view Kafka topics as Hive tables. This enables BI developers to perform SQL queries and analytics on streaming data in Kafka. It enables connecting to Kafka topics to create external tables, analyzing data through time travel and partition pruning capabilities, and transforming the data through SQL queries, joins, and aggregations. Key features mentioned include creating views of data within time windows, optimizing queries using timestamps and offsets, and joining streaming data with static tables.

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Nitin Kumar

Westpac Bank Tech Talk 1: Dive into Apache Kafka

confluent

Similar to Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams (20)

Learn from HomeAway Hadoop Development and Operations Best Practices

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

SQL Engines for Hadoop - The case for Impala

Go Big or Go Home: Approaching Kafka Replication at Scale

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Real time Messages at Scale with Apache Kafka and Couchbase

Capital One Delivers Risk Insights in Real Time with Stream Processing

Strimzi - Where Apache Kafka meets OpenShift - OpenShift Spain MeetUp

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Building Scalable Big Data Infrastructure Using Open Source Software Presenta...

Flink Forward Berlin 2017: Fabian Hueske - Using Stream and Batch Processing ...

Unify Enterprise Data Processing System Platform Level Integration of Flink a...

Flink and Hive integration - unifying enterprise data processing systems

Fast federated SQL with Apache Calcite

Real time data pipline with kafka streams

Real time fraud detection at 1+M scale on hadoop stack

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Integration for real-time Kafka SQL

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Westpac Bank Tech Talk 1: Dive into Apache Kafka

More from confluent

Building API data products on top of your real-time data infrastructure

confluent

This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products. You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering. You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks. Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!

Speed Wins: From Kafka to APIs in Minutes

confluent

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

Workshop híbrido: Stream Processing con Flink

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

AWS Immersion Day Mapfre - Confluent

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

This document discusses networking options and best practices for Confluent Cloud. It provides an overview of public endpoints, private link, and peering options. It then discusses best practices for private networking architectures on Azure using hub-and-spoke and private link designs. Finally, it addresses networking considerations and challenges for Kafka Connect managed connectors, as well as planned enhancements for DNS peering and outbound private link support.

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Build real-time streaming data pipelines to AWS with Confluent

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

Confluent & GSI Webinars series - Session 3

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Citi Tech Talk: Messaging Modernization

confluent

This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Data In Motion Paris 2023

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

More from confluent (20)

Building API data products on top of your real-time data infrastructure

Speed Wins: From Kafka to APIs in Minutes

Evolving Data Governance for the Real-time Streaming and AI Era

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Recently uploaded

Mechanical Engineering on AAI Summer Training Report-003.pdf

21UME003TUSHARDEB

Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...

Transcat

Join us for this solutions-based webinar on the tools and techniques for commissioning and maintaining PV Systems. In this session, we'll review the process of building and maintaining a solar array, starting with installation and commissioning, then reviewing operations and maintenance of the system. This course will review insulation resistance testing, I-V curve testing, earth-bond continuity, ground resistance testing, performance tests, visual inspections, ground and arc fault testing procedures, and power quality analysis. Fluke Solar Application Specialist Will White is presenting on this engaging topic: Will has worked in the renewable energy industry since 2005, first as an installer for a small east coast solar integrator before adding sales, design, and project management to his skillset. In 2022, Will joined Fluke as a solar application specialist, where he supports their renewable energy testing equipment like IV-curve tracers, electrical meters, and thermal imaging cameras. Experienced in wind power, solar thermal, energy storage, and all scales of PV, Will has primarily focused on residential and small commercial systems. He is passionate about implementing high-quality, code-compliant installation techniques.

Accident detection system project report.pdf

Kamal Acharya

The Rapid growth of technology and infrastructure has made our lives easier. The advent of technology has also increased the traffic hazards and the road accidents take place frequently which causes huge loss of life and property because of the poor emergency facilities. Many lives could have been saved if emergency service could get accident information and reach in time. Our project will provide an optimum solution to this draw back. A piezo electric sensor can be used as a crash or rollover detector of the vehicle during and after a crash. With signals from a piezo electric sensor, a severe accident can be recognized. According to this project when a vehicle meets with an accident immediately piezo electric sensor will detect the signal or if a car rolls over. Then with the help of GSM module and GPS module, the location will be sent to the emergency contact. Then after conforming the location necessary action will be taken. If the person meets with a small accident or if there is no serious threat to anyone’s life, then the alert message can be terminated by the driver by a switch provided in order to avoid wasting the valuable time of the medical rescue team.

OOPS_Lab_Manual - programs using C++ programming language

PreethaV16

NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT

Addu25809

一比一原版(USF毕业证)旧金山大学毕业证如何办理

uqyfuc

原件一模一样【微信：95270640】【旧金山大学毕业证USF学位证成绩单】【微信：95270640】（留信学历认证永久存档查询）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。【业务选择办理准则】一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信：95270640】文凭即可二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信：95270640】即可三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。留信网认证的作用: 1:该专业认证可证明留学生真实身份【微信：95270640】 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才 → 【关于价格问题（保证一手价格）我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子我给客户的都是第一手的代理价格，因为我想坦诚对待大家不想跟大家在价格方面浪费时间对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。选择实体注册公司办理，更放心，更安全！我们的承诺：可来公司面谈，可签订合同，会陪同客户一起到教育部认证窗口递交认证材料，客户在教育部官方认证查询网站查询到认证通过结果后付款，不成功不收费！办理旧金山大学毕业证毕业证学位证USF学位证【微信：95270640 】外观非常精致，由特殊纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。办理旧金山大学毕业证USF学位证毕业证学位证【微信：95270640 】格式相对统一，各专业都有相应的模板。通常包括以下部分：校徽：象征着学校的荣誉和传承。校名:学校英文全称授予学位：本部分将注明获得的具体学位名称。毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。办理旧金山大学毕业证毕业证学位证USF学位证【微信：95270640 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。综上所述，办理旧金山大学毕业证毕业证学位证USF学位证【微信：95270640 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。

Height and depth gauge linear metrology.pdf

q30122000

Height gauges may also be used to measure the height of an object by using the underside of the scriber as the datum. The datum may be permanently fixed or the height gauge may have provision to adjust the scale, this is done by sliding the scale vertically along the body of the height gauge by turning a fine feed screw at the top of the gauge; then with the scriber set to the same level as the base, the scale can be matched to it. This adjustment allows different scribers or probes to be used, as well as adjusting for any errors in a damaged or resharpened probe.

一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理

sydezfe

原版一模一样【微信：741003700 】【(uoft毕业证书)加拿大多伦多大学毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原。 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 【主营项目】一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】外观非常简单，由纸质材料制成，上面印有校徽、校名、毕业生姓名、专业等信息。办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】格式相对统一，各专业都有相应的模板。通常包括以下部分：校徽：象征着学校的荣誉和传承。校名:学校英文全称授予学位：本部分将注明获得的具体学位名称。毕业生姓名：这是最重要的信息之一，标志着该证书是由特定人员获得的。颁发日期：这是毕业正式生效的时间，也代表着毕业生学业的结束。其他信息：根据不同的专业和学位，可能会有一些特定的信息或章节。办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】价值很高，需要妥善保管。一般来说，应放置在安全、干燥、防潮的地方，避免长时间暴露在阳光下。如需使用，最好使用复印件而不是原件，以免丢失。综上所述，办理(uoft毕业证书)加拿大多伦多大学毕业证【微信：741003700 】是证明身份和学历的高价值文件。外观简单庄重，格式统一，包括重要的个人信息和发布日期。对持有人来说，妥善保管是非常重要的。

smart pill dispenser is designed to improve medication adherence and safety f...

um7474492

Supermarket Management System Project Report.pdf

Kamal Acharya

Supermarket management is a stand-alone J2EE using Eclipse Juno program. This project contains all the necessary required information about maintaining the supermarket billing system. The core idea of this project to minimize the paper work and centralize the data. Here all the communication is taken in secure manner. That is, in this application the information will be stored in client itself. For further security the data base is stored in the back-end oracle and so no intruders can access it.

AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...

Paris Salesforce Developer Group

Zener Diode and its V-I Characteristics and Applications

Shiny Christobel

Data Driven Maintenance | UReason Webinar

UReason

Discover the latest insights on Data Driven Maintenance with our comprehensive webinar presentation. Learn about traditional maintenance challenges, the right approach to utilizing data, and the benefits of adopting a Data Driven Maintenance strategy. Explore real-world examples, industry best practices, and innovative solutions like FMECA and the D3M model. This presentation, led by expert Jules Oudmans, is essential for asset owners looking to optimize their maintenance processes and leverage digital technologies for improved efficiency and performance. Download now to stay ahead in the evolving maintenance landscape.

5G Radio Network Througput Problem Analysis HCIA.pdf

AlvianRamadhani5

Software Engineering and Project Management - Introduction, Modeling Concepts...

Prakhyath Rai

Introduction, Modeling Concepts and Class Modeling: What is Object orientation? What is OO development? OO Themes; Evidence for usefulness of OO development; OO modeling history. Modeling as Design technique: Modeling, abstraction, The Three models. Class Modeling: Object and Class Concept, Link and associations concepts, Generalization and Inheritance, A sample class model, Navigation of class models, and UML diagrams Building the Analysis Models: Requirement Analysis, Analysis Model Approaches, Data modeling Concepts, Object Oriented Analysis, Scenario-Based Modeling, Flow-Oriented Modeling, class Based Modeling, Creating a Behavioral Model.

Mechatronics material . Mechanical engineering

sachin chaurasia

Mechatronics is a multidisciplinary field that refers to the skill sets needed in the contemporary, advanced automated manufacturing industry. At the intersection of mechanics, electronics, and computing, mechatronics specialists create simpler, smarter systems. Mechatronics is an essential foundation for the expected growth in automation and manufacturing. Mechatronics deals with robotics, control systems, and electro-mechanical systems.

AI-Based Home Security System : Home security

AIRCC Publishing Corporation

Home security is of paramount importance in today's world, where we rely more on technology, home security is crucial. Using technology to make homes safer and easier to control from anywhere is important. Home security is important for the occupant’s safety. In this paper, we came up with a low cost, AI based model home security system. The system has a user-friendly interface, allowing users to start model training and face detection with simple keyboard commands. Our goal is to introduce an innovative home security system using facial recognition technology. Unlike traditional systems, this system trains and saves images of friends and family members. The system scans this folder to recognize familiar faces and provides real-time monitoring. If an unfamiliar face is detected, it promptly sends an email alert, ensuring a proactive response to potential security threats.

一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理

ecqow

CalArts毕业证学历书【微信95270640】CalArts毕业证’圣力嘉学院毕业证《Q微信95270640》办理CalArts毕业证√文凭学历制作{CalArts文凭}购买学历学位证书本科硕士,CalArts毕业证学历学位证【实体公司】办毕业证、成绩单、学历认证、学位证、文凭认证、办留信网认证、（网上可查，实体公司，专业可靠） (诚招代理)办理国外高校毕业证成绩单文凭学位证,真实使馆公证（留学回国人员证明）真实留信网认证国外学历学位认证雅思代考国外学校代申请名校保录开请假条改GPA改成绩ID卡 1.高仿业务:【本科硕士】毕业证,成绩单（GPA修改）,学历认证（教育部认证）,大学Offer,,ID,留信认证,使馆认证,雅思,语言证书等高仿类证书； 2.认证服务: 学历认证（教育部认证）,大使馆认证（回国人员证明）,留信认证（可查有编号证书）,大学保录取,雅思保分成绩单。 3.技术服务：钢印水印烫金激光防伪凹凸版设计印刷激凸温感光标底纹镭射速度快。办理加利福尼亚艺术学院加利福尼亚艺术学院毕业证文凭证书流程： 1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）； 2开始安排制作毕业证成绩单电子图； 3毕业证成绩单电子版做好以后发送给您确认； 4毕业证成绩单电子版您确认信息无误之后安排制作成品； 5成品做好拍照或者视频给您确认； 6快递给客户（国内顺丰国外DHLUPS等快读邮寄） -办理真实使馆公证（即留学回国人员证明） -办理各国各大学文凭（世界名校一对一专业服务,可全程监控跟踪进度） -全套服务：毕业证成绩单真实使馆公证真实教育部认证。让您回国发展信心十足！（详情请加一下文凭顾问+微信:95270640）欢迎咨询！子小伍玩小伍比山娃小一岁虎头虎脑的很霸气父亲让山娃跟小伍去夏令营听课山娃很高兴夏令营就设在附近一所小学山娃发现那所小学比自己的学校更大更美操场上还铺有塑胶跑道呢里面很多小朋友一班一班的快快乐乐原来城里娃都藏这儿来了怪不得平时见不到他们山娃恍然大悟起来吹拉弹唱琴棋书画山娃都不懂却什么都想学山娃怨自己太笨什么都不会斟酌再三山娃终于选定了学美术当听说每月要交元时父亲犹豫了山娃也说爸算了吧咱学校一学期才转

4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf

Gino153088

UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS

vmspraneeth

Recently uploaded (20)

Mechanical Engineering on AAI Summer Training Report-003.pdf

Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...

Accident detection system project report.pdf

OOPS_Lab_Manual - programs using C++ programming language

NATURAL DEEP EUTECTIC SOLVENTS AS ANTI-FREEZING AGENT

一比一原版(USF毕业证)旧金山大学毕业证如何办理

Height and depth gauge linear metrology.pdf

一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理

smart pill dispenser is designed to improve medication adherence and safety f...

Supermarket Management System Project Report.pdf

AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...

Zener Diode and its V-I Characteristics and Applications

Data Driven Maintenance | UReason Webinar

5G Radio Network Througput Problem Analysis HCIA.pdf

Software Engineering and Project Management - Introduction, Modeling Concepts...

Mechatronics material . Mechanical engineering

AI-Based Home Security System : Home security

一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理

4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf

UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS

Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

1. Hunter Kelly @retnuh Real-Time Domain Rankings with Kafka Streams

2. What are we doing? ● Discovering relevant domains in the Fashion Web ● Use modified HITS - a.k.a. Hubs & Authorities Why use Kafka Streams? ● Why not Spark, Hadoop, Whatevs? ● Advantages of Kafka Streams Show me the nitty gritty! ● Okay!

3. The Fashion Web

4. Curated list of Fashion related sites ● Initial lists from domain experts ● Support multiple languages ● What are the next “best” domains?

5. cosmopolitan stylecaster instyle racked harpersbazaar elle vogue hypebeast

6. http://www.cs.cornell.edu/home/kleinber/auth.pdf http://en.wikipedia.org/wiki/HITS_algorithm * Hyperlink-Induced Topic Search, but no-one seems to call it that Need Data-driven method ● Kleinberg’s HITS* algorithm ● Allows us to discover Hubs and Authorities ● Flattened to domain level

7. HITS in a nutshell

8. HITS in a nutshell Adjacency Matrix Image courtesy of http://faculty.ycp.edu/~dbabcock/PastCourses/cs360/lectures/lecture15.html

9. HITS in a nutshell ● Why not just counts? ● If you point to good sites, you’re a better Hub ● If good sites point to you, you’re a better Authority ● Each iteration, the weights change, until they reach convergence

10. Common Questions ● What about non-fashion domains? ● Why not PageRank?

11. Kafka Streams

12. Please tell me you aren’t using Kafka to do Matrix multiplication!

13. Please tell me you aren’t using Kafka to do Matrix multiplication! No, I am not crazy enough* to use Kafka to do matrix multiplication!

14. Please tell me you aren’t using Kafka to do Matrix multiplication! No, I am not crazy enough* to use Kafka to do matrix multiplication! *Although, I probably did spend too much time thinking about it on the bus!

15. Why not use Map-Reduce? Basically what it was invented for, right?

16. You could use Hadoop...

17. You could use Hadoop...

18. You could use Spark… ● High infrastructure overhead if not using it for anything else ● Bad initial experience w/ Spark Streaming snapshotting and recovery ● Already using Kafka!

19. Why use Kafka Streams? ● Has primitives necessary for Map-Reduce ○ Map step groupBy groupByKey ○ Reduce step reduce aggregate ● Focus is on your data not distributed computing machinery ● Streaming allows us to have (near) real-time, up-to-date data

20. The Nitty Gritty

21. Kafka Streams 101 ● No explicit consumers/producers - plumbing handled for you ● Topics still a fundamental communication piece ● Think of individual datum flowing through KStreams & KTables

22. Kafka Streams 101 - KSTreams ● Focus is on specific functional transformations - map, filter, flatMap ● Also supports various flavours of joins with other KStreams ● Usually created from one or more topics or a transformation on another KStream

23. Kafka Streams 101 - KTable ● Still offers functional transforms, but on a primary-keyed table ● Offers persistent storage ● Created from aggregations on a KStream or transforms on other KTable

24. ● The bridge between KStream and KTable ● Created by doing a groupBy or groupByKey on a KStream ● Create KTables by doing reduce, aggregate, count Kafka Streams 101 - KGroupedStream

25. Output Topic (log compacted) KTable ops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API Data Flow

26. Output Topic (log compacted) KTable ops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API

27. Output Topic (log compacted) KTable ops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API

28. Output Topic (log compacted) KTable ops: groupByKey, reduce, toStream KStream Input Topic KStream ops: flatMapDomain Link Extractor Domain Reducer HITS Calculator & API

29. Current Domain Link Extractor Input Topic Fetch Content External Links Resolve Links Extract Domains KStream KStream ops: flatMap

30. Extracted Link Resolver Domain Link Extractor Input Topic Fetch Content External Links Resolve Links Extract Domains Output Topic KStream ops: flatMap KStream ops: flatMap KStream ops: flatMap

31. Domain Link Extractor Input Topic Fetch Content External Links Resolve Links Extract Domains KStream Other Function(s) KStream KStream ops: flatMap KStream Topic Whatev’s With Fork

32. So what’s the point?

33. TL;DR ● Kafka Streams can help solve problems in your application domain ● Focus on your data! ● Naturally decompose the problem into flexible microservices

34.

35. Hunter Kelly @retnuh https://github.com/retnuh

Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams

Similar to Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Kafka Summit SF 2017 - Real-Time Document Rankings with Kafka Streams