Nielsen Presents: Fun with Kafka, Spark and Offset Management

•

3 likes•472 views

1) The document discusses how Nielsen moved from using Kafka's offset manager to store offsets to using an RDS database for offset storage. This provided more consistent committing of offsets and allowed subscribing to topics with different offset strategies. 2) Key aspects of the new approach included building an offsets map with topic and partition information, getting current offsets from the database, and implementing a "two phase commit" to the database and Kafka in a consistent way. 3) Benefits of the new offset storage approach included faster restart times, improved monitoring, better disaster recovery, and unified configurations.

Nielsen Presents: Fun
with Kafka, Spark and
Offset Management
or how we stopped worrying and started to enjoy offset
management

Agenda
• How we used to work
• What we wanted to do and why
• What we ended up doing
• Extra Benefits

• Simona
• Big Data Engineer at Nielsen Marketing Cloud
• Data lover
• Concert Goer
• Japan enthusiast
whoami

How we used to
work
• Kafka Spark Consumer
• Offsets Strategies
• Kafka Offset Manager
• TEAM A vs TEAM B Flows

Our Different Flows
TEAM RON TEAM ITAI
Micro Batch
Length
4-6 minutes 1 hour
Output to
Commit
Kafka
Offsets
Kafka
Offsets, File
names

Defining Starting
Offsets
• Kafka Offset Manager
• earliest
• latest
• XML

kafka offset
manager
• Self made
• Offsets stored in kafka
• Everything is against brokers

What we wanted to
do and why
• JUST A SIMPLE UPGRADE
• Two main goals to achieve with our
infrastructure
• Some of the problems
we encountered

Just a Simple
Upgrade
● New Kafka Consumer API
● New consumer config
● Subscribing to Kafka

SOME OF THE PROBLEMS
WE ENCOUNTERED
Committing offsets to Kafka fails
Longer timeouts then!
Spark graceful shutdown
Spark Kafka aSync Commit
……?

what we ended up
doing
• RDS Offset Store
• Consistent Committing
• Subscribing to topics (Offsets
Strategies)
• Unified infrastructure
• Timeouts

RDS Offset Store
• Table Structure
• Constraints and upserts
• Triggers

Building the
Offsets Map
● What is the offsets Map[TopicPartition,Long]
● Getting the number of partitions
● Constructing the offsets map

Getting Current
Offsets
● Each map represents a row
● Reduce rows into the offsets map

“Two Phase”
Commit
● What is a “two phase” commit?
● Why do we need it?
● How did we end up implementing it?

Extra benefits
• Restart Time
• Monitoring
• Disaster Recovery
• Unified Configurations

Committing to
Kafka
● Why?
● Two options of committing to Kafka

New Offsets
Strategies
• DataBase
• Earliest
• Latest

Nielsen Presents: Fun with Kafka, Spark and Offset Management

At Nielsen, data is very important. Being the core of our business, we love it and there’s lots of it. We don’t want to lose it, and at the same time, we don’t want to duplicate it. Our data goes through a robust Kafka architecture, into several ETLs, receiving, transforming and storing the data. While we clearly understood our ETLs’ workflow, we had no visibility into what parts of the data, if any, were lost or duplicated, and in which stage or stages of the workflow, from source to destination. But how much do we know about the way our data makes though our systems? And what about the life long question, is it the end of the day yet? In this talk I’m going to present to you the design process behind our Data Auditing system, Life Line. From tracking and producing , to analysing and storing auditing information, using technologies such as Kafka, Avro, Spark, Lambda functions and complex SQL queries. We’re going to cover: * AVRO Audit header * Auditing heart beat - designing your metadata * Designing and optimising your auditing table - what does this data look like anyway? * Creating an alert based monitoring system * Answering the most important question of all - is it the end of the day yet?

Spark logs made easy

Simona Meriam

This document summarizes how logging was improved for Apache Spark jobs on Amazon EMR. It discusses: 1) Previously, logs had to be accessed directly on servers or downloaded, which was difficult. A solution was developed using bootstrap actions to install Filebeat and Metricbeat, shipping logs to Redis then to Elasticsearch for analysis in Kibana. 2) Main obstacles included resistance to change from data engineers. 3) Future additions could replace Redis with Kafka and collect logs from other services like Druid and Kafka.

Serverless Code Deployments in AWS

Marko Tomic

Front End Orlando - AWS Lambda

Brandon Westcott

AWS Lambda is a serverless compute service that runs code in response to events, automatically managing resources. It handles capacity, scaling, deployment, monitoring, logging, and security patching. Developers pay only for the compute time used to run their code, avoiding the need to provision and manage servers. Lambda functions can be triggered by events from services like API Gateway or S3 and are written in languages like Node.js, Python, Java, and C#.

How and why test Azure Front Door with AWS Lambda & PowerShell? | Osman Sahin...

UK DevOps Collective

Osman Sahin, a regular attendee of our events, explains how and why he is using AWS Lambda & PowerShell to test the new Azure Front Door service. Presented Wednesday 28th July 2019 at the London PowerShell User Group Meetup hosted by dotdigital Group. Connect with Osman Sahin: - LinkedIn: https://www.linkedin.com/in/osmanysahin/ Thanks to dotdigital Group (https://dotdigital.com / https://twitter.com/dotdigital) for providing the venue, food and drinks. We very much appreciate your continued support of our community of PowerShell & DevOps tech enthusiasts. Join our next event at https://www.meetup.com/PowerShell-London-UK/. We are running at least one Meetup every month. #PowerShell #Azure #AWS #DevOps

Fury road to a worldwide API - API Days - December 2015

Julien Lemoine

Algolia's Fury Road to a Worldwide API - Take Off Conference 2016

Olivier Lance

Algolia - Hosted Search API

enterprisesearchmeetup

Algolia is a distributed Search-as-a-Service API that processes more than 4 billions user-generated queries per month. Algolia’s DNA is performance: Algolia's service is optimized to reply in milliseconds from any location worldwide while maintaining high availability. Sylvain will provide you details on Algolia architecture, showing how they have designed their fault tolerant service and how they cracked the website & in-app search. http://www.meetup.com/Enterprise-Search-and-Analytics-Meetup/events/223926122/

This document discusses infrastructure as code using Terraform on AWS. It introduces infrastructure as code and why it is important compared to traditional infrastructure deployment methods. Some key benefits highlighted are having code to represent infrastructure that can be version controlled, allowing infrastructure to be deployed in an automated and repeatable way. The document then demonstrates deploying AWS infrastructure like VPCs, subnets, security groups, load balancers etc. using Terraform code. It shows the workflow of initializing, planning and applying Terraform code to provision the AWS resources.

The great migration embracing serverless first

AngelaTimofte1

Serverless architectures can reduce operational cost and development time but it does come with an architectural overhead and complexity. In this talk you'll get an insight on how Trustpilot adopted a serverless mindset. The benefits and challenges of migrating and refactoring our systems having a serverless first approach. Learn how we mitigated the complexity of our services and cut down the cost on our infrastructure.

Best practices for running Windows workloads on AWS - AWS Summit Stockholm (M...

T. Alexander Lystad

Serverless Computing @ x-celerate 2018

Nils Rhode

Serverless Computing: Run code, not servers

x-celerate

Durch den Einzug von Cloud Computing erleben wir in der IT ein Wandel von Infrastructure as a Serivce (IaaS) über Software as a Serivce (SaaS) & Platform as a Service (PaaS) bis hin zu Function as a Service (FaaS) – auch bekannt unter dem Begriff Serverless Computing. Die Vorteile liegen dabei auf der Hand: bedarfsgerechte Abrechnung, schnelle und hohe Skalierbarkeit und eine einfache Möglichkeit den Ansatz von Microservices schnell und einfach umzusetzen. Der Einsatz von FaaS nimmt stark zu – auch wenn gleich viele diese Funktionen nur als kleine Helferlein benutzen und selten fester Bestandteil in einer Anwendung sind. Wie die Haufe.Group es geschafft hat, ganze Workflows und eine ganze Applikation rein auf FaaS aufzubauen und über 100 verschiedene Funktionen orchestriert bekommt, wird der Fokus des Vortrags sein. Dabei steht im Mittelpunkt das Analysieren & Importieren mehrere tausend Dokumente pro Sekunde. Zur Bewältigung dieser Aufgabe setzt die Haufe.Group auf die AWS Cloud und deren Services, wie auch das OpenSource Framework „Serverless Framework“ auf die im Vortrag eingegangen wird.

Fall in Love with Graphs and Metrics using Grafana

torkelo

MongoDB World 2018: Solving Your Backup Needs Using MongoDB Ops Manager, Clou...

MongoDB

This document discusses MongoDB's cloud database offerings including MongoDB Atlas, Ops Manager, and Cloud Manager. It provides an overview of key features such as automated backups, point-in-time restore, queryable snapshots, global availability, security, and elastic scaling. The document also demonstrates MongoDB's managed backup capabilities in Atlas including cloud provider snapshots on AWS and Azure, as well as a roadmap for future disaster recovery features.

How to move a mission critical system to 4 AWS regions in one year?

Wojciech Gawroński

A year ago our team was challenged to enhance the scope and scale of an existing platform, that is providing significant revenue for our client. As the designers and maintainers of that solution, we decided to leverage AWS cloud during that transition. In the presentation, I would like to discuss how we have tackled that migration - with the assumption that we had to move in a limited resource, hybrid cloud environment - working in close cooperation with teams responsible for other parts of the system. As I stated previously - it was a challenge - and I would like to talk what problems we have solved during that process. Also, what services we have leveraged to smooth the transition. And last, but not least - I would like to present how we have maintained the delivery pipeline, automation and massive pile of CloudFormation templates and why AWS Lambda is an excellent glue for any operational work you have to do in the cloud. Our hard work paid off. In October 2017 we have deployed our system into 4th AWS region. Bare with me during the talk, and you will learn how we achieved that

MongoDB World 2018: Using Puppet, Ansible and Ops Manager to Create Your Own ...

MongoDB

Using Puppet, Ansible, and Ops Manager, Paychex automated the provisioning of MongoDB databases to address the challenges of a manual process that took 1-2 weeks. Puppet is used for OS configuration and standardization. Ansible is used for orchestration to make API calls to Ops Manager for database provisioning in 15-25 minutes. Ops Manager provides management, automation, monitoring, and backups. Future work includes further integrating backups and restores using Ansible and integrating with additional tools like ServiceNow.

What is new in pass summit 2014

Harry Zheng

This document provides information on what's new at the PASS Summit 2014 conference. It includes details on the conference schedule from Monday to Friday with preconference sessions on Monday and Tuesday and main summit sessions from Wednesday to Friday. There will be evening receptions on Tuesday, Wednesday and Thursday. The keynote speakers and topics are outlined for Wednesday and Thursday morning sessions. New features and services for Microsoft's Azure cloud platform being highlighted include Azure Document DB, Azure Stream Analytics, and Azure Machine Learning. A session highlight discusses changing the default statistics update behavior in SQL Server 2014 using a trace flag.

The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018

iguazio

Serverless architectures allow automatic scaling of compute resources and pay-per-use pricing models. While scalability is often a key benefit, serverless functions have limitations regarding state management. Event-driven serverless platforms can be viewed as a form of stream processing. To handle stateful operations, the data model should be designed based on how data will be queried. An "inside out database" approach treats each event as data and action, and uses the database primarily for data storage rather than querying. This mirrors an CQRS pattern and allows scaling to infinite event streams.

Algolia's Fury Road to a Worldwide API

Paul-Louis NECH

1. Algolia started as a beta with two machines in 2013 and has since expanded to 15 regions across 50 data centers, serving over 2600 customers globally. 2. Early challenges included risky deployments without easy rollbacks and intermittent slowness caused by DNS issues. 3. Algolia now has a distributed search network for worldwide synchronization and spreads clusters across independent providers and datacenters for high reliability.

Elk meetup

Asaf Yigal

Choosing the right messaging service for your serverless app [with lumigo]

Dhaval Nagar

This document summarizes a presentation about choosing the right messaging service for serverless applications. It discusses serverless and event-driven architectures, and how functions are executed in response to events through messaging services. It then covers the main AWS messaging services: Amazon SQS for message queues, Amazon SNS for publish/subscribe, and Amazon EventBridge for managing events. It provides examples of how these services can be used with Lambda and discusses factors for selecting the appropriate service. Monitoring serverless applications is also discussed.

How we use the play framework

Itai Gilo

Autodesk's Homestyler group uses Scala and the Play Framework to build backends that power their home design mobile and web applications serving millions of users. They chose Scala and Play because it allows them to build robust, scalable, and agile services in a fun and modern way. Their architecture includes technologies like Couchbase, MySQL, ElasticSearch, and services running on AWS. Play provides features like routing, MVC structure, and testing infrastructure that help them build new projects and modules quickly while its async abilities improve performance. They have replaced their old backend with a new one using Play that was easy to structure and has allowed them to re-route a live application while adopting technologies like Swagger for integration and Grafana

CICD in the World of Serverless

Srushith Repakula

In the ever fast paced software development, Serverless came in as a boon, enabling developers concentrate primarily on the business logic and nothing more! This only calls for a stringent process flow, making Continuous Development, Countinous Integration and Continuous Delivery highly advisable. This talk will focus on local testing of the serverless functions and their associated services locally.

Tracing Java Applications on Azure

CodeOps Technologies LLP

FME Cloud Tips for Success

Safe Software

FME Cloud provides tips for successfully using their hosted version of FME Server in the cloud. Key benefits include automated backups, real-time monitoring, pay-as-you-go pricing, and no responsibility for hardware. New features include trial instances without credit cards and instance scheduling. Tips include saving money with annual subscriptions and scheduled instances, maximizing security, and leveraging AWS data storage services like S3 for object storage and RDS for scalable relational databases. Demonstrations show uploading data to S3 and using AWS services can provide more robust, scalable and cost effective deployments than on-premises solutions.

Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)

DataArt

Serverless architectures allow developers to run code without provisioning or managing servers. With serverless, code runs in ephemeral containers that are managed by third-party cloud providers. Key benefits include automatic scaling, high availability, and paying only for the resources consumed. Some limitations are vendor lock-in, memory and computing limits per function, and potential cold start delays. Serverless functions can be triggered synchronously, asynchronously via events, or via streaming data sources. Common event sources that trigger serverless functions include S3, API Gateway, IoT, and SQS. Serverless is well suited for building event-driven architectures and processing streaming data at scale.

Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...

iguazio

The document discusses combining functions as a service with containers to build a serverless container experience. Some key points made include: - Containers and functions each have their own use cases and patterns that could be combined - Using containers allows reusing existing tools for microservices while getting benefits of functions like scaling to zero and pay per use billing - A serverless experience with containers could bridge the gap between monolithic apps and microservices by allowing no infrastructure management, scaling by request, and utility billing - The future of serverless may involve running functions in containers on any platform for a more flexible serverless experience

Learn from HomeAway Hadoop Development and Operations Best Practices

Driven Inc.

Distributed Kafka Architecture Taboola Scale

Apache Kafka TLV

___________________________________________ Meetup#7 | Session 2 | 21/03/2018 | Taboola _____________________________________________ In this talk, we will present our multi-DC Kafka architecture, and discuss how we tackle sending and handling 10B+ messages per day, with maximum availability and no tolerance for data loss. Our architecture includes technologies such as Cassandra, Spark, HDFS, and Vertica - with Kafka as the backbone that feeds them all.

What's hot

Infrastructure as code (iac) - Terraform for AWS

Johanes Glenn

The great migration embracing serverless first

AngelaTimofte1

Best practices for running Windows workloads on AWS - AWS Summit Stockholm (M...

T. Alexander Lystad

Serverless Computing @ x-celerate 2018

Nils Rhode

Serverless Computing: Run code, not servers

x-celerate

Fall in Love with Graphs and Metrics using Grafana

torkelo

MongoDB World 2018: Solving Your Backup Needs Using MongoDB Ops Manager, Clou...

MongoDB

How to move a mission critical system to 4 AWS regions in one year?

Wojciech Gawroński

MongoDB World 2018: Using Puppet, Ansible and Ops Manager to Create Your Own ...

MongoDB

What is new in pass summit 2014

Harry Zheng

The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018

iguazio

Algolia's Fury Road to a Worldwide API

Paul-Louis NECH

Elk meetup

Asaf Yigal

Choosing the right messaging service for your serverless app [with lumigo]

Dhaval Nagar

How we use the play framework

Itai Gilo

CICD in the World of Serverless

Srushith Repakula

Tracing Java Applications on Azure

CodeOps Technologies LLP

FME Cloud Tips for Success

Safe Software

Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)

DataArt

Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...

iguazio

What's hot (20)

Infrastructure as code (iac) - Terraform for AWS

The great migration embracing serverless first

Best practices for running Windows workloads on AWS - AWS Summit Stockholm (M...

Serverless Computing @ x-celerate 2018

Serverless Computing: Run code, not servers

Fall in Love with Graphs and Metrics using Grafana

MongoDB World 2018: Solving Your Backup Needs Using MongoDB Ops Manager, Clou...

How to move a mission critical system to 4 AWS regions in one year?

MongoDB World 2018: Using Puppet, Ansible and Ops Manager to Create Your Own ...

What is new in pass summit 2014

The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018

Algolia's Fury Road to a Worldwide API

Elk meetup

Choosing the right messaging service for your serverless app [with lumigo]

How we use the play framework

CICD in the World of Serverless

Tracing Java Applications on Azure

FME Cloud Tips for Success

Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)

Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...

Similar to Nielsen Presents: Fun with Kafka, Spark and Offset Management

Learn from HomeAway Hadoop Development and Operations Best Practices

Driven Inc.

Distributed Kafka Architecture Taboola Scale

Apache Kafka TLV

Stream Processing @ Lyft

Jamie Grier

Lyft's streaming platform uses Apache Flink for stream processing and Apache Kafka for messaging. Flink was chosen for its capabilities around state management, exactly-once processing, and flexible APIs. Kafka was chosen for its durability, low latency, and consumer fanout. However, open problems remain around rescaling Kafka while preserving per-key ordering, enabling dynamic stream computations, long-term event storage, and zero downtime deployments. Lyft is working to solve these challenges as it builds out its next generation streaming platform.

Introduction to Kafka

Akash Vacher

(ARC310) Solving Amazon's Catalog Contention With Amazon Kinesis

Amazon Web Services

The Amazon.com product catalog receives millions of updates an hour across billions of products with many of the updates concentrated on comparatively few products. In this session, hear how Amazon.com has used Amazon Kinesis to build a pipeline orchestrator that provides sequencing, optimistic batching, and duplicate suppression whilst at the same time significantly lowering costs. This session covers the architecture of that solution and draws out the key enabling features that Amazon Kinesis provides. This talk is intended for those who are interested in learning more about the power of the distributed log and understanding its importance for enabling OLTP just as DHT is for storage.

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

ScyllaDB

The RCP/Rakuten Catalog Platform has been growing at a brisk speed over the last couple of years. Our original backbone was Cassandra. However, as they continued their growth, they internally started realizing that it was not suitable for our next stage of growth. As such, they started looking into ScyllaDB as a better ROI solution as well as a much more stable backend. The migration itself was challenging since this has to be done for a production live data processing pipeline with minimal impact on customers. In this talk, Hitesh Shah, Engineering Manager at Rakuten USA will dive deeper into challenges and takeaways. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Apache spark y cómo lo usamos en nuestros proyectos

OpenSistemas

This document discusses Apache Spark and how it is used in projects at OpenSistemas. It provides a brief history of Hadoop and the limitations that led to the development of Apache Spark. It then explains what Apache Spark is, how it was designed from the ground up to take advantage of memory, and how it is influenced by Scala. It also introduces the Kappa architecture as an alternative to the Lambda architecture that only requires maintaining a single stream processing codebase. Finally, it provides examples of how OpenSistemas uses the Kappa architecture with Apache Spark streaming for various clients in domains like telecommunications, insurance, energy, and more.

High Performance Object Pascal Code on Servers (at EKON 22)

Arnaud Bouchez

This EKON 22 conference is about high performance on servers, written in the object pascal (Delphi / FPC) language. Profiling should be the first step to avoid premature optimization, which is the root of all evil (Knuth). But when some bottlenecks are identified, we introduce some simple architecture patterns (like caching or microservices), data structures and algorithms to make process actually faster, with minimal refactoring. It was a fun session about how to write faster code, ending up by looking at the Delphi CPU view – even if you don’t know assembly.

Moving to microservices – a technology and organisation transformational journey

Boyan Dimitrov

Moving to microservices was a transformational journey for the company as their system grew rapidly. They started with a monolithic architecture which became difficult to maintain and scale. This led them to redesign their system using microservices built with Go running on AWS (Amazon Web Services). They developed core platform capabilities to support automated provisioning, routing, discovery, monitoring and more. This allowed them to deploy new services rapidly and operate their distributed system more efficiently. The transition required changes to both their technology and organizational culture.

KoprowskiT_SQLRelay2014#4_Caerdydd_MaintenancePlansForBeginners

Tobias Koprowski

Rubyslava + PyVo #48

Jozef Képesi

Skypicker is a flight ticket search and booking engine that processes hundreds of terabytes of airline data monthly and sells thousands of tickets daily in Europe, Russia, and China. It uses PostgreSQL extensively to power its API and databases. The document discusses Skypicker's PostgreSQL infrastructure, which includes 5 database clusters with over 0.5TB of memory each that handle 20 million updates per hour. It also covers optimizations made to scale the databases, such as table partitioning, cascading replication, and query tuning. Skypicker is currently preparing for a large increase in data and is hiring database masters and developers.

Cassandra Day SV 2014: Spark, Shark, and Apache Cassandra

DataStax Academy

Evan Chan from Ooyala presents on integrating Apache Spark and Apache Cassandra for interactive analytics. He discusses how Ooyala uses Cassandra for analytics and is becoming a major Spark user. The talk focuses on using Spark to generate dynamic queries over Cassandra data, as precomputing all possible aggregates is infeasible at Ooyala's scale. Chan describes Ooyala's architecture that uses Spark to generate materialized views from Cassandra for fast querying, and demonstrates running queries over a Spark/Cassandra dataset.

The Metamorphosis of Database Changes With Tim Steinbach | Current 2022

HostedbyConfluent

The Metamorphosis of Database Changes With Tim Steinbach | Current 2022 This talk describes our journey of ingesting multiple Kafka data streams from thousands of topics and about half a million partitions, storing Apache Iceberg datasets and explaining the issues along the way. We will take a look at CDC streams produced from our MySQL databases by Debezium, how we decided to process and store the data, and how our data teams now access the information. Join us on a whirlwind tour through Kafka Connect, Avro schemas, Iceberg tables, table evolutions, breaking schema changes, recurring exceptions, fun bugs, and why timestamps are hard. Finally, we will discuss some of the solutions these datasets have enabled for us and how the data is now used.

Building a derived data store using Kafka

Venu Ryali

LinkedIn built a new derived data store called Venice to address limitations of their previous system Voldemort. Venice uses Kafka to enable scalable, fault-tolerant processing and replication of both batch-processed and incrementally updated derived data. It processes data through Hadoop jobs to Kafka topics, from which both batch-stored and real-time copies are maintained through Venice and Samza respectively. Kafka Mirror Maker replicates data across data centers for high availability.

023 Quick Deploy GraphQL API With SST - NODES2022 AMERICAS Intermediate 9 - S...

Neo4j

This document discusses using the Serverless Stack Platform (SST) to quickly deploy a Neo4j GraphQL serverless application. SST allows deploying AWS resources across stacks with local debugging capabilities. It has templates that can stand up proof of concepts within 10 minutes by importing a template, updating code files, and deploying locally and to AWS. This provides advantages like simplified authentication, no server administration, high availability, scalability, and lower costs through pay per use of AWS Lambda.

Modern MySQL Monitoring and Dashboards.

Mydbops

2013 CPM Conference, Nov 6th, NoSQL Capacity Planning

asya999

This document discusses MongoDB capacity planning. It begins with a brief history of databases and the factors driving NoSQL adoption. It then discusses MongoDB's origins and key features like document storage, auto-sharding, and high availability. The document emphasizes that capacity planning requires understanding an application's requirements, resources used, and monitoring metrics over time. It provides examples of measuring and planning for storage, memory, CPU, and network resources as applications and data change. The goal of capacity planning is to continuously and proactively scale resources to meet evolving needs.

Inside Wordnik's Architecture

Tony Tam

Wordnik's architecture is built around a large English word graph database and uses microservices and ephemeral Amazon EC2 storage. Key aspects include: 1) The system is built as independent microservices that communicate via REST APIs documented using Swagger specifications. 2) Databases for each microservice are kept small by design to facilitate operations like backups, replication, and index rebuilding. 3) Services are deployed across multiple Availability Zones and regions on ephemeral Amazon EC2 storage for high availability despite individual host failures.

Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016

Monal Daxini

Building and Deploying Large Scale SSRS using Lessons Learned from Customer D...

Denny Lee

This document discusses lessons learned from deploying large scale SQL Server Reporting Services (SSRS) environments based on customer scenarios. It covers the key aspects of success, scaling out the architecture, performance optimization, and troubleshooting. Scaling out involves moving report catalogs to dedicated servers and using a scale out deployment architecture. Performance is optimized through configurations like disabling report history and tuning memory settings. Troubleshooting utilizes logs, monitoring, and diagnosing issues like out of memory errors.

Similar to Nielsen Presents: Fun with Kafka, Spark and Offset Management (20)

Learn from HomeAway Hadoop Development and Operations Best Practices

Distributed Kafka Architecture Taboola Scale

Stream Processing @ Lyft

Introduction to Kafka

(ARC310) Solving Amazon's Catalog Contention With Amazon Kinesis

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

Apache spark y cómo lo usamos en nuestros proyectos

High Performance Object Pascal Code on Servers (at EKON 22)

Moving to microservices – a technology and organisation transformational journey

KoprowskiT_SQLRelay2014#4_Caerdydd_MaintenancePlansForBeginners

Rubyslava + PyVo #48

Cassandra Day SV 2014: Spark, Shark, and Apache Cassandra

The Metamorphosis of Database Changes With Tim Steinbach | Current 2022

Building a derived data store using Kafka

023 Quick Deploy GraphQL API With SST - NODES2022 AMERICAS Intermediate 9 - S...

Modern MySQL Monitoring and Dashboards.

2013 CPM Conference, Nov 6th, NoSQL Capacity Planning

Inside Wordnik's Architecture

Netflix keystone streaming data pipeline @scale in the cloud-dbtb-2016

Building and Deploying Large Scale SSRS using Lessons Learned from Customer D...

Recently uploaded

Applications of artificial Intelligence in Mechanical Engineering.pdf

Atif Razi

Historically, mechanical engineering has relied heavily on human expertise and empirical methods to solve complex problems. With the introduction of computer-aided design (CAD) and finite element analysis (FEA), the field took its first steps towards digitization. These tools allowed engineers to simulate and analyze mechanical systems with greater accuracy and efficiency. However, the sheer volume of data generated by modern engineering systems and the increasing complexity of these systems have necessitated more advanced analytical tools, paving the way for AI. AI offers the capability to process vast amounts of data, identify patterns, and make predictions with a level of speed and accuracy unattainable by traditional methods. This has profound implications for mechanical engineering, enabling more efficient design processes, predictive maintenance strategies, and optimized manufacturing operations. AI-driven tools can learn from historical data, adapt to new information, and continuously improve their performance, making them invaluable in tackling the multifaceted challenges of modern mechanical engineering.

Rainfall intensity duration frequency curve statistical analysis and modeling...

bijceesjournal

Using data from 41 years in Patna’ India’ the study’s goal is to analyze the trends of how often it rains on a weekly, seasonal, and annual basis (1981−2020). First, utilizing the intensity-duration-frequency (IDF) curve and the relationship by statistically analyzing rainfall’ the historical rainfall data set for Patna’ India’ during a 41 year period (1981−2020), was evaluated for its quality. Changes in the hydrologic cycle as a result of increased greenhouse gas emissions are expected to induce variations in the intensity, length, and frequency of precipitation events. One strategy to lessen vulnerability is to quantify probable changes and adapt to them. Techniques such as log-normal, normal, and Gumbel are used (EV-I). Distributions were created with durations of 1, 2, 3, 6, and 24 h and return times of 2, 5, 10, 25, and 100 years. There were also mathematical correlations discovered between rainfall and recurrence interval. Findings: Based on findings, the Gumbel approach produced the highest intensity values, whereas the other approaches produced values that were close to each other. The data indicates that 461.9 mm of rain fell during the monsoon season’s 301st week. However, it was found that the 29th week had the greatest average rainfall, 92.6 mm. With 952.6 mm on average, the monsoon season saw the highest rainfall. Calculations revealed that the yearly rainfall averaged 1171.1 mm. Using Weibull’s method, the study was subsequently expanded to examine rainfall distribution at different recurrence intervals of 2, 5, 10, and 25 years. Rainfall and recurrence interval mathematical correlations were also developed. Further regression analysis revealed that short wave irrigation, wind direction, wind speed, pressure, relative humidity, and temperature all had a substantial influence on rainfall. Originality and value: The results of the rainfall IDF curves can provide useful information to policymakers in making appropriate decisions in managing and minimizing floods in the study area.

LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant

Anant Corporation

2008 BUILDING CONSTRUCTION Illustrated - Ching Chapter 02 The Building.pdf

Yasser Mahgoub

原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样

ydzowc

原件一模一样【微信：bwp0011】《(Humboldt毕业证书)柏林大学毕业证学位证》【微信：bwp0011】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原。 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问微bwp0011 【主营项目】一.毕业证【微bwp0011】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【微bwp0011】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...

shadow0702a

This document serves as a comprehensive step-by-step guide on how to effectively use PyCharm for remote debugging of the Windows Subsystem for Linux (WSL) on a local Windows machine. It meticulously outlines several critical steps in the process, starting with the crucial task of enabling permissions, followed by the installation and configuration of WSL. The guide then proceeds to explain how to set up the SSH service within the WSL environment, an integral part of the process. Alongside this, it also provides detailed instructions on how to modify the inbound rules of the Windows firewall to facilitate the process, ensuring that there are no connectivity issues that could potentially hinder the debugging process. The document further emphasizes on the importance of checking the connection between the Windows and WSL environments, providing instructions on how to ensure that the connection is optimal and ready for remote debugging. It also offers an in-depth guide on how to configure the WSL interpreter and files within the PyCharm environment. This is essential for ensuring that the debugging process is set up correctly and that the program can be run effectively within the WSL terminal. Additionally, the document provides guidance on how to set up breakpoints for debugging, a fundamental aspect of the debugging process which allows the developer to stop the execution of their code at certain points and inspect their program at those stages. Finally, the document concludes by providing a link to a reference blog. This blog offers additional information and guidance on configuring the remote Python interpreter in PyCharm, providing the reader with a well-rounded understanding of the process.

spirit beverages ppt without graphics.pptx

Madan Karki

integral complex analysis chapter 06 .pdf

gaafergoudaay7aga

4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf

Gino153088

Computational Engineering IITH Presentation

co23btech11018

Curve Fitting in Numerical Methods Regression

Nada Hikmah

People as resource Grade IX.pdf minimala

riddhimaagrawal986

一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理

ecqow

CalArts毕业证学历书【微信95270640】CalArts毕业证’圣力嘉学院毕业证《Q微信95270640》办理CalArts毕业证√文凭学历制作{CalArts文凭}购买学历学位证书本科硕士,CalArts毕业证学历学位证【实体公司】办毕业证、成绩单、学历认证、学位证、文凭认证、办留信网认证、（网上可查，实体公司，专业可靠） (诚招代理)办理国外高校毕业证成绩单文凭学位证,真实使馆公证（留学回国人员证明）真实留信网认证国外学历学位认证雅思代考国外学校代申请名校保录开请假条改GPA改成绩ID卡 1.高仿业务:【本科硕士】毕业证,成绩单（GPA修改）,学历认证（教育部认证）,大学Offer,,ID,留信认证,使馆认证,雅思,语言证书等高仿类证书； 2.认证服务: 学历认证（教育部认证）,大使馆认证（回国人员证明）,留信认证（可查有编号证书）,大学保录取,雅思保分成绩单。 3.技术服务：钢印水印烫金激光防伪凹凸版设计印刷激凸温感光标底纹镭射速度快。办理加利福尼亚艺术学院加利福尼亚艺术学院毕业证文凭证书流程： 1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）； 2开始安排制作毕业证成绩单电子图； 3毕业证成绩单电子版做好以后发送给您确认； 4毕业证成绩单电子版您确认信息无误之后安排制作成品； 5成品做好拍照或者视频给您确认； 6快递给客户（国内顺丰国外DHLUPS等快读邮寄） -办理真实使馆公证（即留学回国人员证明） -办理各国各大学文凭（世界名校一对一专业服务,可全程监控跟踪进度） -全套服务：毕业证成绩单真实使馆公证真实教育部认证。让您回国发展信心十足！（详情请加一下文凭顾问+微信:95270640）欢迎咨询！子小伍玩小伍比山娃小一岁虎头虎脑的很霸气父亲让山娃跟小伍去夏令营听课山娃很高兴夏令营就设在附近一所小学山娃发现那所小学比自己的学校更大更美操场上还铺有塑胶跑道呢里面很多小朋友一班一班的快快乐乐原来城里娃都藏这儿来了怪不得平时见不到他们山娃恍然大悟起来吹拉弹唱琴棋书画山娃都不懂却什么都想学山娃怨自己太笨什么都不会斟酌再三山娃终于选定了学美术当听说每月要交元时父亲犹豫了山娃也说爸算了吧咱学校一学期才转

ITSM Integration with MuleSoft.pptx

VANDANAMOHANGOUDA

Software Quality Assurance-se412-v11.ppt

TaghreedAltamimi

哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样

insn4465

原版一模一样【微信：741003700 】【(csu毕业证书)查尔斯特大学毕业证硕士学历】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Certificates - Mahmoud Mohamed Moursi Ahmed

Mahmoud Morsy

Advanced control scheme of doubly fed induction generator for wind turbine us...

IJECEIAES

This paper describes a speed control device for generating electrical energy on an electricity network based on the doubly fed induction generator (DFIG) used for wind power conversion systems. At first, a double-fed induction generator model was constructed. A control law is formulated to govern the flow of energy between the stator of a DFIG and the energy network using three types of controllers: proportional integral (PI), sliding mode controller (SMC) and second order sliding mode controller (SOSMC). Their different results in terms of power reference tracking, reaction to unexpected speed fluctuations, sensitivity to perturbations, and resilience against machine parameter alterations are compared. MATLAB/Simulink was used to conduct the simulations for the preceding study. Multiple simulations have shown very satisfying results, and the investigations demonstrate the efficacy and power-enhancing capabilities of the suggested control system.

Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...

IJECEIAES

Medical image analysis has witnessed significant advancements with deep learning techniques. In the domain of brain tumor segmentation, the ability to precisely delineate tumor boundaries from magnetic resonance imaging (MRI) scans holds profound implications for diagnosis. This study presents an ensemble convolutional neural network (CNN) with transfer learning, integrating the state-of-the-art Deeplabv3+ architecture with the ResNet18 backbone. The model is rigorously trained and evaluated, exhibiting remarkable performance metrics, including an impressive global accuracy of 99.286%, a high-class accuracy of 82.191%, a mean intersection over union (IoU) of 79.900%, a weighted IoU of 98.620%, and a Boundary F1 (BF) score of 83.303%. Notably, a detailed comparative analysis with existing methods showcases the superiority of our proposed model. These findings underscore the model’s competence in precise brain tumor localization, underscoring its potential to revolutionize medical image analysis and enhance healthcare outcomes. This research paves the way for future exploration and optimization of advanced CNN models in medical imaging, emphasizing addressing false positives and resource efficiency.

Welding Metallurgy Ferrous Materials.pdf

AjmalKhan50578

Recently uploaded (20)

Applications of artificial Intelligence in Mechanical Engineering.pdf

Rainfall intensity duration frequency curve statistical analysis and modeling...

LLM Fine Tuning with QLoRA Cassandra Lunch 4, presented by Anant

2008 BUILDING CONSTRUCTION Illustrated - Ching Chapter 02 The Building.pdf

原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样

Use PyCharm for remote debugging of WSL on a Windo cf5c162d672e4e58b4dde5d797...

spirit beverages ppt without graphics.pptx

integral complex analysis chapter 06 .pdf

4. Mosca vol I -Fisica-Tipler-5ta-Edicion-Vol-1.pdf

Computational Engineering IITH Presentation

Curve Fitting in Numerical Methods Regression

People as resource Grade IX.pdf minimala

一比一原版(CalArts毕业证)加利福尼亚艺术学院毕业证如何办理

ITSM Integration with MuleSoft.pptx

Software Quality Assurance-se412-v11.ppt

哪里办理(csu毕业证书)查尔斯特大学毕业证硕士学历原版一模一样

Certificates - Mahmoud Mohamed Moursi Ahmed

Advanced control scheme of doubly fed induction generator for wind turbine us...

Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...

Welding Metallurgy Ferrous Materials.pdf

Nielsen Presents: Fun with Kafka, Spark and Offset Management

1. Nielsen Presents: Fun with Kafka, Spark and Offset Management or how we stopped worrying and started to enjoy offset management

2. Agenda

3. Agenda • How we used to work • What we wanted to do and why • What we ended up doing • Extra Benefits

4. • Simona • Big Data Engineer at Nielsen Marketing Cloud • Data lover • Concert Goer • Japan enthusiast whoami

5. How we used to work • Kafka Spark Consumer • Offsets Strategies • Kafka Offset Manager • TEAM A vs TEAM B Flows

6. Our Different Flows TEAM RON TEAM ITAI Micro Batch Length 4-6 minutes 1 hour Output to Commit Kafka Offsets Kafka Offsets, File names

7. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML

8. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML

9. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML

10. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML

11. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML

12. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML

13. kafka offset manager • Self made • Offsets stored in kafka • Everything is against brokers

14. What we wanted to do and why • JUST A SIMPLE UPGRADE • Two main goals to achieve with our infrastructure • Some of the problems we encountered

15. Just a Simple Upgrade ● New Kafka Consumer API ● New consumer config ● Subscribing to Kafka

16. SOME OF THE PROBLEMS WE ENCOUNTERED Committing offsets to Kafka fails Longer timeouts then! Spark graceful shutdown Spark Kafka aSync Commit ……?

17. what we ended up doing • RDS Offset Store • Consistent Committing • Subscribing to topics (Offsets Strategies) • Unified infrastructure • Timeouts

18. RDS Offset Store • Table Structure • Constraints and upserts • Triggers

19. RDS Offset Store

20. RDS Offset Store

21. RDS Offset Store

22. Subscribing to Topics

23. Building the Offsets Map ● What is the offsets Map[TopicPartition,Long] ● Getting the number of partitions ● Constructing the offsets map

24. Getting Number of Partitions

25. Getting Current Offsets

26. Getting Current Offsets

27. Getting Current Offsets

28. Getting Current Offsets

29. Getting Current Offsets