Auditing your data and answering the life long question, is it the end of the day yet? with Simona Meriam | Kafka Summit London 2022

•

0 likes•428 views

Over here at Nielsen, data is very important to us. Being the core of our business, we love it and there’s lots of it. We don’t want to lose it, and at the same time, we don’t want to duplicate it. Our data goes through a robust Kafka architecture, into several ETLs, receiving, transforming and storing the data. While we clearly understood our ETLs’ workflow, we had no visibility into what parts of the data, if any, were lost or duplicated, and in which stage or stages of the workflow, from source to destination. But how much do we know about the way our data makes though our systems? And what about the life long question, is it the end of the day yet? In this talk I’m going to present to you the design process behind our Data Auditing system, Life Line. From tracking and producing , to analysing and storing auditing information, using technologies such as Kafka, Avro, Spark, Lambda functions and complex SQL queries. We’re going to cover: * AVRO Audit header * Auditing heart beat - designing your metadata * Designing and optimising your auditing table - what does this data look like anyway? * Creating an alert based monitoring system * Answering the most important question of all - is it the end of the day yet?

Technology

Auditing data and
answering that life long
question, is it the end of
the day yet?
A true story based off of my endeavors
at Nielsen
Simona Meriam, Aidoc

Agenda
1. Little ﬁres everywhere
2. Designing our auditing data
3. Storing and querying our data
4. Is it the end of the day yet?
5. Alerts and add-ons

● Simona Meriam
● Senior Big Data Engineer @ Aidoc
● Data lover
● Concert goer
● Japan enthusiast
WHOAMI?
LinkedIn:
https://il.linkedin.com/in/simona-meriam-94
68885
Twitter: @simonameriam

Little fires everywhere
● Data arrival pain points and recovering from failures
● When to process data?
● Is it the end of the day yet?
● Some more pain points

Is it the end of the day yet?
● Let’s talk about this question, and possible answers
○ Data granularity
○ Time granularity
● So when do we process data? When is it the end of
the day?
● The implications of processing and reprocessing

Is it the end of the day yet?
Legacy answers to a legacy question
● Fixed time
● AWS S3 ls

Auditing window?
● Several Kafka topics
● Data serving infrastructure
○ Our own “Nielsen Kafka Producer”
○ 2 JVMs on a single machine
○ Each JVM works against several topics
○ SLA’s are very important!!
● The use of AVRO
Then ﬁnally, what is a window?
What should we keep in
mind?

Auditing window
Key
● Topic
● Server
● Process
● Audit time
Value
● Counter

Auditing subsystem
Audit windows Audit headers
● Open window
● Close window
● Increment counter
● Inject audit header
Nielsen Kafka Producer

● S3 Consumer ● Kafka consumer
● Straight through API
Audit header Audit window
Consuming our data

Designing our output table
● Level of granularity?
● Arrival rates?
● Arrival latency?
Questions we want answered!

● Audit timestamp
● Topic
● Server
● Process
● Event count
● Location - origin of data
How about some add ons?
● Region
● Insert time
Designing our output table

insert_time window_timestamp topic_name server_name region process_id location event_count
2021-08-02T00:05:00.079000 2021-08-02T00:00:00 TOPIC1 server1.ams1.nielsen ams1 19862 kafka_windows 0
2021-08-02T00:05:00.082000 2021-08-02T00:00:00 TOPIC1 server1.slj1.nielsen slj1 4075 kafka_windows 98396
2021-08-02T00:05:00.082000 2021-08-02T00:00:00 TOPIC2 server1.slj1.nielsen slj1 4075 kafka_windows 31805
2021-08-02T00:05:00.082000 2021-08-02T00:00:00 TOPIC1 server1.slj1.nielsen slj1 4075 kafka_windows 98396
2021-08-02T00:32:12.082000 2021-08-02T00:00:00 TOPIC2 server1.slj1.nielsen slj1 4075 rdr_headers 12453
2021-08-02T00:05:00.132000 2021-08-02T00:00:00 TOPIC3 server2.ams1.nielsen ams1 31573 kafka_windows 84924
2021-08-02T00:05:00.131000 2021-08-02T00:00:00 TOPIC1 server2.ams1.nielsen ams1 31573 kafka_windows 0
2021-08-02T00:10:00.009700 2021-08-02T00:05:00 TOPIC2 server2.ams1.nielsen ams1 31571 kafka_windows 3177

Q & A with Superset
SELECT window_timestamp AT TIME ZONE 'UTC' AS window_timestamp,
topic_name,
SUM(CASE WHEN location = 'kafka_windows' THEN event_count ELSE 0 END) AS producer_count,
SUM(CASE WHEN location = 'rdr_headers' AND
(insert_time AT TIME zone 'utc' - window_timestamp AT TIME zone 'utc' <= INTERVAL '3
HOURS') THEN event_count ELSE 0 END) AS rdr_count
FROM audit.audit_data
WHERE window_timestamp <= CURRENT_TIMESTAMP - INTERVAL '2 HOURS' AND
window_timestamp >= CURRENT_TIMESTAMP - INTERVAL '1 MONTH' AND
event_count > 0
GROUP BY window_timestamp, topic_name
HAVING SUM(CASE WHEN location = 'kafka_windows' THEN event_count ELSE 0 END) > 0

Is is the end of the day yet??
1. Data arrival rate for the entire
scope?
3. Arrival rate for the last
window?
2. Number of audit windows
for the entire scope?

Some alerts and add-ons
● Alert granularity for different types of
failures
○ Region
○ Topic
○ Server
● Detecting duplications
● More locations!

CREDITS: This presentation template was
created by Slidesgo, including icons by
Flaticon, and infographics & images by Freepik

Thank you!
@simonameriam
https://il.linkedin.com/in/simona-meriam-9468885

At Nielsen, data is very important. Being the core of our business, we love it and there’s lots of it. We don’t want to lose it, and at the same time, we don’t want to duplicate it. Our data goes through a robust Kafka architecture, into several ETLs, receiving, transforming and storing the data. While we clearly understood our ETLs’ workflow, we had no visibility into what parts of the data, if any, were lost or duplicated, and in which stage or stages of the workflow, from source to destination. But how much do we know about the way our data makes though our systems? And what about the life long question, is it the end of the day yet? In this talk I’m going to present to you the design process behind our Data Auditing system, Life Line. From tracking and producing , to analysing and storing auditing information, using technologies such as Kafka, Avro, Spark, Lambda functions and complex SQL queries. We’re going to cover: * AVRO Audit header * Auditing heart beat - designing your metadata * Designing and optimising your auditing table - what does this data look like anyway? * Creating an alert based monitoring system * Answering the most important question of all - is it the end of the day yet?

Data Stack Considerations: Build vs. Buy at Tout

Looker

THE RISE AND FALL OF SERVERLESS COSTS - TAMING THE (SERVERLESS) BEAST

Opher Dubrovsky

From talk given on June 10, 2020 at the DevTalks Reimagined conference. ABSTRACT: Serverless is an amazing beast of a technology. With it, you can quickly build and deploy incredible systems. You get out-of-the-box scalability and flexibility.Nevertheless, with great power comes great(er) cost!In this talk, you’ll learn about building a huge data pipeline using Serverless architecture, and how to tame the beast.After this session, you will understand the pitfalls to avoid and the great powers to exploit.

3 types of monitoring for 2020

T. Alexander Lystad

Data to the Masses: Automated Word Document Creation with FME

Safe Software

Join us for an exciting webinar featuring special Guest Speaker Mary Brown from the Swiss Agency for Development and Cooperation, who will share her experiences using FME and Microsoft Word to tackle data challenges. Mary will discuss how FME was used to leverage GIS at the core of Swiss Humanitarian Aid, fight fires in combination with ArcGIS Online, and support the Foreign Ministry during the Covid-19 pandemic. Hear her use cases to learn how FME was used to make a real difference in the world. Our Technical Support Specialist Kailin will then show you how to use FME and Microsoft Word Writer to automate the creation of professional documents, making your work faster and more efficient. You'll learn how to design custom templates, merge data from multiple sources, and generate documents with multiple pages and sections. Plus, Kailin will give a special bonus demo on using FME to write PowerPoint documents. Whether you're a beginner or an experienced user, this webinar is a must-see. Sign up now to reserve your free spot and learn how to use FME to create Microsoft Word documents like a pro!

iCare Sales Presentation (PowerPoint - view notes page for notes) 11-11

Daniel Kelly, PSP

The document describes a customer portal that provides (1) secured online access to data and service information, (2) a tool that supports the standard service delivery process, and (3) real-time work order management including status updates. It allows customers to manage service requests, view status updates, and access reports on performance, costs and compliance. The portal uses encryption and access controls to securely store and transmit customer information.

If an Application Fails in the Datacenter and No Users Are On It, Will it Cut...

SolarWinds

Sun modeling

Andy Cobley

Designing systems to use Cassandra can be difficult for programmers used to relational databases, normalisation and entity relationship diagrams. One of the first steps in designing the tables for Cassandra is to model the CQL commands that will be run and use these to designate tables, but how to get these statements from the users requirements? Sun modelling is a technique used by designers of OLAP systems to gather requirements from users without having to explain the intricacies of Star Schema. In this talk we'll introduce Sun modelling and show how it can be simply modified to gather requirements for CQL queries and hence table design.

The document summarizes key points about smart manufacturing and moving to Industry 4.0. Part 1 discusses the value of data collection from machines and processes, and provides case studies on using sensors to monitor machine efficiency and oven temperature. Part 2 defines Industry 4.0 as the fourth industrial revolution, outlines its 6 design principles, and notes benefits for small businesses including faster response, improved productivity and flexibility. It concludes with an interactive quiz to self-assess a company's readiness for Industry 4.0.

Value Stream Mapping: What to Do Before You Dive In

TKMG, Inc.

Recorded webinar: http://slidesha.re/1juuPs4 To subscribe: http://ksmartin.com/subscribe To purchase the book: http://bit.ly/VSMbk Value Stream Mapping is a powerful way to improve performance across a large portion of an enterprise. But, before you dive into mapping, there are significant steps you must take to assure a successful outcome. In this webinar, award-winning author, Karen Martin shares the proper preparation an organization must take to reap the full set of benefits Value Stream Mapping offers. Topics include: Leadership – what is their role? Scoping – how "big” should you go? Team formation – who are the right people to include? Facilitator selection – what traits and skills are needed? Charter development and socialization – it’s far more than a planning tool! Logistics & communication – how do you make sure everyone is engaged and prepared? Watch this webinar for a no-nonsense discussion about the key success factors and common failings in preparing to value stream map.

Data Collection 2.pptx

Esteban López

This document provides guidance on data collection for simulation modeling. It discusses collecting existing data from various sources, performing preliminary data collection to understand the system, and detailed data collection to determine key aspects of how the system operates. Recommendations are provided on focusing data collection, collecting enough data for statistical analysis, continuous data collection over various time periods, and using multiple collectors to observe without interfering with the system. An example of data collection for a single queue system is also included.

Critical incident management.pptx

DavidForeroS

This document outlines Sutherland's major incident management process. It describes who is involved in incident response, including Service Delivery, Workforce Management, and Global Technology Infrastructure teams. It defines incidents versus service requests and outlines service level agreements and incident priority levels. The document provides guidance on reporting priority 1 incidents, including calling the Global Service Desk directly or escalating lower priority tickets. It also describes notifying local IT operations teams and examples of communication emails. Finally, it discusses next steps after resolution like collecting additional metrics and conducting post-mortem reviews.

Our journey with druid - from initial research to full production scale

Itai Yaffe

Here at the Nielsen Marketing Cloud we use druid.io (http://druid.io/) as one of our main data stores, both for simple counts and for approximate count-distinct (DataSketches). It’s been more than a year since we started using it, injecting billions of events each day to multiple druid clusters for different use-cases. In this meet-up, we will share our journey, the challenges we had, the way we overcame them (at least most of them) and the steps we made to optimize the process around Druid to keep the solution cost effective. Before diving into Druid, we will briefly present our data pipeline architecture, starting from the front-end serving system, deployed in number of geo-locations, to a centralized Kafka cluster in the cloud, and give some examples of the different processes that consume from Kafka and feed our different data sources.

Iwsm2014 gathering data on it maintenance work (jos de weerdt) public release

Nesma

Ordina is an IT services company with annual turnover of 185 million euros and 2,890 employees. They implemented an automated process to gather time spent data from their IT maintenance work, logging over 22,000 time entries per month. This solution helped address challenges around tracking over 4,500 monthly service calls manually and officers disliking recording time. The automated data provides accurate and timely aggregated time spent reporting to evaluate contracts and bids.

Analytics in Action: Project Analytics: Visibility that Aids Risk Management

Hannah Flynn

Product Managers face changes that put delivery at risk. Just as you use data from the customer to inform your solutions, transparency during the building of those solutions is critical for making better risk mitigation decisions. Whether your solution has fixed scope, fixed scheduling, fixed resources or fixed level of quality, the earlier you can know when these are at risk (and how) the better. The more clear the picture, the better you can understand the impacts of changes, and the more effectively you can deliver the solution the customer needs, when they need it, at a reasonable cost. This session will focus on the use of a One-Dimensional Product Backlog from a risk management point of view. We will show how this tool can be used to monitor and evaluate how your solution is getting built, and a clear view of any cascading impacts risks have as they surface.

Project Analytics: Visibility that Aids Risk Management

Aggregage

The Scary Truth of Spreadsheets

KeyedIn Projects

Tom Raper, Director of Product Solutions at KeyedIn, presented on how spreadsheets are commonly used for project management but have significant downsides for collaboration, data management, and usability. While spreadsheets are flexible and easy to use, they make collaboration difficult, can only store limited information, require manual updates, and lack version control. This can result in projects taking longer and being harder to manage. Formal project management solutions can help address these issues and improve project success. KeyedIn is a project portfolio management provider that aims to help organizations transition from spreadsheets.

How Graphs are Taming the Complexity of Network & IT Ops

Neo4j

Networks are stressed. Telco networks, data centers, corporate networks, and service providers are all stressed. Network operators who face balancing uptime and service levels with growth and security demands may find that graph technology provided by Neo4j can provide welcome relief. During this webinar presentation, we explore: -How to map, manage and query complex network infrastructures comprised of interrelated devices, systems, clouds, locations, and users. -How to model these interdependencies to help diagnose machine, service and network failures while minimizing service interruption. -How to grow the network by adding physical and virtual nodes, devices, {towers}, servers, or applications without increasing the complexity of managing them. -How to use the power of graph relationships to identify suspicious behaviors or failures without impeding regular operations. -How to use Neo4j to craft your own IT asset and configuration management system that incorporates not only hard assets, but also cloud services, business processes and users.

Unified Operations Vision

Steve Mushero

Cloud Dataflow - A Unified Model for Batch and Streaming Data Processing

DoiT International

AsianGames Security Story - Andika Triwidada

idsecconf

1. The author took over as IT director for the 2018 Asian Games in late March 2018 with minimal budget, staff, security measures, and documentation of existing infrastructure in place. 2. Over the course of the games, the IT team supported over 40 competition venues, 4000+ endpoints, internal backbone bandwidth up to 1 Gbps, and cloud-based services for 11k+ athletes across 45 countries over 3 weeks of competition. 3. Despite having only a small core IT staff of less than 20 people and relying heavily on partners, the IT infrastructure and services performed smoothly with no apparent security breaches during the Asian Games, surprising the author given the minimal security that was in place.

24-HourClock.pdf

AbhishekJha401

The document provides an overview of a 24-hour clock schedule for daily processing in a meter data management system (MDMS). It includes a sample daily schedule showing iterative processing of meter data validation and MRA exchanges throughout the day. The primary objective is continuous publishing of billing data, with the billing cutoff time being a key driver for the schedule. It also allows time for exceptions to be resolved before billing.

B tech project sample- petrol pump management system project in vb.net .

Study Stuff

Agile Data Science

Volodymyr Kazantsev

Challenges of Operationalising Data Science in Production

iguazio

The presentation topic for this meet-up was covered in two sections without any breaks in-between Section 1: Business Aspects (20 mins) Speaker: Rasmi Mohapatra, Product Owner, Experian https://www.linkedin.com/in/rasmi-m-428b3a46/ Once your data science application is in the production, there are many typical data science operational challenges experienced today - across business domains - we will cover a few challenges with example scenarios Section 2: Tech Aspects (40 mins, slides & demo, Q&A ) Speaker: Santanu Dey, Solution Architect, Iguazio https://www.linkedin.com/in/santanu/ In this part of the talk, we will cover how these operational challenges can be overcome e.g. automating data collection & preparation, making ML models portable & deploying in production, monitoring and scaling, etc. with relevant demos.

Benchmarking The Digital Workplace

Unicorn Titans

5 Amazing Reasons DBAs Need to Love Extended Events

Jason Strate

Solving the Database Problem

Jay Gordon

In 2018 we've seen a huge uptick in applications using Kubernetes for their deployment method. Many times your persistent data layer is a difficult decision. What will you store data in, how long will you need to access this data and who will manage the lifecycle of this data? These are important questions many developers and ops teams have taken to heart. In this talk we'll review how the data layer is managed for high availability and reliability in modern application deployment. The attendee should leave having a better understanding of the options in front of them and their ability to build applications in any hosting environment.

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

Generating privacy-protected synthetic data using Secludy and Milvus

Zilliz

During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.

A Comprehensive Guide to DeFi Development Services in 2024

Intelisync

DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum. In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance. In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape. At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology. Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!

TrustArc Webinar - 2024 Global Privacy Survey

TrustArc

How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024? In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores. See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe. This webinar will review: - The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey - The top challenges for privacy leaders, practitioners, and organizations in 2024 - Key themes to consider in developing and maintaining your privacy program

Introduction of Cybersecurity with OSS at Code Europe 2024

Hiroshi SHIBATA

I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems. The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS. Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application. I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.

Letter and Document Automation for Bonterra Impact Management (fka Social Sol...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on automated letter generation for Bonterra Impact Management using Google Workspace or Microsoft 365. Interested in deploying letter generation automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Serial Arm Control in Real Time Presentation

tolgahangng

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

Chart Kalyan

UI5 Controls simplified - UI5con2024 presentation

Wouter Lemaire

Nordic Marketo Engage User Group_June 13_ 2024.pptx

MichaelKnudsen27

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...

Tatiana Kojar

Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI. With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.

Operating System Used by Users in day-to-day life.pptx

Pravash Chandra Das

Dive into the realm of operating systems (OS) with Pravash Chandra Das, a seasoned Digital Forensic Analyst, as your guide. 🚀 This comprehensive presentation illuminates the core concepts, types, and evolution of OS, essential for understanding modern computing landscapes. Beginning with the foundational definition, Das clarifies the pivotal role of OS as system software orchestrating hardware resources, software applications, and user interactions. Through succinct descriptions, he delineates the diverse types of OS, from single-user, single-task environments like early MS-DOS iterations, to multi-user, multi-tasking systems exemplified by modern Linux distributions. Crucial components like the kernel and shell are dissected, highlighting their indispensable functions in resource management and user interface interaction. Das elucidates how the kernel acts as the central nervous system, orchestrating process scheduling, memory allocation, and device management. Meanwhile, the shell serves as the gateway for user commands, bridging the gap between human input and machine execution. 💻 The narrative then shifts to a captivating exploration of prominent desktop OSs, Windows, macOS, and Linux. Windows, with its globally ubiquitous presence and user-friendly interface, emerges as a cornerstone in personal computing history. macOS, lauded for its sleek design and seamless integration with Apple's ecosystem, stands as a beacon of stability and creativity. Linux, an open-source marvel, offers unparalleled flexibility and security, revolutionizing the computing landscape. 🖥️ Moving to the realm of mobile devices, Das unravels the dominance of Android and iOS. Android's open-source ethos fosters a vibrant ecosystem of customization and innovation, while iOS boasts a seamless user experience and robust security infrastructure. Meanwhile, discontinued platforms like Symbian and Palm OS evoke nostalgia for their pioneering roles in the smartphone revolution. The journey concludes with a reflection on the ever-evolving landscape of OS, underscored by the emergence of real-time operating systems (RTOS) and the persistent quest for innovation and efficiency. As technology continues to shape our world, understanding the foundations and evolution of operating systems remains paramount. Join Pravash Chandra Das on this illuminating journey through the heart of computing. 🌟

AWS Cloud Cost Optimization Presentation.pptx

HarisZaheer8

This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

Best 20 SEO Techniques To Improve Website Visibility In SERP

Pixlogix Infotech

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on integration of Salesforce with Bonterra Impact Management. Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

WeTestAthens: Postman's AI & Automation Techniques

Postman

Recently uploaded (20)

Generating privacy-protected synthetic data using Secludy and Milvus

A Comprehensive Guide to DeFi Development Services in 2024

TrustArc Webinar - 2024 Global Privacy Survey

Introduction of Cybersecurity with OSS at Code Europe 2024

Letter and Document Automation for Bonterra Impact Management (fka Social Sol...

Serial Arm Control in Real Time Presentation

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

UI5 Controls simplified - UI5con2024 presentation

Nordic Marketo Engage User Group_June 13_ 2024.pptx

Artificial Intelligence for XMLDevelopment

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...

Operating System Used by Users in day-to-day life.pptx

AWS Cloud Cost Optimization Presentation.pptx

Presentation of the OECD Artificial Intelligence Review of Germany

Best 20 SEO Techniques To Improve Website Visibility In SERP

Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...

Columbus Data & Analytics Wednesdays - June 2024

HCL Notes and Domino License Cost Reduction in the World of DLAU

WeTestAthens: Postman's AI & Automation Techniques

Auditing your data and answering the life long question, is it the end of the day yet? with Simona Meriam | Kafka Summit London 2022

1. Auditing data and answering that life long question, is it the end of the day yet? A true story based off of my endeavors at Nielsen Simona Meriam, Aidoc

2. Agenda 1. Little ﬁres everywhere 2. Designing our auditing data 3. Storing and querying our data 4. Is it the end of the day yet? 5. Alerts and add-ons

3. ● Simona Meriam ● Senior Big Data Engineer @ Aidoc ● Data lover ● Concert goer ● Japan enthusiast WHOAMI? LinkedIn: https://il.linkedin.com/in/simona-meriam-94 68885 Twitter: @simonameriam

4. Nielsens’ architecture (AT THE TIME)

5. Little fires everywhere

6. Little fires everywhere ● Data arrival pain points and recovering from failures ● When to process data? ● Is it the end of the day yet? ● Some more pain points

7. Data arrival pain points

8. Data arrival pain points

10. Recovering from failures

11. Recovering from failures

12. Is it the end of the day yet? ● Let’s talk about this question, and possible answers ○ Data granularity ○ Time granularity ● So when do we process data? When is it the end of the day? ● The implications of processing and reprocessing

13. Is it the end of the day yet? Legacy answers to a legacy question ● Fixed time ● AWS S3 ls

14. Back to pain points

15. And some more

16. Little fires everywhere

17. Designing our auditing data

18. Auditing window? ● Several Kafka topics ● Data serving infrastructure ○ Our own “Nielsen Kafka Producer” ○ 2 JVMs on a single machine ○ Each JVM works against several topics ○ SLA’s are very important!! ● The use of AVRO Then ﬁnally, what is a window? What should we keep in mind?

19. Auditing window Key ● Topic ● Server ● Process ● Audit time Value ● Counter

20. Auditing header

21.

22. Auditing header injection

23. Auditing subsystem Audit windows Audit headers ● Open window ● Close window ● Increment counter ● Inject audit header Nielsen Kafka Producer

24. In context

25. ● S3 Consumer ● Kafka consumer ● Straight through API Audit header Audit window Consuming our data

26. In context

27. Storing and querying our data

28. Designing our output table ● Level of granularity? ● Arrival rates? ● Arrival latency? Questions we want answered!

29. ● Audit timestamp ● Topic ● Server ● Process ● Event count ● Location - origin of data How about some add ons? ● Region ● Insert time Designing our output table

30. insert_time window_timestamp topic_name server_name region process_id location event_count 2021-08-02T00:05:00.079000 2021-08-02T00:00:00 TOPIC1 server1.ams1.nielsen ams1 19862 kafka_windows 0 2021-08-02T00:05:00.082000 2021-08-02T00:00:00 TOPIC1 server1.slj1.nielsen slj1 4075 kafka_windows 98396 2021-08-02T00:05:00.082000 2021-08-02T00:00:00 TOPIC2 server1.slj1.nielsen slj1 4075 kafka_windows 31805 2021-08-02T00:05:00.082000 2021-08-02T00:00:00 TOPIC1 server1.slj1.nielsen slj1 4075 kafka_windows 98396 2021-08-02T00:32:12.082000 2021-08-02T00:00:00 TOPIC2 server1.slj1.nielsen slj1 4075 rdr_headers 12453 2021-08-02T00:05:00.132000 2021-08-02T00:00:00 TOPIC3 server2.ams1.nielsen ams1 31573 kafka_windows 84924 2021-08-02T00:05:00.131000 2021-08-02T00:00:00 TOPIC1 server2.ams1.nielsen ams1 31573 kafka_windows 0 2021-08-02T00:10:00.009700 2021-08-02T00:05:00 TOPIC2 server2.ams1.nielsen ams1 31571 kafka_windows 3177

31.

32. Q & A with Superset

33. Q & A with Superset

34. Q & A with Superset SELECT window_timestamp AT TIME ZONE 'UTC' AS window_timestamp, topic_name, SUM(CASE WHEN location = 'kafka_windows' THEN event_count ELSE 0 END) AS producer_count, SUM(CASE WHEN location = 'rdr_headers' AND (insert_time AT TIME zone 'utc' - window_timestamp AT TIME zone 'utc' <= INTERVAL '3 HOURS') THEN event_count ELSE 0 END) AS rdr_count FROM audit.audit_data WHERE window_timestamp <= CURRENT_TIMESTAMP - INTERVAL '2 HOURS' AND window_timestamp >= CURRENT_TIMESTAMP - INTERVAL '1 MONTH' AND event_count > 0 GROUP BY window_timestamp, topic_name HAVING SUM(CASE WHEN location = 'kafka_windows' THEN event_count ELSE 0 END) > 0

35. Is it the end of the day yet?

36.

37. Is is the end of the day yet?? 1. Data arrival rate for the entire scope? 3. Arrival rate for the last window? 2. Number of audit windows for the entire scope?

38. Some alerts and add-ons

39. Some alerts and add-ons ● Alert granularity for different types of failures ○ Region ○ Topic ○ Server ● Detecting duplications ● More locations!

40.

41.

42. CREDITS: This presentation template was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik

43. Thank you! @simonameriam https://il.linkedin.com/in/simona-meriam-9468885

Auditing your data and answering the life long question, is it the end of the day yet? with Simona Meriam | Kafka Summit London 2022

Recommended

Recommended

More Related Content

Similar to Auditing your data and answering the life long question, is it the end of the day yet? with Simona Meriam | Kafka Summit London 2022

Similar to Auditing your data and answering the life long question, is it the end of the day yet? with Simona Meriam | Kafka Summit London 2022 (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Auditing your data and answering the life long question, is it the end of the day yet? with Simona Meriam | Kafka Summit London 2022