Samza tech talk_2015 - strata

•Download as PPTX, PDF•

2 likes•1,227 views

This document discusses stream processing at scale using Samza at LinkedIn. It describes how LinkedIn uses Samza for stream processing tasks like tracking analytics from 1.4 trillion events per day and content standardization. Samza provides scalable, fault-tolerant stream processing by partitioning streams and replicating state across clusters of hundreds of nodes. The document outlines LinkedIn's large Samza deployments and upcoming features to improve local state storage, configuration, and development.

Software

Stream Processing @Scale in
LinkedIn
Yi Pan
Data Infrastructure
Samza Team @LinkedIn
Databus

• What is Stream Processing?
• What is Samza?
• Stream Processing @LinkedIn
• Upcoming features
Overview

• What’s stream processing
– Input: an unbounded sequence of events
• E.g. web server logs, user activity tracking events,
database changelogs, etc.
– Latency: near real-time
• From milliseconds to minutes, instead of hours to days
– Output: an unbounded sequence of changes to
the derived dataset
• The derived dataset is usually the final or partial
analytic results that can either be in another stream, or
a serving data store
Stream Processing

Response latency
Milliseconds to minutes
Synchronous Later. Possibly much later.
0 ms
Stream Processing

• What are the application requirements?
– Scalable, fast, stateful stream processing
– What scale should we operate at?
• Traffic Volume: 1.4 Trillion events/day
• Intermediate State Size: multi TB / colo (*)
– Why is it expensive to run stream processing at
scale?
• Intermediate data set needs to be stored to allow low
latency processing
• Large volume of data needs to be pulled and pushed
via network
Stream Processing

• Samza is a distributed Turing machine
– Single Task Samza Job is a stateful Turing
machine
What’s Samza
Samza Task
Input stream Output stream
State
changelog
checkpoint

– Scaling a Samza job: partition the streams
What’s SamzaInputstreamA
partition 0
partition 1
partition 2
partition 3
partition n

– Scaling a Samza job: partition the streams
What’s SamzaInputstreamB
partition 0
partition 1
partition 2
partition 3
partition n

– Scaling a Samza job: replicating the state
machine
What’s Samza
shared checkpoint
Job

• Samza Execution in Yarn
What’s Samza
Host 1 Host 2 Host 3
Application
Master
Samza container Samza container
Samza container
Deploy Samza job

• States in Samza
– Checkpoints
• Offsets per input stream partitions
– State Stores
• In-memory or on-disk (RocksDB) derived data set
What’s Samza
Samza Task
Output stream partitions
State
changelogpartitions
checkpoint Host 1

• States in Samza
– Checkpoints and local state stores are backed by
distributed logs
What’s Samza
Samza Task
Output stream partitions
State
changelogpartitions
checkpoint Host 1

Stream Processing @
LinkedIn
WebServers
WebServers
WebServers
WebServers
WebServers
WebServers
WebServersMonitor
Servers
Oracle
Espresso
Kafka Databus
Tracking
events
Metrics
changelog
changelog
Samza
Jobs
Samza
Jobs
Samza
Jobs
Samza
Jobs
bootstrap
bootstrap
Voldemort
Derived
Data Derived
Data

Stream Processing @
LinkedIn
• Tracking aggregate/analysis (ACG)

Stream Processing @
LinkedIn
• Kafka Deployment
– 1.1 Trillion messages / day
• Databus Deployment
– 300 Billion messages / day
• Samza Deployment
– multiple colos
– 10+ Yarn clusters
– 200+ nodes
– 100+ Jobs in production

• What is Stream Processing?
• What’s Samza
• Stream Processing @LinkedIn
• Upcoming features
Overview

• New features
– Local state store improvements
• RocksDB TTL support
• Fast recovery
– Dynamic configuration
– Easier deployment w/ standalone jobs
– High-level query language for faster
development
Upcoming Features

Contact Us / Get Involved
• Open Source
–Documentation: samza.apache.org
–Mailing list: dev@samza.apache.org
–JIRA:
https://issues.apache.org/jira/browse/SA
MZA

The Marketplace data team at Uber has built a scalable complex event processing platform to solve many challenging real time data needs for various Uber products. This platform has been in production for almost a year and it has proven to be very flexible to solve many use cases. In this talk, we will share in detail the design and architecture of the platform, and how we employ Samza, Kafka, and Siddhi at scale. This slides was presented at Stream Processing Meetup @ LinkedIn on June 15 2016.

Samza la hug

Sriram Subramanian

Air traffic controller - Streams Processing meetup

Ed Yakabosky

Event Stream Processing with Kafka and Samza

Zach Cox

Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber

confluent

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...

confluent

The Netflix Way to deal with Big Data Problems

Monal Daxini

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1eGbVJv. Chris Riccomini discusses: Samza's feature set, how Samza integrates with YARN and Kafka, how it's used at LinkedIn, and what's next on the roadmap. Filmed at qconsf.com. Chris Riccomini is a Staff Software Engineer at LinkedIn, where he's is currently working as a committer and PMC member for Apache Samza. He's been involved in a wide range of projects at LinkedIn, including, "People You May Know", REST.li, Hadoop, engineering tooling, and OLAP systems. Prior to LinkedIn, he worked on data visualization and fraud modeling at PayPal.

High cardinality time series search: A new level of scale - Data Day Texas 2016

Eric Sammer

Modern search systems provide incredible feature sets, developer-friendly APIs, and low latency indexing and query response. By some measures, these systems operate "at scale," but rarely is that quantified. Customers of Rocana typically look to push ingest rates in excess of 1 million events per second, retaining years of data online for query, with the expectation of sub-second response times for any reasonably sized subset of data. We quickly found that the tradeoffs made by general purpose search systems, while right for common use cases, were less appropriate for these high cardinality, large scale use cases. This session details the architecture, tradeoffs, and interesting implementation decisions made in building a new time series optimized distributed search system using Apache Lucene, Kafka, and HDFS. Data ingestion and durability, index and metadata organization, storage, query scheduling and optimization, and failure modes will be covered. Finally, a summary of the results achieved will be shown.

Easily Build a Smart Pulsar Stream Processor_Simon Crosby

StreamNative

For organizations with boundless data sources, it is important to analyze, learn, predict and even respond in real time – directly from streaming data. This is important when: •Data volumes are large, or moving raw data is expensive, •Data is generated by widely distributed assets (eg: mobile devices), •Data is of ephemeral value and analysis can’t wait, or •It is critical to always have the latest insight and extrapolation won’t do. Use cases include prediction of failures on assembly lines, prediction of traffic flows in cities, predicting demand placed in power grids, detection of hackers, and understanding connection quality in mobile networks. They are characterized by a need to know – now – and require real-time processing of streaming data. Our goal is to enable real-time stream processing for Apache Pulsar in which analysis, learning and prediction are done on-the-fly, with continuous insights streamed back to the broker.

Flink forward-2017-netflix keystones-paas

Monal Daxini

The need for gleaning answers from unbounded data streams is moving from nicety to a necessity. Netflix is a data driven company, and has a need to process over 1 trillion events a day amounting to 3 PB of data to derive business insights. To ease extracting insight, we are building a self-serve, scalable, fault-tolerant, multi-tenant "Stream Processing as a Service" platform so the user can focus on data analysis. I'll share our experience using Flink to help build the platform.

DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...

Hakka Labs

By Alan Gardner (Platform Engineer, Rocana) Rocana Ops is designed to handle terabytes a day of application logs and system metrics from across multiple data centres. We use Apache Kafka as a durable, high-throughput message bus at the centre of our application architecture. This talk will discuss the design and features of Kafka, its operational characteristics, and why we chose it as the backbone of our data pipeline.

Netflix Data Pipeline With Kafka

Allen (Xiaozhong) Wang

Will it Scale? The Secrets behind Scaling Stream Processing Applications

Navina Ramesh

Principles in Data Stream Processing | Matthias J Sax, Confluent

HostedbyConfluent

Data stream processing is, for many of us, a new paradigm with which you process data and build applications. In this talk, we will take you on a journey through the theoretical foundations of stream processing and discuss the underlying principles and unique problems that need to be addressed. What actually is a data stream anyway? And how do I use it? How do streams relate to application state and when do I use the one or the other? ksqlDB and Kafka Streams are both, at their core, designed to help build stream processing applications and we will explain how stream processing principles are reflected in the design of each system and what trade-offs were chosen (and - more importantly! - why). Finally, we take a look into the future how the stream processing space, and in particular ksqlDB and Kafka Streams, may evolve over the next few years as we outline extensions and improvements to the underlying conceptual model. So, bring your thinking hats and notepads and prepare to learn WHY these systems are the way they are!

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

Siphon is a highly available and reliable distributed pub/sub system built using Apache Kafka. It is used to publish, discover and subscribe to near real-time data streams for operational and product intelligence. Siphon is used as a “Databus” by a variety of producers and subscribers in Microsoft, and is compliant with security and privacy requirements. It has a built-in Auditing and Quality control. This session will provide an overview of the use of Kafka at Microsoft, and then deep dive into Siphon. We will describe an important business scenario and talk about the technical details of the system in the context of that scenario. We will also cover the design and implementation of the service, the scale, and real world production experiences from operating the service in the Microsoft cloud environment.

A Marriage of Lambda and Kappa: Supporting Iterative Development of an Event ...

confluent

With increasing data volumes typically comes a corresponding increase in (non-windowed) batch processing times, and many companies have looked to streaming as a way of delivering data processing results faster and more reliably. Event Driven Architectures further enhance these offerings by breaking centralised Data Platforms into loosely coupled and distributed solutions, supported by linearly scalable technologies such as Apache Kafka(TM) and Apache Kafka Streams(TM). However, there remains a problem of how to handle changes to operational systems: if a record is the result of business logic, and that business logic changes, what do we do? Do we recalculate everything on the fly, adding in additional latencies for all data requests and potentially breaching non-functional requirements? Or do we run a batch job, risking that incorrect data will be served whilst the job is running? This talk covers how 6point6 leveraged Kafka and Kafka Streams to transition a customer from a traditional business flow onto an Event Driven Architecture, with business logic triggered directly by real-time events across over 3000 loosely coupled business services, whilst ensuring that the active development of these services (and their containing logic and models) would not affect components which relied on data served by the platform. Learn how we: – Used versioning of topics, data and business logic to facilitate iterative development, ensuring that the reprocessing of large volumes of data would not result in incorrect or stale data being delivered. – Handled distributed versioning of JSON event messages between separate teams/services, using discovery, automated contract negotiation and version porting. – Developed technical patterns and libraries to allow rapid development and deployment of new event driven services using Kafka Streams. – Developed functionality and approaches for deploying defensive services, including strategies for event retry and failure.

Robust stream processing with Apache Flink

Aljoscha Krettek

Aljoscha Krettek offers a very short introduction to stream processing before diving into writing code and demonstrating the features in Apache Flink that make truly robust stream processing possible, with a focus on correctness and robustness in stream processing. All of this will be done in the context of a real-time analytics application that we’ll be modifying on the fly based on the topics we’re working though, as Aljoscha exercises Flink’s unique features, demonstrates fault recovery, clearly explains why event time is such an important concept in robust, stateful stream processing, and covers the features you need in a stream processor to do robust, stateful stream processing in production. We’ll also use a real-time analytics dashboard to visualize the results we’re computing in real time, allowing us to easily see the effects of the code we’re developing as we go along. Topics include: * Apache Flink * Stateful stream processing * Event time versus processing time * Fault tolerance * State management in the face of faults * Savepoints * Data reprocessing

Apache Spark Streaming - www.know bigdata.com

knowbigdata

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

confluent

Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN

blueboxtraveler

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

confluent

Zurich Flink Meetup

Konstantinos Kloudas

Kappa Architecture on Apache Kafka and Querona: datamass.io

Piotr Czarnas

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Monal Daxini

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

Data pipeline with kafka

Mole Wong

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

ApacheCon BigData - What it takes to process a trillion events a day?

Jagadish Venkatraman

Cloud Security Monitoring and Spark Analytics

amesar0

What's hot

Samza: Real-time Stream Processing at LinkedIn

C4Media

High cardinality time series search: A new level of scale - Data Day Texas 2016

Eric Sammer

Easily Build a Smart Pulsar Stream Processor_Simon Crosby

StreamNative

Flink forward-2017-netflix keystones-paas

Monal Daxini

DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...

Hakka Labs

Netflix Data Pipeline With Kafka

Allen (Xiaozhong) Wang

Will it Scale? The Secrets behind Scaling Stream Processing Applications

Navina Ramesh

Principles in Data Stream Processing | Matthias J Sax, Confluent

HostedbyConfluent

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

A Marriage of Lambda and Kappa: Supporting Iterative Development of an Event ...

confluent

Robust stream processing with Apache Flink

Aljoscha Krettek

Apache Spark Streaming - www.know bigdata.com

knowbigdata

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

confluent

Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN

blueboxtraveler

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

confluent

Zurich Flink Meetup

Konstantinos Kloudas

Kappa Architecture on Apache Kafka and Querona: datamass.io

Piotr Czarnas

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Monal Daxini

Data pipeline with kafka

Mole Wong

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

What's hot (20)

Samza: Real-time Stream Processing at LinkedIn

High cardinality time series search: A new level of scale - Data Day Texas 2016

Easily Build a Smart Pulsar Stream Processor_Simon Crosby

Flink forward-2017-netflix keystones-paas

DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...

Netflix Data Pipeline With Kafka

Will it Scale? The Secrets behind Scaling Stream Processing Applications

Principles in Data Stream Processing | Matthias J Sax, Confluent

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

A Marriage of Lambda and Kappa: Supporting Iterative Development of an Event ...

Robust stream processing with Apache Flink

Apache Spark Streaming - www.know bigdata.com

Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming

Apache Samza: Reliable Stream Processing Atop Apache Kafka and Hadoop YARN

Hadoop made fast - Why Virtual Reality Needed Stream Processing to Survive

Zurich Flink Meetup

Kappa Architecture on Apache Kafka and Querona: datamass.io

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015

Data pipeline with kafka

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

Similar to Samza tech talk_2015 - strata

ApacheCon BigData - What it takes to process a trillion events a day?

Jagadish Venkatraman

Cloud Security Monitoring and Spark Analytics

amesar0

Play With Streams

Tianjian Chen

This tutorial gives out an brief and interesting introduction to modern stream computing technologies. The participants can learn the essential concepts and methodologies for designing and building a advanced stream processing system. The tutorial unveils the key fundamentals behind various kinds of design choices. Some forecast of technology developments in this domain is also introduced at the last section of this tutorial.

Unified Batch & Stream Processing with Apache Samza

DataWorks Summit

The traditional lambda architecture has been a popular solution for joining offline batch operations with real time operations. This setup incurs a lot of developer and operational overhead since it involves maintaining code that produces the same result in two, potentially different distributed systems. In order to alleviate these problems, we need a unified framework for processing and building data pipelines across batch and stream data sources. Based on our experiences running and developing Apache Samza at LinkedIn, we have enhanced the framework to support: a) Pluggable data sources and sinks; b) A deployment model supporting different execution environments such as Yarn or VMs; c) A unified processing API for developers to work seamlessly with batch and stream data. In this talk, we will cover how these design choices in Apache Samza help tackle the overhead of lambda architecture. We will use some real production use-cases to elaborate how LinkedIn leverages Apache Samza to build unified data processing pipelines. Speaker Navina Ramesh, Sr. Software Engineer, LinkedIn

Distributed monitoringLeon Torres

Apache samza past, present and future

Ed Yakabosky

Flink Streaming @BudapestData

Gyula Fóra

Apache Spark Components

Girish Khanzode

BDA403 The Visible Network: How Netflix Uses Kinesis Streams to Monitor Appli...

Amazon Web Services

Thousands of services work in concert to deliver millions of hours of video streams to Netflix customers every day. These applications vary in size, function, and technology, but they all make use of the Netflix network to communicate. Understanding the interactions between these services is a daunting challenge both because of the sheer volume of traffic and the dynamic nature of deployments. In this talk, we’ll first discuss why Netflix chose Amazon Kinesis Streams over other data streaming solutions like Kafka to address these challenges at scale. We’ll then dive deep into how Netflix uses Amazon Kinesis Streams to enrich network traffic logs and identify usage patterns in real time. Lastly, we will cover how Netflix uses this system to build comprehensive dependency maps, increase network efficiency, and improve failure resiliency. From this talk, you’ll take away techniques and processes that you can apply to your large-scale networks and derive real-time, actionable insights.

John adams talk cloudy

John Adams

Introduction to Apache Apex

Apache Apex

AWS re:Invent 2016: JustGiving: Serverless Data Pipelines, Event-Driven ETL, ...

Amazon Web Services

Organizations need to gain insight and knowledge from a growing number of Internet of Things (IoT), application programming interfaces (API), clickstreams, unstructured and log data sources. However, organizations are also often limited by legacy data warehouses and ETL processes that were designed for transactional data. Building scalable big data pipelines with automated extract-transform-load (ETL) and machine learning processes can address these limitations. JustGiving is the world’s largest social platform for online giving. In this session, we describe how we created several scalable and loosely coupled event-driven ETL and ML pipelines as part of our in-house data science platform called RAVEN. You learn how to leverage AWS Lambda, Amazon S3, Amazon EMR, Amazon Kinesis, and other services to build serverless, event-driven, data and stream processing pipelines in your organization. We review common design patterns, lessons learned, and best practices, with a focus on serverless big data architectures with AWS Lambda.

Real-time Data Processing Using AWS Lambda

Amazon Web Services

Shared Personalization Service - How To Scale to 15K RPS, Patrice PellandFuenteovejuna

Aerospike Hybrid Memory Architecture

Aerospike, Inc.

Learn how Aerospike's Hybrid Memory Architecture brings transactions and analytics together to power real-time Systems of Engagement ( SOEs) for companies across AdTech, financial services, telecommunications, and eCommerce. We take a deep dive into the architecture including use cases, topology, Smart Clients, XDR and more. Aerospike delivers predictable performance, high uptime and availability at the lowest total cost of ownership (TCO).

Apache Samza Past, Present and Future

Kartik Paramasivam

Discretized Stream - Fault-Tolerant Streaming Computation at Scale - SOSP

Tathagata Das

Netflix Keystone Pipeline at Samza Meetup 10-13-2015

Monal Daxini

Beam me up, Samza!

Xinyu Liu

Next Gen Big Data Analytics with Apache Apex

DataWorks Summit/Hadoop Summit

Similar to Samza tech talk_2015 - strata (20)

ApacheCon BigData - What it takes to process a trillion events a day?

Cloud Security Monitoring and Spark Analytics

Play With Streams

Unified Batch & Stream Processing with Apache Samza

Distributed monitoring

Apache samza past, present and future

Flink Streaming @BudapestData

Apache Spark Components

BDA403 The Visible Network: How Netflix Uses Kinesis Streams to Monitor Appli...

John adams talk cloudy

Introduction to Apache Apex

AWS re:Invent 2016: JustGiving: Serverless Data Pipelines, Event-Driven ETL, ...

Real-time Data Processing Using AWS Lambda

Shared Personalization Service - How To Scale to 15K RPS, Patrice Pelland

Aerospike Hybrid Memory Architecture

Apache Samza Past, Present and Future

Discretized Stream - Fault-Tolerant Streaming Computation at Scale - SOSP

Netflix Keystone Pipeline at Samza Meetup 10-13-2015

Beam me up, Samza!

Next Gen Big Data Analytics with Apache Apex

Recently uploaded

Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better

XfilesPro

Globus Connect Server Deep Dive - GlobusWorld 2024

Globus

Graphic Design Crash Course for beginners

e20449

Large Language Models and the End of Programming

Matt Welsh

Lecture 1 Introduction to games development

abdulrafaychaudhry

Cyaniclab : Software Development Agency Portfolio.pdf

Cyanic lab

CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.

Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx

ShamsuddeenMuhammadA

APIs for Browser Automation (MoT Meetup 2024)

Boni García

Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL

Natan Silnitsky

In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey. Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience. Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system. Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.

Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...

informapgpstrackings

Quarkus Hidden and Forbidden Extensions

Max Andersen

Introduction to Pygame (Lecture 7 Python Game Development)

abdulrafaychaudhry

Enhancing Research Orchestration Capabilities at ORNL.pdf

Globus

Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...

Globus

The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.

Top 7 Unique WhatsApp API Benefits | Saudi Arabia

Yara Milbes

Discover the transformative power of the WhatsApp API in our latest SlideShare presentation, "Top 7 Unique WhatsApp API Benefits." In today's fast-paced digital era, effective communication is crucial for both personal and professional success. Whether you're a small business looking to enhance customer interactions or an individual seeking seamless communication with loved ones, the WhatsApp API offers robust capabilities that can significantly elevate your experience. In this presentation, we delve into the top 7 distinctive benefits of the WhatsApp API, provided by the leading WhatsApp API service provider in Saudi Arabia. Learn how to streamline customer support, automate notifications, leverage rich media messaging, run scalable marketing campaigns, integrate secure payments, synchronize with CRM systems, and ensure enhanced security and privacy.

Pro Unity Game Development with C-sharp Book

abdulrafaychaudhry

A Sighting of filterA in Typelevel Rite of Passage

Philip Schwarz

Providing Globus Services to Users of JASMIN for Environmental Data Analysis

Globus

JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.

Globus Compute Introduction - GlobusWorld 2024

Globus

Cracking the code review at SpringIO 2024

Paco van Beckhoven

Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production. Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process? In this session we will cover: - The Art of Effective Code Reviews - Streamlining the Review Process - Elevating Reviews with Automated Tools By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces

Recently uploaded (20)

Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better

Globus Connect Server Deep Dive - GlobusWorld 2024

Graphic Design Crash Course for beginners

Large Language Models and the End of Programming

Lecture 1 Introduction to games development

Cyaniclab : Software Development Agency Portfolio.pdf

Text-Summarization-of-Breaking-News-Using-Fine-tuning-BART-Model.pptx

APIs for Browser Automation (MoT Meetup 2024)

Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL

Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...

Quarkus Hidden and Forbidden Extensions

Introduction to Pygame (Lecture 7 Python Game Development)

Enhancing Research Orchestration Capabilities at ORNL.pdf

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...

Top 7 Unique WhatsApp API Benefits | Saudi Arabia

Pro Unity Game Development with C-sharp Book

A Sighting of filterA in Typelevel Rite of Passage

Providing Globus Services to Users of JASMIN for Environmental Data Analysis

Globus Compute Introduction - GlobusWorld 2024

Cracking the code review at SpringIO 2024

Samza tech talk_2015 - strata

1. Stream Processing @Scale in LinkedIn Yi Pan Data Infrastructure Samza Team @LinkedIn Databus

2. • What is Stream Processing? • What is Samza? • Stream Processing @LinkedIn • Upcoming features Overview

3. • What’s stream processing – Input: an unbounded sequence of events • E.g. web server logs, user activity tracking events, database changelogs, etc. – Latency: near real-time • From milliseconds to minutes, instead of hours to days – Output: an unbounded sequence of changes to the derived dataset • The derived dataset is usually the final or partial analytic results that can either be in another stream, or a serving data store Stream Processing

4. Response latency Milliseconds to minutes Synchronous Later. Possibly much later. 0 ms Stream Processing

5. • What are the application requirements? – Scalable, fast, stateful stream processing – What scale should we operate at? • Traffic Volume: 1.4 Trillion events/day • Intermediate State Size: multi TB / colo (*) – Why is it expensive to run stream processing at scale? • Intermediate data set needs to be stored to allow low latency processing • Large volume of data needs to be pulled and pushed via network Stream Processing

6. • What is Stream Processing? • What is Samza? • Stream Processing @LinkedIn • Upcoming features Overview

7. • Samza is a distributed Turing machine – Single Task Samza Job is a stateful Turing machine What’s Samza Samza Task Input stream Output stream State changelog checkpoint

8. – Scaling a Samza job: partition the streams What’s SamzaInputstreamA partition 0 partition 1 partition 2 partition 3 partition n

9. – Scaling a Samza job: partition the streams What’s SamzaInputstreamB partition 0 partition 1 partition 2 partition 3 partition n

10. – Scaling a Samza job: replicating the state machine What’s Samza shared checkpoint Job

11. • Samza Execution in Yarn What’s Samza Host 1 Host 2 Host 3 Application Master Samza container Samza container Samza container Deploy Samza job

12. • States in Samza – Checkpoints • Offsets per input stream partitions – State Stores • In-memory or on-disk (RocksDB) derived data set What’s Samza Samza Task Output stream partitions State changelogpartitions checkpoint Host 1

13. • States in Samza – Checkpoints and local state stores are backed by distributed logs What’s Samza Samza Task Output stream partitions State changelogpartitions checkpoint Host 1

14. • What is Stream Processing? • What is Samza? • Stream Processing @LinkedIn • Upcoming features Overview

15. Stream Processing @ LinkedIn WebServers WebServers WebServers WebServers WebServers WebServers WebServersMonitor Servers Oracle Espresso Kafka Databus Tracking events Metrics changelog changelog Samza Jobs Samza Jobs Samza Jobs Samza Jobs bootstrap bootstrap Voldemort Derived Data Derived Data

16. Stream Processing @ LinkedIn • Tracking aggregate/analysis (ACG)

17. Stream Processing @ LinkedIn • Content standardization w/ adjunct data set Member Profile DB Bootstrap Job Databus Kafka Content Standardization Kafka Kafka

18. Stream Processing @ LinkedIn • Kafka Deployment – 1.1 Trillion messages / day • Databus Deployment – 300 Billion messages / day • Samza Deployment – multiple colos – 10+ Yarn clusters – 200+ nodes – 100+ Jobs in production

19. • What is Stream Processing? • What’s Samza • Stream Processing @LinkedIn • Upcoming features Overview

20. • New features – Local state store improvements • RocksDB TTL support • Fast recovery – Dynamic configuration – Easier deployment w/ standalone jobs – High-level query language for faster development Upcoming Features

21. Contact Us / Get Involved • Open Source –Documentation: samza.apache.org –Mailing list: dev@samza.apache.org –JIRA: https://issues.apache.org/jira/browse/SA MZA

Editor's Notes

Rpc – bulk of what we do, we expect immediate response (web page, bunch of requests sent) Other extreme is batch processing, typically done in hadoop (order of hours if not days) Samza fits in middle, async, relatively quickly, Order of ms to minute stream processing for us = anything asynchronous, but not batch computed.

Samza tech talk_2015 - strata

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Samza tech talk_2015 - strata

Similar to Samza tech talk_2015 - strata (20)

Recently uploaded

Recently uploaded (20)

Samza tech talk_2015 - strata

Editor's Notes