Stream upload and asynchronous job processing in large scale systems

•Download as PPTX, PDF•

0 likes•1,061 views

This document discusses an asynchronous job processing system developed by VNG for Zalo. It allows for parallel stream uploads and background processing of large amounts of data. The key points are: 1. It uses a job server and distributed worker model to process jobs asynchronously in a reliable, scalable, and high performance manner. 2. Jobs are collected, then processed by workers and responded to. The system supports both single and batch jobs to efficiently handle large volumes of uploads. 3. It was implemented using C/C++ for high performance and includes features like load balancing, failover, recovery from failures, and a job state system to reliably process all jobs.

Technology Business

Stream Upload And
Asynchronous Job
Processing System
Lê Bá Minh – minhlb@vng.com.vn
Technical Manager – Zalo Team - VNG

Agenda
• 1/ Why we need an Asynchronous Job Processing
System?
• 2/ How it works ?
• 3/ Application
• 4/ Q &A

Parallel Stream Upload
• Data is separated in chunks

Facts
• Zalo Stream Upload
• Background continuous Voice Upload
• Background Image upload
• …
• Facts (now)
• 1M voices /day
• 800K images /day
• Peak: 500 Chunks/second
• Expect:
• Scalable (more than 5000 chunks/second)
• High performance

What we need
• Asynchronous Job processing System
Collect Data
Processing Data
Response
Collect Data
Processing DataResponse
Workers

What we need
• Asynchronous Job processing System
• Batch Job
• Big data job
• High Reliable: No job missed
• Distributed job processing workers
• High performance
• Persistent
• Load balancing, Failed over, Recoverable

Open-source solutions
• Share-memory workers
• All workers in one physical server
• No fail-over
• Un-scalable
• Gearman
• Good but not completely fit our requirement
• No Batch Job support
• Not full reliable (lost job)
• Not full load-balance
• Un-stable if more than 2000 jobs/second

Zalo Asyn Job Processing
System
Client
Client
Worker 1
Worker 2
Worker 3
Z Database
Short Connection
Long Connection
TCP
TCP
Worker
Manager
Job
Caching
Job
Manager
Persistent
Manager
Job
Clean-Up
Job Server
TCP
TCP
TCP

Implementation
• C/C++ for Job Server
• C/C++, Java for client and workers
• Binary Protocol
• Z-Database

Job State
Queuing
Processing
Failed Time Out
Finished
Deliver to Worker
Worker ACK Failed
Worker ACK Finished
No ACK
Started

Job Type
• Single Job
• Simple task
• Immediately deliver
• Batch Job
• Multiple tasks
• Deliver when received all tasks

Deployment
Job Server 1
Job Server 2
Synchronized
Business Server
Worker 1
Worker 2
Worker 3

Applications
• Using for all Asynchronous job processing in Zalo: voice
upload, image upload, feed processing…
• Benchmark (single server)
• 50K images/seconds (640x480)
• 50k voices/seconds (30s)
• Advantages
• Batch Jobs
• Never lost job
• Worker can restart or stop any time
• Fail-over, Load Balancing, Quick recover in failure
• Issue
• Job duplication (handled by worker)

Stream upload and asynchronous job processing in large scale systems

What's hot

When the connection fails

Barbara Fusinska

A Bird and the Web

Mind The Firebird

Building rich interface components with SharePoint

Louis-Philippe Lavoie

Virtual Reference

SharonYang

MobileClient

rpatil82

PLNOG 13: Grzegorz Janoszka: Peering vs Tranzyt – Czy peering jest naprawdę s...

PROIDEA

Wa mw 2013

Nina McHale

What's hot (7)

When the connection fails

A Bird and the Web

Building rich interface components with SharePoint

Virtual Reference

MobileClient

PLNOG 13: Grzegorz Janoszka: Peering vs Tranzyt – Czy peering jest naprawdę s...

Wa mw 2013

Similar to Stream upload and asynchronous job processing in large scale systems

Management Data Warehouse

Matt Horn

Ahmed Jassat Oracle Customer Day Presentation at Monte Casino

Zahid02

Moving from Snapshot to Snapshot

sysnickm

Hands-on Performance Tuning Lab - Devoxx Poland

C2B2 Consulting

Data Care, Feeding, and Maintenance

Mercedes Coyle

Real time monitoring of hadoop and spark workflows

Shankar Manian

Maxis Alchemize imug 2017

BrandonWilhelm4

Hangfire An easy way to perform background processing in .NET and .NET Core applications. No Windows Service or separate process required. Why Background Processing? Lengthy operations like updating lot of records in DB Checking every 2 hours for new data or files Invoice generation at the end of every billing period Monthly Reporting Rebuild data, indexes or search-optimized index after data change Automatic subscription renewal Regular Mailings Send an email due to an action Background service provisioning

Background processing with hangfire

Aleksandar Bozinovski

Monitoring at Facebook - Ran Leibman, Facebook - DevOpsDays Tel Aviv 2015

DevOpsDays Tel Aviv

Building data intensive applications

Amit Kejriwal

Priority enabled wps

52North

DevOpsDays Austin: Helping Horses Become Unicorns, Chef's Operations Maturity...

Matt Ray

Scott Callaghan from the Southern California Earthquake Center presented this deck in a recent Blue Waters Webinar. "I will present an overview of scientific workflows. I'll discuss what the community means by "workflows" and what elements make up a workflow. We'll talk about common problems that users might be facing, such as automation, job management, data staging, resource provisioning, and provenance tracking, and explain how workflow tools can help address these challenges. I'll present a brief example from my own work with a series of seismic codes showing how using workflow tools can improve scientific applications. I'll finish with an overview of high-level workflow concepts, with an aim to preparing users to get the most out of discussions of specific workflow tools and identify which tools would be best for them." Watch the video: http://wp.me/p3RLHQ-gtH Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Overview of Scientific Workflows - Why Use Them?

inside-BigData.com

In Transit Images Drives Online Photography Business Forward with DAM

Extensis

ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server

Serdar Basegmez

EPUG UKI - Lancaster Analytics

jhkrug

Hadoop bangalore-meetup-dec-2011-yoda

InMobi

Engage 2013 - Leveraging Ad Hoc Analysis

Webtrends

Join a real uplift experience with Hagay Albo, the CTO of the Zap/Yellow Page Group in Israel, in which he explains how his team was able to take a legacy (slow and hard to modify) group of sites and make them easier to work with, MUCH faster and greatly simplified the operational environment. By prioritizing high availability, flexible data modeling and focusing on raw speed Zap was able to reduce its load times by Two Orders of Magnitudes. Using RavenDB as the core engine behind Zap's new sites had improved site traffic, reduced time to market and made it possible to implement the next-gen features that were previously beyond reach.

Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB

Oren Eini

Mentor Graphics Customer Presentation

Splunk

Similar to Stream upload and asynchronous job processing in large scale systems (20)

Management Data Warehouse

Ahmed Jassat Oracle Customer Day Presentation at Monte Casino

Moving from Snapshot to Snapshot

Hands-on Performance Tuning Lab - Devoxx Poland

Data Care, Feeding, and Maintenance

Real time monitoring of hadoop and spark workflows

Maxis Alchemize imug 2017

Background processing with hangfire

Monitoring at Facebook - Ran Leibman, Facebook - DevOpsDays Tel Aviv 2015

Building data intensive applications

Priority enabled wps

DevOpsDays Austin: Helping Horses Become Unicorns, Chef's Operations Maturity...

Overview of Scientific Workflows - Why Use Them?

In Transit Images Drives Online Photography Business Forward with DAM

ICONUK 2016: Back From the Dead: How Bad Code Kills a Good Server

EPUG UKI - Lancaster Analytics

Hadoop bangalore-meetup-dec-2011-yoda

Engage 2013 - Leveraging Ad Hoc Analysis

Zapping ever faster: how Zap sped up by two orders of magnitude using RavenDB

Mentor Graphics Customer Presentation

Recently uploaded

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

This talk focuses on the practical aspects of integrating various telephony systems with Salesforce, drawing on examples from implementations in the Czech scene. It aims to inform attendees about the spectrum of telephony solutions available, from small to large scale, and their compatibility with Salesforce. The presentation will highlight key considerations for selecting a telephony provider that integrates smoothly with Salesforce, including important questions to support the decision-making process. It will also discuss methods for integrating existing telephony systems with Salesforce, aimed at companies contemplating or in the process of adopting this CRM platform. The discussion is designed to provide a straightforward overview of the steps and considerations involved in telephony and Salesforce integration, with an emphasis on functionality, compatibility, and the practical experiences of Czech companies.

Integrating Telephony Systems with Salesforce: Insights and Considerations, B...

CzechDreamin

The standard Salesforce Approval process can be limiting in many ways, especially in complex scenarios. What if there was a way to implement very flexible approvals where one can use Apex code to make data updates in unrelated records, dynamically generate next steps details, and compute assignees on the fly? And still use UI-based configurations to implement concrete approval processes. In this session, we will share ideas behind such a solution and show a few lines of code to get you started.

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

CzechDreamin

Already know how to write a basic SOQL query? Great! But what about an *aggregate* SOQL query? You know, the kind that uses aggregate functions like COUNT & MAX along with GROUP BY and HAVING clauses? No? Well, get ready to learn how to slice & dice your org’s data right inside your own dev console. From finding duplicate records to prototyping summary & matrix reports, learn the ins and outs of aggregate queries during this fast-paced but admin-friendly session on advanced SOQL concepts.

SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...

CzechDreamin

New customer? New industry? New cloud? New team? A lot to handle! How to ensure the success of the project? Start it well! I've created the 3 areas of focus at the beginning of the project that helped me in multiple roles (BA, PO, and Consultant). Learn from real-world experiences and discover how these insights can empower you to deliver unparalleled value to your customers right from the project's start.

Powerful Start- the Key to Project Success, Barbara Laskowska

CzechDreamin

Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application. In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics. Length: 30 minutes Session Overview ------------------------------------------- During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana: - What out-of-the-box solutions are available for real-time monitoring JMeter tests? - What are the benefits of integrating InfluxDB and Grafana into the load testing stack? - Which features are provided by Grafana? - Demonstration of InfluxDB and Grafana using a practice web application To view the webinar recording, go to: https://www.rttsweb.com/jmeter-integration-webinar

JMeter webinar - integration with InfluxDB and Grafana

RTTS

What's New in Teams Calling, Meetings and Devices April 2024

Stephanie Beckett

Discover the essentials of performance testing in the IT sector with our concise guide. Learn about various testing types such as load, stress, endurance, spike, scalability, and volume testing. Understand key performance metrics like response time, throughput, CPU and memory utilization, and error rate. Explore top tools like Apache JMeter, LoadRunner, Gatling, Neoload, and BlazeMeter. Gain insights into best practices for defining objectives, creating realistic scenarios, automating tests, and optimizing performance to ensure user satisfaction, reliability, scalability, and cost efficiency. Ideal for developers, QA engineers, and IT professionals. Visit Expeed Software for more information. https://expeed.com/

In-Depth Performance Testing Guide for IT Professionals

Expeed Software

ODC, Data Fabric and Architecture User Group

CatarinaPereira64715

In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring. Learn about: • The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks. • Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective. • Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification. • Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process. Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Inflectra

IoT Analytics Company Presentation May 2024

IoTAnalytics

Generative AI architecture, at its core, revolves around the concept of machines being able to generate content autonomously, mimicking human-like creativity and decision-making processes. Unlike traditional AI systems that rely on predefined rules and data inputs, generative AI leverages deep learning techniques to produce new, original outputs based on patterns and examples it has learned from vast datasets. This capability opens up a multitude of possibilities across various domains within an enterprise.

The architecture of Generative AI for enterprises.pdf

alexjohnson7307

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

I'm excited to share my latest predictions on how AI, robotics, and other technological advancements will reshape industries in the coming years. The slides explore the exponential growth of computational power, the future of AI and robotics, and their profound impact on various sectors. Why this matters: The success of new products and investments hinges on precise timing and foresight into emerging categories. This deck equips founders, VCs, and industry leaders with insights to align future products with upcoming tech developments. These insights enhance the ability to forecast industry trends, improve market timing, and predict competitor actions. Highlights: ▪ Exponential Growth in Compute: How $1000 will soon buy the computational power of a human brain ▪ Scaling of AI Models: The journey towards beyond human-scale models and intelligent edge computing ▪ Transformative Technologies: From advanced robotics and brain interfaces to automated healthcare and beyond ▪ Future of Work: How automation will redefine jobs and economic structures by 2040 With so many predictions presented here, some will inevitably be wrong or mistimed, especially with potential external disruptions. For instance, a conflict in Taiwan could severely impact global semiconductor production, affecting compute costs and related advancements. Nonetheless, these slides are intended to guide intuition on future technological trends.

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl

Peter Udo Diehl

Screen flow is a powerful automation tool that is commonly designed for internal and external users. However, what about the guest users? We will dive into various methods of launching screen flows and understand how to make them publicly accessible, extending their usability to a broader audience. The presentation will also cover the implementation of security layers and highlight best practices for a smooth and protected user experience. Discover the potential of screen flows beyond conventional use and learn how to leverage them effectively.

Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade

CzechDreamin

The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.

Search and Society: Reimagining Information Access for Radical Futures

Bhaskar Mitra

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

How to differentiate Sales Cloud and CPQ on first glance might be tricky if you do not know where to look and what to look at. You will know :-) Managing the sales process within Salesforce is a common use case that can be managed with standart Sales Cloud. If you want to do entire quoting process you will find out Salesforce CPQ solution exists. What is then the difference if both can handle selling products? You will see comparison of 10 different features, which Sales Cloud and Salesforce CPQ handle differently. Simple question you will always remember if you should consider using Salesforce CPQ will be a cherry on top.

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

CzechDreamin

Ever caught yourself nodding along when someone mentions "delivering value" in Agile, but secretly wondering what the heck they actually mean? You're not alone! Join us for an eye-opening session where we'll strip away the buzzwords and dive into the heart of Agile—value delivery. But what is "value"? Is it a mythical unicorn in the world of software development, or is there more to this overused term? This isn't going to be a sit-and-get lecture. We're talking about a face-to-face, interactive meetup where YOU play a crucial role. Come along to: Define It: What does "value" really mean? We’ll build a definition that’s not just words, but a compass for your Agile journey. Contextualise It: Discover what value means specifically to you, your team, your company, and your industry. Because one size does not fit all. Deliver It: Share strategies and gather new ones for uncovering and delivering true value—no more shooting in the dark!

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx

David Michel

Recently uploaded (20)