2015-10-20 Processed Billions of job by using Beanstalkd Queue

•Download as PPTX, PDF•

1 like•981 views

This document discusses AfterShip's use of Beanstalkd, an open source queue system. AfterShip uses Beanstalkd to process billions of jobs per day for shipment tracking across 290+ carriers. Beanstalkd provides first-in, first-out processing of jobs and supports features like priority, delaying jobs, and load balancing across 4 Beanstalkd servers with very high uptime. The document also compares Beanstalkd to other queue systems like Amazon SQS and outlines the typical lifecycle of a job within Beanstalkd, from putting a job onto the queue to reserving, deleting, burying, kicking, and delaying jobs.

Technology

teddy@aftership.com
Teddy Chan
CEO & Co-Founder

Shipment tracking API
Supporting 290 310+ carriers

Free Shipping APIs
Single API for 5 19 carriers

Beanstalkd
The queue system processed billions of job

Queue
- Buffer
- Load Balancing
- Scale Horizontal
- Async process

Beanstalkd
How AfterShip use queue?
- Ten of millions of job per day
- Billions of job so far
- 4 beanstalkd servers (AWS m3.large)
- 4 down incidents in last 3 years

Beanstalkd Protocol
Characteristics
- Guarantee FIFO
- Each job is uniquely assigned to one worker.
- Each connection can only reserve one job at a time
- Support jobs persistent (“-b” option)
- Support priority, delay, ttr (time to run), bury
- No HA

Queue systems
References:
http://queues.io/
http://www.sitepoint.com/message-queues-comparing-beanstalkd-ironmq-amazon-sqs/
Beanstalkd Amazon SQS
Server Setup Self Host Full Managed
Response Time single digit ms hundred ms
First In First Out (FIFO) Guarantee & unique Does not guarantee
Priority Yes No
Delay Yes Yes
Pub / Sub TCP pub/sub https polling

Beanstalkd Protocol
Action you can do
Status Description
put (with delay) Add a new job in the queue
reserve Take the job from queue
delete Delete the job after reserved.
bury Bury the job after reserved.
kick Kick the job from buried status to ready status.
release (with delay) Release the job back to ready status, after reserved it.
The job id, pri, will be same.

Beanstalkd Protocol
Typical life cycle for a job
put
--------> [READY] -------> [RESERVED]-------> *proof*
Ready
Tube
Job 1
put

Beanstalkd Protocol
Put
put
--------> [READY] -------> [RESERVED]-------> *proof*
Ready
Tube
Job 1

Beanstalkd Protocol
Reserve
put reserve
--------> [READY] -------> [RESERVED]-------> *proof*
Ready
Tube
Job 1
reserve

Beanstalkd Protocol
Delete
put reserve delete
--------> [READY] -------> [RESERVED]-------> *proof*
Ready
Tube
Job 1
delete

Beanstalkd Protocol
Bury
put reserve bury
--------> [READY] -------> [RESERVED]-------> [BURIED]
Ready
Tube
Job 1
bury
Buried
Tube

Beanstalkd Protocol
Kick
put reserve bury
--------> [READY] -------> [RESERVED]-------> [BURIED]
KICK
Ready
Tube
kick
Buried
Tube
Job 1

Beanstalkd Protocol
Delay
Job 1
Job 2
Ready Job
Delay Job (with 3s)

Beanstalkd Protocol
Delay
Job 1
Job 2
Ready Job
Delay Job (After 3s)

Beanstalkd Protocol
Full picture of the life cycle

Beanstalkd
So how much?
- 15 servers @ USD0.05 / hr
- 2 workers @ 0.05 USD per hr
- some traffic… $0.12 / GB

Beanstalkd
So how much?
- 15 servers @ USD0.05 / hr
- 2 workers @ 0.05 USD per hr
- some traffic… $0.12 / GB
- $0.97 in total

This document discusses refactoring the installer modules for Katello. It provides an overview of Foreman and its installers, including the Kafo library used. It then discusses refactoring the modules to make them more reusable outside of the installer for higher quality and community usage. It highlights some specific modules that have been refactored like puppet-certs or need more work like puppet-katello. The goal is to allow deployment on separate machines and improve usability and quality.

Arnold Bechtoldt, Inovex GmbH Linux systems engineer - Configuration Manageme...

SaltStack

Foreman - Advanced use cases - Timo Goebel

NETWAYS

High availability for puppet - 2016

Zack Smith

This session will be an overview of highly available components that can be deployed with Puppet Enterprise. It will focus on some of the current Beta support in PuppetDB as well as tips and tricks from the professional services department. The session will cover field solutions ( both supported and unsupported ) that allow architectures to be designed that align with different levels of high availability across the services that support running puppet on agent nodes during an outage of your primary puppet infrastructure.

Provisioning on Libvirt with Foreman

Nikhil Kathole

A user's perspective on SaltStack and other configuration management tools

SaltStack

Network Automation with Salt and NAPALM: a self-resilient network

APNIC

This document discusses using Salt and NAPALM for network automation. Salt is used as an orchestrator to manage network devices at scale through NAPALM, which provides vendor-agnostic APIs and drivers. Key points include: - Salt and NAPALM allow Cloudflare to automate tasks like deploying new network locations, reducing human errors and speeding recovery. - NAPALM integrates with Salt to provide vendor-agnostic methods for tasks like configuration management, CLI execution, and collecting operational data from devices. - Together Salt and NAPALM allow Cloudflare to manage thousands of devices through a single framework, deploy configurations consistently, and monitor network performance through

Contrail at AllegroGroup

Krzysztof Kowalik

This document summarizes AllegroGroup's implementation of an OpenContrail network fabric to replace separate cloud environments with network complexity. They deployed OpenContrail version 1.20 across two regions with puppet manifests for ease of maintenance and growth. Implementation involved migrating tenants between regions while addressing problems like Cassandra tuning, collector flooding, and flow issues. Their environment now supports 1350 VMs across 150 hosts in one data center and they plan further expansion and upgrades.

The document discusses Foreman and Puppet for deploying OpenStack. It provides an introduction to Foreman and Puppet, and describes how they can be used together to automatically deploy and configure OpenStack on bare metal servers. The Foreman server acts as a Puppet master and PXE server, installing Ubuntu 14.04 on the bare metal nodes and configuring them using Puppet modules for OpenStack components. It outlines the network requirements and provides an example Puppet manifest for configuring OpenStack networks, services, and nodes.

First hop redundancy

Mohamed Gamel

This document compares three first-hop redundancy protocols: HSRP, VRRP, and GLBP. HSRP and VRRP provide default gateway redundancy with one active and one standby router, while GLBP additionally supports load balancing across gateways. Key differences include that HSRP is Cisco proprietary, VRRP is an open standard, and GLBP is Cisco proprietary but supports load balancing. All three protocols elect an active router and use multicast for communication.

Linux host orchestration with Foreman, Puppet and Gitlab

Ben Tullis

A brief look at the Foreman host lifecycle management system, beginning with its rapid provisioning features and moving onto its integration with the Puppet configuration management system. GItlab is introduced to the mix and an example is given of how it can be integrated with Forman and Puppet to form an on-premise configuration versioning component. This configuration, which builds upon the Puppet multiple environments feature, is currently being employed in the task of building a test-driven continuous delivery system for the OpenCorporates project.

Apache Provisionr (incubating) - Bucharest JUG 10

Andrei Savu

Openstack il2014 staypuft- your friendly foreman openstack installer

Arthur Berezin

Project Staypuft is a plugin for the Foreman OpenStack Installer that provides an intuitive deployment tool for production grade OpenStack installations. It includes features such as a live CD/USB, host auto-discovery via PXE, and an OpenStack-oriented user interface to orchestrate deployments through steps like first deploying controller nodes and then compute nodes. The goal is to simplify the complex OpenStack installation process that typically requires manual configuration and troubleshooting of packages.

OpenNebula, the foreman and CentOS play nice, too

inovex GmbH

This document discusses setting up a private cloud using OpenNebula and the Foreman. It begins with an introduction and agenda. It then covers installing CentOS, setting up a local YUM repository using Pulp, installing the Foreman for bare metal provisioning, and using Puppet modules. It demonstrates deploying OpenNebula nodes using the Foreman and provides an overview of accessing the new cloud. It notes there are some rough edges to address but the modules are minor. It concludes by thanking the audience and providing contact information.

Connecting AWS and Katello/The Foreman

Lukas Kallies

Configuration Management with Saltstack

inovex GmbH

This document discusses configuration management with SaltStack. It begins with an introduction to configuration management systems and their traditional limitations. It then provides an overview of SaltStack fundamentals, including why SaltStack is useful and flexible. The document dives into details of SaltStack architecture, terminology, targeting, states, pillars, templates and more. It concludes that SaltStack can help manage large infrastructures and make customers happy.

PuppetConf 2016: High Availability for Puppet – Russ Mull & Zack Smith, Puppet

Puppet

Managing your SaltStack Minions with Foreman

Stephen Benjamin

Jenkins as Code

CloudOps2005

Full Stack Automation with Katello & The Foreman

Weston Bassler

This document summarizes a presentation about FullStackAutomationwithKatelloandForeman. It introduces the presenters Justin Miller and Weston Bassler and their backgrounds. The presentation covers the Foreman project, Katello project, and how they work together. Foreman is an open source tool for provisioning, configuring, and managing servers. Katello adds content management capabilities like repositories and subscriptions. The document outlines key features of provisioning, configuration, repositories, subscriptions, and more. It includes links to demo videos showcasing functions like host discovery, bulk actions, and Docker integration.

Containerize ovs ovn components

Aliasgar Ginwala

The document discusses containerizing OVS/OVN components to make them easier to manage, move around, and run on Kubernetes. Containerizing them optimizes development efforts by removing the need for kernel modules in OVN and not tying OVN container components to the host OS. It also allows running OVS/OVN as pods in Kubernetes through combining multiple containers in the same pod. The document demonstrates building, pushing and using containerized OVS/OVN and references next steps like running the OVN control plane in a container cluster mode with SSL support.

Configuration and lifecycle in Mixed environments

Dmitry Kireev

PuppetCamp Sydney 2012 - Building a Multimaster Environment

Greg Cockburn

This document discusses a solution for providing Puppet services globally across multiple regions with poor WAN connectivity. The solution involves building a "Puppeteer" master that acts as a central point of entry for code updates and certificate management. It ensures Puppet masters in each region are in sync. LDAP is used as an external node classifier to provide node definitions across regions. The Puppet file server replicates configuration between masters. F5 load balancers route clients to the nearest master and provide high availability if any master fails. Workflows for adding new servers and masters are also summarized.

How to Save, backup and restore IOS on Cisco router

tcpipguru

This document provides instructions for saving, backing up, and restoring configurations on a Cisco 1751 router. It describes how to save the running configuration to the startup configuration, backup the running configuration to a TFTP server, and restore a running configuration stored on a TFTP server. The TFTP server IP address is 192.168.1.15. To backup the configuration, the router copies the running configuration file to the TFTP server and names it "run-config backup". To restore, it copies the "run-config" file from the TFTP server to the router's running configuration.

[Js hcm] Deploying node.js with Forever.js and nginx

Nicolas Embleton

[Intuit] Control Everything

Perforce

Deploying Foreman in Enterprise Environments

inovex GmbH

Beanstalkg

Vimukthi Wickramasinghe

This document provides an overview of BeanstalkG, an open source work queue system written in Go that is based on the Beanstalkd work queue but aims to add additional features like high availability. It discusses the motivation for creating BeanstalkG due to issues with lack of high availability in Beanstalkd. It then covers key aspects of work queues, examples of existing work queue systems, the state model and commands of Beanstalkd, and the architecture and workflow of BeanstalkG including how producers, workers, tubes and backends are handled. It concludes with a demo, roadmap which includes implementing high availability, and a call for contributions.

Scaling mysql with python (and Docker).

Roberto Polli

What's hot

Foreman in your datacenter

lzap

Foreman-and-Puppet-for-Openstack-Audo-Deployment

yating yang

First hop redundancy

Mohamed Gamel

Linux host orchestration with Foreman, Puppet and Gitlab

Ben Tullis

Apache Provisionr (incubating) - Bucharest JUG 10

Andrei Savu

Openstack il2014 staypuft- your friendly foreman openstack installer

Arthur Berezin

OpenNebula, the foreman and CentOS play nice, too

inovex GmbH

Connecting AWS and Katello/The Foreman

Lukas Kallies

Configuration Management with Saltstack

inovex GmbH

PuppetConf 2016: High Availability for Puppet – Russ Mull & Zack Smith, Puppet

Puppet

Managing your SaltStack Minions with Foreman

Stephen Benjamin

Jenkins as Code

CloudOps2005

Full Stack Automation with Katello & The Foreman

Weston Bassler

Containerize ovs ovn components

Aliasgar Ginwala

Configuration and lifecycle in Mixed environments

Dmitry Kireev

PuppetCamp Sydney 2012 - Building a Multimaster Environment

Greg Cockburn

How to Save, backup and restore IOS on Cisco router

tcpipguru

[Js hcm] Deploying node.js with Forever.js and nginx

Nicolas Embleton

[Intuit] Control Everything

Perforce

Deploying Foreman in Enterprise Environments

inovex GmbH

What's hot (20)

Foreman in your datacenter

Foreman-and-Puppet-for-Openstack-Audo-Deployment

First hop redundancy

Linux host orchestration with Foreman, Puppet and Gitlab

Apache Provisionr (incubating) - Bucharest JUG 10

Openstack il2014 staypuft- your friendly foreman openstack installer

OpenNebula, the foreman and CentOS play nice, too

Connecting AWS and Katello/The Foreman

Configuration Management with Saltstack

PuppetConf 2016: High Availability for Puppet – Russ Mull & Zack Smith, Puppet

Managing your SaltStack Minions with Foreman

Jenkins as Code

Full Stack Automation with Katello & The Foreman

Containerize ovs ovn components

Configuration and lifecycle in Mixed environments

PuppetCamp Sydney 2012 - Building a Multimaster Environment

How to Save, backup and restore IOS on Cisco router

[Js hcm] Deploying node.js with Forever.js and nginx

[Intuit] Control Everything

Deploying Foreman in Enterprise Environments

Similar to 2015-10-20 Processed Billions of job by using Beanstalkd Queue

Beanstalkg

Vimukthi Wickramasinghe

Scaling mysql with python (and Docker).

Roberto Polli

Golang Performance : microbenchmarks, profilers, and a war story

Aerospike

Torquebox OSCON Java 2011

tobiascrawley

The document summarizes TorqueBox, which allows Ruby applications to run on the JBoss Application Server. TorqueBox combines JRuby and JBoss AS to provide features like clustering, load balancing, high availability, messaging, background jobs, and long-running services to Ruby applications. It allows Ruby applications to leverage Java libraries and tools while retaining the simplicity and flexibility of Ruby.

Tunning mobicent-jean deruelle

Ivelin Ivanov

Building a FaaS with pulsar

StreamNative

At Clever Cloud, we are working on extremely light virtual machines to run WebAssembly binaries. As it’s WASM, we can write code using a lot of languages. We use a custom unikernel to run this WASM as Function-as-a-Service, using one VM per function execution. These VM can run on events from messages coming through Pulsar, or from HTTP invocation, the run is on-demand as only the consumers stay up. This can be a new model: Pulsar functions for real isolation in multi-tenancy use cases. This talk will show the use case, explain the virtualization underneath and demonstrate the multi-tenancy use case.

New features in Ruby 2.5

Ireneusz Skrobiś

Ruby 2.5 includes new features such as allowing rescue/else/ensure blocks directly in do/end, yield_self, Hash slicing methods, and Struct classes that accept keyword arguments. It also provides performance improvements like removing trace instructions for 5-10% faster execution and optimizing block passing. Additionally, there are other notable changes like Thread exceptions defaulting to report, SecureRandom preferring OS sources, and updates to standard libraries, Unicode version, RubyGems, and RDoc.

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

Natan Silnitsky

Kafka is the bedrock of Wix’s distributed Mega Microservices system. Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices. In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These Include: * How to increase dev velocity of event-driven style code. * How to optimize working with Kafka in polyglot setting * How to migrate from request-reply to event-driven * How to tackle multiple DCs environment.

Continuous Deployment at Spreaker

Marco Pracucci

This document discusses the continuous deployment process used by Marco Pracucci's audio platform. It deploys new code to its EC2 instances running PHP every 1.5 minutes on average through Fabric scripts. The instances are tagged and mapped to DNS names by role (web, api, worker) through a custom DNS service to address them individually or as groups. The deployment process checks for issues, warms caches, and switches to the new code in under 90 seconds, allowing for frequent deployments while maintaining a robust and scalable infrastructure.

Scaling Flink in Cloud

Steven Wu

Quantifying Container Runtime Performance: OSCON 2017 Open Container Day

Phil Estes

A talk given at Open Container Day at O'Reilly's OSCON convention in Austin, Texas on May 9th, 2017. This talk describes an open source project, bucketbench, which can be used to compare performance, stability, and throughput of various container engines. Bucketbench currently supports docker, containerd, and runc, but can be extended to support any container runtime. This work was done in response to performance investigations by the Apache OpenWhisk team in using containers as the execution vehicle for functions in their "Functions-as-a-Service" runtime. Find out more about bucketbench here: https://github.com/estesp/bucketbench

Kamailioworld 2018 - Modular and test driven SIP Routing with Lua

Sebastian Damm

Specs2 3.4

specs2

This document provides an overview of changes and new features in Specs2 3.x, including updates to the user guide, API, and support for ScalaCheck and HTML reporting. It highlights new capabilities like nesting specifications, linking related specs, running specs in parallel, and composing custom notifiers. It also requests help from the community on additional enhancements like JUnit category support, code coverage, and creating plugins for other tools.

Specs2 3.4

Eric Torreborre

Acceptance testing in php with Codeception - Techmeetup Edinburgh

Engineor

Transition to high-speed WORDPRESS using KUSANAGI

Sumito Tsukada

The document discusses transitioning a WordPress environment to a high-speed configuration using the KUSANAGI image. It summarizes the previous and current environments, stress test results showing improved performance, and issues encountered with certain plugins that were addressed. In conclusion, KUSANAGI can help speed up WordPress but care needs to be taken with plugins, and sites should be checked before enabling caching.

What’s new in 9.6, by PostgreSQL contributor

Masahiko Sawada

The document is a presentation about new features in PostgreSQL 9.6. It discusses several major new features including parallel queries, avoiding VACUUM on all-frozen pages using freeze maps, monitoring the progress of VACUUM, phrase full text search, multiple synchronous replication, remote_apply synchronous commit, and improved capabilities of the postgres_fdw extension including pushing down sorts, joins, updates and deletes to remote servers.

One commit, one release. Continuously delivering a Symfony project.

Javier López

For the last few months we've been implementing a Continuous Delivery pipeline for the redesign of Time Out. In this talk I will demonstrate a real life example of what our pipeline looks like, the different tools we've used to get it done (phing, github, jenkins, ansible, AWS S3, ...), and peculiarities for PHP and Symfony2 projects. Most importantly, I'll be looking at things we've struggled with along the way and the lessons we've learnt.

CCNA Routing and Switching Lesson 06 - IOS Basics - Eric Vanderburg

Eric Vanderburg

This document provides an overview of IOS basics, ICMP, and NAT configuration for Cisco routers. It includes commands for viewing router information and configuration, enabling CDP, checking line status, viewing ICMP codes, and configuring static and dynamic NAT with overload. Static NAT maps internal IP addresses to external IP addresses in a 1-to-1 ratio, while dynamic NAT uses port address translation for a 1-to-many mapping when addresses are limited.

Migrating to a bazel based CI system: 6 learnings

Or Shachar

Two years ago, we were given a big challenge - Transform Wix Build System, then based on Maven and Teamcity, to a new system that will support our exponentially growing scale. But, how could we move to a system so different in so many ways than the existing one? Furthermore, we were required not to break the current build system, as we migrate to the new one. Fast forward to today: Wix backend CI system is fully migrated to Bazel! The system builds in a fracture of the time - even with our largest codebases. In this talk, we will describe how we achieved this, why it took us so long, what tools we had to build on the way (and what we already have, and will, open source!), and share the principles that helped us.

Similar to 2015-10-20 Processed Billions of job by using Beanstalkd Queue (20)

Beanstalkg

Scaling mysql with python (and Docker).

Golang Performance : microbenchmarks, profilers, and a war story

Torquebox OSCON Java 2011

Tunning mobicent-jean deruelle

Building a FaaS with pulsar

New features in Ruby 2.5

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

Continuous Deployment at Spreaker

Scaling Flink in Cloud

Quantifying Container Runtime Performance: OSCON 2017 Open Container Day

Kamailioworld 2018 - Modular and test driven SIP Routing with Lua

Specs2 3.4

Acceptance testing in php with Codeception - Techmeetup Edinburgh

Transition to high-speed WORDPRESS using KUSANAGI

What’s new in 9.6, by PostgreSQL contributor

One commit, one release. Continuously delivering a Symfony project.

CCNA Routing and Switching Lesson 06 - IOS Basics - Eric Vanderburg

Migrating to a bazel based CI system: 6 learnings

Recently uploaded

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Safe Software

Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency. During the hour, we’ll take you through: Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board. Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes. Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI. We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI. This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!

“I’m still / I’m still / Chaining from the Block”

Claudio Di Ciccio

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

Programming Foundation Models with DSPy - Meetup Slides

Zilliz

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

June Patch Tuesday

Ivanti

Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Speck&Tech

ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune. Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile. BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Tosin Akinosho

Monitoring and Managing Anomaly Detection on OpenShift Overview Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices. Key Topics Covered 1. Introduction to Anomaly Detection - Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems. 2. Understanding Edge (IoT) - Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source. 3. What is ArgoCD? - Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices. 4. Deployment Using ArgoCD for Edge Devices - Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD. 5. Introduction to Apache Kafka and S3 - Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions. 6. Viewing Kafka Messages in the Data Lake - Learn how to view and analyze Kafka messages stored in a data lake for better insights. 7. What is Prometheus? - Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices. 8. Monitoring Application Metrics with Prometheus - Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system. 9. What is Camel K? - Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes. 10. Configuring Camel K Integrations for Data Pipelines - Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow. 11. What is a Jupyter Notebook? - Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text. 12. Jupyter Notebooks with Code Examples - Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.

National Security Agency - NSA mobile device best practices

Quotidiano Piemontese

Generating privacy-protected synthetic data using Secludy and Milvus

Zilliz

During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.

CAKE: Sharing Slices of Confidential Data on Blockchain

Claudio Di Ciccio

Presented at the CAiSE 2024 Forum, Intelligent Information Systems, June 6th, Limassol, Cyprus. Synopsis: Cooperative information systems typically involve various entities in a collaborative process within a distributed environment. Blockchain technology offers a mechanism for automating such processes, even when only partial trust exists among participants. The data stored on the blockchain is replicated across all nodes in the network, ensuring accessibility to all participants. While this aspect facilitates traceability, integrity, and persistence, it poses challenges for adopting public blockchains in enterprise settings due to confidentiality issues. In this paper, we present a software tool named Control Access via Key Encryption (CAKE), designed to ensure data confidentiality in scenarios involving public blockchains. After outlining its core components and functionalities, we showcase the application of CAKE in the context of a real-world cyber-security project within the logistics domain. Paper: https://doi.org/10.1007/978-3-031-61000-4_16

Choosing The Best AWS Service For Your Website + API.pptx

Brandon Minnick, MBA

Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API? Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose? Which one is cheapest? Which one is fastest? Which one will scale to meet our needs? Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!

Presentation of the OECD Artificial Intelligence Review of Germany

innovationoecd

Fueling AI with Great Data with Airbyte Webinar

Zilliz

How to use Firebase Data Connect For Flutter

Daiki Mogmet Ito

UI5 Controls simplified - UI5con2024 presentation

Wouter Lemaire

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx

SitimaJohn

Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Malak Abu Hammad

Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers: * What is Vector Search? * Importance and benefits of vector search * Practical use cases across various industries * Step-by-step implementation guide * Live demos with code snippets * Enhancing LLM capabilities with vector search * Best practices and optimization strategies Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications. #MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology

Recently uploaded (20)

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

Climate Impact of Software Testing at Nordic Testing Days

Driving Business Innovation: Latest Generative AI Advancements & Success Story

“I’m still / I’m still / Chaining from the Block”

Removing Uninteresting Bytes in Software Fuzzing

Programming Foundation Models with DSPy - Meetup Slides

HCL Notes and Domino License Cost Reduction in the World of DLAU

June Patch Tuesday

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Monitoring and Managing Anomaly Detection on OpenShift.pdf

National Security Agency - NSA mobile device best practices

Generating privacy-protected synthetic data using Secludy and Milvus

CAKE: Sharing Slices of Confidential Data on Blockchain

Choosing The Best AWS Service For Your Website + API.pptx

Presentation of the OECD Artificial Intelligence Review of Germany

Fueling AI with Great Data with Airbyte Webinar

How to use Firebase Data Connect For Flutter

UI5 Controls simplified - UI5con2024 presentation

Ocean lotus Threat actors project by John Sitima 2024 (1).pptx

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

2015-10-20 Processed Billions of job by using Beanstalkd Queue

1. teddy@aftership.com Teddy Chan CEO & Co-Founder

2. Shipment tracking API Supporting 290 310+ carriers

3. Free Shipping APIs Single API for 5 19 carriers

4. Beanstalkd The queue system processed billions of job

5. Why we love queue?

6. Queue - Buffer - Load Balancing - Scale Horizontal - Async process

7. Beanstalkd How AfterShip use queue? - Ten of millions of job per day - Billions of job so far - 4 beanstalkd servers (AWS m3.large) - 4 down incidents in last 3 years

8. Beanstalkd Protocol Characteristics - Guarantee FIFO - Each job is uniquely assigned to one worker. - Each connection can only reserve one job at a time - Support jobs persistent (“-b” option) - Support priority, delay, ttr (time to run), bury - No HA

9. Queue systems References: http://queues.io/ http://www.sitepoint.com/message-queues-comparing-beanstalkd-ironmq-amazon-sqs/ Beanstalkd Amazon SQS Server Setup Self Host Full Managed Response Time single digit ms hundred ms First In First Out (FIFO) Guarantee & unique Does not guarantee Priority Yes No Delay Yes Yes Pub / Sub TCP pub/sub https polling

10. Beanstalkd Protocol Action you can do Status Description put (with delay) Add a new job in the queue reserve Take the job from queue delete Delete the job after reserved. bury Bury the job after reserved. kick Kick the job from buried status to ready status. release (with delay) Release the job back to ready status, after reserved it. The job id, pri, will be same.

11. Beanstalkd Protocol Typical life cycle for a job put --------> [READY] -------> [RESERVED]-------> *proof* Ready Tube Job 1 put

12. Beanstalkd Protocol Put put --------> [READY] -------> [RESERVED]-------> *proof* Ready Tube Job 1

13. Beanstalkd Protocol Reserve put reserve --------> [READY] -------> [RESERVED]-------> *proof* Ready Tube Job 1 reserve

14. Beanstalkd Protocol Delete put reserve delete --------> [READY] -------> [RESERVED]-------> *proof* Ready Tube Job 1 delete

15. Beanstalkd Protocol Bury put reserve bury --------> [READY] -------> [RESERVED]-------> [BURIED] Ready Tube Job 1 bury Buried Tube

16. Beanstalkd Protocol Kick put reserve bury --------> [READY] -------> [RESERVED]-------> [BURIED] KICK Ready Tube kick Buried Tube Job 1

17. Beanstalkd Protocol Delay Job 1 Job 2 Ready Job Delay Job (with 3s)

18. Beanstalkd Protocol Delay Job 1 Job 2 Ready Job Delay Job (After 3s)

19. Beanstalkd Protocol Full picture of the life cycle

20. Beanstalkd Let’s do something fun

21. Beanstalkd So how much? - 15 servers @ USD0.05 / hr - 2 workers @ 0.05 USD per hr - some traffic… $0.12 / GB

22. Beanstalkd So how much? - 15 servers @ USD0.05 / hr - 2 workers @ 0.05 USD per hr - some traffic… $0.12 / GB - $0.97 in total

23. http://www.slideshare.net/aftership FAQ

2015-10-20 Processed Billions of job by using Beanstalkd Queue

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 2015-10-20 Processed Billions of job by using Beanstalkd Queue

Similar to 2015-10-20 Processed Billions of job by using Beanstalkd Queue (20)

Recently uploaded

Recently uploaded (20)

2015-10-20 Processed Billions of job by using Beanstalkd Queue