Learn the ins and outs of running background tasks with the popular python module Celery. We'll hit the ground running. With everything you need to know to run your first task, to scaling your stack to run millions each day.
We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.
Finally, we wrap up with a quick example of a task queue in PHP using Redis.
https://github.com/bryanhelmig/phqueue
Learn the ins and outs of running background tasks with the popular python module Celery. We'll hit the ground running. With everything you need to know to run your first task, to scaling your stack to run millions each day.
We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.
Finally, we wrap up with a quick example of a task queue in PHP using Redis.
https://github.com/bryanhelmig/phqueue
Celery is a really good framework for doing background task processing in Python (and other languages). While it is ridiculously easy to use celery, doing complex task flow has been a challenge in celery. (w.r.t task trees/graphs/dependecies etc.)
This talk introduces the audience to these challenges in celery and also explains how these can be fixed programmatically and by using latest features in Celery (3+)
Getting Started with Spring Authorization ServerVMware Tanzu
SpringOne 2021
Title: Getting Started with Spring Authorization Server
Speakers: Joe Grandja, Spring Security Engineer at VMware; Steve Riesenberg, Software Engineer at VMware
This is an interactive PowerPoint presentation I made for my seminar at my workplace. It talks and illustrates how the JavaScript engine works when executing code. It also contains a step-by-step animation which demonstrates the usage of the JavaScript event loop.
Feel free to use and refer content. Copyrights reserved.
A comparison of different solutions for full-text search in web applications using PostgreSQL and other technology. Presented at the PostgreSQL Conference West, in Seattle, October 2009.
PromQL Deep Dive - The Prometheus Query Language Weaveworks
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
[2019] 바르게, 빠르게! Reactive를 품은 Spring KafkaNHN FORWARD
※다운로드하시면 더 선명한 자료를 보실 수 있습니다.
Spring Kafka 2.3에 추가된 Reactive API를 소개합니다.
모니터링시스템에서 감지한 이상 현상을 담당자들에게 통지하는 실제 사례를 중심으로 설명합니다.
Reactive 방식으로 메시지를 발행하고 소비하는 방법을 소개하고, 읽어 들인 이벤트 메시지에 적용해야 할 여러 복잡한 요구 사항을 Rx의 연산자들을 통해 간결하게 구현하는 예제를 공유합니다.
Publisher와 Subscriber 간의 동작 구조를 통해 여러 시스템 그리고 저장소와 연계할 때 주의할 점을 되짚어보고, 특히 Kafka를 이용해서 생길 수 있는 문제와 이를 해결할 방법을 제안합니다.
목차
1. Kafka 메시지를 비동기로 처리하는 방법
2. ReactiveX에서 제공하는 연산자를 활용하는 사례
3. Project Reactor의 내부 구조(Publisher-Subscriber 간 처리 흐름)
대상
- Reactive Programming에 관심 있는 분
- Kafka 등 스트리밍 플랫폼의 메시지 처리량을 높이고 싶은 분
■관련 동영상: https://youtu.be/HzQfJNusnO8
This session will give attendees an overview of the new testing features in Spring 3.1 as well the new Spring MVC test support. Sam Brannen will demonstrate how to use the Spring TestContext Framework to write integration tests for Java-based Spring configuration using @Configuration classes. He'll then compare and contrast this approach with XML-based configuration and follow up with a discussion of the new testing support for bean definition profiles. Next, Rossen Stoyanchev will show attendees how testing server-side code with annotated controllers and client-side code with the RestTemplate just got a whole lot easier with the new Spring MVC test support. Come to this session to see these new Spring testing features in action and learn how you can get involved in the Spring MVC Test Support project.
Asynchronous API in Java8, how to use CompletableFutureJosé Paumard
Slides of my talk as Devoxx 2015. How to set up asynchronous data processing pipelines using the CompletionStage / CompletableFuture API, including how to control threads and how to handle exceptions.
Celery is a really good framework for doing background task processing in Python (and other languages). While it is ridiculously easy to use celery, doing complex task flow has been a challenge in celery. (w.r.t task trees/graphs/dependecies etc.)
This talk introduces the audience to these challenges in celery and also explains how these can be fixed programmatically and by using latest features in Celery (3+)
Getting Started with Spring Authorization ServerVMware Tanzu
SpringOne 2021
Title: Getting Started with Spring Authorization Server
Speakers: Joe Grandja, Spring Security Engineer at VMware; Steve Riesenberg, Software Engineer at VMware
This is an interactive PowerPoint presentation I made for my seminar at my workplace. It talks and illustrates how the JavaScript engine works when executing code. It also contains a step-by-step animation which demonstrates the usage of the JavaScript event loop.
Feel free to use and refer content. Copyrights reserved.
A comparison of different solutions for full-text search in web applications using PostgreSQL and other technology. Presented at the PostgreSQL Conference West, in Seattle, October 2009.
PromQL Deep Dive - The Prometheus Query Language Weaveworks
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
[2019] 바르게, 빠르게! Reactive를 품은 Spring KafkaNHN FORWARD
※다운로드하시면 더 선명한 자료를 보실 수 있습니다.
Spring Kafka 2.3에 추가된 Reactive API를 소개합니다.
모니터링시스템에서 감지한 이상 현상을 담당자들에게 통지하는 실제 사례를 중심으로 설명합니다.
Reactive 방식으로 메시지를 발행하고 소비하는 방법을 소개하고, 읽어 들인 이벤트 메시지에 적용해야 할 여러 복잡한 요구 사항을 Rx의 연산자들을 통해 간결하게 구현하는 예제를 공유합니다.
Publisher와 Subscriber 간의 동작 구조를 통해 여러 시스템 그리고 저장소와 연계할 때 주의할 점을 되짚어보고, 특히 Kafka를 이용해서 생길 수 있는 문제와 이를 해결할 방법을 제안합니다.
목차
1. Kafka 메시지를 비동기로 처리하는 방법
2. ReactiveX에서 제공하는 연산자를 활용하는 사례
3. Project Reactor의 내부 구조(Publisher-Subscriber 간 처리 흐름)
대상
- Reactive Programming에 관심 있는 분
- Kafka 등 스트리밍 플랫폼의 메시지 처리량을 높이고 싶은 분
■관련 동영상: https://youtu.be/HzQfJNusnO8
This session will give attendees an overview of the new testing features in Spring 3.1 as well the new Spring MVC test support. Sam Brannen will demonstrate how to use the Spring TestContext Framework to write integration tests for Java-based Spring configuration using @Configuration classes. He'll then compare and contrast this approach with XML-based configuration and follow up with a discussion of the new testing support for bean definition profiles. Next, Rossen Stoyanchev will show attendees how testing server-side code with annotated controllers and client-side code with the RestTemplate just got a whole lot easier with the new Spring MVC test support. Come to this session to see these new Spring testing features in action and learn how you can get involved in the Spring MVC Test Support project.
Asynchronous API in Java8, how to use CompletableFutureJosé Paumard
Slides of my talk as Devoxx 2015. How to set up asynchronous data processing pipelines using the CompletionStage / CompletableFuture API, including how to control threads and how to handle exceptions.
Life in a Queue - Using Message Queue with djangoTareque Hossain
Brief introduction on message queue and how its relevant in web applications
How to tell if your web application could benefit from message queue
Common example of tasks that could benefit from message queues
Choosing a broker/protocol
What broker/protocol PBS Education chose and why
Message queue solution architecture
Brief introduction on celery/carrot
Writing a message queue task using celery
How to invoke a message queue taks
What happens when you invoke a task (walk through architecture)
How to write tasks efficiently
What are the things that are good to know when writing tasks (things we experienced at PBS Education)
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016Wei Lin
In this talk, the basic mechanisms of Celery and Docker-Swarm will be explained. With Docker-Swarm , a cluster was built upon two Raspberry Pi machines. Hadoop entry-level "Word Count" program could be re-written in Python and executed parallelly via Celery on the cluster. An example of distributed system modeling neural-network will also be explained.
Understanding Non Blocking I/O with PythonVaidik Kapoor
Video recording of the talk: https://www.youtube.com/watch?v=t0VljE9kq1Q
As an engineer working on any web stack, you may have heard about Blocking and Non-Blocking IO. You may as well have used any framework or library that supports Non-Blocking IO. After all, they are very useful as you don’t want to block execution of other tasks while one task is waiting to complete a network call to another service (like HTTP call to an API or may be a TCP call to your database). Non-Blocking IO while doing tasks and not wait for IO. This also helps us handle a lot many connections than we possibly could with Blocking IO. Python supports Non-Blocking IO, but we always use some existing 3rd party library that hides all the gory details and makes it all look like black magic to the uninitiated. But there is nothing like black magic.
This presentation will be an introductory talk focused at explaining how Non-Blocking IO works, which is the basis of libraries like Gevent, Tornado and Twisted. We will learn about how Non-Blocking IO can be implemented using the most basic modules that form the base for the above mentioned libraries. Hopefully after this talk, Non-Blocking IO will not be an unsolved mystery for you anymore.
Когда тестировать, что тестировать, как тестировать, Как ускорить тесты и упростить их написание. Отказываемся от классических фикстур в пользу динамически создаваемых моделей.
Introduction on animations for Angular applications + heads ups for some tricky bits.
Don't forget to visit the included links:
- https://medium.com/ux-in-motion/creating-usability-with-motion-the-ux-in-motion-manifesto-a87a4584ddc
- https://medium.com/frontend-coach/angular-in-motion-4-approaches-to-animation-1aa7426aae5a
- https://medium.com/frontend-coach/angular-router-animations-what-they-dont-tell-you-3d2737a7f20b
... and follow me on twitter https://twitter.com/sulco for more on Angular, TypeScript and frontend in general.
Some javascript testing patterns that has made writing tests at Refinery29 more maintainable. Covered in this presentation is userContext (or let) in Jasmine, lazy evaluation and shared behavior.
Similar to Europython 2011 - Playing tasks with Django & Celery (20)
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
20. Web monitoring (django-celery and others) “ Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well”
21. AMPQ The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for Message Oriented Middleware.
30. Defining a simple task from celery.decorators import task @task def make_money (how_much): logger = make_money.get_logger() logger.info( "Congratulation, you earned %s$" % how_much) if how_much>1000000: return "Bora Bora" return "Keep working" >>> result = make_money.delay(200) >>> result.get() “ Keep working”
31. Retrying a task if something fails from celery.decorators import task @task def make_money_real_life (how_much, wife=True): try : logger = make_money.get_logger() if wife: raise Exception( "Incompatibility exception" ) logger.info( "Congratulation, you earned %s$" % how_much) if how_much>1000000: return "Bora Bora" return "Keep working" except Exception,exc: make_money_real_life.retry(exc=exc, countdown=60, args=[how_much,False])
32. Task set example def run (self, setid=None, subtasks=None, **kwargs): … if not setid or not subtasks: … tasks = [] for slice in slices: tasks.append(uploadTrackSlice.subtask((slice,folder_name))) job = TaskSet(tasks=tasks) task_set_result = job.apply_async() setid = task_set_result.taskset_id subtasks = [result.task_id for result in task_set_result.subtasks] self.incrementalRetry("Result not ready", args=[setid,subtasks]) #Is a retry than we just have to check the results tasks_result = TaskSetResult(setid, map(AsyncResult,subtasks)) if not tasks_result.ready(): self.incrementalRetry( "Result not ready" , args=[setid,subtasks]) else : if tasks_result.successful(): return tasks_result.join() else : raise Exception( "Some of the tasks was failing" ) Extract from a jamendo task that upload track metadata in xml format to an ftp server for music analysis
33. The Jamendo Task class class JamTask (Task): def __call__ (self, *args, **kwargs): """This method is in charge of call the run method of the task""" self.max_retries = 30 self.sandbox = SandBox(self.name, self.request.id, settings.PATH_SANDBOX, settings.DEBUG) self.taskLogger = TaskLogger(args, kwargs) self.taskLogger.__enter__() . . return self.run(*args, **kwargs) . . def after_return (self, status, retval, task_id, args, kwargs, einfo): """This method is called when the tasks end, on whatever return state""" self.taskLogger.__exit__(status, retval, args, kwargs, einfo) self.cleanTaskSandBox(status,kwargs) self.closeAllConnections() The way for define common behaviour to all your tasks is to override __call__ and after_return methods of the celery Task class