Celery is a really good framework for doing background task processing in Python (and other languages). While it is ridiculously easy to use celery, doing complex task flow has been a challenge in celery. (w.r.t task trees/graphs/dependecies etc.)
This talk introduces the audience to these challenges in celery and also explains how these can be fixed programmatically and by using latest features in Celery (3+)
Learn the ins and outs of running background tasks with the popular python module Celery. We'll hit the ground running. With everything you need to know to run your first task, to scaling your stack to run millions each day.
Learn the ins and outs of running background tasks with the popular python module Celery. We'll hit the ground running. With everything you need to know to run your first task, to scaling your stack to run millions each day.
A basic Introduction to Rust. Rust is a modern system programming language which offering different approach than other existing new modern system programming languages to deal with: memory safety without GC, abstraction without overhead, and concurrency without data races.
Une riche présentation de Mapping Objet Relationnel qui traite le standard JPA et l’implémentation Hibernate en les intégrant avec le frammework IOC spring.
We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.
Finally, we wrap up with a quick example of a task queue in PHP using Redis.
https://github.com/bryanhelmig/phqueue
Asynchronous JavaScript Programming with Callbacks & PromisesHùng Nguyễn Huy
This presentation is about JavaScript Promise. Topics covered in this session are:
1. Asynchronous processing in JavaScript
2. Callbacks and Callback hell
3. Promises arrive in JavaScript!
4. Constructing a Promise
5. Promise states
6. Promises chaining and transformation
7. Error handling
8. Promise.all() and Promise.race()
we will see internal architecture of spark cluster i.e what is driver, worker,executer and cluster manager, how spark program will be run on cluster and what are jobs,stages and task.
A basic Introduction to Rust. Rust is a modern system programming language which offering different approach than other existing new modern system programming languages to deal with: memory safety without GC, abstraction without overhead, and concurrency without data races.
Une riche présentation de Mapping Objet Relationnel qui traite le standard JPA et l’implémentation Hibernate en les intégrant avec le frammework IOC spring.
We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.
Finally, we wrap up with a quick example of a task queue in PHP using Redis.
https://github.com/bryanhelmig/phqueue
Asynchronous JavaScript Programming with Callbacks & PromisesHùng Nguyễn Huy
This presentation is about JavaScript Promise. Topics covered in this session are:
1. Asynchronous processing in JavaScript
2. Callbacks and Callback hell
3. Promises arrive in JavaScript!
4. Constructing a Promise
5. Promise states
6. Promises chaining and transformation
7. Error handling
8. Promise.all() and Promise.race()
we will see internal architecture of spark cluster i.e what is driver, worker,executer and cluster manager, how spark program will be run on cluster and what are jobs,stages and task.
We will see internal architecture of spark cluster i.e what is driver, worker, executor and cluster manager, how spark program will be run on cluster and what are jobs,stages and task.
In this iteration of iOS Meetup, The experts from Seven Peaks Software will walk you through on the Swift programming language, Giving you the latest tips and tricks for you to be success on the iOS development
Rupendra opened up the meetup with Concurrency in Swift. Concurrency allows programs to deal with multiple tasks at once. But writing a concurrent program is not as easy as it seems. Dealing with threads and locks can be quite cumbersome, making concurrent programs difficult to write. His Topic will focus on making it straightforward and understandable so that anyone who is an intermediate to advanced Swift developer can apply these concepts to their projects.
Trevor McDonald - Nagios XI Under The Hood - What happens when a check is run? What are the parts that move behind the scenes to turn a service check into a notification? In this talk, Trevor will walk through the check process from start to finish, giving an overview of the components involved at each step.
Robust C++ Task Systems Through Compile-time ChecksStoyan Nikolov
Task-based (aka job systems) engine architectures are becoming the de-facto standard for AAA game engines and software solutions. The talk explains how the task system in the Hummingbird game UI engine was designed to both be convenient and to avoid common programmer pitfalls. Advanced C++ techniques are employed to warn and shield the developer from errors at compile time.
Async and parallel patterns and application design - TechDays2013 NLArie Leeuwesteijn
TechDays2013 NL session on async and parallel programming. Gives an overview of todays relevant .net technologies, examples and tips and tricks. This session will help you to understand and select and use the right async/parallel technology to use in your .net application. (arie@macaw.nl)
Troubleshooting MySQL from a MySQL Developer PerspectiveMarcelo Altmann
Working as a MySQL Developer as part of the Bugs committee exposes you to a variety of bugs, such as server crashes, memory leaks, wrong query results, internal thread deadlocks, and others. In this talk, I will cover some of the technics we utilize to troubleshoot MySQL when things are not working as expected.
Some of the topics covered include:
Reproducible test cases
Git Bisect
Stack Traces
GDB
Record and Replay
By the end of this session, attendees will grasp how to tackle analyses of when software is not working as expected.
NovaProva, a new generation unit test framework for C programsGreg Banks
I wrote NovaProva because I was sick of writing unit tests using the venerable but clunky CUnit library. CUnit looked easy when I started writing unit tests but I soon discovered it's limitations and ended up screaming in frustration. Meanwhile over 18 months the Cyrus IMAP server project has gone from 0 unit tests to 461, of which 277 are written in C with CUnit. So that's a big itch!
Kernel Recipes 2018 - KernelShark 1.0; What's new and what's coming - Steven ...Anne Nicolas
Ftrace is the official tracer of the Linux kernel. It was added in 2008, and in 2009 came trace-cmd which was a command line tool that would make interaction with ftrace easier. Shortly after that, KernelShark was created as a GUI for trace-cmd interface. But as KernelShark and trace-cmd were mostly side projects, there wasn't as much activity that they deserved. trace-cmd was updated more often, but KernelShark has suffered with bit-rot for some time. But all that has changed recently as VMware has active developers working on it.
KernelShark has been completely rewritten from scratch and version 1.0 is due to be released in August of 2018 (has already been released as of this talk). This will discuss what changed, how to use the new tool and what is coming in the future.
In this presentation , i provide in depth information about the how MapReduce works. It contains many details about the execution steps , Fault tolerance , master / worker responsibilities.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
2. @mahendra
● Python developer for 6 years
● FOSS enthusiast/volunteer for 14 years
● Bangalore LUG and Infosys LUG
● FOSS.in and LinuxBangalore/200x
● Celery user for 3 years
● Contributions
● patches, testing new releases
● Zookeeper msg transport for kombu
● Kafka support (in-progress)
3. Quick Intro to Celery
● Asynchronous task/job queue
● Uses distributed message passing
● Tasks are run asynchronously on worker nodes
● Results are passed back to the caller (if any)
6. Uses of Celery
● Asynchronous task processing
● Handling long running / heavy jobs
● Image resizing, video transcode, PDF generation
● Offloading heavy web backend operations
● Scheduling tasks to be run at a particular time
● Cron for python
7. Advanced Uses
● Task Routing
● Task retries, timeout and revoking
● Task Canvas – combining tasks
● Task co-ordination
● Dependencies
● Task trees or graphs
● Batch tasks
● Progress monitoring
● Tricks
● DB conflict management
8. Sending tasks to a particular worker
Worker 1
(Windows)
windows
Worker 2
windows (Windows)
Sender Msg Q
.
linux
.
.
Worker N
(Linux)
9. Routing tasks – Use cases
● Priority execution
● Based on hardware capabilities
● Special cards available for video capture
● Making use of GPUs (CUDA)
● Based on OS (for eg. Playready encryption)
● Based on location
● Moving compute closer to data (Hadoop-ish)
● Sending tasks to different data centers
● Sequencing operations (CouchDB conflicts)
10. Sample Code
from celery.task import task
@task(queue = 'windows')
def drm_encrypt(audio_file, key_phrase):
...
r = drm_encrypt.apply_async( args = [afile, key],
queue = 'windows' )
#Start celery worker with queues options
$ celery worker -Q windows
12. Retrying tasks
● You can specify the number of times a task can
be retried.
● The cases for retrying a task must be handled
within code. Celery will not do it automatically
● The tasks should be designed to be idempotent
13. Handling worker failures
@task( acks_late = True )
def drm_encrypt(audio_file, key_phrase):
try:
playready.encrypt(...)
except Exception, exc:
raise drm_encrypt.retry(exc=exc, countdown=5)
● This is used where the task must be resend in case of
worker or node failure
● The ack message to the message queue is sent after the
task finishes executing
14. Worker processes
Worker 1
(Windows)
windows
Worker 2
windows (Windows)
Sender Msg Q
.
linux
.
.
Worker N
(Linux)
Process 1
Process 2
Process N
15. Worker processes
Worker 1
(Windows)
windows
Worker 2
windows (Windows)
Sender Msg Q
.
linux
.
.
Worker N
(Linux)
Process 1
Process 2
Process N
16. Worker process
● In every worker node, celery starts a pool of
worker processes
● The number is determined by the concurrency
setting (or autodetected – for full CPU usage)
● Each processes can be configured to restart
after running x number of tasks
● Disabled by default
● Alternately eventlet can be used instead of
processes (discuss later)
17. Revoking tasks
celery.control.revoke( task_id,
terminate = False,
signal = 'SIGKILL' )
●
revoke() works by sending a broadcast
message to all workers
● If a task has not yet run, workers will keep this
task_id in memory and ensure that it does not
run
● If a task is running, revoke() will not work
unless terminate = True
18. Task expiration
task.apply_async( expires = x )
x can be
* in seconds
* a specific datetime()
● Global time limits can be configured in settings
● Soft time limit – the task receives an exception
which can be used to cleanup
● Hard time limit – the worker running the task is
killed and is replaced with another one.
20. Task Canvas
● Chains – Linking one task to another
● Groups – Execute several tasks in parallel
● Chord – execute a task after a set of tasks has
finished
● Map and starmap – Similar to map() function
● Chunks – divide an iterable of work into chunks
● Chunks + Chord/chain can be used for map-
reduce
Best shown in a demo
22. Task Trees
● Home grown solution (our current approach)
● Use db models and keep track of trees
● Better approach
● Use celery-tasktree
● http://pypi.python.org/pypi/celery-tasktree
23. Celery Batches
● Collect jobs and execute it in a batch.
● Can be used for stats collection
● Batch execution is done once
● a configured timeout is reached OR
● a configured number of tasks have been received
● Useful for reducing n/w and db loads