We upgraded from Backgroundrb to Resque. The pagers have stopped buzzing, and we are very pleased with the migration.
Resque was a little tricky to get the last 5% complete. This presentation shares some of the implementation details (code and config files) to help others make their Resque setup rock solid.
Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture.
This talk will show examples of how to use Gearman for remote installation and how to call functions written in Perl from any other language or from inside MySQL server, with no knowledge of Perl at all.
We use Gearman for managing queue system. This covers why we should use a queue in many situations on web-based interface as well as server-side application.
A presentation with tips and tools on how to integrate batch and asynchronous operations in a generic ruby on rails application.
Did this at rubyday.it 2011
A Spring Batch bootcamp! Spring Batch is the open source batch processing framework from SpringSource, makes of the Spring framework. http://www.springsource.org/spring-batch
The MEW Workshop is now established as a leading national event dedicated to distributed high performance scientific computing. The principle objective is to encourage close contact between the research communities from the Mathematics, Chemistry, Physics and Materials Programmes of EPSRC and the major vendors.
A short description of Perly grammar processors leading up to Regexp::Grammars. Develops two R::G modules, one for single-line logfile entries, another for larger FASTA format entries in the NCBI "nr.gz" file. The second example shows how to derive one grammar from another by overriding tags in the base grammar.
Async and parallel patterns and application design - TechDays2013 NLArie Leeuwesteijn
TechDays2013 NL session on async and parallel programming. Gives an overview of todays relevant .net technologies, examples and tips and tricks. This session will help you to understand and select and use the right async/parallel technology to use in your .net application. (arie@macaw.nl)
Hanborq Optimizations on Hadoop MapReduceHanborq Inc.
A Hanborq optimized Hadoop Distribution, especially with high performance of MapReduce. It's the core part of HDH (Hanborq Distribution with Hadoop for Big Data Engineering).
Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture.
This talk will show examples of how to use Gearman for remote installation and how to call functions written in Perl from any other language or from inside MySQL server, with no knowledge of Perl at all.
We use Gearman for managing queue system. This covers why we should use a queue in many situations on web-based interface as well as server-side application.
A presentation with tips and tools on how to integrate batch and asynchronous operations in a generic ruby on rails application.
Did this at rubyday.it 2011
A Spring Batch bootcamp! Spring Batch is the open source batch processing framework from SpringSource, makes of the Spring framework. http://www.springsource.org/spring-batch
The MEW Workshop is now established as a leading national event dedicated to distributed high performance scientific computing. The principle objective is to encourage close contact between the research communities from the Mathematics, Chemistry, Physics and Materials Programmes of EPSRC and the major vendors.
A short description of Perly grammar processors leading up to Regexp::Grammars. Develops two R::G modules, one for single-line logfile entries, another for larger FASTA format entries in the NCBI "nr.gz" file. The second example shows how to derive one grammar from another by overriding tags in the base grammar.
Async and parallel patterns and application design - TechDays2013 NLArie Leeuwesteijn
TechDays2013 NL session on async and parallel programming. Gives an overview of todays relevant .net technologies, examples and tips and tricks. This session will help you to understand and select and use the right async/parallel technology to use in your .net application. (arie@macaw.nl)
Hanborq Optimizations on Hadoop MapReduceHanborq Inc.
A Hanborq optimized Hadoop Distribution, especially with high performance of MapReduce. It's the core part of HDH (Hanborq Distribution with Hadoop for Big Data Engineering).
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
2. Summary
! Background queues let us defer logic outside the browser request and
response.
! Background.rb was crashing for us often. Moved to resque and it
hasn't crashed since.
! Background.rb is easier to run out of the box.
! Adding just a little code makes Resque just as easy without sacrificing
all the added flexibility.
3. Why we upgraded?
! bdrb pages Boss 4 times my first weekend
! memory leaks caused crashes
! monit can't restart workers in backgroundrb
! move to active project (ala heroku, github, redis)
4. What do each bring to the table
bdrb resque
adhoc (out of request) ! !
delay (run/remind) ! resque-schedule
schedule (cron) ! resque-schedule
mail (invisible/out of req) code resque_mailer
status reporting code resque-meta, web
backgroundrb does most of what we need out of the box
resque has plugins to make up the difference
5. Bdrb Components
scheduler
workers
rails main queue
work
enqueue queue manager
mailer
we started Monitored data
bdrb yml
simple w/ 1 queue (add started_at for delayed jobs)
scheduler is a special worker - managed by 1 process (is a runner/worker)
6. Resque Components
delayed scheduler schedule
queue
2
rails
enqueue
1
workers rake
4
resque main
main work
web main
queue
queue
queue workers
6
3
mailer
we started Monitored data
5
many moving parts
simplified in all workers are the same
scheduler simply adds entries in the queue (instead of MetaWorker/running jobs)
web ui is a nice touch
7. 1. Ad-hoc Enqueuing
bdrb resque
args hash ruby, checked
enqueue AR objects !
mail(invisible) ! !
AR objects - creeped up in the action_mailer deliver calls
Looks like bdrb wins here, but not enqueuing AR objects is best practice
8. Ad-hoc/Delayed (bdrb)
class JobWorker < BackgrounDRb::MetaWorker
set_worker_name :job_worker
def purge_job_logs()
JobLog.purge_expired!
persistent_job.finish!
end
def self.perform_later(*args)
MiddleMan.worker(:job_worker).enq_purge_job_logs(
:job_key => new_job_key, :arg => args)
end
def self.perform_at(*args)
time=args.shift
MiddleMan.worker(:job_worker).enq_purge_job_logs(
:job_key => new_job_key, :arg => *args,:scheduled_at => time)
end
def self.new_job_key()
"purge_job_logs_#{ActiveSupport::SecureRandom.hex(8)}"
end
end
don't need to do a command pattern (our code didn't)
scheduled_at = beauty of SQL
parent class
enqueue knows queue name (code not loaded)
9. Ad-hoc/Delayed (resque)
class PurgeJobLogs
@queue = :job_worker
def self.process()
JobLog.purge_expired!
end
def self.perform_later(*args)
Resque.enqueue(self, *args)
end
def self.perform_at(*args)
time=args.shift
Resque.enqueue_at(time, self, *args)
end
end
Enqueue needs worker class to know the name of the queue
(even if called directly into Resque)
interface only (perform_{at,later}) -> abstracted out to parent?
10. 2. Scheduled Enqueuing
bdrb resque
sched any method !x2 command
scheduler ! !+
adhoc jobs !
Need to define schedule in 2 places. yml and ruby.
We ran into case where this caused a problem
web ui for easy adhoc kicking off of resque commands. (very useful in staging)
11. Scheduled (bdrb)
:backgroundrb:
:ip: 127.0.0.1
:port: 11006
:environment: development
:schedules:
:scheduled_worker:
:purge_job_logs:
:trigger_args: 0 */5 * * * *
Evidence of framework - scheduled_worker defined here, need meta worker (so it can be run)
12. Scheduled (bdrb)
class ScheduledWorker < BackgrounDRb::MetaWorker
extend BdrbUtils::CronExtensions
set_worker_name :scheduled_worker
threaded_cron_job(:purge_job_logs) { JobLog.purge_expired! }
end
scheduler = MetaWorker. Defined 2 times - so it calls your code, so can call "any static method"
13. Scheduled (resque)
---
clear_logs:
cron: "*/10 * * * *"
class: PurgeJobLogs
queue: job_worker
description: Remove old logs
queue_name (so scheduler does not need to load worker into memory to enqueue)
cron is standard format (remove 'seconds') - commands
scheduler in separate process. (can run when workers are stopped / changed) - minimal env
scheduler injects into queue (vs runs jobs) - so can adhoc inject via web
no ruby code for this
15. worker list (resque)
primary:
queues: background,mail
secondary:
queues: mail,background
can have multiple workers running the same queues
can have multiple queues in 1 worker
worker pool can be * generalized, * response focused, * schedule focused, *changed at runtime
inverted priority list - prevents starvation
16. 4. Running Workers
namespace :resque do
desc 'start all background resque daemons'
task :start_daemons do
mrake_start "resque_scheduler resque:scheduler"
workers_config.each do |worker, config|
mrake_start "resque_#{worker} resque:work QUEUE=#{config['queues']}"
end
end
desc 'stop all background resque daemons'
task :stop_daemons do
sh "./script/monit_rake stop resque_scheduler"
workers_config.each do |worker, config|
sh "./script/monit_rake stop resque_#{worker} -s QUIT"
end
end
def self.workers_config
YAML.load(File.open(ENV['WORKER_YML'] || 'config/resque_workers.yml'))
end
def self.mrake_start(task)
sh "nohup ./script/monit_rake start #{task} RAILS_ENV=#{ENV['RAILS_ENV']} >> log/daemons.log &"
end
end
17. Deploying (cap)
namespace :resque do
desc "Stop the resque daemon"
task :stop, :roles => :resque do
run "cd #{current_path} && RAILS_ENV=#{rails_env} WORKER_YML=#{resque_workers_yml} rake
resque:stop_daemons; true"
end
desc "Start the resque daemon"
task :start, :roles => :resque do
run "cd #{current_path} && RAILS_ENV=#{rails_env} WORKER_YML=#{resque_workers_yml} rake
resque:start_daemons"
end
end
18. 5. Monitoring Workers (monit.erb)
check process resque_scheduler
with pidfile <%= @rails_root %>/tmp/pids/resque_scheduler.pid
group resque
alert errors@domain.com
start program = "/bin/sh -c 'cd <%= @rails_root %>; RAILS_ENV=production ./script/monit_rake
start resque_scheduler resque:scheduler'"
stop program = "/bin/sh -c 'cd <%= @rails_root %>; RAILS_ENV=production ./script/monit_rake
stop resque_scheduler'"
<% YAML.load(File.open(Rails.root+'/config/production/resque/resque_workers.yml')).each_pair do
|worker, config| %>
check process resque_<%=worker%>
with pidfile <%= @rails_root %>/tmp/pids/resque_<%=worker%>.pid
group resque
alert errors@domain.com
start program = "/bin/sh -c 'cd <%= @rails_root %>; RAILS_ENV=production ./script/monit_rake
start resque_<%=worker%> resque:work QUEUE=<%=config['queues']%>'"
stop program = "/bin/sh -c 'cd <%= @rails_root %>; RAILS_ENV=production ./script/monit_rake
stop resque_<%=worker%>'"
<% end %>
use template to generate monit file
19. Monitoring Rake Processes
#!/bin/sh
# wrapper to daemonize rake tasks: see also http://mmonit.com/wiki/Monit/FAQ#pidfile
usage() {
echo "usage: ${0} [start|stop] name target [arguments]"
echo "tname is used to create or read the log and pid file names"
echo "tfor start: target and arguments are passed to rake"
echo "tfor stop: target and arguments are passed to kill (e.g.: -n 3)"
exit 1
}
[ $# -lt 2 ] && usage
cmd=$1
name=$2
shift ; shift
pid_file=./tmp/pids/${name}.pid
log_file=./log/${name}.log
# ...
20. Monitoring Processes
case $cmd in
start)
if [ ${#} -eq 0 ] ; then
echo -e "nERROR: missing targetn"
usage
fi
pid=`cat ${pid_file} 2> /dev/null`
if [ -n "${pid}" ] ; then
ps ${pid}
if [ $? -eq 0 ] ; then
echo "ensure process ${name} (pid: ${pid_file}) is not running"
exit 1
fi
fi
echo $$ > ${pid_file}
exec 2>&1 rake $* 1>> ${log_file} ;;
stop)
pid=`cat ${pid_file} 2> /dev/null`
[ -n "${pid}" ] && kill $* ${pid}
rm -f ${pid_file} ;;
*) usage ;;
esac
22. 6. Running Web
namespace :resque do
task :setup => :environment
desc 'kick off resque-web'
task :web => :environment do
$stdout.sync=true
$stderr.sync=true
puts `env RAILS_ENV=#{RAILS_ENV} resque-web #{RAILS_ROOT}/config/initializers/resque.rb`
end
end
24. 5. Monitoring Work
bdrb resque
ad-hoc queries SQL redis query
did it run? custom resque-meta
did it fail? hoptoad !
rerun !
have id ! resque-meta
que health sample controller !
Did the job run?
resque assumes all worked - only tells you failures. not good enough for us
25. Pausing Workers
signal what happens when to use
quit wait for child & exit gracefully shutdown
term / int immediately kill child & exit shutdown now
usr1 immediately kill child stale child
usr2 don't start any new jobs
cont start to process new jobs
26. Testing Worker
bdrb resque
testing queue mid-easy resque_unit
testing command !
all workers same !
interface only !
28. Extending with Hooks
resque hooks
around_enqueue "
after_enqueue !
before_perform !
around_perform !/"
after_perform !
all plugins want to extend enqueue - not compatible
need to be able to alter arguments (e.g.: add id for meta plugins)
29. Conclusion
! Boss got no pages in first month of implementation
! no memory leaks, great uptime (don't need monit...)
! Fast
! generalized workers increases throughput (nightly vs 1 hour)
! minimal custom code
! still some intimidation
! Eating flavor of the month