Learn the ins and outs of running background tasks with the popular python module Celery. We'll hit the ground running. With everything you need to know to run your first task, to scaling your stack to run millions each day.
Celery is a really good framework for doing background task processing in Python (and other languages). While it is ridiculously easy to use celery, doing complex task flow has been a challenge in celery. (w.r.t task trees/graphs/dependecies etc.)
This talk introduces the audience to these challenges in celery and also explains how these can be fixed programmatically and by using latest features in Celery (3+)
Learn the ins and outs of running background tasks with the popular python module Celery. We'll hit the ground running. With everything you need to know to run your first task, to scaling your stack to run millions each day.
Celery is a really good framework for doing background task processing in Python (and other languages). While it is ridiculously easy to use celery, doing complex task flow has been a challenge in celery. (w.r.t task trees/graphs/dependecies etc.)
This talk introduces the audience to these challenges in celery and also explains how these can be fixed programmatically and by using latest features in Celery (3+)
Surviving the Java Deserialization Apocalypse // OWASP AppSecEU 2016Christian Schneider
The hidden danger of Java deserialization vulnerabilities – which often lead to remote code execution – has gained extended visibility in the past year. The issue has been known for years; however, it seems that the majority of developers were unaware of it until recent media coverage around commonly used libraries and major products. This talk aims to shed some light about how this vulnerability can be abused, how to detect it from a static and dynamic point of view, and -- most importantly -- how to effectively protect against it. The scope of this talk is not limited to the Java serialization protocol but also other popular Java libraries used for object serialization.
The ever-increasing number of new vulnerable endpoints and attacker-usable gadgets has resulted in a lot of different recommendations on how to protect your applications, including look-ahead deserialization and runtime agents to monitor and protect the deserialization process. Coming at the problem from a developer’s perspective and triaging the recommendations for you, this talk will review existing protection techniques and demonstrate their effectiveness on real applications. It will also review existing techniques and present new gadgets that demonstrates how attackers can actually abuse your application code and classpath to craft a chain of gadgets that will allow them to compromise your servers.
This talk will also present the typical architectural decisions and code patterns that lead to an increased risk of exposing deserialization vulnerabilities. Mapping the typical anti-patterns that must be avoided, through the use of real code examples we present an overview of hardening techniques and their effectiveness. The talk will also show attendees what to search the code for in order to find potential code gadgets the attackers can leverage to compromise their applications. We’ll conclude with action items and recommendations developers should consider to mitigate this threat.
--
This talk was presented by Alvaro Muñoz & Christian Schneider at the OWASP AppSecEU 2016 conference in Rome.
(video of these slides available here http://fsharpforfunandprofit.com/fppatterns/)
In object-oriented development, we are all familiar with design patterns such as the Strategy pattern and Decorator pattern, and design principles such as SOLID.
The functional programming community has design patterns and principles as well.
This talk will provide an overview of some of these, and present some demonstrations of FP design in practice.
While most bugs reveal their cause within their stack trace, Java’s OutOfMemoryError is less talkative and therefore regarded as being difficult to debug by a majority of developers. With the right techniques and tools, memory leaks in Java programs can however be tackled like any other programming error. This talks discusses how a JVM stores data, categorizes different types of memory leaks that can occur in a Java program and presents techniques for fixing such errors. Furthermore, we will have a closer look at lambda expressions and their considerable potential of introducing memory leaks when they are used incautiously.
Slides for a lightning talk on Java 8 lambda expressions I gave at the Near Infinity (www.nearinfinity.com) 2013 spring conference.
The associated sample code is on GitHub at https://github.com/sleberknight/java8-lambda-samples
Here I discuss about reactive programming, observable, observer and difference between observable and promise.
Also discuss some of important operators like forkJoin, switchMap, from, deboucneTime, discardUntilChanged, mergeMap. I discuss some of observable creation function.
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
ProxySQL is well-affirmed into thousands of production environments for the features we all know: multiplexing, query routing, and rewriting to name a few.
Let's go through those use cases which maybe are the least common: from keeping malicious eyes away from your production data to rebuilding your non-production environment, from stopping having hanging transactions to monitor your instance, from query firewalling to changing ProxySQL configuration without a single line of SQL!
When the sky is the limit, ProxySQL gives you some boost!
Threads, Queues, and More: Async Programming in iOSTechWell
To keep your iOS app running butter-smooth at 60 frames per second, Apple recommends doing as many tasks as possible asynchronously or “off the main thread.” Joe Keeley introduces you to some basic concepts of asynchronous programming in iOS. He discusses what threads and queues are, how they are related, and the special significance of the main queue to iOS. Look at what options are available in the iOS SDK to work asynchronously, including NSOperationQueues and Grand Central Dispatch. Take an in depth look at how to implement some common use cases for those options in Swift. Joe pays special attention to networking, one of the most common asynchronous use cases. Spend some time discussing common asynchronous programming pitfalls—and how to avoid them. Leave this session ready to try out asynchronous programming in your iOS app.
Surviving the Java Deserialization Apocalypse // OWASP AppSecEU 2016Christian Schneider
The hidden danger of Java deserialization vulnerabilities – which often lead to remote code execution – has gained extended visibility in the past year. The issue has been known for years; however, it seems that the majority of developers were unaware of it until recent media coverage around commonly used libraries and major products. This talk aims to shed some light about how this vulnerability can be abused, how to detect it from a static and dynamic point of view, and -- most importantly -- how to effectively protect against it. The scope of this talk is not limited to the Java serialization protocol but also other popular Java libraries used for object serialization.
The ever-increasing number of new vulnerable endpoints and attacker-usable gadgets has resulted in a lot of different recommendations on how to protect your applications, including look-ahead deserialization and runtime agents to monitor and protect the deserialization process. Coming at the problem from a developer’s perspective and triaging the recommendations for you, this talk will review existing protection techniques and demonstrate their effectiveness on real applications. It will also review existing techniques and present new gadgets that demonstrates how attackers can actually abuse your application code and classpath to craft a chain of gadgets that will allow them to compromise your servers.
This talk will also present the typical architectural decisions and code patterns that lead to an increased risk of exposing deserialization vulnerabilities. Mapping the typical anti-patterns that must be avoided, through the use of real code examples we present an overview of hardening techniques and their effectiveness. The talk will also show attendees what to search the code for in order to find potential code gadgets the attackers can leverage to compromise their applications. We’ll conclude with action items and recommendations developers should consider to mitigate this threat.
--
This talk was presented by Alvaro Muñoz & Christian Schneider at the OWASP AppSecEU 2016 conference in Rome.
(video of these slides available here http://fsharpforfunandprofit.com/fppatterns/)
In object-oriented development, we are all familiar with design patterns such as the Strategy pattern and Decorator pattern, and design principles such as SOLID.
The functional programming community has design patterns and principles as well.
This talk will provide an overview of some of these, and present some demonstrations of FP design in practice.
While most bugs reveal their cause within their stack trace, Java’s OutOfMemoryError is less talkative and therefore regarded as being difficult to debug by a majority of developers. With the right techniques and tools, memory leaks in Java programs can however be tackled like any other programming error. This talks discusses how a JVM stores data, categorizes different types of memory leaks that can occur in a Java program and presents techniques for fixing such errors. Furthermore, we will have a closer look at lambda expressions and their considerable potential of introducing memory leaks when they are used incautiously.
Slides for a lightning talk on Java 8 lambda expressions I gave at the Near Infinity (www.nearinfinity.com) 2013 spring conference.
The associated sample code is on GitHub at https://github.com/sleberknight/java8-lambda-samples
Here I discuss about reactive programming, observable, observer and difference between observable and promise.
Also discuss some of important operators like forkJoin, switchMap, from, deboucneTime, discardUntilChanged, mergeMap. I discuss some of observable creation function.
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
ProxySQL is well-affirmed into thousands of production environments for the features we all know: multiplexing, query routing, and rewriting to name a few.
Let's go through those use cases which maybe are the least common: from keeping malicious eyes away from your production data to rebuilding your non-production environment, from stopping having hanging transactions to monitor your instance, from query firewalling to changing ProxySQL configuration without a single line of SQL!
When the sky is the limit, ProxySQL gives you some boost!
Threads, Queues, and More: Async Programming in iOSTechWell
To keep your iOS app running butter-smooth at 60 frames per second, Apple recommends doing as many tasks as possible asynchronously or “off the main thread.” Joe Keeley introduces you to some basic concepts of asynchronous programming in iOS. He discusses what threads and queues are, how they are related, and the special significance of the main queue to iOS. Look at what options are available in the iOS SDK to work asynchronously, including NSOperationQueues and Grand Central Dispatch. Take an in depth look at how to implement some common use cases for those options in Swift. Joe pays special attention to networking, one of the most common asynchronous use cases. Spend some time discussing common asynchronous programming pitfalls—and how to avoid them. Leave this session ready to try out asynchronous programming in your iOS app.
MongoDB is the trusted document store we turn to when we have tough data store problems to solve. For this talk we are going to go a little bit off the path and explore what other roles we can fit MongoDB into. Others have discussed how to turn MongoDB’s capped collections into a publish/subscribe server. We stretch that a little further and turn MongoDB into a full fledged broker with both publish/subscribe and queue semantics, and a the ability to mix them. We will provide code and a running demo of the queue producers and consumers. Next we will turn to coordination services: We will explore the fundamental features and show how to implement them using MongoDB as the storage engine. Again we will show the code and demo the coordination of multiple applications.
Slides from my talk on #ruby-mg meeting.
Intro about how we in catars.me are using postgREST to create fast and simple API that can be represented with various mithril.js components.
Unit testing in JavaScript? There is no such thing.” This is something most of the Java developers would say. With AngularJS coming more and more to scene and Google standing behind it, testing is starting to be core part of all AngularJS project. I would like to show how you can do unit testing in pure JavaScript (AngularJS) application (together with backend mocking…).
apidays LIVE Australia 2020 - Building distributed systems on the shoulders o...apidays
apidays LIVE Australia 2020 - Building Business Ecosystems
Building distributed systems on the shoulders of giants
Dasith Wijesiriwardena, Telstra Purple (Readify)
Forge - DevCon 2016: Building a Drone Imagery ServiceAutodesk
3D Robotics builds drones and drone imagery services for construction, inspection, and surveying. Each drone survey can result in hundreds of high resolution images that must be uploaded to the Autodesk Reality Capture API to be converted into a textured three-dimensional mesh. This talk will discuss the challenges faced by 3D Robotics in managing such large datasets, and explain how they architected their server to solve these challenges by integrating Scala, Couchbase, and an array of other technologies with the Reality Capture API.
Javascript and first-class citizenry: require.js & node.js
Javascript on web pages is ubiquitous and its problems are legendary. Javascript, seen as a second-class code citizen, is usually hacked together even by seasoned developers. New libraries (jQuery, prototype, backbone, knockout, underscore) and runtime tools (firebug, jasmine) look like they solve many problems - and they do. But they still leave poorly written code as just that. One key problem is that all javascript code lives globally and this results in poorly managed, tested and delivered code.
In this session, I will illustrate that we can treat javascript as a first-class citizen using with require.js and node.js: it can be modular, encapsulated and easily unit tested and added to continuous integration cycle. The dependencies between javascript modules can also be managed and packaged just like in C# and Java. In the end, we can resolve many javascript difficulties at compile time rather than waiting until runtime.
An overview of key CSS concepts that are often misunderstood. These are things I wish somebody had sat me down and explained when I was just getting started.
Slides from a lecture given to 3rd-year interactive design students at the Holon Institute of Technology (IsraelI)
A wide-ranging overview of important CSS concepts for people new to CSS. These are the things I wish somebody sat me down and explained to me when I was first starting out with CSS.
Includes some material recycled from my other talks, including a little about selectors and typography.
Lecture given to 3rd year students of interactive design at the Holon Institute of Technology (Israel).
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
3. Celery is a...
Distributed
Asynchronous
Task Queue
For Django
4. Celery is a...
Distributed
Asynchronous
Task Queue
For Django
5. Celery is a...
Distributed
Asynchronous
Task Queue
For Django
6. Celery is a...
Distributed
Asynchronous
Task Queue
For Django
7. Celery is a...
Distributed
Asynchronous
Task Queue
sin
For Django 0.8 ce
8. What can I use it for?
http://www.flickr.com/photos/jabzg/2145312172/
9. Potential Uses
» Anything that needs to run
asynchronously, e.g. outside of the
request-response cycle.
» Background computation of ‘expensive
queries’ (ex. denormalized counts)
» Interactions with external API’s
(ex. Twitter)
» Periodic tasks (instead of cron & scripts)
» Long-running actions with results
displayed via AJAX.
10. How does it work?
http://www.flickr.com/photos/tomypelluz/14638999/
15. Celery Architecture
celery task result
workers store
task result (tombstone)
is written to task store:
‣RDBMS
‣memcached
‣Tokyo Tyrant
‣MongoDB
‣AMQP (new in 0.8)
24. Adding Celery
1. get & install an AMQP broker
(pay attention to vhosts, permissions)
2. add Celery to INSTALLED_APPS
3. add a few settings:
AMQP_SERVER = "localhost"
AMQP_PORT = 5672
AMQP_USER = "myuser"
AMQP_PASSWORD = "mypassword"
AMQP_VHOST = "myvhost"
4. manage.py syncdb
25. Celery Workers
» Run at least 1 celery worker server
» manage.py celeryd
(--detatch for production)
» Can be on different machines
» Celery guarantees that tasks are only
executed once
27. Tasks
» Define tasks in your app
» app_name/tasks.py
» register & autodiscovery
(like admin.py)
28. Task
from celery.task import Task
from celery.registry import tasks
class FetchUserInfoTask(Task):
def run(self, screen_name, **kwargs):
logger = self.get_logger(**kwargs)
try:
user = twitter.users.show(id=screen_name)
logger.debug("Successfully fetched {0}".format(screen_name))
except TwitterError:
logger.error("Unable to fetch {0}: {1}".format(
screen_name, TwitterError))
raise
return user
tasks.register(FetchUserInfoTask)
29. Run It!
>>> from myapp.tasks import FetchUserInfoTask
>>> result = FetchUserInfoTask.delay('idangazit')
30. Task Result
» result.ready()
true if task has finished
» result.result
the return value of the task or exception
instance if the task failed
» result.get()
blocks until the task is complete then
returns result or exception
» result.successful()
returns True/False of task success
41. Celery.Views
» Celery ships with some django views for
launching /getting the status of tasks.
» JSON views perfect for use in your AJAX
(err, AJAJ) calls.
» celery.views.apply(request, task_name, *args)
» celery.views.is_task_done(request, task_id)
» celery.views.task_status(request, task_id)
43. Routable Tasks
» "I want tasks of type X to only execute on
this specific server"
» Some extra settings in settings.py:
CELERY_AMQP_EXCHANGE = "tasks"
CELERY_AMQP_PUBLISHER_ROUTING_KEY = "task.regular"
CELERY_AMQP_EXCHANGE_TYPE = "topic"
CELERY_AMQP_CONSUMER_QUEUE = "foo_tasks"
CELERY_AMQP_CONSUMER_ROUTING_KEY = "foo.#"
» set the task's routing key:
class MyRoutableTask(Task):
routing_key = 'foo.bars'