MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZENorvald Ryeng
This presentation focuses on two of the new features in MySQL 8.0.18: hash joins and EXPLAIN ANALYZE. It covers how these features work, both on the surface and on the inside, and how you can use them to improve your queries and make them go faster.
Both features are the result of major refactoring of how the MySQL executor works. In addition to explaining and demonstrating the features themselves, the presentation looks at how the investment in a new iterator based executor prepares MySQL for a future with faster queries, greater plan flexibility and even more SQL features.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
The Query Optimizer is the “brain” of your Postgres database. It interprets SQL queries and determines the fastest method of execution. Using the EXPLAIN command , this presentation shows how the optimizer interprets queries and determines optimal execution.
This presentation will give you a better understanding of how Postgres optimally executes their queries and what steps you can take to understand and perhaps improve its behavior in your environment.
To listen to the webinar recording, please visit EnterpriseDB.com > Resources > Ondemand Webcasts
If you have any questions please email sales@enterprisedb.com
Database in Kubernetes: Diagnostics and MonitoringSveta Smirnova
Kubernetes is the new cool in 2023. Many database installations are on Kubernetes now. And this creates challenges for Support engineers because traditional monitoring and diagnostic tools work differently on bare hardware and Kubernetes. In this session, I will focus on differences in methods we use to collect metrics, describe challenges that Percona Support hits when working with database installations on Kubernetes, and discuss how we resolve them. This talk will cover all database technologies we support: MySQL, MongoDB, and PostgreSQL.
Presented at Percona Live 2023
PACKAGES, Package Specification and Scope, Create Package Syntax, Declaring Procedures and Functions within a Package, Package Body, Create Package Body Syntax,Example –Package, Example– Package Body, Example – Calling Package Procedure, mResults of Calling Package Procedure, Cursors in Packages , cursor Example – Package Body, Example – Use Cursor Variable
About Flexible Indexing
Postgres’ rich variety of data structures and data-type specific indexes can be confusing for newer and experienced Postgres users alike who may be unsure when and how to use them. For example, gin indexing specializes in the rapid lookup of keys with many duplicates — an area where traditional btree indexes perform poorly. This is particularly useful for json and full text searching. GiST allows for efficient indexing of two-dimensional values and range types.
To listen to the recorded presentation with Bruce Momjian, visit Enterprisedb.com > Resources > Webcasts > Ondemand Webcasts.
For product information and subscriptions, please email sales@enterprisedb.com.
This presentation gives a clear and concise description of joins in sql and several types of sql joins.
These slides also contains the pictorial representation as well as syntax for each type of joins.
MySQL 8.0.18 latest updates: Hash join and EXPLAIN ANALYZENorvald Ryeng
This presentation focuses on two of the new features in MySQL 8.0.18: hash joins and EXPLAIN ANALYZE. It covers how these features work, both on the surface and on the inside, and how you can use them to improve your queries and make them go faster.
Both features are the result of major refactoring of how the MySQL executor works. In addition to explaining and demonstrating the features themselves, the presentation looks at how the investment in a new iterator based executor prepares MySQL for a future with faster queries, greater plan flexibility and even more SQL features.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
The Query Optimizer is the “brain” of your Postgres database. It interprets SQL queries and determines the fastest method of execution. Using the EXPLAIN command , this presentation shows how the optimizer interprets queries and determines optimal execution.
This presentation will give you a better understanding of how Postgres optimally executes their queries and what steps you can take to understand and perhaps improve its behavior in your environment.
To listen to the webinar recording, please visit EnterpriseDB.com > Resources > Ondemand Webcasts
If you have any questions please email sales@enterprisedb.com
Database in Kubernetes: Diagnostics and MonitoringSveta Smirnova
Kubernetes is the new cool in 2023. Many database installations are on Kubernetes now. And this creates challenges for Support engineers because traditional monitoring and diagnostic tools work differently on bare hardware and Kubernetes. In this session, I will focus on differences in methods we use to collect metrics, describe challenges that Percona Support hits when working with database installations on Kubernetes, and discuss how we resolve them. This talk will cover all database technologies we support: MySQL, MongoDB, and PostgreSQL.
Presented at Percona Live 2023
PACKAGES, Package Specification and Scope, Create Package Syntax, Declaring Procedures and Functions within a Package, Package Body, Create Package Body Syntax,Example –Package, Example– Package Body, Example – Calling Package Procedure, mResults of Calling Package Procedure, Cursors in Packages , cursor Example – Package Body, Example – Use Cursor Variable
About Flexible Indexing
Postgres’ rich variety of data structures and data-type specific indexes can be confusing for newer and experienced Postgres users alike who may be unsure when and how to use them. For example, gin indexing specializes in the rapid lookup of keys with many duplicates — an area where traditional btree indexes perform poorly. This is particularly useful for json and full text searching. GiST allows for efficient indexing of two-dimensional values and range types.
To listen to the recorded presentation with Bruce Momjian, visit Enterprisedb.com > Resources > Webcasts > Ondemand Webcasts.
For product information and subscriptions, please email sales@enterprisedb.com.
This presentation gives a clear and concise description of joins in sql and several types of sql joins.
These slides also contains the pictorial representation as well as syntax for each type of joins.
Have you ever wondered about the internal structure of indexes in postgres? What's the difference between a btree and a gin index? How are indexes searches etc. ? This talk is about this.
Required for certification, window functions are powerful analytic functions built into SQL Server. SQL Server 2005 introduced the core window ranking functions, and SQL Server 2012 added time and statistical percentage window functions. These functions allow for advanced variable creation, and are of direct benefit to people creating features for data science. This talk will also recommend further reading on this topic.
Let's get into several common types of queries that developers struggle with, showing SQL solutions, and then analyze them for optimal efficiency. I'll cover Exclusion Join, Random Selection, Greatest-Per-Group, Dynamic Pivot, and Relational Division.
Top 65 SQL Interview Questions and Answers | EdurekaEdureka!
** MYSQL DBA Certification Training https://www.edureka.co/mysql-dba **
This Edureka PPT on Top 65 SQL Interview Question and Answers will help you to prepare yourself for Database Administrators Interviews. It covers questions for beginners, intermediate and experienced professionals.
Follow us to never miss an update in the future.
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
Tras muchos años de llevar adelante procesos de Automatización, hemos ayudado a muchos departamentos de IT a hacer más eficientes sus propios procesos. En este webinar, presentamos los casos más frecuentes de procesos de IT que se pueden automatizar utilizando diferentes tecnologías, así como también ejemplos reales de proyectos innovadores de automatización.
Conozca los tipos de proyectos que están llevando a cabo sus colegas de IT, cómo lo hacen y qué softwares utilizan.
Vea este webinar en el que un experto en Automatización le explicará:
Diferencias y usos de distintas tecnologías de automatización: RPA, scheduling, MFT, scripts y más.
Ideas de automatización de procesos de IT: generación de reportes, procesos nocturnos, resolución automática de incidencias, chequeos manuales de operación y muchos más!
Ejemplos de proyectos reales de equipos de IT en todo el mundo.
Consejos a tener en cuenta para que un proyecto de automatización sea exitoso.
A presentation about new features and enhancements related to indexes and indexing in Oracle 12c.
See also the related post: http://db-oriented.com/2015/07/03/indexes-and-indexing-in-oracle-12c
Have you ever wondered about the internal structure of indexes in postgres? What's the difference between a btree and a gin index? How are indexes searches etc. ? This talk is about this.
Required for certification, window functions are powerful analytic functions built into SQL Server. SQL Server 2005 introduced the core window ranking functions, and SQL Server 2012 added time and statistical percentage window functions. These functions allow for advanced variable creation, and are of direct benefit to people creating features for data science. This talk will also recommend further reading on this topic.
Let's get into several common types of queries that developers struggle with, showing SQL solutions, and then analyze them for optimal efficiency. I'll cover Exclusion Join, Random Selection, Greatest-Per-Group, Dynamic Pivot, and Relational Division.
Top 65 SQL Interview Questions and Answers | EdurekaEdureka!
** MYSQL DBA Certification Training https://www.edureka.co/mysql-dba **
This Edureka PPT on Top 65 SQL Interview Question and Answers will help you to prepare yourself for Database Administrators Interviews. It covers questions for beginners, intermediate and experienced professionals.
Follow us to never miss an update in the future.
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
Tras muchos años de llevar adelante procesos de Automatización, hemos ayudado a muchos departamentos de IT a hacer más eficientes sus propios procesos. En este webinar, presentamos los casos más frecuentes de procesos de IT que se pueden automatizar utilizando diferentes tecnologías, así como también ejemplos reales de proyectos innovadores de automatización.
Conozca los tipos de proyectos que están llevando a cabo sus colegas de IT, cómo lo hacen y qué softwares utilizan.
Vea este webinar en el que un experto en Automatización le explicará:
Diferencias y usos de distintas tecnologías de automatización: RPA, scheduling, MFT, scripts y más.
Ideas de automatización de procesos de IT: generación de reportes, procesos nocturnos, resolución automática de incidencias, chequeos manuales de operación y muchos más!
Ejemplos de proyectos reales de equipos de IT en todo el mundo.
Consejos a tener en cuenta para que un proyecto de automatización sea exitoso.
A presentation about new features and enhancements related to indexes and indexing in Oracle 12c.
See also the related post: http://db-oriented.com/2015/07/03/indexes-and-indexing-in-oracle-12c
This talk covers a successful utilization of Rails Engines to share features that cut across the layers of MVC in different Rails 3 projects. Rails Engines thus provide the best of both worlds: improved productivity by reusing MVC code (including assets like Javascript, CSS, and Images) and better flexibility by allowing different applications to customize behavior as needed without reliance on application-dependent conditionals. Rails Engine patterns will be provided to guide developers on how to leverage Rails Engines' reusability and flexibility without sacrificing maintainability.
ActiveWarehouse/ETL - BI & DW for Ruby/RailsPaul Gallagher
Presentation delivered at the Singapore Ruby Brigade meetup 6-Jan-2010 (at hackerspace.sg). Discusses BI and DW in the Rails context, and test drives ActiveWarehouse and ActiveWarehouse/ETL with a "Cupcakes Inc" sample application.
6 reasons Jubilee could be a Rubyist's new best friendForrest Chang
(Video here: http://confreaks.com/videos/5014-RubyConf2014-6-reasons-jubilee-could-be-a-rubyist-s-new-best-friend or https://www.youtube.com/watch?feature=player_embedded&v=FFR0G89WXI8)
Rubyconf 2014 talk on Jubilee, a Vert.x module that runs rack apps.
Alternate titles
Beyond Rails while using Rails
Rails can't do everything I want and <fill> makes me want to cry
Rubyconf abstract
Do you do web development in Ruby? Have you been forced to go to node or other technologies just for concurrency/websockets etc. Do miss your gems, and tire of functionality you have to implement from scratch? Do you hate javascript?
Well no need to switch languages/platforms, Jubilee could be your new best friend.
Jubilee, a rack server on top of Vert.x gives you
* Concurrency
* Speed
* Easy Websockets support
* Shared Memory
* Access to the JVM ecosystem
* Ability to reuse your existing Ruby knowledge and gems
"Say Hello to your new friend" - Al Pacino
- Ruby on Rails (RoR) is a platform of choice for emerging technology firms in eCommerce, Cloud, e-Business and mobile space.
- Neev is a leading provider of technology services in the emerging technology space and has developed a strong competency and resource pool with RoR skills.
- Neev also offers its RoR development services through turn-key projects or Outsourced Product Development platforms.
- Neev can help setup teams of 1 to 20 resources for a duration of 3 months to 12 months through a dedicated team model.
Ruby on Rails versus Django - A newbie Web Developer's Perspective -Shreyank...ThoughtWorks
As I have been developing Web Applications for just more than an year now, I can safely introduce myself as a Newbie Web Developer. A year ago, when I started out, my first choice when it came to building Web Apps was Ruby on Rails. The reason was the fascination. There was a steep learning curve from my Python background, but it was all worth it.
Today, working in the industry as a Web Programmer, I program applications in both Ruby on Rails and Django. And during my coffee-breaks, when I sit down and retrospect, I start comparing. And when I compare, I come up with advantages and disadvantages of both the web frameworks.
Both Ruby on Rails and Django are no doubt (in my opinion) the two best Web Frameworks in existence. Both have their own bits of Superiority and Inferiority when compared with the other. My talk aims to put these bits in front of the audience and have a little discussion on the areas of improvements.
Compared to 2 - 3 years back today's eco-system around IT Operations has changed significantly, The definitive responsibility line between infrastructure administrators and developers has been blurring continuously. We have experienced that developers want to experience same level of ease as its publicly available in form of commodity hardware, computing, monitoring, deployment and source control systems. Lots of IT organizations still go through lot of manual process for provisioning, monitoring, analysis, setup and do not take advantage new age tools. We had these issues in our existing environment as well. WE have been managing our servers using new age configuration management tools but thats not sufficient as we did not bring that advantage to our end users.
Apache thrift-RPC service cross languagesJimmy Lai
This slides illustrate how to use Apache Thrift for building RPC service and provide demo example code in Python. The example scenario is: we have a prepared machine learning model, and we'd like to load the model in advance as a server for providing prediction service.
Talk given at ClueCon 2016 that discusses FreeSWITCH and its place in a microservices architecture. Covers a specific deployment case using Docker and Adhearsion, along with certain features that make FreeSWITCH a model use-case for such a technology stack.
A fun filled tour through distributed programming with the Ruby standard library.
Presented on February 2nd, 2012 at RubyFuza in Cape Town, South Africa.
A language for the Internet: Why JavaScript and Node.js is right for Internet...Tom Croucher
Increasingly we want to do more with the web and Internet applications we build. We have more features, more data, more users, more devices and all of it needs to be in real-time. With all of these demands how can we keep up? The answer is choosing a language and a platform that are optimized for the kind of architecture Internet and web applications really have. The traditional approach prioritises computation, assigning server resources before they are actually needed. JavaScript and Node.js both take an event driven approach only assigning resources to events as they happen. This allows us to make dramatic gains in performance and resource utilization while still having an environment which is fun and easy to program.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
2. About Me
• a.k.a. ihower
• http://ihower.tw
• http://twitter.com/ihower
• http://github.com/ihower
• Ruby on Rails Developer since 2006
• Ruby Taiwan Community
• http://ruby.tw
3. Agenda
• Distributed Ruby
• Distributed Message Queues
• Background-processing in Rails
• Message Queues for Rails
• SOA for Rails
• Distributed Filesystem
• Distributed database
5. DRb
• Ruby's RMI system
(remote method invocation)
• an object in one Ruby process can invoke
methods on an object in another Ruby
process on the same or a different machine
6. DRb (cont.)
• no defined interface, faster development time
• tightly couple applications, because no
defined API, but rather method on objects
• unreliable under large-scale, heavy loads
production environments
7. server example 1
require 'drb'
class HelloWorldServer
def say_hello
'Hello, world!'
end
end
DRb.start_service("druby://127.0.0.1:61676",
HelloWorldServer.new)
DRb.thread.join
10. server example 2
require 'drb'
require 'user'
class UserServer
attr_accessor :users
def find(id)
self.users[id-1]
end
end
user_server = UserServer.new
user_server.users = []
5.times do |i|
user = User.new
user.username = i + 1
user_server.users << user
end
DRb.start_service("druby://127.0.0.1:61676", user_server)
DRb.thread.join
13. Why? DRbUndumped
• Default DRb operation
• Pass by value
• Must share code
• With DRbUndumped
• Pass by reference
• No need to share code
14. Example 2 Fixed
# user.rb
class User
include DRbUndumped
attr_accessor :username
end
# <DRb::DRbObject:0x1003b84f8 @ref=2149433940,
@uri="druby://127.0.0.1:61676">
# Username: 2
# Username: ihower
15. Why use DRbUndumped?
• Big objects
• Singleton objects
• Lightweight clients
• Rapidly changing software
16. ID conversion
• Converts reference into DRb object on server
• DRbIdConv (Default)
• TimerIdConv
• NamedIdConv
• GWIdConv
17. Beware of garbage
collection
• referenced objects may be collected on
server (usually doesn't matter)
• Building Your own ID Converter if you want
to control persistent state.
18. DRb security
require 'drb'
ro = DRbObject.new_with_uri("druby://127.0.0.1:61676")
class << ro
undef :instance_eval
end
# !!!!!!!! WARNING !!!!!!!!! DO NOT RUN
ro.instance_eval("`rm -rf *`")
20. DRb security (cont.)
• Access Control Lists (ACLs)
• via IP address array
• still can run denial-of-service attack
• DRb over SSL
21. Rinda
• Rinda is a Ruby port of Linda distributed
computing paradigm.
• Linda is a model of coordination and communication among several parallel processes
operating upon objects stored in and retrieved from shared, virtual, associative memory. This
model is implemented as a "coordination language" in which several primitives operating on
ordered sequence of typed data objects, "tuples," are added to a sequential language, such
as C, and a logically global associative memory, called a tuplespace, in which processes
store and retrieve tuples. (WikiPedia)
22. Rinda (cont.)
• Rinda consists of:
• a TupleSpace implementation
• a RingServer that allows DRb services to
automatically discover each other.
23. RingServer
• We hardcoded IP addresses in DRb
program, it’s tight coupling of applications
and make fault tolerance difficult.
• RingServer can detect and interact with
other services on the network without
knowing IP addresses.
24. 1. Where Service X?
RingServer
via broadcast UDP
address
2. Service X: 192.168.1.12
Client
@192.1681.100
3. Hi, Service X @ 192.168.1.12
Service X
@ 192.168.1.12
4. Hi There 192.168.1.100
25. ring server example
require 'rinda/ring'
require 'rinda/tuplespace'
DRb.start_service
Rinda::RingServer.new(Rinda::TupleSpace.new)
DRb.thread.join
26. service example
require 'rinda/ring'
class HelloWorldServer
include DRbUndumped # Need for RingServer
def say_hello
'Hello, world!'
end
end
DRb.start_service
ring_server = Rinda::RingFinger.primary
ring_server.write([:hello_world_service, :HelloWorldServer, HelloWorldServer.new,
'I like to say hi!'], Rinda::SimpleRenewer.new)
DRb.thread.join
27. client example
require 'rinda/ring'
DRb.start_service
ring_server = Rinda::RingFinger.primary
service = ring_server.read([:hello_world_service, nil,nil,nil])
server = service[2]
puts server.say_hello
puts service.inspect
# Hello, world!
# [:hello_world_service, :HelloWorldServer, #<DRb::DRbObject:0x10039b650
@uri="druby://fe80::21b:63ff:fec9:335f%en1:57416", @ref=2149388540>, "I like
to say hi!"]
28. TupleSpaces
• Shared object space
• Atomic access
• Just like bulletin board
• Tuple template is
[:name, :Class, object, ‘description’ ]
29. 5 Basic Operations
• write
• read
• take (Atomic Read+Delete)
• read_all
• notify (Callback for write/take/delete)
30. Starfish
• Starfish is a utility to make distributed
programming ridiculously easy
• It runs both the server and the client in
infinite loops
• MapReduce with ActiveRecode or Files
31. starfish foo.rb
# foo.rb
class Foo
attr_reader :i
def initialize
@i = 0
end
def inc
logger.info "YAY it incremented by 1 up to #{@i}"
@i += 1
end
end
server :log => "foo.log" do |object|
object = Foo.new
end
client do |object|
object.inc
end
32. starfish server example
ARGV.unshift('server.rb')
require 'rubygems'
require 'starfish'
class HelloWorld
def say_hi
'Hi There'
end
end
Starfish.server = lambda do |object|
object = HelloWorld.new
end
Starfish.new('hello_world').server
33. starfish client example
ARGV.unshift('client.rb')
require 'rubygems'
require 'starfish'
Starfish.client = lambda do |object|
puts object.say_hi
exit(0) # exit program immediately
end
Starfish.new('hello_world').client
34. starfish client example (another way)
ARGV.unshift('server.rb')
require 'rubygems'
require 'starfish'
catch(:halt) do
Starfish.client = lambda do
|object|
puts object.say_hi
throw :halt
end
Starfish.new
('hello_world').client
end
puts "bye bye"
35. MapReduce
• introduced by Google to support
distributed computing on large data sets on
clusters of computers.
• inspired by map and reduce functions
commonly used in functional programming.
37. starfish client example
ARGV.unshift('client.rb')
require 'rubygems'
require 'starfish'
Starfish.client = lambda { |logs|
logs.each do |log|
puts "Processing #{log}"
sleep(1)
end
}
Starfish.new("log_server").client
38. Other implementations
• Skynet
• Use TupleSpace or MySQL as message queue
• Include an extension for ActiveRecord
• http://skynet.rubyforge.org/
• MRToolkit based on Hadoop
• http://code.google.com/p/mrtoolkit/
39. MagLev VM
• a fast, stable, Ruby implementation with
integrated object persistence and
distributed shared cache.
• http://maglev.gemstone.com/
• public Alpha currently
42. Why not DRb?
• DRb has security risk and poorly designed APIs
• distributed message queue is a great way to do
distributed programming: reliable and scalable.
43. Starling
• a light-weight persistent queue server that
speaks the Memcache protocol (mimics its
API)
• Fast, effective, quick setup and ease of use
• Powered by EventMachine
http://eventmachine.rubyforge.org/EventMachine.html
• Twitter’s open source project, they use it
before 2009. (now switch to Kestrel, a port of Starling from Ruby
to Scala)
45. Starling set example
require 'rubygems'
require 'starling'
starling = Starling.new('192.168.1.4:22122')
100.times do |i|
starling.set('my_queue', i)
end
append to the queue, not
overwrite in Memcached
46. Starling get example
require 'rubygems'
require 'starling'
starling = Starling.new('192.168.2.4:22122')
loop do
puts starling.get("my_queue")
end
47. get method
• FIFO
• After get, the object is no longer in the
queue. You will lost message if processing
error happened.
• The get method blocks until something is
returned. It’s infinite loop.
48. Handle processing
error exception
require 'rubygems'
require 'starling'
starling = Starling.new('192.168.2.4:22122')
results = starling.get("my_queue")
begin
puts results.flatten
rescue NoMethodError => e
puts e.message
Starling.set("my_queue", [results])
rescue Exception => e
Starling.set("my_queue", results)
raise e
end
49. Starling cons
• Poll queue constantly
• RabbitMQ can subscribe to a queue that
notify you when a message is available for
processing.
50. AMQP/RabbitMQ
• a complete and highly reliable enterprise
messaging system based on the emerging
AMQP standard.
• Erlang
• http://github.com/tmm1/amqp
• Powered by EventMachine
51. Stomp/ActiveMQ
• Apache ActiveMQ is the most popular and
powerful open source messaging and
Integration Patterns provider.
• sudo gem install stomp
• ActiveMessaging plugin for Rails
52. beanstalkd
• Beanstalk is a simple, fast workqueue
service. Its interface is generic, but was
originally designed for reducing the latency
of page views in high-volume web
applications by running time-consuming tasks
asynchronously.
• http://kr.github.com/beanstalkd/
• http://beanstalk.rubyforge.org/
• Facebook’s open source project
53. Why we need asynchronous/
background-processing in Rails?
• cron-like processing
text search index update etc)
(compute daily statistics data, create reports, Full-
• long-running tasks (sending mail, resizing photo’s, encoding videos,
generate PDF, image upload to S3, posting something to twitter etc)
• Server traffic jam: expensive request will block
server resources(i.e. your Rails app)
• Bad user experience: they maybe try to reload
and reload again! (responsive matters)
57. cron
• Cron is a time-based job scheduler in Unix-
like computer operating systems.
• crontab -e
58. Whenever
http://github.com/javan/whenever
• A Ruby DSL for Defining Cron Jobs
• http://asciicasts.com/episodes/164-cron-in-ruby
• or http://cronedit.rubyforge.org/
every 3.hours do
runner "MyModel.some_process"
rake "my:rake:task"
command "/usr/bin/my_great_command"
end
60. rufus-scheduler
http://github.com/jmettraux/rufus-scheduler
• scheduling pieces of code (jobs)
• Not replacement for cron/at since it runs
inside of Ruby.
require 'rubygems'
require 'rufus/scheduler'
scheduler =
Rufus::Scheduler.start_new
scheduler.every '5s' do
puts 'check blood pressure'
end
scheduler.join
61. Daemon Kit
http://github.com/kennethkalmer/daemon-kit
• Creating Ruby daemons by providing a
sound application skeleton (through a
generator), task specific generators (jabber
bot, etc) and robust environment
management code.
62. Monitor your daemon
• http://mmonit.com/monit/
• http://github.com/arya/bluepill
• http://god.rubyforge.org/
66. run_later plugin
http://github.com/mattmatt/run_later
• Borrowed from Merb
• Uses worker thread and a queue
• Simple solution for simple tasks
run_later do
AccountMailer.deliver_signup(@user)
end
68. spawn (cont.)
• By default, spawn will use the fork to spawn
child processes.You can configure it to do
threading.
• Works by creating new database
connections in ActiveRecord::Base for the
spawned block.
• Fock need copy Rails every time
69. threading vs. forking
• Forking advantages:
• more reliable? - the ActiveRecord code is not thread-safe.
• keep running - subprocess can live longer than its parent.
• easier - just works with Rails default settings. Threading
requires you set allow_concurrency=true and. Also,
beware of automatic reloading of classes in development
mode (config.cache_classes = false).
• Threading advantages:
• less filling - threads take less resources... how much less?
it depends.
• debugging - you can set breakpoints in your threads
70. Okay, we need
reliable messaging system:
• Persistent
• Scheduling: not necessarily all at the same time
• Scalability: just throw in more instances of your
program to speed up processing
• Loosely coupled components that merely ‘talk’
to each other
• Ability to easily replace Ruby with something
else for specific tasks
• Easy to debug and monitor
72. Rails only?
• Easy to use/write code
• Jobs are Ruby classes or objects
• But need to load Rails environment
73. ar_mailer
http://seattlerb.rubyforge.org/ar_mailer/
• a two-phase delivery agent for ActionMailer.
• Store messages into the database
• Delivery by a separate process, ar_sendmail
later.
74. BackgroundDRb
http://backgroundrb.rubyforge.org/
• BackgrounDRb is a Ruby job server and
scheduler.
• Have scalability problem due to
Mark Bates)
(~20 servers for
• Hard to know if processing error
• Use database to persist tasks
• Use memcached to know processing result
75. workling
http://github.com/purzelrakete/workling
• Gives your Rails App a simple API that you
can use to make code run in the
background, outside of the your request.
• Supports Starling(default), BackgroundJob,
Spawn and AMQP/RabbitMQ Runners.
77. Workling example
class EmailWorker < Workling::Base
def deliver(options)
user = User.find(options[:id])
user.deliver_activation_email
end
end
# in your controller
def create
EmailWorker.asynch_deliver( :id => 1)
end
78. delayed_job
• Database backed asynchronous priority
queue
• Extracted from Shopify
• you can place any Ruby object on its queue
as arguments
• Only load the Rails environment only once
80. delayed_job example
send_later
def deliver
mailing = Mailing.find(params[:id])
mailing.send_later(:deliver)
flash[:notice] = "Mailing is being delivered."
redirect_to mailings_url
end
81. delayed_job example
custom workers
class MailingJob < Struct.new(:mailing_id)
def perform
mailing = Mailing.find(mailing_id)
mailing.deliver
end
end
# in your controller
def deliver
Delayed::Job.enqueue(MailingJob.new(params[:id]))
flash[:notice] = "Mailing is being delivered."
redirect_to mailings_url
end
82. delayed_job example
always asynchronously
class Device
def deliver
# long running method
end
handle_asynchronously :deliver
end
device = Device.new
device.deliver
83. Running jobs
• rake jobs:works
(Don’t use in production, it will exit if the database has any network connectivity
problems.)
• RAILS_ENV=production script/delayed_job start
• RAILS_ENV=production script/delayed_job stop
84. Priority
just Integer, default is 0
• you can run multipie workers to handle different
priority jobs
• RAILS_ENV=production script/delayed_job -min-
priority 3 start
Delayed::Job.enqueue(MailingJob.new(params[:id]), 3)
Delayed::Job.enqueue(MailingJob.new(params[:id]), -3)
85. Scheduled
no guarantees at precise time, just run_after_at
Delayed::Job.enqueue(MailingJob.new(params[:id]), 3, 3.days.from_now)
Delayed::Job.enqueue(MailingJob.new(params[:id]),
3, 1.month.from_now.beginning_of_month)
86. Configuring Dealyed
Job
# config/initializers/delayed_job_config.rb
Delayed::Worker.destroy_failed_jobs = false
Delayed::Worker.sleep_delay = 5 # sleep if empty queue
Delayed::Worker.max_attempts = 25
Delayed::Worker.max_run_time = 4.hours # set to the amount of time
of longest task will take
87. Automatic retry on failure
• If a method throws an exception it will be
caught and the method rerun later.
• The method will be retried up to 25
(default) times at increasingly longer
intervals until it passes.
• 108 hours at most
Job.db_time_now + (job.attempts ** 4) + 5
88. Capistrano Recipes
• Remember to restart delayed_job after
deployment
• Check out lib/delayed_job/recipes.rb
after "deploy:stop", "delayed_job:stop"
after "deploy:start", "delayed_job:start"
after "deploy:restart", "delayed_job:restart"
89. Resque
http://github.com/defunkt/resque
• a Redis-backed library for creating background jobs,
placing those jobs on multiple queues, and processing
them later.
• Github’s open source project
• you can only place JSONable Ruby objects
• includes a Sinatra app for monitoring what's going on
• support multiple queues
• you expect a lot of failure/chaos
90. My recommendations:
• General purpose: delayed_job
(Github highly recommend DelayedJob to anyone whose site is not 50% background work.)
• Time-scheduled: cron + rake
91. 5. SOA for Rails
• What’s SOA
• Why SOA
• Considerations
• The tool set
92. What’s SOA
Service oriented architectures
• “monolithic” approach is not enough
• SOA is a way to design complex applications
by splitting out major components into
individual services and communicating via
APIs.
• a service is a vertical slice of functionality:
database, application code and caching layer
93. a monolithic web app example
request
Load
Balancer
WebApps
Database
94. a SOA example
request
Load
request
Balancer
WebApp WebApps
for Administration for User
Services A Services B
Database Database
96. Shared Resources
• Different front-web website use the same
resource.
• SOA help you avoiding duplication databases
and code.
• Why not only shared database?
• code is not DRY WebApp
for Administration
WebApps
for User
• caching will be problematic
Database
97. Encapsulation
• you can change underly implementation in
services without affect other parts of system
• upgrade library
• upgrade to Ruby 1.9
• you can provide API versioning
98. Scalability1: Partitioned
Data Provides
• Database is the first bottleneck, a single DB
server can not scale. SOA help you reduce
database load
• Anti-pattern: only split the database WebApps
• model relationship is broken
• referential integrity Database
A
Database
B
• Myth: database replication can not help you
speed and consistency
99. Scalability 2: Caching
• SOA help you design caching system easier
• Cache data at the right times and expire
at the right times
• Cache logical model, not physical
• You do not need cache view everywhere
100. Scalability 3: Efficient
• Different components have different task
loading, SOA can scale by service.
WebApps
Load
Balancer Load
Balancer
Services A Services A Services B Services B Services B Services B
101. Security
• Different services can be inside different
firewall
• You can only open public web and
services, others are inside firewall.
102. Interoperability
• HTTP is the common interface, SOA help
you integrate them:
• Multiple languages
• Internal system e.g. Full-text searching engine
• Legacy database, system
• External vendors
103. Reuse
• Reuse across multiple applications
• Reuse for public APIs
• Example: Amazon Web Services (AWS)
105. Reduce Local
Complexity
• Team modularity along the same module
splits as your software
• Understandability: The amount of code is
minimized to a quantity understandable by
a small team
• Source code control
107. How to partition into
Separate Services
• Partitioning on Logical Function
• Partitioning on Read/Write Frequencies
• Partitioning by Minimizing Joins
• Partitioning by Iteration Speed
108. API Design
• Send Everything you need
• Parallel HTTP requests
• Send as Little as Possible
• Use Logical Models
109. Physical Models &
Logical Models
• Physical models are mapped to database
tables through ORM. (It’s 3NF)
• Logical models are mapped to your
business problem. (External API use it)
• Logical models are mapped to physical
models by you.
110. Logical Models
• Not relational or normalized
• Maintainability
• can change with no change to data store
• can stay the same while the data store
changes
• Better fit for REST interfaces
• Better caching
116. XML parser
• http://nokogiri.org/
• Nokogiri ( ) is an HTML, XML, SAX, and
Reader parser. Among Nokogiri’s many
features is the ability to search documents
via XPath or CSS3 selectors.
119. Tips
• Define your logical model (i.e. your service
request result) first.
• model.to_json and model.to_xml is easy to
use, but not useful in practice.
120. 6.Distributed File System
• NFS not scale
• we can use rsync to duplicate
• MogileFS
• http://www.danga.com/mogilefs/
• http://seattlerb.rubyforge.org/mogilefs-client/
• Amazon S3
• HDFS (Hadoop Distributed File System)
• GlusterFS
123. References
• Books&Articles:
• Distributed Programming with Ruby, Mark Bates (Addison Wesley)
• Enterprise Rails, Dan Chak (O’Reilly)
• Service-Oriented Design with Ruby and Rails, Paul Dix (Addison Wesley)
• RESTful Web Services, Richardson&Ruby (O’Reilly)
• RESTful WEb Services Cookbook, Allamaraju&Amundsen (O’Reilly)
• Enterprise Recipes with Ruby on Rails, Maik Schmidt (The Pragmatic Programmers)
• Ruby in Practice, McAnally&Arkin (Manning)
• Building Scalable Web Sites, Cal Henderson (O’Reilly)
• Background Processing in Rails, Erik Andrejko (Rails Magazine)
• Background Processing with Delayed_Job, James Harrison (Rails Magazine)
• Bulinging Scalable Web Sites, Cal Henderson (O’Reilly)
• Web 点 ( )
• Slides:
• Background Processing (Rob Mack) Austin on Rails - April 2009
• The Current State of Asynchronous Processing in Ruby (Mathias Meyer, Peritor GmbH)
• Asynchronous Processing (Jonathan Dahl)
• Long-Running Tasks In Rails Without Much Effort (Andy Stewart) - April 2008
• Starling + Workling: simple distributed background jobs with Twitter’s queuing system, Rany Keddo 2008
• Physical Models & Logical Models in Rails, dan chak
125. Todo (maybe next time)
• AMQP/RabbitMQ example code
• How about Nanite?
• XMPP
• MagLev VM
• More MapReduce example code
• How about Amazon Elastic MapReduce?
• Resque example code
• More SOA example and code
• MogileFS example code