https://youtu.be/9yYwTPMRSOY Nathan Fluegel, head of Customer Success at Rundeck, talks clustering and high availability. We'll show how to deploy Rundeck servers in a clustered configuration with Rundeck Enterprise.

1. Rundeck How-To:
Advanced Cluster Settings
Nathan Fluegel May 30, 2019

2. Agenda
● Cluster Architecture
● Job Ownership
○ Scheduled jobs
○ One-off jobs
● Heartbeat Setting
● Cluster Remote Execution Policy
● Load Policy
● Auto-Takeover Policy
● Do you need these settings?

5. Job Ownership:
Scheduled Jobs
Scheduled jobs are always owned by a
single Rundeck server cluster member.
The default job owner is the cluster where
the schedule was first created.
If I had been logged in to Server1 when I
first set a schedule for JobA, JobA’s
scheduled runs will happen on Server1.
This behavior can be over-ridden through Cluster
Manager.

6. Job Ownership:
One-off Jobs
Jobs that are run at a point in time are owned
by a single Rundeck server instance within the
cluster.
By default, the job owner is the server instance
a user is connected to when the job run is
initiated.
If I had been logged in to Server1 when I
initiated a job run for JobA, that run happened
on Server1.
This default behavior can be over-ridden by
Cluster Remote Execution Policies.

7. Heartbeat
Settings
“Heartbeat” is a setting used with all
Rundeck servers to measure whether a
another server is still alive and able to
execute jobs.
The results of a heartbeat check can be
used to determine logically whether a
particular server can be depended on by
other settings or processes, such as
remote execution or autotakeover policies.

8. # heartbeat interval in seconds
rundeck.clusterMode.heartbeat.interval=30
# initial delay after startup to send heartbeat
rundeck.clusterMode.heartbeat.delay=10
# remote execute/abort message processing interval
in seconds
rundeck.clusterMode.remoteExec.process.interval=10
# age in seconds since last heartbeat to consider
another member inactive
rundeck.clusterMode.heartbeat.considerInactive=150
# age in seconds since last heartbeat to consider
another member dead
rundeck.clusterMode.heartbeat.considerDead=300
Heartbeat Settings
Configuration

9. Cluster Remote
Execution Policy
Based on policy configurations, Rundeck
Enterprise cluster members may forward
job executions to other cluster members.
The default policy is to have each job
executed locally. If a job is executed in a
project which is not assigned to a specific
profile, the default policy is used.
You can define multiple profiles and assign
different projects to different profiles.
Execution policies are defined on each
server in rundeck-config.properties.

10. rundeck.clusterMode.remoteExecution.policy = <Policy>
Valid settings for <Policy>:
None
Default. Executes locally only
Random
Executes randomly among allowed members
RoundRobin
Executes round-robin style among allowed members
Preset
Executes on one other preset member
Load
Executes on a member based on load (Cluster 2.3.0+)
Cluster Remote
Execution
Configuration
Policy
Policy setting indicates what method of
assignment will be used for jobs in a
particular policy.

11. rundeck.clusterMode.remoteExecution.config.allowedTags =
<List of Member Tags>
“Allowed” tags indicate any machines that could have a job assigned
to them as part of this policy.
rundeck.clusterMode.remoteExecution.config.preferredTags =
<List of Member Tags>
“Preferred” tags indicate machines that should be given job executions
by preference as part of this policy. Preferred machines should be a
subset of your allowed machine. If your policy has both allowed and
preferred machines, allowed tags would usually only receive a job if
there are no Preferred+Allowed tagged machines available.
Cluster Remote
Execution
Configuration
Member Tags
Tags are used to restrict which Cluster
Members could be assigned to run a
remote execution.
Tags are not the only way to define which
cluster members could receive jobs but
they are the most flexible and are
commonly used.
Tags are assigned to individual servers
by being added in each
framework.properties file as
desired.

12. rundeck.clusterMode.remoteExecution.config.allowed = <List
of Allowed Members>
Self - Execute locally
Other - Any other member except this one
UUID - A particular UUID
/regex/ - A regular expression matching a UUID
rundeck.clusterMode.remoteExecution.config.activeOnly =
true/false
Cluster Remote
Execution
Configuration
List of Allowed Members
Rather than using tags, you can define
allowed members statically.
If used in conjunction with tags, the
tagged servers would be used by
preference.
Active Only
If set to true, jobs will only be delegated
to an active machine (one with a
heartbeat).

13. rundeck.clusterMode.remoteExecution.profiles = profile1,
profile2
This setting is un-necessary if you’re only using one defined profile.
rundeck.clusterMode.remoteExecution.profile.profile1.projec
ts=projectA, projectB
The optional project name setting is provided in the context of a
specific profile. Typically, this would only occur if you have more than
one profile in play.
Cluster Remote
Execution
Configuration
Profile Name
Profile names are used to distinguish
multiple profiles from one another for
more complex configurations.
Project Name
Each policy can be associated with one
or more projects, if desired. This can be
used to determine if jobs in a specific
project should always be executed on
specific machines.

14. rundeck.clusterMode.remoteExecution.config.criteria =
threadRatio,load
This setting identifies which criteria will be used to compute load for
this policy.
rundeck.clusterMode.remoteExecution.config.weights =
1.0,1.5
Defines relative computational value of the criteria. In this example,
load is weighted at 50% more than threadRatio.
rundeck.clusterMode.remoteExecution.config.groupWeight=1,0,
0,0
Cluster members are sorted by weighted load and placed into groups.
Each group has a weight and the policy randomly chooses a group
based on the proportional weight of the group. A group member is
then chosen randomly and used.
This setting defines four groups, each with 25% of available members.
Weights define 100% chance of using group 1.
Cluster Remote
Execution
Configuration
Load Balanced Policy
When a remote execution configuration
policy is set to load, jobs will be assigned
to servers based on statistics calculated
through the heartbeat process. Load is
calculated based on thread ratio and
percentage of CPU for each member.

15. Example #1
rundeck.clusterMode.remoteExecution.policy = Random
rundeck.clusterMode.remoteExecution.config.allowed
= other
rundeck.clusterMode.remoteExecution.config.activeOn
ly = true
Example #2
rundeck.clusterMode.remoteExecution.policy =
RoundRobin
rundeck.clusterMode.remoteExecution.config.allowed
= self,/1C519C5A-4E78-4BE9-85EC-.+/
rundeck.clusterMode.remoteExecution.config.activeOn
ly = true
Cluster Remote
Execution
Configuration
Example #1
A simple policy that sends all jobs to
a random peer machine as long as
that machine has a heartbeat.
Example #2
Round robin assignment among self
and any instances with a UUID that
matches the regex.

16. rundeck.clusterMode.remoteExecution.profiles = profile1, profile2
Example #3
rundeck.clusterMode.remoteExecution.profile.profile1.policy= RoundRobin
rundeck.clusterMode.remoteExecution.profile.profile1.projects=projectA,projectB
rundeck.clusterMode.remoteExecution.profile.profile1.config.allowed = other
rundeck.clusterMode.remoteExecution.profile.profile1.config.allowedTags = *
rundeck.clusterMode.remoteExecution.profile.profile1.config.preferredTags =
worker,secondary
rundeck.clusterMode.remoteExecution.profile.profile1.config.activeOnly = true
Example #4
rundeck.clusterMode.remoteExecution.profile.profile2.policy = Preset
rundeck.clusterMode.remoteExecution.profile.profile2.config.allowed = eff4405a-
58aa-4f14-b7bd-68e1e44b53d4
rundeck.clusterMode.remoteExecution.profile.profile2.config.activeOnly = true
Cluster Remote Execution Configuration - Examples

17. Auto-Takeover
Policy
If a cluster member goes down, an auto-
takeover policy helps moved scheduled
jobs to another cluster member.
Autotakeover uses heartbeat settings to
determine whether a machine is alive or
not for purposes of triggering autotakeover.
For non-scheduled jobs, autotakeover will
not kick in. Instead, your remote execution
policies can account for this situation.

18. # enables autotakeover for members detected as "dead"
rundeck.clusterMode.autotakeover.enabled=true
# policy indicates which nodes to take over. "Any": all dead nodes. "Static": only allowed uuids
rundeck.clusterMode.autotakeover.policy=any
# delay in seconds to wait after sending autotakeover proposal
rundeck.clusterMode.autotakeover.delay = 60
# sleep in minimum seconds between autotakeover attempts for a particular destination
rundeck.clusterMode.autotakeover.sleep = 300
# Name servers to act as takeover destinations, by UUID
rundeck.clusterMode.autotakeover.config.allowed=<uuid1>,<uuid2>,...
Auto-Takeover Configuration

19. Do you need these settings?