As part of Rundeck Enterprise*, most users install and enable Rundeck in a “cluster” mode.  Depending on the license, this is a set of 2 or more servers that work together against a common database to provide robust usability for Rundeck users.  There are also other elements that should be shared as well, such as shared execution log storage.  At base, applying the right license and using a shared database gets you the basic working cluster.

To really get the most out of clustering, there are a few related settings that may be configured in config files on each server.  The settings aren’t always easy to process the first time through so we’ll provide some context here on the settings and also provide some clarifying examples.

Remember as a Rundeck Enterprise customer, our Support and Customer Success Teams are here to assist you at any time

The following video is a summary of this article:

A word of note, before we dive further: Rundeck users have found that the default clustering settings are quite effective for a vast majority of environments. We recommend analyzing your use patterns before embarking on any major customizations.

Architecture

Before we get into details, let’s review the overall architecture and the pieces that are relevant to these settings.

 

Primarily, the settings we’ll describe here are related exclusively to the Rundeck servers themselves (inside the blue box in the image).  Jobs will always run on one of these servers and we’ll be tweaking the settings to determine which server should own the job, depending on circumstances.

Job Ownership

Scheduled Jobs

Scheduled jobs are always owned by a single Rundeck server cluster member.
The default job owner is the cluster member where the schedule was first created.  If you were logged in to Server1 when you first set a schedule for JobA, JobA’s scheduled runs will happen on Server1.

This behavior can be over-ridden through Cluster Manager.

One-off Jobs

Jobs that are run at a single point in time (rather than on a schedule) are also owned by a single Rundeck cluster member.  By default, the job owner is the cluster member a user is connected to when the job run is initiated.  If you were logged in to Server1 when you initiated a job run for JobA, that run happened on Server1.

This default behavior can be over-ridden by Cluster Remote Execution Policies.

Heartbeat Settings

“Heartbeat” is a setting used with all Rundeck servers to measure whether another server is still alive and able to execute jobs.
The results of a heartbeat check can be used to determine logically whether a particular server can be depended on by other settings or processes, such as remote execution or autotakeover policies.

Heartbeat Settings Configuration

# heartbeat interval in seconds
rundeck.clusterMode.heartbeat.interval=30
# initial delay after startup to send heartbeat
rundeck.clusterMode.heartbeat.delay=10
# remote execute/abort message processing interval in seconds 
rundeck.clusterMode.remoteExec.process.interval=10
# seconds since heartbeat to consider another member inactive 
rundeck.clusterMode.heartbeat.considerInactive=150
# seconds since heartbeat to consider another member dead 
rundeck.clusterMode.heartbeat.considerDead=300
# settings should be added to rundeck-config.properties

Cluster Remote Execution Policies

Based on policy configurations, Rundeck Enterprise cluster members may forward job executions to other cluster members.  The default policy is to have each job executed locally.  If a job is executed in a project which is not assigned to a specific profile, that default policy is used.  You can define multiple profiles and assign different projects to different profiles.

Execution policies are defined on each server in rundeck-config.properties.

Cluster Remote Execution Configuration – Policy

The policy setting indicates what method of assignment will be used for jobs in a particular policy.

rundeck.clusterMode.remoteExecution.policy=<Policy>
# settings should be added to rundeck-config.properties

Valid settings for <Policy>:

• None – Default. Executes locally only
• Random – Executes randomly among allowed members
• RoundRobin – Executes round-robin style among allowed members
• Preset – Executes on one other preset member
• Load – Executes on a member based on load (Cluster 2.3.0+)

Cluster Remote Execution Configuration – Member Tags

Tags are used to restrict which Cluster Members could be assigned to run a remote execution.  Tags are not the only way to define which cluster members could receive jobs but they are the most flexible and are commonly used.

Tags are assigned to individual servers by being added in each framework.properties file as desired.

rundeck.clusterMode.remoteExecution.config.allowedTags=tag1
# settings should be added to rundeck-config.properties

“Allowed” tags indicate any machines that could have a job assigned to them as part of this policy.

rundeck.clusterMode.remoteExecution.config.preferredTags=tag1
# settings should be added to rundeck-config.properties

“Preferred” tags indicate machines that should be given job executions by preference as part of this policy.  Preferred machines should be a subset of your allowed machine.  If your policy has both allowed and preferred machines, allowed tags would usually only receive a job if there are no Preferred+Allowed tagged machines available.

Cluster Remote Execution Configuration – List of Allowed Members

Rather than using tags, you can define allowed members statically.  If used in conjunction with tags, the tagged servers would be used by preference.

rundeck.clusterMode.remoteExecution.config.allowed=Other,Self
# settings should be added to rundeck-config.properties

Valid values for config.allowed

• Self – Execute locally
• Other – Any other member except this one
• UUID – A particular UUID
• /regex/ – A regular expression matching a UUID

Cluster Remote Execution Configuration – Active Only

If set to true, jobs will only be delegated to an active machine (one with a heartbeat).

rundeck.clusterMode.remoteExecution.config.activeOnly=true
# settings should be added to rundeck-config.properties

Cluster Remote Execution Configuration – Profile Name

Profile names are used to distinguish multiple profiles from one another for more complex configurations.

rundeck.clusterMode.remoteExecution.profiles=profile1,profile2
# settings should be added to rundeck-config.properties

This setting is un-necessary if you’re only using one defined profile.

Cluster Remote Execution Configuration – Project Name

Each policy can be associated with one or more projects, if desired.  This can be used to determine if jobs in a specific project should always be executed on specific machines.

rundeck.clusterMode.remoteExecution.profile.profile1.projects=BlueProject,GreenProject
# settings should be added to rundeck-config.properties

The optional project name setting is provided in the context of a specific profile.  Typically, this would only occur if you have more than one profile in play.

Cluster Remote Execution Configuration – Load Balanced Policy

When a remote execution configuration policy is set to load, jobs will be assigned to servers based on statistics calculated through the heartbeat process. Load is calculated based on thread ratio and percentage of CPU for each member.

rundeck.clusterMode.remoteExecution.config.criteria=threadRatio,load
# settings should be added to rundeck-config.properties

This setting identifies which criteria will be used to compute load for this policy.

rundeck.clusterMode.remoteExecution.config.weights=1.0,1.5
# settings should be added to rundeck-config.properties

Defines relative computational value of the criteria.  In this example, load is weighted at 50% more than threadRatio.

rundeck.clusterMode.remoteExecution.config.groupWeight=1,0,0,0
# settings should be added to rundeck-config.properties

Cluster members are sorted by weighted load and placed into groups.  Each group has a weight and the policy randomly chooses a group based on the proportional weight of the group. A group member is then chosen randomly and used.

This setting defines four groups, each with 25% of available members.  Weights define 100% chance of using group 1.

Cluster Remote Execution Configuration – Examples

# Example #1
# Set method as Random
rundeck.clusterMode.remoteExecution.policy=Random
# Assign jobs to any other cluster member
rundeck.clusterMode.remoteExecution.config.allowed=other
# Only assign to active cluster members
rundeck.clusterMode.remoteExecution.config.activeOnly=true
# settings should be added to rundeck-config.properties

# Example #2
# Set method as RoundRobin
rundeck.clusterMode.remoteExecution.policy=RoundRobin
# Assign jobs to the server initiating the job or UUIDs that match regex
rundeck.clusterMode.remoteExecution.config.allowed=self,/1C519C5A-4E78-4BE9-85EC-.+/
# Only assign to active cluster members
rundeck.clusterMode.remoteExecution.config.activeOnly = true
# settings should be added to rundeck-config.properties

# Example Profile Names
# Indicate how many profiles are in use by naming them
rundeck.clusterMode.remoteExecution.profiles=profile1,profile2
# settings should be added to rundeck-config.properties

# Example #3
# All jobs from these projects should follow this policy.
rundeck.clusterMode.remoteExecution.profile.profile1.projects=projectA,projectB
# First priority is to assign jobs preferred tag cluster members
rundeck.clusterMode.remoteExecution.profile.profile1.config.preferredTags=worker,secondary
# Second priority is to assign jobs to other cluster members with any tags
rundeck.clusterMode.remoteExecution.profile.profile1.config.allowedTags=*
# If no tagged cluster members are available, assign to any other cluster member
rundeck.clusterMode.remoteExecution.profile.profile1.config.allowed=other
# For the available server set, use RoundRobin as method.
rundeck.clusterMode.remoteExecution.profile.profile1.policy=RoundRobin
# Always assign to any active cluster members
rundeck.clusterMode.remoteExecution.profile.profile1.config.activeOnly=true
# settings should be added to rundeck-config.properties

# Example #4
# Method of assignment for this policy is preset
rundeck.clusterMode.remoteExecution.profile.profile2.policy=Preset
# Indicate the preset assignee by UUID
rundeck.clusterMode.remoteExecution.profile.profile2.config.allowed=eff4405a-58aa-4f14-b7bd-68e1e44b53d4
# Always assign to any active cluster members 
rundeck.clusterMode.remoteExecution.profile.profile2.config.activeOnly=true
# settings should be added to rundeck-config.properties

Auto-Takeover Policy

If a cluster member goes down, an auto-takeover policy helps moved scheduled jobs to another cluster member.

Autotakeover uses heartbeat settings to determine whether a machine is alive or not for purposes of triggering autotakeover.

For non-scheduled jobs, autotakeover will not kick in.  Instead, your remote execution policies can account for this situation.

Auto-Takeover Configuration

# enables autotakeover for members detected as "dead"
rundeck.clusterMode.autotakeover.enabled=true
# policy indicates which nodes to take over. 
# "Any": all dead nodes. "Static": only allowed uuids 
rundeck.clusterMode.autotakeover.policy=any
# delay in seconds to wait after sending autotakeover proposal
rundeck.clusterMode.autotakeover.delay=60
# sleep in minimum seconds between autotakeover attempts
# for a particular destination
rundeck.clusterMode.autotakeover.sleep=300
# Name servers to act as takeover destinations, by UUID
rundeck.clusterMode.autotakeover.config.allowed=<uuid1>,<uuid2>,...
# settings should be added to rundeck-config.properties

*Clustering is a Rundeck Enterpise exclusive feature, and is not supported in the free version of Rundeck