5. When
When is an option to steps which defines rules for jobs to run. The rules are checked against output and result codes of previous job steps.
5.1. Some facts about the when
statement
When
statements are:
the most complex parts of job definitions.
configured at the step level
checks against results of other steps
decisions that define if a step should be run
defined as go templates
5.2. Special methods
PgQuartz has implementations of special methods to be used as part of the when
statements:
5.2.1. Rc
PgQuartz keeps track of all Return Codes of all commands and has a special method to collect the return codes of all commands.
the method is called Rc
and is implemented on Steps, Instances, and Commands, where:
Step.RC returns the sum of all Rc values of all instances
Instance.Rc returns the sum of all Rc values of all Commands
Command.Rc:
shell: Is set to the return code of the execution
postgresql: Is set to 0 on success, is set to 1 of failure
Note that Rc is implemented as an integer, which means that it could overflow but only with more than 16843009 commands ending in exit code 255, or even more with lower exit codes, which probably is not a realistic use case.
5.2.2. StdOut and StdErr
PgQuartz keeps track of all output of all Commands, and has a special method to collect the return codes of all commands.
the method is called Rc
and is implemented on Steps, Instances, and Commands, where:
Step.StdOut returns a concatination all StdOut values of all instances ; similar for StdErr
Instance.StdOut concatination all StdOut values of all Commands ; similar for StdErr
for shell scripts, StdOut contains the stdOut of the execution and StdErr (as expected) contains the stdErr of the execution
for postgresql queries, StdOut contains a special formatted output of the table where
every row ends on a new line
every line is compiled version of {name}={value} where name is the columns name and value is the column value (with ‘ replaced by ‘’ for values)
for postgresql queries, StdErr is empty
Note Internally PgQuartz works with a special type Result, which are meant for efficient handling of concatination of stdout and stdErr. But the less output (rows or lines) the better, so make sure as a developer that Commands produce as little output as required.
5.2.3. StdOut and StdErr
5.3. Example
We make the ‘When concept’ more tangible with an example:
5.3.1. Example config
steps:
step 1:
commands:
- name: Check if /etc/fstab exists
type: shell
inline: test -x /etc/fstab
step 2:
commands:
- name: Run command 2.1
type: pg
inline: select datname from pg_database
step 3:
commands:
- name: Run command 3.1
type: pg
file: ./sql/step_3.1.sql
depends:
- step 1
- step 2
when:
- "eq .Steps["step 1"].Instances.Rc 0"
- '.Steps.["step 2"].Instances.StdOut.Contains "{datname}={mydb}"'
parallel: 2
5.4. Graphical representation
5.4.1. What does it do?
When running a job with a specification as shown in the example, PgQuartz will do the following:
PgQuartz will create a work queue and add
step 1
, andstep 2
to that queue.step 3
needs to wait for its dependencies to resolve.
PgQuartz will create 2 Runners (due to
parallel: 2
).Both Runners start processing a step (both
step 1
andstep 2
are processed).
Should command 1.1 fail (see commands on what can make a command enter failed state):
RC of
step 1
would not be 0the first
when
statement onstep 3
will preventstep 3
from running
Should command 2.1 fail, or not find any database
mydb
:the stdout will not contain
{datname}={mydb}
the second
when
statement onstep 3
will preventstep 3
from running
note that step 3
also has dependencies on step 1
and step 2
.
These are currently required because without them:
step 3
would be scheduled beforestep 1
andstep 2
would have been runningat schedule time,
step 1
would not have an RC set. RC defaults to 0, sostep 3
would still be scheduled regardless of failure of step 1.at schedule time
step3
would not have a stdout set. StdOut defaults to empty string, which does not contain{datname}={mydb}
so step 3 would not be run, even ifmydb
would existstep 3
would not be run (prevented by second when statement ‘always failing’ without dependencies)