How to automatically resubmit jobs

powerJobs Processor jobs can sometimes fail because of instability errors. In those cases the jobs work when resubmitting them.

This article shows how to create a job that is automatically queued on regular intervals using time triggered jobs.

The job checks if failed jobs in the job queue failed with specific error messages and resubmits them if they did.

It also keeps track of how many times a job was already resubmitted and stops resubmitting them after a configurable amount of times.

Create the job script

To create the job script open your preferred scripting environment and paste the code below in a new file.

Then save the file in the powerJobs Processor job directory ("C:\ProgramData\coolOrange\powerJobs\Jobs").

In the beginning of the script you can set some Options:

  • $maxResubmit: Controls how many times a job can be resubmitted
  • $JobMaxCount: Controls the maximum number of jobs that are retrieved in the Resubmit job
  • $Errors: Contains the error messages that cause the job to be resubmit.

The script will get the failed jobs in the queue and check if they failed with one of the errors in $Errors. If it did it will update the job description to keep count of the times the job was resubmitted. The job also adds the user that originally queued the job because when resubmitting it the user will be the one logged into powerJobs Processor and not the user that originally queued  the job.

The AddJob API call will resubmit the job if the job parameters are the same as the existing job instead of queuing a new job

$maxResubmit = 5
$JobMaxCount = 100
$Errors = @(
"The remote procedure call failed",
"1013"
)

$JobList = $vault.JobService.GetJobsByDate($JobMaxCount, [DateTime]::MinValue)
$FailedJobs = $JobList | Where-Object {$_.StatusCode -eq "Failure"}

foreach($Job in $FailedJobs ){
if($Errors -notcontains $job.StatusMsg){
continue
}
if($Job.Descr.StartsWith("Resubmit")) {
$originalDesc = $job.Descr.Substring(13)
[int]$SubmitCount = $Job.Descr.Substring(9,1)
$nextSubmitCount = $SubmitCount + 1
if ($nextSubmitCount -eq $maxResubmit) {
$description = ("Resubmit {0} Limit reached | {1}" -f $nextSubmitCount, $originalDesc)
} else {
$description = ("Resubmit {0} | {1}" -f $nextSubmitCount, $originalDesc)
}
}else{
$SubmitCount = 1
$jobCreator = $job.CreateUserName
$description = ("Resubmit {0} | JobCreator {1} | {2}" -f $SubmitCount, $Job.CreateUserName, $job.Descr)
}

if($maxResubmit -gt $SubmitCount){
$AddJob = $vault.JobService.AddJob($Job.Typ, $description, $Job.ParamArray, $Job.Priority )
}
}
Create the time based job trigger

To allow the job to be automatically triggered at a configurable interval a SETTINGS file needs to be created in the powerJobs Processor job directory that has the same name as the job. For Example if the job script is called "ResubmitJob.ps1" the SETTINGS file must be called "ResubmitJob.settings".

The settings file defines when the job should be queued. This example queues the job every 10 minutres.

If you are not familiar with CRON you can create CRON expressions using a website. For example CRONMaker.

{
"Trigger":
{
"TimeBased": "0 0/10 * 1/1 * ? *",

// This is the name of the Vault you want to trigger the job
"Vault":"Vault",

// And this two parameters are optional and self explaining:
"Priority":10,
"Description":"This job is triggered weekdays at 8:00 am"
}
}
See Also