How to automatically resubmit jobs

powerJobs Processor jobs can sometimes fail because of instability errors. In those cases the jobs work when resubmitting them.

This article shows how to create a job that is automatically queued on regular intervals using time triggered jobs.

The job checks if failed jobs in the job queue failed with specific error messages and resubmits them if they did.

It also keeps track of how many times a job was already resubmitted and stops resubmitting them after a configurable amount of times.

Create the job script

To create the job script open your preferred scripting environment and paste the code below in a new file.

Then save the file in the powerJobs Processor job directory ("C:\ProgramData\coolOrange\powerJobs\Jobs").

In the beginning of the script you can set some Options:

• $maxResubmit: Controls how many times a job can be resubmitted
• $JobMaxCount: Controls the maximum number of jobs that are retrieved in the Resubmit job
• $Errors: Contains the error messages that cause the job to be resubmit.

The script will get the failed jobs in the queue and check if they failed with one of the errors in $Errors. If it did it will update the job description to keep count of the times the job was resubmitted. The job also adds the user that originally queued the job because when resubmitting it the user will be the one logged into powerJobs Processor and not the user that originally queued  the job.

$maxResubmit = 3
$Errors = @(
    "The remote procedure call failed",
    "1013",
    "The RPC server is unavailable"
)

$JobList = $vault.JobService.GetJobsByDate([int]::MaxValue, [DateTime]::MinValue)
$FailedJobs = $JobList | Where-Object {$_.StatusCode -eq "Failure"}

foreach($Job in $FailedJobs ){
    if($Errors -notcontains $job.StatusMsg){
        continue
    }
    if($Job.Descr.StartsWith("Resubmit")) {
        $originalDesc = $job.Descr.Substring(13)
        [int]$SubmitCount = $Job.Descr.Substring(9,1)
        $nextSubmitCount = $SubmitCount + 1
        if ($nextSubmitCount -eq $maxResubmit) {
            $description = ("Resubmit {0}: Limit reached | {1}" -f $nextSubmitCount, $originalDesc)      
        } else {
            $description = ("Resubmit {0} | {1}" -f $nextSubmitCount, $originalDesc)
        }
    }else{
        $SubmitCount = 1
        $jobCreator = $job.CreateUserName
        $description = ("Resubmit {0} | JobCreator {1} | {2}" -f $SubmitCount, $Job.CreateUserName, $job.Descr)
    }

    if($maxResubmit -gt $SubmitCount){
        $params = @{}
        foreach ($param in $Job.ParamArray) {
            $params.add($param.Name, $param.Val)
        }
        Add-VaultJob -Name $job.Typ -Description $description -Parameters $params -Priority $job.Priority
    }
}

{
 "Trigger": 
  {
    "TimeBased": "0 0/10 * 1/1 * ? *",

    // This is the name of the Vault you want to trigger the job
    "Vault":"Vault",

    // And this two parameters are optional and self explaining:
    "Priority":10,
    "Description":"This job is triggered weekdays at 8:00 am"
  }
}