odinserj

on development

HangFire 0.8.3 Released

23 May 2014 edit on github

Cancellation tokens

This release comes with great feature for long-running jobs: cancellation tokens. They are like CancellationToken class, but for background jobs.

Previously, you was not able to stop a running job by using Requeue or Delete methods of the BackgroundJob class, or by clicking a button in HangFire Monitor. Its state was changed, but the job is still running. And running… And running… Nobody waits for it, why it is not stopped yet?

Job cancellation tokens provide ThrowIfCancellationRequested method that throws OperationCanceledException if a job was canceled due to:

To use cancellation tokens, you need to add just one parameter of type IJobCancellationToken to a target method:

public static void Cancelable(
    int iterationCount, 
    IJobCancellationToken token)
{
    try
    {
        for (var i = 1; i <= iterationCount; i++)
        {
            // Loop breaker
            token.ThrowIfCancellationRequested();

            Thread.Sleep(1000);

            Console.WriteLine(
                "Performing step {0} of {1}...", 
                i, 
                iterationCount);            
        }
    }
    catch (OperationCanceledException)
    {
        Console.WriteLine("Cancellation requested, exiting...");
        throw;
    }
}

Then, create a background job:

BackgroundJob.Enqueue(
    () => Cancelable(1000, JobCancellationToken.Null));

To be able to test target methods, or to add the support of cancellation token to your old jobs, you can use the JobCancellationToken class:

public void Test()
{
    var token = new JobCancellationToken(true);
    Cancelable(10, token);
}

Hanging jobs

This release solved another problem. On some job queues (Redis, SQL Server, but not MSMQ) it is not possible to apply transaction just to fetch a job from a queue, hide it from another workers, and delete it on successful processing or place it back to a queue on failure or process termination.

To mimic this behavior, HangFire uses atomic get/set commands: update with output clause in SQL Server and BRPOPLPUSH in Redis. That is why other workers don’t see just fetched job. But in case of process termination, all jobs will remain in invisible state. To fight with this, there is a component on a server, who seeks invisible jobs and decides what to do with them.

Unfortunately, there is no way to determine whether a job was aborted by a process termination, or it is still working. To separate these cases, the component checks the fetching time of a job and compares it with current time. If the result is greater than InvisibilityTimeout, then we should trait the job as hanged, and return it to its queue. InvisibilityTimeout defaults to 30 minutes to be sure that the job processing was aborted.

When ASP.NET issues shutdown request and gives background jobs 30 seconds to die, some of them may be aborted by ThreadAbortException. In previous releases this lead to the fact that background job may stay in Invisible state on regular shutdown, and this sometimes increase the latency of job processing.

In this release HangFire make an attempt to place aborted job back to the start of its queue on this exception, and it will be started again immediately after application restart on success, and after InvisibilityTimeout on failure.

When the feature of re-queueing jobs on ThreadAbortException plays together with job cancellation tokens, it means that you greatly decrease the latency of background job processing, because the probability of using InvisibilityTimeout is greatly decreased.

TL;DR

Use job cancellation tokens where possible to ensure that your jobs are shutting down gracefully, and you greatly decrease the probability of high latencies in your background job processing.

Changes

Related Posts

Comments