Hangfire Docker with Multiple Servers

I’ve been using Hangfire for almost 2 years. It’s a wonderful job schedule API with persistent storage. Community have a lot of examples about how to use Hangfire. However, almost all of them use the application (UI) as a hangfire server. Here, I will explain how to use Hangfire with docker and multiple servers. Let’s begin.

UI

First, let’s create an API. I will show my example on ASP.NET Core 2.2 with PostgreSQL. Open your terminal.

$ mkdir Hangfire.UI
$ cd Hangfire.UI
$ dotnet new webapi

Then we need the Hangfire packages.

$ dotnet add package Hangfire.Core
$ dotnet add package Hangfire.AspNetCore
$ dotnet add package Hangfire.PostgreSql

Open Startup.cs and add those lines into ConfigureServices

Next, open appsettings.json and add the connection string

"ConnectionStrings": { 
     "HangfireConnection": "Server=postgresql;Port=5432;Database=demo;User Id=demo; Password=111111;"
},

Don’t change the server part of connection string, PostgreSQL will be exposed with that name.

Next and the most important thing, dashboard settings. By default, dashboard can only be reached by localhost. Since we are using docker, it will stay in the image. Therefore, we have to expose the dashboard. Open Startup.cs and copy following code.

As I mentioned in the code, do not use this code in the production directly. You have to create your own scenario (e.g. only admin roles can reach hangfire.). Also, we have disabled anti forgery token for now, the security is not our first concern here. UI part is done.

Background Server

Secondly, we need a separate background server project. Create this project besides Hangfire.UI project.

$ mkdir Hangfire.Server
$ cd Hangfire.Server
$ dotnet new console

Run following package commands

$ dotnet add package Hangfire.Core
$ dotnet add package Hangfire.PostgreSql

The console must stay alive, all the time. Therefore we will use one of the best solutions of .NET CoreHostBuilder.

We are setting the connection string and creating a background server with respect to it. I have setWorkerCount as 1 for demonstration, you can use Environment.ProcessorCount * 5 at production. Also, BackgroundServerOptions class has a Activatorproperty which allows you to use Dependency Injection with your jobs (e.g. Example).RunConsoleAsync will suspend the main thread and prevent the server termination.

Output of the Background Server;

hang.server1_1  | Application started. Press Ctrl+C to shut down.
hang.server1_1  | Hosting environment: Production
hang.server1_1  | Content root path: /app/

Jobs

The UI and Server projects must share the same code base for the Jobs. Therefore, I will create a new library project beside those projects.

$ mkdir Hangfire.Jobs
$ cd Hangfire.Jobs
$ dotnet new classlib

A sample job using Thread.Sleep(ms)

This common project can be added to the UI and Server projects with following command.

$ dotnet add reference ../Hangfire.Jobs

We are done with the infrastructure. Additionally, we need to enqueue the example job (MyJob). At Hangfire.UI there should be ValuesController where you can put an example job there. Or you can create your own controller for jobs.

The above code will enqueue the example job with random interval when anyone hit the Get() action.

We are done with the codebase. We have separated the Hangfire Background Server and Dashboard, and created a common class library for jobs. What we have to do in the next part is, dockerize the projects and create environment with those containers.

Docker

Since, the both UI and Server projects are .NET Core 2.2 projects. The Dockerfile should be almost same. (I tried my best while creating these dockerfiles). Hangfire.UI ‘s Dockerfile is shown below. Open a new file named Dockerfile and copy following commands into it.

Only difference between UI and Server is EXPOSE 80 command on the file. The server project does not need any TCP port exposing.

Since we obtained multiple dockerized applications, we are almost done. Next, run the images using docker-compose command.

Docker-Compose

Docker-compose creates an environment which run multiple images and enables them to communicate with each other. Go to main folder and create docker-compose.yml file and copy following content into it.

The compose file has a network namesvcnw . This network will connect the images with each other. On the other hand, the dependency between images should be in a particular order.

PostgreSQL > UI > Servers.

Docker-compose up multiple servers which are called as hang.server1 and hang.server2. Server 2 will use the same image with Server 1, therefore we can up more than one server using the same image. Keep in mind that using docker-compose is not ideal in every scenario, we need an orchestrator such as Kubernetes so that the number of servers can be increased on demand easily.

Let’s see the results. Go to project’s root folder and run

$ docker-compose up --build

The docker-compose will build all images and run them. Next, you can go to

http://localhost:5005/hangfire/

As you can see here, there are two servers running. Let’s try the job scheduling. Load twice the following endpoint http://localhost:5005/api/values to enqueue new jobs.

The image shows that jobs are distributed to the servers. Remember, we have set the worker count to 1 above.

Conclusion

In this post, we have successfully showed how to run multiple Hangfire servers using docker images. We have separated the UI and Server part of the Hangfire, also dockerized those applications. To make it clear, you should distribute your Hangfire servers with kubernetes or service fabric. It is important to use advantages of the distributing jobs and dockerizing. See you in next post.

You can get the source files below.

GitHub - alimozdemir/Hangfire.Docker: An experimental hangfire docker project with multiple servers…

References

Documentation - Hangfire Documentation