Testing Sidekiq jobs: Which mode should I choose?

Sidekiq is a background job processor that helps Rails developers to increase the efficiency and responsiveness of an application.

The brilliant Sidekiq gem provides built-in tools for testing the various aspects of your worker’s lifecycle. The most important setting that you may choose as a developer is which testing mode to apply in a particular test. The documentation describes the behavior for both modes pretty clearly. But, besides that, it would be great to have an understanding of which aspects any particular mode suits best.

In this article, we will consider using Sidekiq for optimizing the Rails application performance. For you, we have prepared a brief guide on integrating Sidekiq with your project and testing Sidekiq worker in the different modes. This material will help you to evaluate each testing mode and understand how to work effectively with each of them.

Let’s dive in and try to sort this out.

Note: we’re going to use RSpec as a testing framework.

Using Sidekiq in Rails applications

Sidekiq is used for background processing of the tasks that can take a lot of time. This increases the responsiveness of the application, allowing user not to have to wait for the work to be completed.

This is very convenient for processing the complex tasks (for example, scanning the data from a big document and recording this to the database). When the tasks are launched they are queued. For example, if a user sends a letter from a Rails application that uses Sidekiq, there is no necessity to wait for the actual completion of the tasks. The user sends the letter, Sidekiq gets the task for sending and the browser displays the message: “Sending the letter”. Meanwhile, Sidekiq sends the letter in the background mode.

Further we are going to operate with two common Sidekiq terms: job and worker. For your convenience, let’s define their meanings.

A Sidekiq job is an operation that is processed in the background mode. The gem manages the queue of jobs in the database as JSON data sets. A worker is a Ruby class responsible for executing a job. When Sidekiq is ready to start job processing, the responsible worker is launched.

How to set up Sidekiq for your project

Sidekiq is very easy to set up – it is enough just to add the gem to the Gemfile and run installation of the new gem in Rails Console:

gem 'sidekiq'
bundle install

Then, specify the queue server in config/application.rb and create a task:

config.active_job.queue_adapter = :sidekiq
rails generate job Some

This will generate the file app/jobs/some_job.rb with the following contents:

class SomeJob < ApplicationJob
queue_as :default


def perform(*args)
# Do something later
end
end

To work with Sidekiq, we will need the set up and launched redis-server. For example, on Ubuntu 14.04, it is easy to set up with sudo apt-get install redis-server command. Having launched Sidekiq in the development environment, you are now ready to work.

Types of testing Sidekiq jobs

Fake it until you make it

Sidekiq::Testing.fake! is the default testing mode. When used, Sidekiq pushes all jobs queueing up into the internal array for each worker class.

require "sidekiq/testing" # you would likely add this to your rails_helper.rb


expect { MyWorker.perform_async(:my_arg) }.to change { MyWorker.jobs.size }.by(1)

Your jobs get queued but are not actually executed. To execute them, you need to drain the queues:

Sidekiq::Worker.drain_all

Unfortunately, this means that you would have to repeat the Sidekiq::Worker.drain_all snippet before each test in your worker’s set of examples:

describe MyWorker do
it "does something useful" do
MyWorker.perform_async(:my_arg)
Sidekiq::Worker.drain_all
# expect(...).to ...
end


it "does something useful again" do
MyWorker.perform_async(:my_arg, :additional_arg)
Sidekiq::Worker.drain_all
# expect(...).to ...
end
end

Also, the control exerted by draining all of the queues after each example seems a bit too inspecific for me. What if we could execute each “worker” line of code for real without any additional hassle or messing with draining queues? The next section describes the Sidekiq::Testing mode that works exactly this way.

To summarize the fake mode, its purpose is to test the actual behavior of the “worker’s” performance and how correctly it executes. In contrast, testing to see if the worker queued up should just be a sanity check and could be extracted into a shared example:

shared_examples "a worker with args of" do |*args|
it "gets enqueued" do
expect { described_class.perform_async(*args) }.to change { described_class.jobs.size }.by(1) }
end
end


describe MyWorker do
it_behaves_like "a worker with args of", :my_arg
end


Getting real

The Sidekiq::Testing.inline! testing mode executes the jobs immediately when they are placed in the queue. In my opinion, this should be the default testing mode for Sidekiq. Of course, you can test your workers directly:

worker = MyWorker.new
worker.perform(:my_arg)

But this makes your test invocations inconsistent compared to the invocations in real code (perform_async).

With Sidekiq inline mode as the default, your worker’s logic is tested right out of the box. This will cause you to think about the dependencies of your workers to be able to mock them effectively and even make expectations for calls to them.

# implementation
class DeleteFromRemoteApiWorker
include Sidekiq::Worker

def perform(item_ids)
ApiWrapper.delete_items(item_ids) # dependency
end
end


# test
describe DeleteFromRemoteApiWorker do
let(:items) do
# ...
end

it "delegates the work to the API wrapper as expected" do
allow(ApiWrapper).to receive(:delete_items)
item_ids = items.map(&:id)

described_class.perform_async(item_ids)


expect(ApiWrapper).to(
have_received(:delete_items).with(item_ids)
)
end
end

To simplify the usage for the cases where you still need fake mode, I suggest adding the following snippet to your rails_helper.rb:

require "sidekiq/testing"
Sidekiq::Testing.inline!

config.around(type: :worker) do |example|
if example.metadata[:sidekiq_fake] == true
Sidekiq::Testing.fake! { example.run }
end

# and its usage
# (actually a good example of when you just need to test if that job has queued up)


describe "rake my_project:my_task" do
it "adds new jobs digest job to the queue", :sidekiq_fake do
task.execute # a part of the custom functionality that executes the Rake task mentioned in the example description
expect(MyWorker).to have_enqueued_sidekiq_job
end
end

Bonus: make your worker tests read more naturally

But wait a minute! What was that have_enqueued_sidekiq_job?

In order to test Sidekiq jobs more naturally, I recommend using the beautiful rspec-sidekiq gem, which contains plenty of matchers with “speaking” names. Go on and discover them!

Conclusion

Having read this article, you are now aware of the different modes of Sidekiq testing.

During the everyday testing work, you always test your application at the unit level. Later, you test how it interacts with the other parts of the program. Finally, you test the integration of the app with the database and the other services. In this way, you also should also work with Sidekiq to:

• use unit tests to evaluate the worker behavior;
• test interaction of your jobs while testing the interaction of the different parts of your app;
• test workers inline when it is necessary to test the performance of the overall system.

Happy Sidekiq testing!