Implementing solid_queue in an existing Rails app for database-backed job execution

By introducing solid_queue, one of the solid_* libraries introduced in Rails 8, into an existing Rails app, we can now run ActiveJob using a database (SQLite in this case) instead of Redis.
https://github.com/rails/solid_queue

I introduced solid_queue in the following environment:

Rails 8.0.1
Ruby 3.4.1
cssbundling-rails 1.4.1
jsbundling-rails 1.3.1
solid_queue 1.1.2

Installation

First, add it to your Gemfile and install it.

Gemfile

gem 'solid_queue'

Next, install the gem and generate the initial files for solid_queue.

$ bundle
$ bin/rails solid_queue:install
      create  config/queue.yml
      create  config/recurring.yml
      create  db/queue_schema.rb
      create  bin/jobs
        gsub  config/environments/production.rb

Four files were created and one file was modified.

File Contents

In config/queue.yml, you can configure settings such as the number of threads and processes for each environment.
config/recurring.yml is commented out by default, but it allows you to specify jobs or commands to be executed periodically for each environment. I found it convenient to be able to specify arguments and execution timing for jobs.
db/queue_schema.rb describes the database schema required to run solid_queue.
As shown below, there is a version specification, so I expect this will be updated if the database structure changes in future solid_queue updates.

db/queue_schema.rb

+ ActiveRecord::Schema[7.1].define(version: 1) do
...
+ end

bin/jobs starts solid_queue via the CLI. Its content is as follows:

bin/jobs

#!/usr/bin/env ruby

require_relative "../config/environment"
require "solid_queue/cli"

SolidQueue::Cli.start(ARGV)

In config/environments/production.rb, settings such as the adapter have been replaced with solid_queue as shown below.

config/environments/production.rb

# Replace the default in-process and non-durable queuing backend for Active Job.
+  config.active_job.queue_adapter = :solid_queue
+  config.solid_queue.connects_to = { database: { writing: :queue } }

Running in a Separate Process with Procfile

In my existing app, I use a Procfile to launch multiple processes, so I'll add the following to Procfile.dev so it can be run as a process locally.

Procfile.dev

+ job: bin/jobs

It is also possible to add solid_queue as a plugin to Puma and start it alongside the Rails server.
There is a sample provided in the README here.
https://github.com/rails/solid_queue/blob/main/README.md?plain=1#L108-L111

database.yml Configuration

Since I want to use a separate SQLite file this time, I'll add additional entries to config/database.yml.

config/database.yml

development:
+  primary:
+    <<: *default
+    database: storage/development.sqlite3
+  queue:
+    <<: *default
+    database: storage/development_queue.sqlite3
+    migrations_paths: db/queue_migrate

Now that the setup is complete, let's proceed with an operation test.

Operation Test

I'll create a simple job for now since it's just a test.

$ bin/rails g job TestJob
      invoke  rspec
      create    spec/jobs/test_job_spec.rb
      create  app/jobs/test_job.rb

I'll write the content of TestJob like this for now.

app/jobs/test_job.rb

class TestJob < ApplicationJob
  queue_as :default

  def perform(*args)
    puts "Hello, SolidQueue! Args: #{args}"
  end
end

After this, I'll start bin/jobs. In my case, I added it to Procfile.dev, so I'll start the process with bin/dev.

$ bin/dev

Then, register the job to the queue via the console.

$ bin/rails c
Loading development environment (Rails 8.0.1)
hoge-app(dev)> TestJob.perform_later("Argument")

By doing this, you can see the following entry in the logs of the started process:

00:00:00 job.1  | Hello, SolidQueue! Args: ["Argument"]

I successfully executed the job through SQLite! 🎉

I will also verify that it can be executed with perform_later. Similarly, register the job in the console to run after one minute, as follows:

hoge-app(dev)> TestJob.set(wait: 1.minute).perform_later("Delayed")

After one minute, the following log is output:

00:00:00 job.1  | Hello, SolidQueue! Args: ["Delayed"]

The operation check was successful!

Conclusion

I felt that setting up Redis to run jobs for personal development or small-scale apps often seemed like overkill, so being able to run jobs using a database feels very convenient.

Also, using SQLite is attractive because everything can be contained within the same machine when deploying to platforms like Fly.io or Render, which helps keep costs down.

I was worried that SQLite might lead to more contention issues, but if it can be run in a separate file, that concern should be mitigated. I feel like it's a great new option.

Since I've only run it locally so far, I'd like to see what kind of challenges I might face when running it in production or deploying to Fly.io.