Background jobs

Most apps need to do work outside the request cycle: sending email, processing uploads, calling external APIs. Feather provides three job backends.

Choosing a backend

This is not a development versus production choice. All three run in production. Pick based on what you need.

Goal	Backend	Trade-off
Simplicity, no infrastructure	`sync`	Blocks the request
Speed, return fast and process later	`thread`	Jobs lost on restart
Reliability, never lose a job	`rq`	Requires Redis and workers

sync, when blocking the request is acceptable

Simple apps where jobs finish fast enough
Debugging job logic, since errors surface in the request
Cases where infrastructure simplicity matters more than response time

thread, for fast responses without infrastructure

Jobs run in a Python thread pool. Choose it when:

You want sub-second response times
You would rather not run Redis
Jobs are fire-and-forget, and losing a few on a crash is acceptable
You need concurrency limits for memory-hungry work such as ML or transcription

rq, when reliability matters

Jobs persist to Redis before acknowledgement, and workers run as independent services. Choose it when:

Losing a job would cause real problems, as with payments or notifications
You need to retry failures and inspect history
You run multiple servers
You need scheduled or recurring tasks
Background processing has to survive deploys and crashes

Configuration

JOB_BACKEND=thread
JOB_MAX_WORKERS=4              # thread pool size
# JOB_ENABLE_MONITORING=true   # psutil resource tracking

JOB_BACKEND=sync

JOB_BACKEND=rq
REDIS_URL=redis://localhost:6379/0

When JOB_BACKEND is unset, REDIS_URL selects rq. See automatic backends.

Using the thread backend in development? Set FLASK_DEBUG=0 in .env. Flask’s auto-reloader restarts the process on every file change and kills running threads, so jobs die mid-execution each time you save.

Defining and enqueuing

from feather import job

@job
def send_welcome_email(user_id, email):
    send_email(email, 'Welcome!', render_template('emails/welcome.html'))

@api.post('/users')
@inject(UserService)
def create_user(user_service, email: str):
    user = user_service.create(email=email)
    send_welcome_email.enqueue(user.id, user.email)   # returns immediately
    return {'user': user.to_dict()}, 201

# with a delay, in seconds
send_welcome_email.enqueue(user.id, user.email, delay=60)

Concurrency control

Cap concurrent executions to keep memory-hungry work from exhausting the machine.

@job(concurrency=2)   # max 2 concurrent executions
def transcribe_audio(file_path):
    """Whisper transcription, memory intensive."""
    result = whisper.transcribe(file_path)
    return result['text']

@job(concurrency=1)   # singleton, one at a time
def rebuild_search_index():
    pass

Jobs queue when the limit is reached, first in first out within each task type. Different tasks have independent limits. This matters for audio and video transcription, ML inference, rate-limited external APIs, and database work bounded by the connection pool.

Retries

Failed jobs retry with exponential backoff.

@job(retry=3)   # backoff: 2s, 4s, 8s
def call_external_api(data):
    response = requests.post('https://api.example.com', json=data)
    response.raise_for_status()

@job(concurrency=2, retry=2)
def transcribe_with_retry(video_id):
    pass

Resource monitoring

.env

JOB_ENABLE_MONITORING=true

pip install psutil

Error logs for failed jobs then carry memory in MB, memory percent, CPU percent and thread count.

Scheduled tasks

Recurring jobs need the RQ backend and rq-scheduler.

from feather import scheduled

@scheduled(cron='0 9 * * *')   # every day at 9am
def daily_digest():
    send_daily_digest_emails()

@scheduled(interval=3600)      # every hour
def cleanup_temp_files():
    delete_old_temp_files()

pip install rq-scheduler
rqscheduler --url redis://localhost:6379/0

From a platform scheduler

When your host runs cron jobs as commands, point it at feather jobs run. It runs one @job function in the current process, on any backend, and exits non-zero if the job raises.

feather jobs run daily_digest

Workers as services

On the RQ backend, workers are independent processes that share your codebase but run separately from the web server. They have full access to your models, services and config, on their own lifecycle. That separation is what makes them recoverable. If the web server crashes, jobs already in Redis keep waiting, and a restarted worker picks up where it left off. If a worker crashes mid-job, RQ marks the job failed so it can be retried. Nothing disappears quietly. That suits work which has to complete eventually: billing cycles, subscription renewals, webhook delivery, report generation. Workers also replace cron. Rather than configuring an external scheduler, you enqueue delayed or recurring work from application code, and the worker’s built-in scheduler promotes delayed jobs when they come due. A billing job enqueued with delay=55 fires on time even if the web server restarted in between.

pip install rq

feather worker                    # process the default queue
feather worker high default low   # specific queues, in priority order
feather worker --burst            # exit when the queue is empty

feather worker covers the setup that would otherwise need a custom script. It creates the Flask app and pushes app context so jobs can reach the database and config, uses SimpleWorker on macOS to avoid the fork() crash with the Obj-C runtime, and enables the built-in scheduler by default.

Flag	Description
`--burst`	Exit when the queue is empty
`--simple`	Force SimpleWorker, one process, no fork. Default on macOS
`--fork`	Force the forking worker, one process per job, so a crashing job cannot take the worker down. Default on Linux, and what the Docker `worker` target runs
`--no-scheduler`	Disable the delayed job scheduler
`--name`	Worker name, for identification in logs
`--log-level`	DEBUG, INFO, WARNING, ERROR. Default INFO

Deploying workers

In production, workers run as separate services from the same Docker image as the web server, with a different start command. The generated docker-compose.yml already includes one when background jobs are enabled.

docker-compose.yml

services:
  web:
    build:
      context: .
      target: web
    env_file: .env
    environment:
      DATABASE_URL: postgresql://myapp:${POSTGRES_PASSWORD}@db:5432/myapp
      REDIS_URL: redis://redis:6379/0

  worker:
    build:
      context: .
      target: worker      # same Dockerfile, CMD ["feather", "worker", "--fork"]
    env_file: .env        # same secrets, same database
    environment:
      JOB_BACKEND: rq
      JOB_SERIALIZER: json
      DATABASE_URL: postgresql://myapp:${POSTGRES_PASSWORD}@db:5432/myapp
      REDIS_URL: redis://redis:6379/0

Scale with docker compose up -d --scale worker=3, unless your jobs include a singleton loop such as a scheduler or a billing tick, which must not run twice.

Workers share the image, environment variables and database with the web service
Scale them independently, or dedicate workers to specific queues
Each worker connects to Redis for job pickup and to your database for business logic
Restarting the web service doesn’t interrupt running jobs

Set JOB_SERIALIZER=json for RQ. Pickle payloads are code execution for anyone who can write to your Redis.

Managing the queue

feather jobs status              # queue status and counts
feather jobs list                # all jobs
feather jobs list --status failed
feather jobs list --queue high   # jobs in a specific queue (RQ)
feather jobs list --stuck        # jobs running too long (thread)
feather jobs info <job_id>
feather jobs failed
feather jobs retry <job_id>
feather jobs clear

Get started

Tutorial: build a Kanban app

Building the UI

Backend

Features

Tooling

Testing

Deployment

Background jobs

Choosing a backend

Configuration

Defining and enqueuing

Concurrency control

Retries

Resource monitoring

Scheduled tasks

From a platform scheduler

Workers as services

Deploying workers

Managing the queue

​Choosing a backend

​Configuration

​Defining and enqueuing

​Concurrency control

​Retries

​Resource monitoring

​Scheduled tasks

​From a platform scheduler

​Workers as services

​Deploying workers

​Managing the queue

Choosing a backend

Configuration

Defining and enqueuing

Concurrency control

Retries

Resource monitoring

Scheduled tasks

From a platform scheduler

Workers as services

Deploying workers

Managing the queue