Django, Celery and Redis: Background Tasks Done Right

What happens when a user fills in a form and hits Submit? Ideally the page responds instantly. But if an email goes out in the same request, a slow SMTP server means the user stares at a spinner for three seconds. If SMTP is down, they get an error page — even though the submission was saved successfully.

The fix is to move slow, externally-dependent work out of the request cycle. In the Django ecosystem, the standard tool for that is Celery.

Why Redis?

Celery is a task queue, and it needs somewhere to store tasks. That place is called a broker. Redis and RabbitMQ are the usual choices.

For most web projects Redis is enough: it's simple to run, lightweight, and you're probably already using it for caching. RabbitMQ offers stronger delivery guarantees at the cost of complexity — worth it when no task may ever be lost, such as financial processing.

Setup

pip install celery redis

Define the Celery app at the project root:

# config/celery.py
import os
from celery import Celery
 
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
 
app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

autodiscover_tasks() finds tasks.py in every installed app automatically — no manual registration.

Load Celery when Django starts:

# config/__init__.py
from .celery import app as celery_app
 
__all__ = ("celery_app",)

Settings:

# settings.py
CELERY_BROKER_URL = os.environ["REDIS_URL"]
CELERY_RESULT_BACKEND = os.environ["REDIS_URL"]
CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TIMEZONE = "Europe/Istanbul"
 
# Kill tasks running longer than 5 minutes
CELERY_TASK_TIME_LIMIT = 300

Don't skip CELERY_TASK_SERIALIZER = "json". Celery's legacy default, pickle, serializes Python objects in an executable form; anyone with broker access could use it to run code on your server. JSON carries no such risk.

Your first task

# notifications/tasks.py
from celery import shared_task
from django.core.mail import send_mail
 
@shared_task
def send_contact_notification(name: str, email: str, message: str) -> None:
    send_mail(
        subject=f"New contact form: {name}",
        message=message,
        from_email="noreply@example.com",
        recipient_list=["info@example.com"],
    )

Calling it:

# views.py
send_contact_notification.delay(name, email, message)

.delay() queues the task and returns immediately. The user doesn't wait.

Critical rule: pass IDs, not objects

The most common beginner mistake is passing a model instance to a task:

# ❌ Don't
process_order.delay(order)

Two problems. First, the object must be serialized and the JSON serializer can't. Second — and more insidious — by the time the task runs, that data may be stale. Another process could have changed the record while it sat in the queue.

Pass the ID and fetch fresh data inside the task:

# ✅ Correct
process_order.delay(order.id)
 
@shared_task
def process_order(order_id: int) -> None:
    order = Order.objects.get(pk=order_id)
    # ...

The transaction trap

If you queue a task inside a database transaction, the worker can start before the record is committed. The result: Order.objects.get(pk=order_id) raises DoesNotExist.

Django's on_commit hook solves it:

from django.db import transaction
 
with transaction.atomic():
    order = Order.objects.create(...)
    transaction.on_commit(lambda: process_order.delay(order.id))

Now the task is only queued once the transaction commits successfully. Make this a habit everywhere you dispatch tasks.

Retries

External services fail intermittently. Letting a task retry itself usually resolves it:

@shared_task(
    bind=True,
    autoretry_for=(requests.RequestException,),
    retry_backoff=True,
    retry_kwargs={"max_retries": 5},
)
def notify_external_service(self, payload_id: int) -> None:
    payload = Payload.objects.get(pk=payload_id)
    response = requests.post(EXTERNAL_URL, json=payload.as_dict(), timeout=10)
    response.raise_for_status()

retry_backoff=True grows the delay exponentially (1s, 2s, 4s, 8s…). This matters: retrying at a fixed interval piles more load onto a service that is already struggling.

Don't omit timeout=10 either. An HTTP request without a timeout can hang indefinitely and block a worker.

Tasks should be idempotent

A task can run more than once — after a retry, after a worker crash, or if a message is delivered twice. Running twice must not cause harm.

A "send welcome email" task running twice means the user gets two emails. A simple flag prevents it:

@shared_task
def send_welcome_email(user_id: int) -> None:
    user = User.objects.get(pk=user_id)
 
    if user.welcome_email_sent_at:
        return  # already sent
 
    send_mail(...)
    User.objects.filter(pk=user_id).update(welcome_email_sent_at=timezone.now())

Scheduled tasks

Celery Beat runs periodic, cron-like tasks:

# settings.py
from celery.schedules import crontab
 
CELERY_BEAT_SCHEDULE = {
    "cleanup-expired-sessions": {
        "task": "accounts.tasks.cleanup_expired_sessions",
        "schedule": crontab(hour=3, minute=0),  # 03:00 daily
    },
}

Beat runs as its own process and exactly one instance must be running. Two Beat processes means every scheduled task fires twice.

Running with Docker

Three processes are needed: web, worker, and (if you have periodic tasks) beat.

services:
  redis:
    image: redis:7-alpine
 
  web:
    build: .
    command: gunicorn config.wsgi --bind 0.0.0.0:8000
    depends_on: [redis]
 
  worker:
    build: .
    command: celery -A config worker --loglevel=info --concurrency=2
    depends_on: [redis]
 
  beat:
    build: .
    command: celery -A config beat --loglevel=info
    depends_on: [redis]

Tune --concurrency to your server. Each worker process holds a copy of the Django app in memory; spawning eight on a small box will exhaust it.

If you haven't set up the server itself yet, deploying Django on AWS Lightsail covers getting Docker running on a blank VM. For a fuller stack — Next.js, Postgres and Nginx alongside Django — see a production Docker Compose setup for Next.js + Django.

Monitoring

Tasks failing silently is worse than tasks failing loudly. At minimum, watch the logs:

docker compose logs -f worker

For something more permanent you can run a monitoring tool such as Flower — but don't leave its dashboard publicly reachable, since task names and arguments leak information about your business logic.

When you don't need it

Celery is infrastructure: it brings Redis, worker processes, and monitoring along with it. You may not need it when:

• The work is fast enough (under ~100 ms) to run inline
• You have one scheduled job and system cron will do
• You're on a serverless platform whose own queue service fits better

But for email, external API calls, file processing, or report generation, moving the work out of the request cycle improves the user experience directly.