Abhishek Roka
Abhishek Roka

From 3 to 7 Containers: How My Django Project Architecture Evolved

When I started building my Placement Portal, I intentionally kept the backend architecture simple.

I had three containers:

For the initial version of the application, that was more than enough.

Django handled the application logic.

PostgreSQL handled the database.

Adminer gave me a convenient way to inspect and manage the database.

Simple.

Then the application started growing.

I introduced background task processing.

That introduced Celery.

Celery needed a message broker.

That introduced RabbitMQ.

I needed to monitor background tasks.

That introduced Flower.

I started thinking about scheduled jobs and caching.

That brought Celery Beat and Redis into the development environment.

And suddenly, my project had gone from three containers to seven.

At first, that sounds like the architecture became much more complicated.

But that's not how I see it.

Each new container exists because a new responsibility appeared.

My Architecture Started With Three Containers

The initial architecture was straightforward:

┌──────────────┐
│    Django    │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│  PostgreSQL  │
└──────────────┘

┌──────────────┐
│    Adminer   │
└──────────────┘

This was enough for the core Placement Portal functionality.

At this stage, I didn't need a separate background-processing system.

The application was relatively simple, and keeping the infrastructure small made development easier.

But then I encountered a problem with email delivery.

Background Tasks Changed the Architecture

My password reset flow was sending emails directly from the Django request.

That meant the API had to wait for the SMTP operation before responding.

I wanted the API to respond quickly while email delivery happened independently.

That led me to Celery.

But introducing Celery isn't just about installing a Python package.

I needed a process that could actually execute background tasks.

That's where the Celery Worker came in.

The architecture became:

Django
  │
  ▼
Background Task
  │
  ▼
Celery Worker

Now background processing had its own runtime.

That was the first major architectural expansion.

Celery Worker: Moving Work Out of Django

The Celery Worker is responsible for executing background tasks.

For my current implementation, one of the main use cases is email delivery.

Instead of making Django perform the email operation itself, Django can delegate the work to Celery.

This creates a cleaner separation:

Django
  │
  │ Create Task
  ▼
Celery Worker
  │
  │ Execute Task
  ▼
Email

The goal isn't to make the architecture complicated.

The goal is to keep the web application focused on handling requests while background workers handle longer-running operations.

RabbitMQ: Connecting Django and Workers

Once I started thinking about the architecture more seriously, another question appeared.

How should tasks move from the Django application to the Celery Worker?

That's where RabbitMQ comes in.

RabbitMQ acts as the message broker between the application and the workers.

The architecture becomes:

Django
  │
  ▼
RabbitMQ
  │
  ▼
Celery Worker

Now each component has a clearer responsibility.

Django creates the work.

RabbitMQ handles the message.

Celery Worker executes the work.

This also means Django doesn't need to know whether one worker or several workers are running.

The application publishes the task and lets the background-processing infrastructure handle the rest.

Flower: Seeing What Is Happening

Once background tasks started running independently, another problem became obvious.

How do I know what's happening?

With a normal Django request, I can inspect the API response and application logs.

Background tasks are different.

A task might be created successfully but fail later inside a worker.

That's why I introduced Flower.

Flower provides a monitoring interface for Celery.

It gives me visibility into things such as:

Monitoring isn't necessarily exciting during development.

But once work starts happening outside the main application process, visibility becomes extremely important.

Celery Beat: Preparing for Scheduled Work

Celery Beat is another service I introduced into my development architecture.

Unlike the Celery Worker, Beat isn't responsible for executing the task itself.

Its purpose is to schedule periodic tasks.

For example, I may eventually need scheduled operations such as:

These aren't features I'm claiming to have implemented yet.

They're future workloads that I expect the Placement Portal may eventually need.

Celery Beat gives me a place to handle that type of scheduled work when the time comes.

Redis: Infrastructure I Don't Use Yet

Redis is probably the most interesting addition because I'm not actively using it yet.

That raises an important question:

Why add a service that currently has no responsibility?

Honestly, I wouldn't recommend adding infrastructure simply because it might be useful someday.

But in my development environment, Redis is there as infrastructure I expect to use as the project evolves.

Potential use cases include:

However, there is an important distinction.

Planned infrastructure isn't the same as implemented functionality.

Right now, Redis isn't solving an active problem in my application.

So I'm not treating its presence as proof that the application already needs it.

Seven Containers, Seven Responsibilities

After these additions, the development environment looks roughly like this:

                    ┌───────────────┐
                    │    Django     │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │   RabbitMQ    │
                    └───────┬───────┘
                            │
                    ┌───────┴───────┐
                    ▼               ▼
             ┌─────────────┐ ┌─────────────┐
             │   Celery    │ │   Celery    │
             │   Worker    │ │    Beat     │
             └─────────────┘ └─────────────┘

                    ┌───────────────┐
                    │    Flower     │
                    │   Monitoring  │
                    └───────────────┘

                    ┌───────────────┐
                    │     Redis     │
                    │ Future Cache  │
                    └───────────────┘

                    ┌───────────────┐
                    │  PostgreSQL   │
                    └───────────────┘

                    ┌───────────────┐
                    │    Adminer    │
                    └───────────────┘

The number seven isn't what matters.

The responsibilities are.

Each service exists for a specific reason.

My Docker Development Workflow Changed Too

Adding more services also changed how I work with Docker.

Initially, I could simply run:

docker compose up

and watch everything directly in my terminal.

With only a few containers, that was convenient.

But as more services started producing logs, the terminal became much harder to work with.

Now I generally prefer running the containers in detached mode and inspecting logs or individual services when necessary.

For example:

docker compose up -d

Then I can inspect the relevant service instead of having every container continuously write into the same terminal.

This may sound like a small workflow change.

But once an application has several continuously running services, development ergonomics become important too.

More Containers Can Actually Reduce Complexity

This sounds contradictory.

How can adding containers reduce complexity?

The answer is responsibility separation.

Imagine putting everything into Django:

Django
 ├── HTTP Requests
 ├── Email Delivery
 ├── Background Jobs
 ├── Scheduling
 ├── Task Monitoring
 ├── Caching
 └── Database Logic

The application now has to handle too many unrelated responsibilities.

Instead, I can separate them:

Django       → Web Application
PostgreSQL   → Database
RabbitMQ     → Message Broker
Celery       → Background Processing
Beat         → Scheduling
Flower       → Monitoring
Redis        → Future Caching
Adminer      → Database Administration

There are more components.

But each component is easier to understand.

That's the trade-off I'm learning to appreciate.

Production Deployment Should Be Different

Another important lesson is that the development environment doesn't need to be identical to the production environment.

Just because a service exists in my development setup doesn't mean I should deploy it immediately.

My current thinking is:

Deploy infrastructure when it provides an actual production responsibility.

If I deployed the Placement Portal today, the services actively required for the current architecture would include the application, database, background worker, message broker, and monitoring components as appropriate to the deployment.

Redis and Celery Beat don't need to be deployed merely because they're present in development if they aren't currently being used.

The same applies to Adminer.

If I need it for administration, it can be useful.

But it doesn't automatically belong in a production deployment just because it's part of my local Docker Compose setup.

This distinction helps prevent infrastructure from becoming unnecessarily expensive or complicated.

Scalability Is About Responsibilities, Not Container Count

Going from three containers to seven might look like a sign that the project became unnecessarily complicated.

But that's not really what happened.

The application gained new responsibilities.

Those responsibilities needed appropriate places to live.

The architecture evolved accordingly.

The important question isn't:

"How many containers does my application have?"

It's:

"Does each component have a clear reason to exist?"

If the answer is yes, adding another service can actually make the system easier to reason about.

Final Thoughts

My Placement Portal started with:

Django
PostgreSQL
Adminer

Today, my development architecture has grown to seven containers as I've introduced background processing, messaging, monitoring, scheduling infrastructure, and future caching capabilities.

The interesting part isn't the number.

It's why the number changed.

Celery appeared because email shouldn't block API requests.

RabbitMQ appeared because task production and task execution needed a communication layer.

Flower appeared because background work needed monitoring.

Celery Beat appeared because scheduled workloads will eventually need a dedicated scheduler.

Redis appeared as infrastructure for potential future caching and related workloads.

Every architectural change came from a problem, requirement, or future responsibility.

That's the lesson I'm taking from this evolution:

Good architecture isn't about having fewer components. It's about making sure every component has a clear responsibility.

Sometimes, adding another container is exactly what makes the overall system simpler.

— Abhishek Roka