Notification System: Technical Specification

Author: Abeleje Olaniyi George
Version: 2.0
Date: May 2026
Context: Designed for a fintech platform serving 1M+ users across push, SMS, and email channels.

1. Goals & Requirements

Functional

-Send push, SMS, and email notifications triggered by financial events (contributions confirmed, payouts sent, group reminders, account alerts, marketing).
-Support per-user channel preferences and opt-outs.
-Deliver notifications with guaranteed at-least-once semantics, deduplicated to exactly-once at the user level.

Non-Functional

-Throughput: sustain 10,000 notifications/minute at baseline; burst to 100,000/minute for broadcast campaigns.
-Reliability: no missed sends for critical financial alerts; no duplicate delivery.
-Degradation: graceful failover when any single provider is unavailable.
-Observability: full delivery audit trail per notification per user.
-Latency: transactional alerts delivered within 5 seconds of trigger; marketing within 10 minutes.

2. High-Level Architecture

Event Sources
(Django signals, Celery tasks, Paystack webhooks, API triggers)
          │
          ▼
┌─────────────────────┐
│  Notification API   │  ← Internal service: accepts notification intents
│  (FastAPI/Django)   │
└────────┬────────────┘
         │  Enqueues to
         ▼
┌─────────────────────┐
│   Celery + Redis    │  ← Redis as broker (one queue per channel + priority)
│   (broker/backend)  │
└────────┬────────────┘
         │  Consumed by
         ▼
┌─────────────────────┐
│  Channel Workers    │  ← Celery workers, one pool per channel
│  (Push/SMS/Email)   │
└────────┬────────────┘
         │  Writes to
         ▼
┌─────────────────────┐      ┌──────────────────────┐
│  Delivery Log (PG)  │      │  Provider Abstraction │
│  + Redis dedup set  │      │  Layer (see §5)       │
└─────────────────────┘      └──────────────────────┘

Why Celery + Redis over SQS:
Celery with Redis as the broker is simpler to operate, cheaper to run, and scales comfortably to millions of tasks. The architecture is identical to an SQS-based system — queues, workers, DLQ handling — without the AWS operational overhead. Migration to SQS is a config-level change if Redis ever becomes the bottleneck, which for most fintech workloads at 1M users it never does.

3. Data Model

`notifications` table

CREATE TABLE notifications (
    id                UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key   TEXT UNIQUE NOT NULL,  -- e.g. "contribution:txn_123:confirmed"
    user_id           UUID NOT NULL REFERENCES users(id),
    channel           TEXT NOT NULL CHECK (channel IN ('push', 'sms', 'email')),
    template_id       TEXT NOT NULL,
    payload           JSONB NOT NULL,
    priority          TEXT NOT NULL DEFAULT 'normal'
                      CHECK (priority IN ('critical', 'normal', 'marketing')),
    status            TEXT NOT NULL DEFAULT 'pending'
                      CHECK (status IN ('pending', 'queued', 'delivered', 'failed', 'suppressed', 'manual_review')),
    provider          TEXT,           -- which provider handled delivery
    provider_ref      TEXT,           -- provider's message ID for reconciliation
    attempts          INT NOT NULL DEFAULT 0,
    last_attempted_at TIMESTAMPTZ,
    delivered_at      TIMESTAMPTZ,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_notifications_user    ON notifications (user_id, created_at DESC);
CREATE INDEX idx_notifications_status  ON notifications (status) WHERE status IN ('pending', 'failed', 'manual_review');
CREATE INDEX idx_notifications_idem    ON notifications (idempotency_key);

`notification_events` table (audit trail + observability)

CREATE TABLE notification_events (
    id              BIGSERIAL PRIMARY KEY,
    notification_id UUID NOT NULL REFERENCES notifications(id),
    event           TEXT NOT NULL,  -- queued, attempted, delivered, failed, retried, suppressed
    provider        TEXT,
    detail          JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

This table is the primary observability layer. Delivered/failed/suppressed ratios per channel per hour are queryable directly:

SELECT
    channel,
    status,
    COUNT(*) AS count,
    date_trunc('hour', created_at) AS hour
FROM notifications
WHERE created_at > now() - interval '24 hours'
GROUP BY channel, status, hour
ORDER BY hour DESC;

`user_notification_preferences` table

CREATE TABLE user_notification_preferences (
    user_id     UUID PRIMARY KEY REFERENCES users(id),
    push        BOOLEAN NOT NULL DEFAULT true,
    sms         BOOLEAN NOT NULL DEFAULT true,
    email       BOOLEAN NOT NULL DEFAULT true,
    marketing   BOOLEAN NOT NULL DEFAULT true,
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

4. Idempotency & Deduplication

Every notification is created with a caller-supplied idempotency_key encoding the event identity:

{event_type}:{entity_id}:{outcome}
e.g. "contribution:txn_abc123:confirmed"
     "payout:cycle_456:sent"
     "reminder:group_789:due:2026-05-27"

Two-layer deduplication:

Layer 1 — Database (durable):
INSERT INTO notifications ... ON CONFLICT (idempotency_key) DO NOTHING
If a key already exists, the insert is silently skipped. No duplicate record is ever created.

Layer 2 — Redis (fast, pre-delivery):
Before sending to any provider, workers check a Redis SET keyed by notif:{idempotency_key} with a 24-hour TTL. If the key exists, the message is dropped before it reaches the provider. This short-circuits duplicates that arise from Celery's at-least-once delivery guarantee.

def is_duplicate(idempotency_key: str) -> bool:
    key = f"notif:{idempotency_key}"
    # SET NX: only sets if key doesn't exist; returns True if newly set
    return not redis.set(key, 1, nx=True, ex=86400)

If the Redis layer is unavailable, the database unique constraint is the fallback — no silent failure.

5. Provider Abstraction & Failover

Each channel has a ProviderRouter wrapping multiple providers behind a common interface:

class NotificationProvider(Protocol):
    def send(self, recipient: str, content: dict) -> ProviderResult: ...
    def health_check(self) -> bool: ...

# Email providers (in priority order)
email_router = ProviderRouter([
    SendGridProvider(),
    MailgunProvider(),      # fallback
])

# SMS providers
sms_router = ProviderRouter([
    TermiiProvider(),       # primary (NG-optimised)
    TwilioProvider(),       # fallback
])

# Push providers
push_router = ProviderRouter([
    FCMProvider(),          # Android
    APNSProvider(),         # iOS
])

ProviderRouter logic

class ProviderRouter:
    def send(self, recipient, content):
        for provider in self.providers:
            if self.circuit_breaker.is_open(provider):
                continue          # skip unhealthy provider
            try:
                result = provider.send(recipient, content)
                self.circuit_breaker.record_success(provider)
                return result
            except ProviderError as e:
                self.circuit_breaker.record_failure(provider)
                log_event(notification_id, "provider_failed", provider, str(e))
                continue          # try next provider
        raise AllProvidersExhausted()

Circuit Breaker (Redis-backed)

Each provider has an independent circuit breaker tracked in Redis:

-Closed (healthy): requests flow normally.
-Open (unhealthy): provider skipped for 60 seconds after 5 consecutive failures.
-Half-open: one probe request allowed; success closes the circuit, failure extends the open window.

This prevents a slow or erroring provider from adding latency to every send.

6. Queue Architecture (Celery + Redis)

Three Celery queues, one per channel, with priority separation for critical vs marketing:

# celery config
celery_app.conf.task_routes = {
    "notifications.tasks.deliver_push":   {"queue": "push"},
    "notifications.tasks.deliver_sms":    {"queue": "sms"},
    "notifications.tasks.deliver_email":  {"queue": "email"},
}

# Worker pools (launched separately)
# celery -A app worker -Q push    --concurrency=10
# celery -A app worker -Q sms     --concurrency=10
# celery -A app worker -Q email   --concurrency=20

Priority handling:
Critical notifications (financial alerts) are enqueued with priority=0 (Celery's highest). Marketing messages use priority=9. Workers process lower numbers first, ensuring financial alerts are never queued behind bulk campaigns.

Dead Letter Queue (DLQ) pattern:
After max_retries (4), the task transitions the notification to manual_review status and fires a Sentry alert. A Celery Beat task runs every 15 minutes to surface manual_review records to ops for human action or bulk requeue.

celery_app.conf.beat_schedule = {
    "dlq-review-every-15min": {
        "task": "notifications.tasks.review_manual_queue",
        "schedule": crontab(minute="*/15"),
    },
}

7. Worker Design

@celery_app.task(
    bind=True,
    max_retries=4,
    acks_late=True,              # only ack after successful processing
    reject_on_worker_lost=True,  # requeue if worker crashes mid-task
)
def deliver_notification(self, notification_id: str):
    notif = Notification.objects.select_for_update().get(id=notification_id)

    # Guard: skip if already terminal
    if notif.status in ("delivered", "suppressed", "manual_review"):
        return

    # Check user preferences
    prefs = get_user_preferences(notif.user_id)
    if not prefs.allows(notif.channel, notif.priority):
        notif.update(status="suppressed")
        log_event(notification_id, "suppressed", reason="user_preference")
        return

    # Redis dedup check
    if is_duplicate(notif.idempotency_key):
        return

    try:
        notif.update(
            status="queued",
            attempts=F("attempts") + 1,
            last_attempted_at=now()
        )
        result = channel_router[notif.channel].send(
            notif.recipient,
            notif.rendered_content
        )
        notif.update(
            status="delivered",
            provider=result.provider,
            provider_ref=result.ref,
            delivered_at=now()
        )
        log_event(notification_id, "delivered", result.provider)

    except AllProvidersExhausted as exc:
        log_event(notification_id, "failed", detail=str(exc))
        if self.request.retries >= self.max_retries:
            notif.update(status="manual_review")
            capture_message(f"Notification {notification_id} exhausted all retries", level="error")
            return
        raise self.retry(
            exc=exc,
            countdown=exponential_backoff(self.request.retries)  # 30s → 2m → 10m → 1h
        )

    except Exception as exc:
        capture_exception(exc)  # Sentry
        raise self.retry(exc=exc, countdown=exponential_backoff(self.request.retries))

acks_late=True ensures Celery does not acknowledge the task until it fully completes. If a worker crashes mid-send, the task becomes visible again and is retried by another worker — no message loss.

8. Graceful Degradation

| Failure Scenario | System Behaviour | |---|---| | Single provider down | Circuit breaker opens; ProviderRouter tries next provider transparently | | All providers for a channel down | Task retries with exponential backoff; Sentry alert fires after final retry | | Redis unavailable | Dedup falls back to DB unique constraint; slight performance degradation only | | Celery broker (Redis) unavailable | Notification API returns 503; caller retries; no data loss | | DB unavailable | Notification API returns 503; nothing is queued; caller retries | | Worker crash mid-send | acks_late + Celery requeue ensures task is reprocessed by another worker | | Max retries exhausted | Notification moves to manual_review; Sentry alert fires; ops reviews |

Critical financial alerts are never silently dropped. Every terminal failure is captured in notification_events and surfaced via Sentry.

9. Observability

Sentry — primary error and performance layer:

-Captures all provider errors and task exceptions with full context (notification ID, user ID, channel, provider, attempt count).
-Performance traces on deliver_notification task duration — surfaces slow providers and latency regressions.
-Alerts when error rates spike above threshold.

notification_events table — audit trail and delivery metrics:

-Complete per-notification event log: queued → attempted → delivered/failed/suppressed.
-Queryable by support for any user complaint ("did my contribution alert send?").
-Delivery rate ratios (delivered/failed/suppressed per channel per hour) queried directly from Postgres — no external dashboard tool required at this scale.

What comes next at scale:
CloudWatch for queue depth and worker throughput metrics, Grafana for delivery rate dashboards, PagerDuty for anomaly alerting. These are the natural next layer once Sentry + structured Postgres logging hits its limits.

10. Load Testing

Load tested using Locust against the notification enqueue endpoint with provider delivery mocked out — isolating queue throughput and worker concurrency as the bottleneck, not provider rate limits.

# locustfile.py
from locust import HttpUser, task, between

class NotificationUser(HttpUser):
    wait_time = between(0.1, 0.5)

    @task
    def enqueue_notification(self):
        self.client.post("/api/v1/notifications/", json={
            "user_id": "test-user-uuid",
            "channel": "email",
            "template_id": "contribution_confirmed",
            "payload": {"amount": 5000, "group": "Family Savings"},
            "idempotency_key": f"contribution:txn_{self.environment.runner.user_count}:confirmed",
            "priority": "critical"
        })

Target: sustain 10,000 enqueues/minute with p95 latency under 200ms, zero task loss under simulated worker crash.

11. Scaling Considerations

-Workers are stateless Celery processes — horizontal scaling is trivial. Add worker pods to increase throughput.
-For broadcast campaigns (1M users), a fan-out Celery task reads user IDs in batches of 1,000 and enqueues individual notification tasks, respecting provider rate limits via Celery's rate_limit setting (rate_limit="500/m" per worker).
-Marketing sends use a dedicated low-priority worker pool so they never contend with transactional alert workers.
-PostgreSQL notifications table partitioned by created_at (monthly) once row count exceeds ~50M — keeps query performance stable without archiving old records.
-Migration from Redis broker to AWS SQS is a Celery config-level change if Redis ever becomes the bottleneck.

12. Build Phases

Phase 1 — Core reliability (week 1–2):
notifications + notification_events tables, idempotency key, single provider per channel, Celery workers with acks_late, manual_review DLQ pattern, Sentry integration.

Phase 2 — Resilience (week 3–4):
Multi-provider fallback, circuit breakers, Redis dedup layer, user preference enforcement, Celery Beat for DLQ review.

Phase 3 — Scale (month 2):
Broadcast fan-out, priority worker pools, Locust load test to 1M virtual users, DB partitioning, CloudWatch metrics.

Authored by Abeleje Olaniyi George · github.com/olaniyigeorge · olaniyigeorge.vercel.app