The_DisExcel_project/CLAUDE.md

The DisExcel Project — Context for Claude Code

Backend: FastAPI + SQLAlchemy (async) + Pydantic v2 + PostgreSQL. JWT auth (access + refresh tokens), RBAC permissions, Redis cache, RabbitMQ background workers. Python >=3.13,<4.0, Poetry for dependency management.

Auth & Permissions

• Users have direct_permissions (list of Permissions) and group (list of PermissionsGroups, each with its own permissions) — many-to-many both ways.
• Effective permissions = direct_permissions ∪ (union of all groups' permissions).
• require_permissions(*permissions) in src/web/protected_routes is a FastAPI dependency factory — wraps CurrentUserService.get_current_user. Call with no args (require_permissions()) for "just authenticated, no specific permission needed".
• Access tokens carry a jti claim. Logout writes revoked_access_token:{jti} to Redis with TTL = remaining token lifetime — get_current_user checks this key before anything else.
• secure flag on refresh_token cookie is driven by env_settings.PROD_MODE (bool) — False locally/tests so cookies work over plain HTTP, True in prod.

Redis (src/cache/)

• RedisClient(redis.Redis) — module-level shared singleton, subclasses redis.Redis directly (inherits all commands, no manual wrapping needed).
• Three uses: permissions is-cache was considered and rejected (no real DB savings — get_user_by_id already eager-loads everything via selectin in one call); rate limiting on login (RateLimit.rate_limit(ip) — INCR + EXPIRE on first attempt, blocks >5/60s); access-token revoke blacklist (see above).
• Rate limit is only triggered inside except HTTPException on /protected/token — i.e. only on failed logins, not successful ones (otherwise legitimate repeated logins would trip it).

RabbitMQ (src/messaging/)

• RabbitMQClient — shared class, lazy connect() (can't be async __init__), holds one connection + one channel, get_channel() ensures setup. connect() retries connect_robust() up to 5 times with exponential backoff (asyncio.sleep(2**attempt)) before raising — needed because when running the app locally (outside Docker Compose) right after docker compose up rabbitmq, the broker's AMQP listener isn't immediately ready and resets the connection mid-handshake (ConnectionResetError). Compose's own depends_on: condition: service_healthy only helps when the app itself also runs as a Compose service — it does nothing when the app is started on the host.
• Topology is centralized, not declared ad-hoc. src/models/rabbitmq_models/email.py defines the whole topology as pydantic data (Topology/ExchangeSpec/QueueSpec, exported as email_topology) — topic exchange "email", one durable quorum queue per message type (queue_welcome_email/email.welcome, queue_reset_email/email.reset), each with a matching dead-letter queue. src/messaging/topology_setup.py::apply_topology(channel, topology) is the only place that actually calls declare_exchange/declare_queue/bind against RabbitMQ. It's called once at startup — in main.py's lifespan (web) and in daemon_run.py's main() (daemon entrypoint) — before any producer/consumer touches the exchange/queues. EmailProducer/ WelcomeEmailConsumer/ResetEmailConsumer no longer declare or bind anything themselves — they only get_exchange("email") / get_queue("queue_..."), assuming apply_topology already ran. Changing a queue's arguments in email_topology requires manually deleting the existing queue in RabbitMQ first — declare_queue on an already-existing queue with different arguments fails with PRECONDITION_FAILED, it doesn't update in place.
• Dead-lettering: each original queue's x-dead-letter-exchange points at email.dlx, a direct exchange (not fanout — fanout would broadcast every dead-lettered message to all DLQs, mixing up welcome/reset failures). Each DLQ is bound to email.dlx with the same routing key as the original queue (email.welcome/email.reset); RabbitMQ preserves a message's original routing key when dead-lettering, so no explicit x-dead-letter-routing-key override is needed — the direct exchange alone routes each dead letter back to the correct DLQ.
• message.process(ignore_processed=True) + manual await message.nack(...) in process_message, not the plain auto-ack/nack that message.process() does by default — needed because business-logic exceptions are caught inside the block (to log them and decide retry vs. dead-letter) rather than left to propagate, so process() itself must be told not to also try to ack/nack on exit. Classification: transient errors (SMTP/network: SMTPServerDisconnected, SMTPConnectError, TimeoutError, ConnectionRefusedError) → nack(requeue=True); everything else (bad message body, SMTPAuthenticationError, bugs) → nack(requeue=False) → DLQ. Note SMTPAuthenticationError is a known rough edge here — it's a systemic config problem (bad credentials), not a per-message one, but is currently DLQ'd per-message like any other permanent error rather than halting the daemon with an alert.
• Email templates: Jinja2 (src/service/email/jinja_env.py — one shared Environment/FileSystemLoader singleton, templates in src/service/email/templates/), inline CSS (email clients don't support <style> reliably), EmailMessage with set_content() (plain-text fallback) + add_alternative(html, subtype="html").
• Reset-password flow mirrors welcome: ResetEmailSender (src/service/email/email_reset.py) renders templates/reset.html ({{ temp_password }}, no longer hardcoded) the same way DaemonEmailSender does welcome.html. ResetEmailConsumer.process_message reads both email and temp_password from the message body and uses the same transient/permanent classification as WelcomeEmailConsumer. EmailProducer.send_reset_email(email, temp_password) takes the password as a second argument now. Still missing: nothing in the app actually calls send_reset_email yet — there's no password-reset route that generates a temp_password and publishes it. Don't assume the reset-password feature is reachable end-to-end until that route exists.

Logging (src/logging/)

• All log output (HTTP endpoints, SQL, daemons) funnels through one module-level asyncio.Queue (log_queue in src/logging/logger.py) and a single LogWriter.log_writer() background task that drains it and appends to ./logs/{type}_log_{month}_{year}.txt — this avoids the original bug of firing an unsupervised asyncio.create_task per log line (unordered writes, lost logs if the process died before the task ran). log_writer() must be started explicitly wherever the process entrypoint is — it's asyncio.create_task(writer.log_writer()) in main.py's lifespan for the web process, and the same call in daemon_run.py's main() for the daemon process. log_queue is process-local (plain in-memory asyncio.Queue, not shared across processes), so every new entrypoint needs its own writer task or its logs silently queue forever and are never written (unbounded memory growth, not a crash).
• logging.Handler subclasses (LoggerDB for sqlalchemy.engine, LoggerDaemon for the "daemon" logger) just push (type, formatted_msg) onto log_queue from emit() — do not give them a custom __init__ that doesn't call super().__init__(); skipping it means self.level/ self.filters/etc. never get set and any log call raises AttributeError: 'LoggerX' object has no attribute 'level'.
• Two ContextVars tag log lines with a correlation id without threading it through every function call: request_id_ctx (set once per HTTP request in LoggingMiddleware.dispatch) and message_id_ctx (meant to be set once per RabbitMQ message in process_message). Only visible within the same async call chain — a ContextVar set in an HTTP request is "-" (the default) inside daemon code, and vice versa; they don't cross the process boundary either.
• src/logging/logger.py vs src/logging/http_logger.py split matters for Docker. logger.py has zero fastapi/starlette imports — intentional, because the daemon Poetry group (and therefore the daemon Docker image) never installs fastapi. http_logger.py holds LoggingMiddleware/ProcessingTimeMiddleware (the only things that actually need FastAPI/Starlette) and is imported only from main.py. src/logging/__init__.py imports only from logger.py. Never import src.logging.http_logger from anything that runs in the daemon process (consumers.py, email_welcome.py, daemon_run.py, etc.) — it would drag in fastapi, which raises ModuleNotFoundError in the daemon container.
• Plain logger.exception(...) must only be called from inside an except block — it pulls the active exception via sys.exc_info() to attach a traceback. Called outside except (e.g. for a routine "message received" log line), it still runs but appends a literal NoneType: None instead of a traceback, since there's no active exception to format.

Daemons / workers (src/daemons/)

• BaseDaemon ABC (name + async run()), one subclass per consumer (WelcomeEmailDaemon, ResetEmailDaemon, more to come — e.g. reports).
• DAEMONS registry dict maps string name → daemon class.
• daemon_run.py (project root) is the single entrypoint: python daemon_run.py <name> runs one daemon, python daemon_run.py --all reads configs/daemons.json ({"daemons": [...]}) and runs all enabled ones concurrently via asyncio.gather. main() also applies the RabbitMQ topology (apply_topology) and starts the log writer task before dispatching to either mode, inside a try/finally so both the single-daemon and --all paths cancel the writer task on the way out.
• Each daemon runs as its own Docker service/container (command: ["python", "daemon_run.py", "<name>"]), same pattern as the migration service.
• daemons.json is read with a plain Pydantic BaseModel + manual json.load, NOT pydantic-settings json_file — that requires wiring settings_customise_sources manually in this pydantic-settings version and isn't worth the complexity here.

Docker / Poetry groups

• pyproject.toml uses PEP 621 [project.dependencies] for shared deps (sqlalchemy, redis, aio-pika, pydantic, bcrypt, jose, aiofiles, asyncpg, psycopg2-binary, alembic, greenlet).
• web group: fastapi, uvicorn, gunicorn, python-multipart — only needed by the API server.
• daemon group: worker-only deps — aiosmtpd, jinja2 (email sending and templating). Must not gain a fastapi/starlette dependency; see the src/logging/logger.py vs http_logger.py split above for why that boundary is enforced deliberately.
• dev group: pytest stack, allure, httpie, requests-async.
• Dockerfile has parallel builder→final stage pairs: builder→prod (installs main,web) and worker-builder→worker (installs main,daemon). Same base pattern: venv builder stage copies /opt/venv into a clean final stage, poetry itself is uninstalled after install to keep the final image lean.
• Alembic runs against a separate sync engine (asyncpg swapped out, psycopg2 used instead) — async SQLAlchemy engine can't drive Alembic directly without the run_sync bridge, and a dedicated sync engine is simpler than that bridge.
• DB_HOST differs between contexts: psql (Docker service name) for containers talking to each other, localhost for anything run on the host (e.g. local alembic revision --autogenerate). Compose services override DB_HOST via environment:; the .env file's own default is for host-side runs.

Testing (tests/unit, tests/integrated, tests/e2e)

• Recurring root cause of "different event loop" / MissingGreenlet-style errors: prod code uses module-level singletons (engine, redis_client) created once at import time and reused for the app's whole lifetime — this is correct for prod (one event loop, whole uptime) but breaks under pytest-asyncio's default function-scoped event loop (a new loop per test, but the singleton's connections stay bound to the first loop). Fix: test fixtures create a fresh engine/RedisClient per test and monkeypatch or inject them in place of the global singleton, then dispose on teardown — not a global session-scoped event loop (that would mask real isolation bugs).
• e2e MySession must subclass httpx.AsyncClient (not requests_async. AsyncSession — that library silently drops cookies between requests, which broke refresh-token-cookie-dependent tests like logout).
• test_user_fixture is indirect=True parametrized with (direct_permissions, group) tuples.
• tests/unit/test_consumers.py covers RabbitMQClient/WelcomeEmailConsumer/ ResetEmailConsumer entirely with mocks — no real broker involved. Pattern: monkeypatch.setattr(rabbitmq_client_module.aio_pika, "connect_robust", ...) patches the module attribute that connect() looks up at call time (not the rabbitmq_client singleton's method — that's a bound method, it has no attribute of its own to patch). aio_pika.connect_robust/asyncio.sleep must both be mocked when testing the retry loop, or the test really sleeps 2**attempt seconds between attempts. message.process(...) is an async context manager, not a plain awaitable — mocking it needs a MagicMock with __aenter__/__aexit__ set to AsyncMocks (see make_fake_message() in that file), not just AsyncMock(). When asserting on what a mocked async method returned, compare against mock.return_value (or a variable captured before assigning it), never against the mock itself — some_mock is some_mock.return_value is never true, they're two different objects.