The_DisExcel_project/CLAUDE.md

The DisExcel Project — Context for Claude Code

Backend: FastAPI + SQLAlchemy (async) + Pydantic v2 + PostgreSQL. JWT auth (access + refresh tokens), RBAC permissions, Redis cache, RabbitMQ background workers. Python >=3.13,<4.0, Poetry for dependency management.

Auth & Permissions

• Users have direct_permissions (list of Permissions) and group (list of PermissionsGroups, each with its own permissions) — many-to-many both ways.
• Effective permissions = direct_permissions ∪ (union of all groups' permissions).
• require_permissions(*permissions) in src/web/protected_routes is a FastAPI dependency factory — wraps CurrentUserService.get_current_user. Call with no args (require_permissions()) for "just authenticated, no specific permission needed".
• Access tokens carry a jti claim. Logout writes revoked_access_token:{jti} to Redis with TTL = remaining token lifetime — get_current_user checks this key before anything else.
• secure flag on refresh_token cookie is driven by env_settings.PROD_MODE (bool) — False locally/tests so cookies work over plain HTTP, True in prod.
• CORS: CORSMiddleware in main.py with allow_origins=[env_settings. FRONT_URL] and allow_credentials=True (needed for the httpOnly refresh_token cookie — with credentials, ["*"] is rejected by browsers). FRONT_URL is a required Env field with no default, so the backend fails to start without it in configs/.env (e.g. http://localhost:5173). It is added last, after the logging middlewares, so it wraps them and CORS headers also appear on error responses (401/429).

Redis (src/cache/)

• RedisClient(redis.Redis) — module-level shared singleton, subclasses redis.Redis directly (inherits all commands, no manual wrapping needed).
• Three uses: permissions is-cache was considered and rejected (no real DB savings — get_user_by_id already eager-loads everything via selectin in one call); rate limiting (RateLimit.rate_limit(ip, action, limit=5, window=60) — INCR + EXPIRE ... NX on first attempt, key rate_limit:{action}:{ip}, so each action has its own counter); access-token revoke blacklist (see above).
• Two rate-limited actions: "login" (5/60s) and "reset" (3/600s). The login limit is only triggered inside except HTTPException on /protected/token — i.e. only on failed logins, not successful ones (otherwise legitimate repeated logins would trip it). /protected/reset counts every call.

RabbitMQ (src/messaging/)

• RabbitMQClient — shared class, lazy connect() (can't be async __init__), holds one connection + one channel, get_channel() ensures setup. connect() retries connect_robust() up to 5 times with exponential backoff (asyncio.sleep(2**attempt)) before raising — needed because when running the app locally (outside Docker Compose) right after docker compose up rabbitmq, the broker's AMQP listener isn't immediately ready and resets the connection mid-handshake (ConnectionResetError). Compose's own depends_on: condition: service_healthy only helps when the app itself also runs as a Compose service — it does nothing when the app is started on the host.
• Topology is centralized, not declared ad-hoc. src/models/rabbitmq_models/email.py defines the whole topology as pydantic data (Topology/ExchangeSpec/QueueSpec, exported as email_topology) — topic exchange "email", one durable quorum queue per message type (queue_welcome_email/email.welcome, queue_reset_email/email.reset), each with a matching dead-letter queue. src/messaging/topology_setup.py::apply_topology(channel, topology) is the only place that actually calls declare_exchange/declare_queue/bind against RabbitMQ. It's called once at startup — in main.py's lifespan (web) and in daemon_run.py's main() (daemon entrypoint) — before any producer/consumer touches the exchange/queues. EmailProducer/ WelcomeEmailConsumer/ResetEmailConsumer no longer declare or bind anything themselves — they only get_exchange("email") / get_queue("queue_..."), assuming apply_topology already ran. Changing a queue's arguments in email_topology requires manually deleting the existing queue in RabbitMQ first — declare_queue on an already-existing queue with different arguments fails with PRECONDITION_FAILED, it doesn't update in place.
• Dead-lettering: each original queue's x-dead-letter-exchange points at email.dlx, a direct exchange (not fanout — fanout would broadcast every dead-lettered message to all DLQs, mixing up welcome/reset failures). Each DLQ is bound to email.dlx with the same routing key as the original queue (email.welcome/email.reset); RabbitMQ preserves a message's original routing key when dead-lettering, so no explicit x-dead-letter-routing-key override is needed — the direct exchange alone routes each dead letter back to the correct DLQ.
• message.process(ignore_processed=True) + manual await message.nack(...) in process_message, not the plain auto-ack/nack that message.process() does by default — needed because business-logic exceptions are caught inside the block (to log them and decide retry vs. dead-letter) rather than left to propagate, so process() itself must be told not to also try to ack/nack on exit. Classification: transient errors (SMTP/network: SMTPServerDisconnected, SMTPConnectError, TimeoutError, ConnectionRefusedError) → nack(requeue=True); everything else (bad message body, SMTPAuthenticationError, bugs) → nack(requeue=False) → DLQ. Note SMTPAuthenticationError is a known rough edge here — it's a systemic config problem (bad credentials), not a per-message one, but is currently DLQ'd per-message like any other permanent error rather than halting the daemon with an alert.
• Email templates: Jinja2 (src/service/email/jinja_env.py — one shared Environment/FileSystemLoader singleton, templates in src/service/email/templates/), inline CSS (email clients don't support <style> reliably), EmailMessage with set_content() (plain-text fallback) + add_alternative(html, subtype="html").
• Reset-password flow mirrors welcome: ResetEmailSender (src/service/email/email_reset.py) renders templates/reset.html ({{ temp_password }}, no longer hardcoded) the same way DaemonEmailSender does welcome.html. ResetEmailConsumer.process_message reads both email and temp_password from the message body and uses the same transient/permanent classification as WelcomeEmailConsumer. EmailProducer.send_reset_email(email, temp_password) takes the password as a second argument.
• Reset route: POST /protected/reset?email=... in auth_routes.py — email is a plain str param, so FastAPI reads it from the query string, not the JSON body. The route rate-limits (3/600s per IP), calls CrudService.reset_password(email) (generates a 16-char temp password via HashService.generate_temp_password() — which is async — hashes and saves it through update_user), then publishes {email, temp_password} to email.reset. A 404 (unknown email) is swallowed so the response is always {"detail": "New password was sent"} — no user enumeration. Known rough edge: the DB password is changed before the email is sent, so a message that ends up in the DLQ leaves the user with a changed password and no email; and anyone who knows an email can reset that account's password. A token-link flow would avoid both.

Logging (src/logging/)

• All log output (HTTP endpoints, SQL, daemons) funnels through one module-level asyncio.Queue (log_queue in src/logging/logger.py) and a single LogWriter.log_writer() background task that drains it and appends to ./logs/{type}_log_{month}_{year}.txt — this avoids the original bug of firing an unsupervised asyncio.create_task per log line (unordered writes, lost logs if the process died before the task ran). log_writer() must be started explicitly wherever the process entrypoint is — it's asyncio.create_task(writer.log_writer()) in main.py's lifespan for the web process, and the same call in daemon_run.py's main() for the daemon process. log_queue is process-local (plain in-memory asyncio.Queue, not shared across processes), so every new entrypoint needs its own writer task or its logs silently queue forever and are never written (unbounded memory growth, not a crash).
• logging.Handler subclasses (LoggerDB for sqlalchemy.engine, LoggerDaemon for the "daemon" logger) just push (type, formatted_msg) onto log_queue from emit() — do not give them a custom __init__ that doesn't call super().__init__(); skipping it means self.level/ self.filters/etc. never get set and any log call raises AttributeError: 'LoggerX' object has no attribute 'level'.
• Two ContextVars tag log lines with a correlation id without threading it through every function call: request_id_ctx (set once per HTTP request in LoggingMiddleware.dispatch) and message_id_ctx (meant to be set once per RabbitMQ message in process_message). Only visible within the same async call chain — a ContextVar set in an HTTP request is "-" (the default) inside daemon code, and vice versa; they don't cross the process boundary either.
• src/logging/logger.py vs src/logging/http_logger.py split matters for Docker. logger.py has zero fastapi/starlette imports — intentional, because the daemon Poetry group (and therefore the daemon Docker image) never installs fastapi. http_logger.py holds LoggingMiddleware/ProcessingTimeMiddleware (the only things that actually need FastAPI/Starlette) and is imported only from main.py. src/logging/__init__.py imports only from logger.py. Never import src.logging.http_logger from anything that runs in the daemon process (consumers.py, email_welcome.py, daemon_run.py, etc.) — it would drag in fastapi, which raises ModuleNotFoundError in the daemon container.
• Plain logger.exception(...) must only be called from inside an except block — it pulls the active exception via sys.exc_info() to attach a traceback. Called outside except (e.g. for a routine "message received" log line), it still runs but appends a literal NoneType: None instead of a traceback, since there's no active exception to format.

Daemons / workers (src/daemons/)

• BaseDaemon ABC (name + async run()), one subclass per consumer (WelcomeEmailDaemon, ResetEmailDaemon, more to come — e.g. reports).
• DAEMONS registry dict maps string name → daemon class.
• daemon_run.py (project root) is the single entrypoint: python daemon_run.py <name> runs one daemon, python daemon_run.py --all reads configs/daemons.json ({"daemons": [...]}) and runs all enabled ones concurrently via asyncio.gather. main() also applies the RabbitMQ topology (apply_topology) and starts the log writer task before dispatching to either mode, inside a try/finally so both the single-daemon and --all paths cancel the writer task on the way out.
• Each daemon runs as its own Docker service/container (command: ["python", "daemon_run.py", "<name>"]), same pattern as the migration service.
• daemons.json is read with a plain Pydantic BaseModel + manual json.load, NOT pydantic-settings json_file — that requires wiring settings_customise_sources manually in this pydantic-settings version and isn't worth the complexity here.

Docker / Poetry groups

• pyproject.toml uses PEP 621 [project.dependencies] for shared deps (sqlalchemy, redis, aio-pika, pydantic, bcrypt, jose, aiofiles, asyncpg, psycopg2-binary, alembic, greenlet).
• web group: fastapi, uvicorn, gunicorn, python-multipart — only needed by the API server.
• daemon group: worker-only deps — aiosmtpd, jinja2 (email sending and templating). Must not gain a fastapi/starlette dependency; see the src/logging/logger.py vs http_logger.py split above for why that boundary is enforced deliberately.
• dev group: pytest stack, allure, httpie, requests-async.
• claude group: graphifyy (the /graphify Claude Code skill) — dev tooling only, never referenced by any Dockerfile stage, install with poetry install --with claude when needed locally.
• Dockerfile has parallel builder→final stage pairs: builder→prod (installs main,web) and worker-builder→worker (installs main,daemon). Same base pattern: venv builder stage copies /opt/venv into a clean final stage, poetry itself is uninstalled after install to keep the final image lean.
• Alembic runs against a separate sync engine (asyncpg swapped out, psycopg2 used instead) — async SQLAlchemy engine can't drive Alembic directly without the run_sync bridge, and a dedicated sync engine is simpler than that bridge.
• DB_HOST differs between contexts: psql (Docker service name) for containers talking to each other, localhost for anything run on the host (e.g. local alembic revision --autogenerate). Compose services override DB_HOST via environment:; the .env file's own default is for host-side runs.

Codebase knowledge graph (graphify)

• /graphify (Claude Code skill, graphifyy package in the claude Poetry group) builds a navigable knowledge graph of this repo into graphify-out/ (graph.html, GRAPH_REPORT.md, graph.json) — gitignored, regenerable, never commit it.
• Its own interpreter/venv detection defaults to uv tool/pipx-style installs; this project has neither — graphifyy is installed straight into the project's own .venv via Poetry, so point graphify at .venv/bin/python3 directly rather than letting it search for uv.
• The sandbox's blanket **/.env/**/.env.* read-deny blocks graphify from reading configs/.env.example too (not just the real .env) — it shows up as skipped_sensitive in detection. This is a sandbox-level block, not graphify's own sensitive-file heuristic; a real .env would be excluded by graphify anyway, but .env.example (no real secrets) would otherwise be safe to include if the sandbox allowed reading it.
• A GRAPH HEALTH WARNING (dangling-endpoint edges, collapsed edges) on a fresh build most likely means the AST extractor's node-ID format changed since a previous partial run — re-run with graphify extract --force if it persists across rebuilds; a one-off warning on first build isn't necessarily a problem.

Frontend (frontend/)

• React 19 + TypeScript + Vite + react-router-dom 7, package manager is yarn (yarn.lock). Run with yarn dev (Vite, default http://localhost:5173). The user writes/edits frontend code themselves — explain rather than edit unless asked (see memory).
• Layout: src/features/auth/components/ (AuthComponent, ResetComponent, each with its own .module.css), src/pages/ (LoginPage at /, ForgotPasswordPage at /forgot-password — thin wrappers), src/layouts/AuthLayout (shared split-screen shell with the marketing block; takes children for the form on the right), src/api/ (token.ts, LoginAPI/Auth.ts, ResetAPI/Reset.ts), src/types/UserTypes.ts (mirrors the Pydantic UserOut/UserCreate). index.css = global reset/font, App.css = #root frame.
• Env: Vite reads .env* from frontend/configs/ (envDir: "./configs" in vite.config.ts), not from frontend/. Only VITE_-prefixed variables reach the browser code, via import.meta.env. Backend address is VITE_BACKEND_URL, exported as API_URL from src/api/token.ts. Restart the dev server after editing .env. There is no Vite server.proxy (removed on purpose) — the browser calls the backend directly, so backend CORS (see Auth) must allow the frontend origin.
• Access token lives in a module variable in src/api/token.ts (setAccessToken/getAccessToken), not in localStorage; it is lost on page reload (a /protected/refresh call on startup is not implemented yet). authFetch(path, init) adds Authorization: Bearer and credentials: "include". Login (POST /protected/token) must be sent as form-data (URLSearchParams with username = email and password), because the route uses OAuth2PasswordRequestForm, and returns {access_token, token_type}; the refresh token arrives as an httpOnly cookie.
• Forms are controlled inputs (useState) with hand-written validation: per-field errors state, checked on blur and on submit; <form noValidate> so browser bubbles don't preempt it. Invalid field gets .inputError, text goes in an absolutely positioned .errorText under the field. Backend errors are logged with console.error and shown as a short generic message.
• CSS Modules gotchas: only class selectors are hashed. Bare element selectors (form {}, input {}) and #id selectors stay global and leak between components (one module's form { gap } restyled the other form). Scope them under the module's own class (.wrapper_1 form {}) — the hashed ancestor makes the rule local. Classes with hyphens are accessed as styles["email-icon"]. Class names shared between two modules are independent copies (duplicate the rules, don't rely on sharing).
• React 19 types: FormEvent is deprecated; use SubmitEvent<HTMLFormElement>. <p> must not contain <div> (hydration warning).
• Login currently redirects to http://localhost:8000/docs after success (placeholder — no post-login page yet); ResetComponent copies that placeholder redirect too.

Testing (tests/unit, tests/integrated, tests/e2e)

• Recurring root cause of "different event loop" / MissingGreenlet-style errors: prod code uses module-level singletons (engine, redis_client) created once at import time and reused for the app's whole lifetime — this is correct for prod (one event loop, whole uptime) but breaks under pytest-asyncio's default function-scoped event loop (a new loop per test, but the singleton's connections stay bound to the first loop). Fix: test fixtures create a fresh engine/RedisClient per test and monkeypatch or inject them in place of the global singleton, then dispose on teardown — not a global session-scoped event loop (that would mask real isolation bugs).
• e2e MySession must subclass httpx.AsyncClient (not requests_async. AsyncSession — that library silently drops cookies between requests, which broke refresh-token-cookie-dependent tests like logout).
• test_user_fixture is indirect=True parametrized with (direct_permissions, group) tuples.
• tests/unit/test_consumers.py covers RabbitMQClient/WelcomeEmailConsumer/ ResetEmailConsumer entirely with mocks — no real broker involved. Pattern: monkeypatch.setattr(rabbitmq_client_module.aio_pika, "connect_robust", ...) patches the module attribute that connect() looks up at call time (not the rabbitmq_client singleton's method — that's a bound method, it has no attribute of its own to patch). aio_pika.connect_robust/asyncio.sleep must both be mocked when testing the retry loop, or the test really sleeps 2**attempt seconds between attempts. message.process(...) is an async context manager, not a plain awaitable — mocking it needs a MagicMock with __aenter__/__aexit__ set to AsyncMocks (see make_fake_message() in that file), not just AsyncMock(). When asserting on what a mocked async method returned, compare against mock.return_value (or a variable captured before assigning it), never against the mock itself — some_mock is some_mock.return_value is never true, they're two different objects.